I tried magic out and I dont really like it. I am level 6 (almost 7) and I am lvl 45 with destruction and have a lot of spells. I want to use melee weapons, but I havent put any skills into them. Could I possibly switch to melee weapons now and it wont affect me? I have put all of my skill points into destruction.

Yes.

Of course, everything is possible. I started of my Argonian as a warrior with a one-handed sword in my right hand and destruction magic in my left hand, but now switched to a sneakier dual-dagger set-up. You can easily switch any time you want, of course. Your final build won't be optimal but who cares.