I am lvl 10 too, and destruction at 30 is my highest skill, with alteration at 28, sneak at 27, restoration, illusion, enchanting and conjuration at 20-25. I have leveled speech a few times like you, and also smithing (twice), alchemy (four times). Plus skillbok levels to two handed, block, heavy armour that does absolutely nothing for me. So i think i am in a similar spot as you, but i dont feel that it is limiting me at all so far.
I've put four perks into destruction already which i guess gives me some punch, but to me the difficulty feels very good. I've found some places that were just too difficult for me forcing me to backtrack, but in general things seem to be moderately difficult. I dont know if it is because i have been towing a follower a round a bit though? It does feel like a follower can help out a ton, and likely i would have had a lot of trouble without a follower with me if the battles would have been the same.