What kind of character are you playing? I also use light armor but I'm an archer so I spend most of my time kiting, either to gain distance to shoot an arrow or to cast healing spells. If I tried to melee those monsters in my paltry armor I'd die very quickly as well. Have you at least perked your light armor?
I steam roll giants, trolls and mammoths as an archer. I dont even need to stand on a mountain or such to take out giants.
By all means, they do 1 hit me, or come very close to it but thats kinda the point. Don't get hit.
I'm really dumb founded how melee are supposed to take out a giant without magic or a bow.
Gaints are well worth it, they will give you alot of income. (100's of gold, decent quality gems, enchanted armor, alchemy ingredients etc)
-they also guard a chest which is generally worth even more, i got something like 3000 gold worth of enchanted items from my last plundering.
Mammoths aren't worth the trouble. the most valuable item they drop is mammoth tusk (worth 150 gold) (i have killed about 15 mammoths thus far, yet to see anything more valuable drop)
How i kill giants as archer:
1) stand as far away as possible, with an escape route incase something goes horribly wrong (such as rocky mountains, which they cannot climb)
2) Open up with a sneak attack (ensure your bow is poisoned)
3) Aim and shoot, ensuring you use eagle eye to slow time
4) when giant gets too close, use whirlwind shout.
If you have decent enough bow and arrows the giant will nearly be dead already, at this point aim and shoot and by the time he gets within 15 feet of you he will drop to his knees Then walk up and put a arrow in his head and LOL
In the item menu it says i do 80 damage, and thats just with an elven bow (improved to epic), i haven't even bothered crafting a glass bow, im just waiting out til dragon smithing.
EDIT: one of the dragon priest handed my bottom to me on a silver platter though, didnt die but had to duck and take cover all the time (which feels like cheating tbh)