If the giants ever decided to band together and attack cities/towns, they would be a far bigger threat than dragons ever could be. They'd just send guards flying with one hit

Giants + Mammoths = End of the World

'Cause the dragons want to kill you. The giants will leave you alone, unless you get too close or mess with them.

EDIT: Grammar

Because Giants are a common sight in Skyrim. People have gotten used to them being around, and know that if you leave them alone, they'll leave you alone as well.

Dragons have been gone for a very, very long time. Legends and stories have built up the associated fears over time.

Because giants are soft targets if they annoys you. Draw agro ride back to city, mages and archers on the wall get some good training.
Now try this with dragons.

Because bethesda decided to program it that way, sorry for breaking your immersion.

Even if your narrative were somehow accurate, why is anyone afraid of any threat when there are even larger ones around? Probably because it's still a threat.

Creatures driven to extinction by men and cliffracers not the ultimate force of destruction. News at 11.

Because the giants are nice. Civil and friendly. Not like, say, dragons. Spriggans, on the other hand, should be feared more than either.

Giants...

Heres me exploring the wilderness when ohh a dragon wants to kill me. So I start fleeing to some cover to get a nice defensive position.

When I come across a nice lonely giant take me under his wing and defend me from this great threat. Well he must have been very protective of this land plot because he starts chasing me as well.

Ohh my savior, my lydia jumps on the offensive and gets shot into the sky like an arrow. Ohh my poor Lydia...I will see you soon.

Because giants are a joke to kill, They are slow and powerful. AKA just use the Default shout you get at rank three.. knock them down and kill them before they can even get up... its sad.