I just met one herding some mammoths. He was quite peaceful towards me.
I finally had some time to wonder around after work and I get it now. Giants have some places marked that are theirs. If you get near they start shaking their clubs. If you pay attention you will back off.
When they aren't guarding their homes from human but out and about doing daily things like herding mammoth they will be peaceful to you.
That is actually kind of awesome. They are selectively peaceful unless they are protecting themselves, or their home.