Trolls are male, so are falmer, Dremora and a great many others. In Oblivion Ogres are male and spriggans are always female. In Morrowind Golden saints were only female and in Shivering Ilses male and female. In Oblivion, Dremora were male and female but the females did not have dialouge options and were relitivly indistinguishable from the males. This is a staple for TES. There are usualy mods that put anatomy on creatures but Non exsist yet for female Giants as far as I know. If you have some time you could tinker with blender and get some cool mods out there. I`m sure they would be popular.
They must be asixual, and thus do not belong to a gender. They may have humanoid characteristics specific to a certain gender, but that doesn't make them male or female.
Wait, maybe like the Entwives? One day they simply walked away, never to be seen again. The Giants are lonely now that their Giantwives are gone, and are agressive nomads herding their animals for that reason.