Think about it, in real life, some people are tall, some people are short. However, in Skyrim, everyone ranges slightly shorter than average to slightly taller than average. I just think this is unrealistic. I mean, sure, height is always based off of race and gender, which isn't how it works in real life, but it still added more diversity to the population.
I mean, especially because sometimes it is hard to differentiate between a nord, an imperial, and a Breton and height added some diversity between those races