Yes bosmer are supposed to be shorter than average, but having the males like midgets and females rather tall is just wrong. And not even "high" elves should be taller than a hulking nord.
I don't know if its lore, but I kind of liked having the typical tall man short woman being switched around. It makes the wood elves more "other" and exotic.
High Elves are everything regal. That includes being tall. They're the oldest, longest-lived, and most intelligent of all the races, so logically by fantasy standards, they're going to be the tallest.