Yes, really. The Dwemer are of average elf size, they're called dwarves because this is what giants called them. To giants, yes, they would be small -- to humans, not so much. So they're not the Tolkein dwarves you're thinking of.
We really should stop linking to a wiki that we wrote as evidence that we are correct. Is it there because it's correct, or is it correct because it's there? We're begging the question here.
One book in game says they were of average height... one. At least one other says they were the size of "large children". None of the in-game appearances of dwemer have been conclusive on this point.
"Dwemer are of normal height, giants called them that because they were short to them" is very widely believed to be true, but it has no more backing that "dwarves were dwarves."