One report states that the aboriginal men were taller by a head and neck, on average, than the Dutch. (The average Dutchman came only to the shoulder of the average native?) Another report describes the aborigines as tall and sturdily built, like semi-giants. This paper on historical Dutch height suggests that 17th century Dutchmen were about 170 cm or so on average. Holland was the richest country in Europe at the time, but nutritional conditions for average people were still not good by modern standards. So how tall were the aborigines? Presumably well above 180cm since "a head and neck" would be at least 20cm! (Some Native Americans were also very tall when the Europeans first encountered them.)
But, strangely, the descendants of these aborigines are not known for being particularly tall. This paper reports that modern day aboriginal children in Taiwan are shorter than their Han counterparts. On the other hand, the Dutch are now the tallest people in the world, with average male height exceeding 6 feet (183 cm). This kind of reversal makes one wonder whether, indeed, most groups of humans have similar potential for height under ideal conditions, as claimed here. (Note the epigenetic effects -- several generations of good nutrition might be required for a group to reach its full height.)
In the nineteenth century, when Americans were the tallest people in the world, the country took in floods of immigrants. And those Europeans, too, were small compared with native-born Americans. Malnourishment in a mother can cause a child not to grow as tall as it would otherwise. But after three generations or so the immigrants catch up. Around the world, well-fed children differ in height by less than half an inch.* In a few, rare cases, an entire people may share the same growth disorder. African Pygmies, for instance, produce too few growth hormones and the proteins that bind them to tissues, so they can’t break five feet even on the best of diets. By and large, though, any population can grow as tall as any other.
* I'm not sure where this statement comes from, since, for example, Japanese still seem to be a few inches shorter than, say Europeans. But it's also true that even the modern Japanese diet is lower in protein and calcium than the corresponding European or American one.