I think it might be both their own height, and the median-ish height of the society they're in.

Like I'd imagine even a small person in a generally-tall society is probably going to eventually stop commenting on the height of everyone else, because they know they're the odd one out (by height)