Your last sentence is correct. We don't have obvious patterns that would cause us any confusion (beyond the confusion we already have with this measurement system that makes no sense), so we simply memorize it.
I can't believe after all these decades, the USA still sticks with the imperial system. It's nonsense. But I grew up with the imperial system so that's what feels natural to me and I can "feel" what a mile feels like, I can "feel" what an inch feels like, but if you speak to me in metric, since we in the USA are not as exposed to it over here, I need to pull out my calculator to make the conversions to understand how a meter relates to a foot and a yard, yes I hate it, I would rather be able to think and feel in metric because it's more logical.