cultural things that similar-sized other towns don’t have
Exactly. Similar sized. College towns punch above their weight when compared to their population peers, but that only goes so far. I have no doubt Pullman, Washington is cooler and more cosmopolitan than Walla Walla (despite the presence of two very small colleges), but it's no Seattle, for good or for ill, depending on your perspective.
Is "college town" agreed to be a denigration? I'd take it as a fairly complex descriptor that could be good or bad depending on your situation. I loved living in college towns. I'm not desperate to move back to one, but I could easily see myself retiring in one, and if you want a small town with more cultural and sporting options and a better educated populace than its peers, then putting up with some rowdy undergrads and a quirky mix of available businesses could be a perfectly sensible tradeoff.