If you do not neglect updates, then by all mean, changing ports does not hurt. :) Sorry if I may have strong reaction on that, but I've seen way too many people in the past couple decades counting on such anecdotal measures and not doing the obvious. I've seen companies doing that. I've seen one changing ports, forcing us to use the company certificate to log in, and then not update their servers in 6 months. I've seen sysadmins who considered that rotating servers every year made it useless to update them, but employees should all use Jumpcloud "for security reasons"! Beware, though, mentioning port changing without saying it's anecdotal and the most important thing is updates, because it will encourage such behaviors. I think the reason is because changing ports sounds cool and smart, while updates just sound boring.
That being said, port scanning is not just about targeted pentesting. You can't just run nmap on a host anymore, because IDS (intrusion detection systems) will detect it, but nowadays automated pentesting tools do distributed port scanning to bypass them : instead of flooding a host to test all their ports, they test a range of hosts for the same port, then start over with a new port. It's half-way classic port scanning and the "let's just test the whole IP range for a single vulnerability" that we more commonly see nowadays. But they are way harder to detect, as they scan smaller sets of hosts, and there can be hours before the same host is tested twice.
I'm not sure about the feasibility of this (my first thought would be that ssh on the host can be accessed directly by IP, unless maybe the VPN software creates its own network interface and sshd binds on it?), but this does not remove the need for frequent updates anyway, as openssh is not the only software that could have bugs : every software that opens a port should be protected as well, and you can't hide your webserver on port 80 behind a VPN if you want it to be public. And it's anyway a way more complicated setup than just doing updates weekly. :)