You can partially use Tailscale but yes, to use it you need to install it in all your devices.
At first I had it installed and just had a few use cases (mainly PiHole with MagicDNS which requires installing it in all your machines). But I still connected to ssh with the ip of the machine.
After I finished that initial set up then I closed all the ports and only allowed ssh from the tailnet.
I think you can get away with installing it just in your reverse proxy and in the machine you want to point to if you have solved the issue of the DNS.
One thing to have in mind with this approach is that you won't benefit from the routing of wireguard, since all your traffic needs to pass through the reverse proxy, a better approach would be to have each service with its own tailnet ip.
The only issue I've found is if you have a container in a machine with tailscale, the container doesn't know about the tailnet unless you have host network type or some other way to share a network which actually knows about tailscale.