Linux

Strange enough TLS 1.3 still doesn't support signed ed25519 certificates :| P‐256, NIST P‐384 or NIST P‐521 curves are known to be "backdoored" or having deliberately chosen mathematical weakness. I'm not an expert and just a noob security/selfhoster enthusiast but I don't want to depend on curves made by NSA or other spy agencies !

I also wondering if the EU isn't going to implement something similar with all their new spying laws currently discussed...

Easiest and most affordable is probably a security key like the Nitrokey or the https://www.yubico.com/. I personally don't like the company behind yubikey much, but if you want something small you can always leave in the device that's pretty much your only option.

For "cheaper, but a bit more effort" would be just getting a smartcard blank, a card reader (if you're not lucky enough to have a notebook or computer with one built in), and then either write your own applet, or use one of the available opensource ones, and upload it to the card. A variant of that would be the Fidesmo card, where you get a card and their applet.

Or you just use the TPM you may have in your system - though you'll need to be careful with that: Typically one reason for using a hardware token is to make sure keys can't get extracted, while TPMs often do allow key extraction. Software to make that work would be opencryptoki.

Generally you'd use PKCS#11 to have the various components talk to each other. On your average Linux pretty much everything but GnuPG place nice. with PKCS#11. Typically you end up with pcscd to interface with the smartcard (the above USB tokens are technically also just USB smartcards), OpenSC as layer to provide PKCS#11 on top, and software (like OpenSSH) then talks to that.

All of that should be available as packages in any Linux distribution nowadays - and typically will also provide p11-kit configured to use a proxy library to make multiple token sources easily available, and avoid blocking on concurrent access.

ssh-add supports adding keys from pkcs#11 providers to the SSH agent (search pkcs11 in ssh-add manpage), with some distribution (like RedHat) also carrying patches allowing you to only select individual tokens for adding.

If you're also using GnuPG it gets more complicated - you pretty much have two options: Stick with PKCS#11, in which case you'd replace GPGs own smartcard agent with gnupg-pkcs11-scd, or you use GPGs own card implementation, in which case you can forget pretty much everything I wrote above, and just follow the security key manual for setting up a GPG card, enable SSH agent support in the GPG agent, and just use that for SSH authentication.

I've just started using SSH inside of Termux, got tired of all the weird pitfalls SSH Clients for Android usually have

Probably. It's in f-droid but increasingly looking not quite unmaintained, but not developed actively enough.