To me this seems like a routing issue. Some things to check:
- Can a port checking service (like canyouseeme) see 443?
- Can you connect to 443 from the internet (use a VPN or separate network)?
- Can you edit your hosts file to point
jellyfin.mydomain.com
to your jellyfin server's LAN IP? - After doing this, can you connect to
jellyfin.mydomain.com
on the LAN?
My guess is the router is routing traffic to it's external IP from the LAN back to itself, without following port forwarding rules. Good luck figuring it out though!