I’ve confirmed there are 2 combined issues, which can be circumvented, but a proper fix would be nice.
TCP must be HTTPS and you must specify your external IP with the UDP port.
1: Some clients (like the PS5) require the TCP connection to be HTTPS when connecting over the internet. The server only speaks http, the client only https, so you’ll need a proxy and a domain/dns with a certificate.
(On the PS5 a local connection does work over http and the windows client has no problem connecting over the internet using http)
if curl https://<yourdomain>:<yourport>/v1/join does not show a proper (json) response, the client PS5 client will not connect to your server over the internet and show a “Door” error.
2: The server is said to advertise its IP address together with the UTP port. I have not verified this, but the solution works, so it might be true. If you don't include an IP address in the server-udp-ports property in server.properties, it will include it’s own ip address, which is a local ip-address if you’re behind a NAT router. That works on your LAN, but not over the internet. The solution is to include your external IP address in the setting.
If the IP address in the UDP port setting is wrong (or absent), the client wont' connect and give a “Boat” error.
Also make sure to set a single port, not a range of ports. The range also seems to contain a bug. I have not found a benefit of setting a range, but if you really want to, you should do so by repeating the property for every port separately. I would think they would overwrite each other, and I have not verified this.
And the PS5 client seems to cache the data for about 5 minutes, so restart it and give it some time. If the client just returns to the menu, or displays the “Terracotta” error, that’s probable caching. Be patient and retry.
PS. online-mode and allow-list should both be set to true.
Another thing which works (at least for PS5 clients) is to put both TCP and UDP on the same port and apply only the second fix (IP-address in server-udp-ports). I don’t know why this works, maybe it’s not requiring HTTPS or maybe it's not using TCP at all, but it saves you having to set up DNS and an SSL certificate.