Yes in my opinion it's crazy to have separate ports for v4 and v6 for exactly the reason the OP describes. The only way around it is to have separate DNS records like v4.example.com resolving to the v4 address and v6.example.com resolving to the v6 address. Nobody does this it's insane. Please try to just have the server run one one port.
Gregory Dunbar
Yes in my opinion it's crazy to have separate ports for v4 and v6 for exactly the reason the OP describes. The only way around it is to have separate DNS records like v4.example.com resolving to the v4 address and v6.example.com resolving to the v6 address. Nobody does this it's insane. Please try to just have the server run one one port.