When catching treasure in non open water, the treasure loot table is still used, but drops nothing.
To reproduce, use the following fishing rod, and fish in "non open" water:
/give @p minecraft:fishing_rod{Enchantments:[{id:luck_of_the_sea,lvl:1000},{id:lure,lvl:5}]}
Notice that you always catch nothing.
Related issues
Comments

It is possible with higher levels of the luck effect itself, which are obtainable using the /effect command. (tested in 20w13a)
I've fished for over an hour in non-open water using a luck of the sea 3 fishing rod, and I didn't fail to catch anything, so this is a duplicate of MC-10755.
The glitch only seems to happen at luck of the sea 85+, upon further testing.

Can confirm.
Is this also possible with lower levels of luck of the sea? Levels higher than 3 are not supported (see MC-10755)