How to reproduce
Give yourself an item with a custom enchantment using:
/give @s diamond{Enchantments:[{lvl:1s,id:"custom:enchantment"}]}
Place it in an anvil and rename it
→ See that the enchantment glow is now gone
Notes
Anvils can also not apply these custom enchantments either, if you combine an item which isn't enchanted with a custom enchanted item. (Enchanted book, same type of tool, etc.)
This is not WAI, as letting players work with custom enchantments without having to use vanilla enchantments is a powerful tool, which is really nice to have. Please keep it!
This issue is related to MC-113127.