I keep getting kicked out of the villager trade menu

When I try to open my custom Villagers GUI it kicks me out.(He is a custom villager that is a nitwit)

What I expected to happen was...:
I would be able to trade with him.

What actually happened was...:
I would see it for a second and then get kicked from the GUI

Steps to Reproduce:
1. Make a custom villager.
2. Try to trade with him.