Villagers finding spots and getting stuck

Villagers tend to find random spots in water or areas I'm building by and they get stuck there, if I move them they go back. 

Villagers should move around and not stay in the same spaces, especially in nearby bodies of water, swimming in circles above kelp. 

I've even tried moving them with fishing poles and pushing them but it rarely works.

1.  Find a village

2. Build near village

3. Villager gets stuck

Update: also seems to only happen in survival?