Cannot fly with elytra

If my crosshair is pointing at a nearby block that I can break, it prevents me from launching into flight with the Elytra. If I somehow manage to start flying, the game teleports me back to the spot where I tried to take off