Animals can climb ladders!

> Any mob can climb a ladder in the same way a player does: by pushing against it. Mobs are not smart enough to deliberately use ladders — they won't stay on a ladder to get somewhere, but a ladder directly in their straight-line path will let them climb up.

http://www.minecraftwiki.net/wiki/Ladder

I got around this problem by putting the ladder 3 blocks above ground level. Players can jump up onto it, but too high for animals.