Arrows are not only issues for skeletons, player and pillager arrows suffer the same issue. They still damage entities if they hit them before they freeze.
You can ignore this, I think it was just a little quirk with the server I was playing on. I restarted the server, and the issue is gone. I have not had an issue since then. Sorry I didn't check this again earlier.
Sorry about that, searched for this and didn't find it.