Steps to reproduce:
Place jukebox
Place record in jukebox
Close world
Reopen world
Result: The music will keep playing, no matter if the record is removed, the block is broken, the game is paused, the world is closed again, etc.
Going away does actually make the sound quieter/makes it stop.
Linked issues
Comments

Cannot reproduce in 1.13-pre5, please check if the issue is persisting.