Most people unconciously associate the pause symbol (Two vertical lines) as currently playing and the playing symbol (sideways triangle) as paused.

That's a result of devs using the "pause" icon to indicate "paused". And when people tap on the "paused" icon to "play" something, it becomes associated with that.

Blame the UI devs who messed this up.