I wouldn't go to one of their shows (it's not really the same band anymore, anyway), but if you stop listening to every band where a member is a piece of shit you're going to miss a lot of really good music.
He's a huge fucking asshole. He's also why I learned in my late 30s that his music WASN'T about being a bisexual atheist who mocked heteronormative Christianity. I was just projecting.
I can only listen to his music when I'm willing to think interpritation is as valid as intent.