By using "that's gay" as a negative, there is an association that gay=bad regardless of context. I don't feel that way but I understand other people's perception of the phrase.
I think also the context genuinely matters. As a straight male, I personally would use “that’s gay” to point out something that is not mainstream normal, yet cool or cute; possibly even kitschy.