Check for the ssh env vars. For example, I use PS1="${SSH_CONNECTION:+\u@\h:}\W\$" to hide the hostname when not on SSH, you could do something similar with the control codes for color.