We'd have to stop using any terms that include colors: redfaced (ashamed), red with rage, blackballed, black humour, black magic, black eye, black sheep, yellow journalist, red line, yellow card, red card, brown-noser, brown thumb, black-and-white thinking, black out, black as night, etc.