When referring to people, I always saw "white" and "black" as a connotation of race, not really color. What we call "white people" aren't really the color white, and what we call "black people" aren't really the color black. Kind of like how in the US the term "brown" has been adopted by people of (I'll call it) "South of the Border" descent.
BTW, who old are you, OP?

Usually, when someone asks this question, it is meant as a put down inferring the other person is young, ignorant, naive, etc... but in this instance, you have to be on the older side of the age spectrum. The last time I heard anyone refer to an person of Asian origin as "yellow" in real life was at the end of the Vietnam War

,when I was old enough to understand
Got in a little hometown jam, so they put a rifle in my hands...
