As a child, we do not recognize. Hence, we do not judge either. Recognition is a must for judgment.
As we grow up and become adults, we learn to recognize. Hence, we start to judge also. We play games with people in order to benefit us someway or the other. Of course, not with everyone and not all the time. Nevertheless, teachers love smart kids, people like to throw names of the powerful people whom they know etc etc. This is definitely a stage that most of us go through. Unfortunately most of us do not go forward and instead remain there.
The next stage is the spiritually awakened state where an individual recognizes but does not judge. Most differences between humans are understood to be arbitrary and dependent on a series of accidents. All relationships are with the purest essence of the individual and hence no differences are recognized between individuals! Of course at this state, we might stay away from a 'troublesome' person but the constant internal anger directed at them will be missing. Unless one spiritually moves forward, it is difficult to do so.