Perception is funny. We have no other way to experience the world then through our own eyes. The things you know about people are true, until they're suddenly not.
My sister makes a concerted effort to not assume people are LDS. Suddenly she goes from everyone having about the same beliefs and morals to very few being like her in that respect.
The first time I realized that other people see the world differently then I do, I was blown away. I had a friend in high school who was very bitter and convinced the world was against her. I couldn't understand how she could see things that way, but because she did, that's how it was.
I used to worry about what the world was really like if everyone saw different things. I still do sometimes. People who who are absolutely convinced that they're doing everything right, when I can plainly see the miss steps taken. Those are the people that worry me now.
If you believe in God, you can at least say to yourself “They'll get their just reward after this life.” but if you don't believe, then it's likely nothing will happen. Justice is a construct of man and has no place in nature, bad people do bad things and get away with it.
Look people in the eye, shake hands with a firm grip and be good to the women in your life. Everything else works itself out.