I think people most always have this misconception that people who are not afraid to voice out how they feel about a matter are being rude. But they don't understand that not everyone will see something going wrong and keep mute about it.
I'm glad you understand that people will always have something to say, you can't always give an explanation to everyone. When we know we're living right, then what people say won't bother us at all.