They usually follow their own policies, so they're, by definition, right. Most things that passengers complain about are, in fact, safety related. People don't get what it is they're actually doing. They're getting into a pressurized aluminum tube and shooting through the sky ~6 miles above the ground at ~500 miles an hour. Things that would be a minor annoyance on foot, or in a car, could kill you in that situation.
Consider an airplane that didn't have any epinephrine on board (this was the case a few years ago when there was a shortage). A passenger with a peanut allergy doesn't have their own epi pen, and is removed from the flight by the captain. What do you think their reaction will be? Probably something like, "OMGWTFBBQPWNSAUCE they discriminated against me and denied me boarding because I have a condition OMGOMGOMGSUESUESUE I HATE THIS AIRLINE THEY'RE THE WORST EVER!!!" What's going to happen to them if their seat neighbor busts open some peanuts and they go into anaphylactic shock? They're probably going to die. There is literally nothing that can be done. You can't pull over, you can't call an ambulance, you can't separate them from the problem. They were removed from the flight because their life would be in danger.
How about the canceled flights due to fog? Invariably there are passengers that complain, "my wife says it's clear there," and things to that effect. Well, it's not clear at the airport, and if we try to land in that fog, YOU COULD DIE.
Or cancelled for thunderstorms, "OMG, YOU'VE RUINED MY LIFE, I HATE THIS AIRLINE, WHY CAN YOU JUST GO." Because that thunderstorm could kill you.
"OMGWTFBBQ they canceled my flight because of snow, I can drive in snow, why can't they fly in it?" Have you tried landing a 150 ton pressurized aluminum tube on a snow or ice covered runway at 150 miles a hour? You could die. They don't make chains for that, and they wouldn't help anyway.
The reason airlines are so safe is because they don't take stupid risks. Most passengers don't even realize what those risks are and don't consider the consequences.