You're right about some of the owners of HE token being able to change the rules of the game. But most times we know who these guys are, the fact that we choose to own their token or buy them in the first place shows just how much we actually trust them.
However, you're right, it's one downside of HE token, the heavy centralization