Voting isn't designed to be popularity-based just to sit there in the comment section of a post, there are actual processes involved in which they play a role in how content is viewable.
I thought you were saying I can just view it differently and that this would solve things. Or if you were saying others could, it's not like they're going to just because they can. That's the problem.