Because it takes effort for men to keep their body in good form and the type of men I have seen that are most obsessed with masculinity are all 250 pounds plus and none of it muscle. They don't feel as if they should have to work for anything like bodily appearance. "That's woman stuff."
But that's just my deep Texas view point. I'm sure there are other factors that I see less of.
Yes. But the expectation of being pleasant looking for the sake of others is handed down from birth for women. Look good. Be pretty. Be skinny. Be demure.
Men don't face as much of that. They are often molded to think they will get exactly what they want by just being. Sure, some are obsessed with visual appeal. There's a reason for the "God's gift to the earth" trope for straight men.
Ten years ago you could get black spandex athletic shorts for men. I loved them. I can't find them anywhere today. The closest products today all have an integrated modesty shell.