The thing is, nobody knows if you write a special character or not. They might try to brute force passwords without first. But the way more important factor is length.
When I wasnt using a password manager for password creation I just did random sentences that were not allowed to have meaning. They were quite easy to memorize because they were so absurd. Usual password check would be 3 trillion trillion years or something.