They did not say that there's no relation to politics at all. But we do not need to hold a particular political position to agree that the government should not censor people's opinions.
It only starts to become a right-wing talking point when liberally applied to everything else. When even government officials argue your opinion should be censored, because it is critical of their opinion. That has nothing to do with the actual free speech principle. Quite the contrary.