For anyone reading this who might have anxiety, I had this problem and it just turned out that I really need better support during the day. Boxers weren’t cutting it.
With that said, yes definitely consider seeing a doctor.
Alternatively, as a form of scrotal elephantiasis, it could be a parasitic infection, swollen lymph nodes from either disease or medical procedures like radiotherapy, or more rarely a hernia that causes the intestines to drop into the scrotum.