The card companies can get data from the Merchant Category Code to infer the nature of purchases, without specifics. The stores also have a record of what items you bought, which could also be sold unless you have a contract with the store that guarantees they won't sell your purchase history (at least in the countries without strong privacy laws)
That's per store though, presumably when they sign up with a payment provider (because there's a lot of rules about e.g. using credit cards to gamble with).
If I buy sex toys from Tesco, it's still showing up as "groceries". If I buy from a sex shop, it's going to be more clear cut.
I can see from my emails that PayPal send out itemised receipts on behalf of their customers, so they're definitely collecting more data than the big two.