It's not that hard to get both for most people. The phone book whitepages used to list those. There are various services that aggregate them now.
Given that it's not all havoc out there, probably not trivially.
But...by the same token, there's nothing intrinsically about knowing your physical address that would let someone do that. But...they could go firebomb your house or something.
Usually they can look up your address if your name is in any way unique or uncommon, especially if they have your phone number as well. Some sites may show associated email addresses, then there’s things like LinkedIn (if you use it) or other social media sites that link to a name.
okay well I'm not worried if anybody finds my address. I live a thousand miles away from that address.
I guess the thing I'm mostly worried about is "phone number spoofing," something related to you know how you get two factor authentification with certain websites where they send you a phone call or a text to make sure it's really you? Well some people have figured out a way to get that 2FA sent to themselves after they spoof your phone number, They can do this with your bank account and drain it, things like that. and this is spooking me out because I want to make sure it's impossible for anyone to do this to me.