Not arguing with the crux of your argument here, but most fast traveling I've done is way more direct than that. New planet, sure there's a few stages, but anywhere you've been before you can pretty much fast travel to directly from anywhere.
Bout the same for me with a 5700xt, 3700x and 16gb ram. Get a solid 35-45 at 1440p on medium/high (wee bit less than I got I'm bg3) though I'm gonna need to dick around with the rendering scale to get rid of distant blurriness. I'm usually a gen or 2 behind on GPU so maybe I'm just used to it at this stage.