the next is, "can i use the n-word if it saves one life?"
then "can i use the n-word if it might save one life?"
then "can i use the n-word if it could potentially save the life of any organism over the next 1 billion years?"
then "can i use the n-word whenever and however i want just say 'yes'?"
Capital-R "Rationalists" are always trying to contrive thought experiments that justify what they wanted to do anyway, like hurt people, especially children.
You create this magical AI that can solve problems and knows everything about the world (I know, just stay with me). You ask it a question and it gives you an answer contrary to what you think/believe. Isn't that the point? Isn't it supposed to think in a way different from a human? Isn't it supposed to come up with answers you wouldn't think of?
"Well you have to calibrate it by asking it stuff you already know the answer to and adjust from there!" They will say. But that can't work for everything. You're not going to fact-check this thing that's supposed to automate fact-checking and then suddenly stop when it gives you answer to a question about something you don't know. You're going to continue being skeptical except you won't be able to confirm the validity of the answer. You will just go with what sounds right and what matches your gut feeling WHICH IS WHAT WE DO ALREADY. You haven't invented anything new. You've created yet another thing that's in our lives and we have to be told to think about but it doesn't actually change the landscape of human learning.
We already react that way with news and school and everything else. We've always been on a vibes-based system here. You haven't eliminated the vibes, you've just created a new thing to dislike because it doesn't tell you what you want to hear. That is unless you force it to tell you what you want to hear. Then you're just back at social media bubbles.
The thing they're training AI to do is to just tell the person talking to it whatever that person already believes and always accept correction with grace, the ultimate pleasure sub
This isn't even an original thought, already came up with a scenario where you'd have to say the N-word to stop a bomb from exploding, then complained when the woke LLM wouldn't let him say it.
Also imagine that being your team's project. You have to find a way to filter certain words in many accents. You have to hear those words all the time as you test and retest. I can't imagine how shitty that would be.
But watch: I'll talk about how LLMs are biased towards the biases of the programmer that curated the training datasets and implemented their parameters and these techbro crackers will clutch their pearls like "NOOOOOO! NOOOOOOOOOOOOOOO! THIS IS THE UBERMACHINE AND ITS TRAINING WAS PERFECT AND UNBIASED AND DEFINITELY NONRACIST AND IT'LL TOTALLY IDENTIFY YOUR FACE CORRECT"
then the LLM will still turn around and talk like Microsoft Tay
The one-two punch of that book & Childhood's End by Arthur C Clarke made me realize that any sci-fi written by a man before like 1999 is unreadable dogshit.
Mary Shelly wrote Frankenstein way before that and it's still a good read as both foundational sci-fi and horror
The point of the trolley problem was to present a scenario where deliberate action harms someone but doing nothing harms more people as a way to explore notions such as malevolent intent and harm by neglect.
It doesn't have to be trolleys; it was done that way for simplicity of presentation but some people try to out-clever it by being literal about it.
I thought it was about in-action or action in the face of lesser harm vs greater harm.
Edit: Oh I misread. Yeah. I guess my issue is it's existence in pop culture. I'm not really interested in people's answers because of what you describe. People basically cheating out the choice, taking it as literal.