AI Lie: Machines Don’t Learn Like Humans (And Don’t Have the Right To)
AI Lie: Machines Don’t Learn Like Humans (And Don’t Have the Right To)
Avram Piltch is the editor in chief of Tom's Hardware, and he's written a thoroughly researched article breaking down the promises and failures of LLM AIs.
I like the point about LLMs interpolating data while humans extrapolate. I think that's sums up a key difference in "learning". It's also an interesting point that we anthropomorphise ML models by using words such as learning or training, but I wonder if there are other better words to use. Fitting?
"Plagiarizing" 😜
Isn't interpolation and extrapolation the same thing effectively, given a complex enough system?
Depending on the geometry of the state space, very literally yes. Think about a sphere, there's a straight line passing from Denver to Guadalajara, roughly hitting Delhi on the way. Is Delhi in between them (interpolation), or behind one from the other (extrapolation)? Kind of both, unless you move the goalposts to add distance limits on interpolation, which could themselves be broken by another geometry
No, repeated extrapolation results in eventually making everything that ever could be made, constant interpolation would result in creating the same "average" work over and over.
The difference is infinite vs zero variety.
What about tuning, to align with "finetuning?"
I also like the point about interpolation vs extrapolation. It's demonstrated when you look at art history (or the history of any other creative field). Humans don't look at paintings and create something that's predictable based on those paintings. They go "what happens when I take that idea and go even further?" An LLM could never have invented Cubism after looking at Paul Cezanne's paintings, but Pablo Picasso did.
Let's be clear on where the responsibility belongs, here. LLMs are neither alive nor sapient. They themselves have no more "rights" than a toaster. The question is whether the humans training the AIs have the right to feed them such-and-such data.
The real problem is the way these systems are being anthropomorphized. Keep your attention firmly on the man behind the curtain.
Yes, these are the same people who are charging a fee to use their AI and profiting. Placing the blame and discussion on the AI itself conveniently overlooks a lot here.
There's a lot of opinion in here written in as if it's fact.
Here I was thinking I could trust Mr Tom
Machines don't Lear like humans yet.
Our brains are a giant electrical/chemical system that somehow creates consciousness. We might be able to create that in a computer. And the day it happens, then what will be the difference between a human and a true AI?
If you read the article, there's "experts" saying that human comprehension is fundamentally computationally intractable, which is basically a religious standpoint. Like, ChatGPT isn't intellegent yet, partly because it doesn't really have long term memory, but yeah, there's overwhelming evidence the brain is a machine like any other.
fundamentally computationally intractable
...using current AI architecture, and the insight isn't new it's maths. This is currently the best idea we have about the subject. Trigger warning: Cybernetics, and lots of it.
Meanwhile yes of course brains are machines like any other claiming otherwise is claiming you can compute incomputable functions which a physical and logical impossibility. And it's fucking annoying to talk about this topic with people who don't understand computability. Usually turns into a shouting match of "you're claiming the existence of something like a soul, some metaphysical origin of the human mind" vs. "no I'm not" vs. "yes you are but you don't understand why".
Bruh, the LLM will be trained on 90% click bait. It's gonna be just as trashy.
🤖 I'm a bot that provides automatic summaries for articles: ::: spoiler Click here to see the summary Unfortunately, many people believe that AI bots should be allowed to grab, ingest and repurpose any data that’s available on the public Internet whether they own it or not, because they are “just learning like a human would.” Once a person reads an article, they can use the ideas they just absorbed in their speech or even their drawings for free.
Iris van Rooj, a professor of computational cognitive science at Radboud University Nijmegen in The Netherlands, posits that it’s impossible to build a machine to reproduce human-style thinking by using even larger and more complex LLMs than we have today.
NY Times Tech Columnist Farhad Manjoo made this point in a recent op-ed, positing that writers should not be compensated when their work is used for machine learning because the bots are merely drawing “inspiration” from the words like a person does.
“When a machine is trained to understand language and culture by poring over a lot of stuff online, it is acting, philosophically at least, just like a human being who draws inspiration from existing works,” Manjoo wrote.
In his testimony before a U.S. Senate subcommittee hearing this past July, Emory Law Professor Matthew Sag used the metaphor of a student learning to explain why he believes training on copyrighted material is usually fair use.
In fact, Microsoft, which is a major investor in OpenAI and uses GPT-4 for its Bing Chat tools, released a paper in March claiming that GPT-4 has “sparks of Artificial General Intelligence” – the endpoint where the machine is able to learn any human task thanks to it having “emergent” abilities that weren’t in the original model.
Saved 93% of original text. :::
That's a philosophical debate we can't really answer and not a lie, the question is if we do anything other than copy. The without any doubt biggest elephant in the room is the fact that AIs don't remember and iterate like we do yet but that's probably just a matter of time, other than that the very different environment we learn in is another huge issue if you try to make any comparison. It's a tricky question that we might never know the answe to but it's also facinating to think about and I don't think rejecting the idea alltogether is a especially good answer.
They have the right to ingest data, not because they're “just learning like a human would". But because I - a human - have a right to grab all data that's available on the public internet, and process it however I want, including by training statistical models. The only thing I don't have a right to do is distribute it (or works that resemble it too closely).
In you actually show me people who are extracting books from LLMs and reading them that way, then I'd agree that would be piracy - but that'd be such a terrible experience if it ever works - that I can't see it actually happening.
Two things:
I'm sick and tired of this "parrots the works of others" narrative. Here's a challenge for you: go to https://huggingface.co/chat/, input some prompt (for example, "Write a three paragraphs scene about Jason and Carol playing hide and seek with some other kids. Jason gets injured, and Carol has to help him."). And when you get the response, try to find the author that it "parroted". You won't be able to - because it wouldn't just reproduce someone else's already made scene. It'll mesh maaany things from all over the training data in such a way that none of them will be even remotely recognizable.
Is there a meaningful difference between reproducing the work and giving a summary? Because I’ll absolutely be using AI to filter all the editorial garbage out of news, setup and trained myself to surface what is meaningful to me stripped of all advertising, sponsorships, and detectable bias
You're making two, big incorrect assumptions:
I know these are really tough pills for AI fans to swallow, but you know what they say... "If it seems too good to be true, it probably is."
One the contrary - the reason copyright is called that is because it started as the right to make copies. Since then it's been expanded to include more than just copies, such as distributing derivative works
But the act of distribution is key. If I wanted to, I could write whatever derivative works in my personal diary.
I also have the right to count the number of occurrences of the letter 'Q' in Harry Potter workout Rowling's permission. This I can also post my count online for other lovers of 'Q', because it's not derivative (it is 'derived', but 'derivative' is different - according to Wikipedia it means 'includes major copyrightable elements').
Or do more complex statistical analysis.