21h ago

Google Co-Scientist AI cracks superbug problem in two days! — because it had been fed the team’s previous paper with the answer in it

pivot-to-ai.com Google Co-Scientist AI cracks superbug problem in two days! — because it had been fed the team’s previous paper with the answer in it

The hype cycle for Google’s fabulous new AI Co-Scientist tool, based on the Gemini LLM, includes a BBC headline about how José Penadés’ team at Imperial College asked the tool about a problem…

TechTakes @awful.systems

David Gerard @awful.systems

21h ago

Google Co-Scientist AI cracks superbug problem in two days! — because it had been fed the team’s previous paper with the answer in it

pivot-to-ai.com /2025/02/22/google-co-scientist-ai-cracks-superbug-problem-in-two-days-because-it-had-been-fed-the-teams-previous-paper-with-the-answer-in-it/

153 7

3 comments

I was always pretty convinced that this was at the root of a lot of those "ChatGPT can pass the bar exam" and similar things that required AGI capabilities which it clearly doesn't possess. I think I actually tried making up similar-style math problems and asking them, and it failed horribly. Clearly I should have written up my results in a paper and been hailed as a visionary.

Perhaps it's not exactly equivalent since this is an LLM, but from what I've learnt in my undergrad machine learning course, shouldn't the test data be separate from the training data?
The train-test (or train-validate-test) split was one of the first few things we learnt to do.
Otherwise, the model can easily get a 100% accuracy (or whatever relevant metric) simply by regurgitating training data, which looks like the case here.
- but that won't trick investors into funding more of it