See Selected Projects

Science & discoveryJapan

Stress-test AI-written research before you trust it

Sakana says a paper written by its AI Scientist-v2 scored above the acceptance threshold (6, 7 and 6) at an ICLR 2025 workshop, then withdrew it before meta-review, as agreed.

Try it withClaudeChatGPT / OpenAI modelsGeminiGrok

Research, Sakana AI, with researchers at the University of British Columbia and the University of Oxford

Why it matters

It is an honest test of where AI-generated science stands: good enough, Sakana says, to score above one workshop's acceptance threshold, with clear limits. Useful for anyone deciding how much to trust AI research.

How it works

With the agreement of ICLR leadership and the workshop organisers, Sakana submitted three AI-generated papers for double-blind review. The one that passed averaged 6.33 and was withdrawn before publication, as planned (Sakana). TechCrunch notes that workshop acceptance rates tend to be higher than for the main conference track.

Who it's for

Researchers, editors and research-integrity teams.

The prompt

You are a skeptical reviewer for [venue]. Review the paper I paste below. Only paste a paper you wrote or have permission to share: many venues forbid putting confidential submissions into AI tools.

1. Summarise its claim in 3 sentences.
2. Check every number in the abstract against the results section and list mismatches.
3. List missing baselines, ablations or statistics, and any claims the experiments do not support.
4. Check that each cited paper exists and says what the text claims. If you cannot look a paper up, mark it 'not verified' and do not guess.
5. Score it from 1 to 10 using the venue's criteria, with your confidence.

Be specific and quote the passages you criticise.

[Paste the paper]

Open inClaude (opens in a new tab)ChatGPT (opens in a new tab)Gemini (opens in a new tab)Grok (opens in a new tab)

Sources

  1. 1. Sakana AI: Scores 6/7/6; withdrawn by protocol; no meta-review. The AI Scientist generates its first peer-reviewed scientific publication (opens in a new tab)

  2. 2. TechCrunch: Three papers submitted; workshop acceptance rates higher; no meta-review. Sakana claims its AI paper passed peer review, but it's a bit more nuanced (opens in a new tab)

Related projects