Stress-test AI-written research before you trust it
Sakana says a paper written by its AI Scientist-v2 scored above the acceptance threshold (6, 7 and 6) at an ICLR 2025 workshop, then withdrew it before meta-review, as agreed.
Try it withClaudeChatGPT / OpenAI modelsGeminiGrok
Research, Sakana AI, with researchers at the University of British Columbia and the University of Oxford
Why it matters
It is an honest test of where AI-generated science stands: good enough, Sakana says, to score above one workshop's acceptance threshold, with clear limits. Useful for anyone deciding how much to trust AI research.
How it works
With the agreement of ICLR leadership and the workshop organisers, Sakana submitted three AI-generated papers for double-blind review. The one that passed averaged 6.33 and was withdrawn before publication, as planned (Sakana). TechCrunch notes that workshop acceptance rates tend to be higher than for the main conference track.
Who it's for
Researchers, editors and research-integrity teams.
The prompt
You are a skeptical reviewer for [venue]. Review the paper I paste below. Only paste a paper you wrote or have permission to share: many venues forbid putting confidential submissions into AI tools. 1. Summarise its claim in 3 sentences. 2. Check every number in the abstract against the results section and list mismatches. 3. List missing baselines, ablations or statistics, and any claims the experiments do not support. 4. Check that each cited paper exists and says what the text claims. If you cannot look a paper up, mark it 'not verified' and do not guess. 5. Score it from 1 to 10 using the venue's criteria, with your confidence. Be specific and quote the passages you criticise. [Paste the paper]
Open inClaude (opens in a new tab)ChatGPT (opens in a new tab)Gemini (opens in a new tab)Grok (opens in a new tab)
Sources
1. Sakana AI: Scores 6/7/6; withdrawn by protocol; no meta-review. The AI Scientist generates its first peer-reviewed scientific publication (opens in a new tab)
2. TechCrunch: Three papers submitted; workshop acceptance rates higher; no meta-review. Sakana claims its AI paper passed peer review, but it's a bit more nuanced (opens in a new tab)