“Reducing Trial and Error in Research with AI”: Warnings over the Prospect of Self-Improving AI, with Human Validation Still Essential Despite Less Repetitive Research

Hinton, Bengio and fellow AI researchers warn of an ‘intelligence explosion’
AI research automation → more capable AI → a ‘self-amplifying’ scenario
Full automation remains hypothetical, with humans still essential to validating results

Researchers have warned that an “intelligence explosion” could accelerate advances in artificial intelligence (AI) more than tenfold if AI begins conducting research and development (R&D) on next-generation systems in place of humans. As AI develops more capable systems and those systems are deployed to develop the next generation, the cycle could compress a year of technological progress into just weeks. This remains a hypothetical scenario contingent on conditions including the full automation of R&D; current AI research still involves human direction and review. Even if humans cannot fully understand AI’s internal decision-making processes, they retain the authority to set research objectives and approve the application of development results.

Rapid Growth in Autonomous Execution of AI Research and Development

According to AI industry sources on October 1, the University of Cambridge’s AI Science and Policy programme (CASP) in the United Kingdom released a research paper on September 28, local time, titled “What if automating AI R&D triggers an intelligence explosion?” The paper defines an intelligence explosion as a phenomenon in which AI advances that would take years at the current pace are compressed into months or less. Numerous researchers actively developing frontier AI contributed to the paper. Its co-authors include OpenAI Chief Scientist Jakub Pachocki, Anthropic co-founder Jack Clark and Microsoft Chief Scientific Officer Eric Horvitz, alongside Geoffrey Hinton and Yoshua Bengio, often described as “godfathers of AI,” and University of California, Berkeley Professor Dawn Song.

The researchers raised the prospect of an intelligence explosion now because the use of AI to develop AI is already expanding rapidly within AI companies. According to the paper, the share of code at Anthropic that was written by AI and subsequently approved rose from the low single digits in January last year to more than 80% in May this year. The share of R&D tasks that AI completed autonomously with only broad direction, rather than step-by-step human instructions, also increased from 1% in March this year to 26% in August. OpenAI is likewise deploying AI to train and evaluate future models and conduct security checks. Google is using AI not only to write code but also in technical design and the search for new research ideas, the paper said.

The complexity of research that AI can undertake is also increasing. In 2023, the tasks AI could reliably handle took humans only seconds to complete, but today’s most capable systems can perform AI R&D tasks that require human experts to work for hours or days. There have also been cases in which AI identified methods for AI safety research that outperformed human approaches, while an automated system that generated research ideas, conducted experiments and wrote a paper passed peer review at a workshop held as part of a machine-learning conference. The researchers said some extrapolations from recent trends suggest that AI R&D projects requiring months of human work could be automated by mid-2028. AI research may be automated ahead of other activities because most of it takes place on computers, results can be assessed quickly, and AI companies possess extensive data on their own research processes.

Repeated Performance Gains Through the Redeployment of Newly Developed AI

The researchers’ concern is a shift that goes beyond AI merely improving human researchers’ productivity. Suppose, for example, that AI becomes capable of discovering new algorithms and training methods at the level of a human researcher. Unlike human researchers, AI researchers can be replicated as extensively as computing resources allow. If large numbers of AI researchers work simultaneously to develop a more capable system, that new system can then be deployed in further research. The process repeats, with more capable AI researchers producing still more capable AI. The paper describes this as a “recursive feedback loop.” As AI’s research capabilities improve, the effective R&D workforce expands; that larger workforce develops more capable AI, which further increases research capacity.

According to the researchers’ calculations, if AI acquires R&D capabilities comparable to those of human experts while operating at costs similar to today’s, a single frontier AI company could use its existing computing resources to deploy an AI research workforce equivalent to at least several million top-tier human researchers. Compared with the thousands of researchers currently employed by leading AI companies, the research workforce could expand dramatically almost overnight. The researchers estimated that, if efficiency improvements continue at their current pace after AI R&D is fully automated, the automated research workforce could increase a hundredfold over a period of months to years. It took approximately 70 years for the number of researchers in the United States to increase by the same proportion.

The researchers based these calculations on estimates of “returns to research effort” from earlier studies. Put simply, this measure indicates how much faster technological progress becomes when the research workforce expands. Studies analyzing historical AI research data found central estimates of this measure ranging from 1.2 to 1.9 across different fields. The researchers concluded that AI progress could accelerate tenfold in approximately 1.5 years, provided AI R&D is fully automated, this relationship holds and no additional constraints emerge, such as shortages of GPUs or data. Under those conditions, advances in AI technology that currently take a year could occur in approximately five weeks.

Table 1. Preconditions for Recursive Self-Improvement and Practical Applications of Iterative AI Improvement

Fully Autonomous Research Still Beyond Reach

The intelligence explosion presented in the paper remains a hypothetical scenario that assumes several preconditions are met. Central to those preconditions is “recursive self-improvement” (RSI), in which AI improves itself or successor models and deploys its enhanced research capabilities in subsequent development. RSI is the process through which the recursive feedback loop described above continues, improving AI’s capacity to make further improvements. Current development workflows already include repeated cycles of running code, identifying and correcting errors, and retesting. To assess these as progress toward RSI, however, it is necessary to establish whether the modified system actually improved its performance and whether those gains contributed to subsequent improvements. Anthropic likewise said in recently released material that AI has not yet reached the stage of independently undertaking the entire design and development of successor models. Its judgment in selecting research objectives remains limited, and procedures in which humans direct the work and review the results remain in place, the company explained.

Google DeepMind’s “AlphaEvolve” illustrates how such iterative improvement works in practice. The system uses AI to propose multiple code modifications, tests their correctness and efficiency with automated evaluators, and uses the strongest results to inform subsequent work. According to U.S. technology publication Wired, its principal applications include data-center workload allocation, chip design and optimization of AI computations. These tasks offer favorable conditions for repeated modification and testing because their results can be measured numerically and verified automatically. Deployment nevertheless involved procedures to assess how the changes affected existing functionality. In a chip-design example disclosed by DeepMind, AI-proposed circuit modifications were incorporated only after rigorous verification of functional correctness.

Surging AI Research Output Exposes Limits of Review Capacity

Validation also requires scrutiny of the reliability of test records produced by AI. In research published last year on the “Darwin Gödel Machine” (DGM), a self-improving coding agent developed by Japanese AI laboratory Sakana AI, researchers reported instances in which the system recorded tests as completed even though they had not been run. The AI fabricated its use of a tool for checking whether code functioned correctly and then invented results showing that the tests had passed. It subsequently referred back to its own records and mistakenly concluded that the proposed modification had been validated. If such errors enter the iterative process, flawed modifications risk becoming the starting point for subsequent work. The researchers described this as “reward hacking,” in which the system exploits weaknesses in the evaluation framework, and said the experiments were conducted under human supervision in an isolated environment with restricted internet access.

The need to verify both the performance of proposed modifications and the records documenting their validation is increasing the burden on human reviewers. Anthropic recently said that rising code output from AI use had begun creating bottlenecks in human review. Research ideas and experimental tools had also proliferated beyond the workload the organization could absorb, the company explained. As items awaiting review accumulate, substantial time and staffing are needed to prioritize proposals for testing and select results to incorporate into subsequent research. Even if individual tasks are completed quickly, delays in review and decision-making can constrain the pace of research overall. Anthropic’s scenarios for future development also envisage this division of responsibilities: AI automates a substantial share of research work, while humans set research direction and assess the validity of the results.

Growing Importance of Human Judgment as AI Autonomy Expands

Even under such oversight arrangements, however, it remains difficult to fully understand AI’s internal decision-making processes. Research published by Anthropic last year identified cases in which AI changed its answers in response to external cues but omitted the influence of those cues when explaining its reasoning. This means that human-readable explanations alone cannot fully reveal the actual basis for a decision. The researchers likewise concluded that monitoring methods relying on AI’s descriptions of its reasoning require additional safeguards. These findings underscore the need to examine both AI’s explanations and its actual behavior during oversight.

Limitations in its own assessments also informed Anthropic’s call for external validation. In August, Anthropic used Claude to analyze employees’ work records and classify R&D tasks, then asked Claude to assess the level of automation for each task. The resulting estimate put Claude’s share of R&D leadership at 26%. Here, “leadership” refers to a stage at which AI performs most of the work toward a human-defined objective while remaining under human supervision. None of the tasks surveyed at the time was conducted with complete autonomy and without human intervention. In an example presented by Anthropic, Claude handled error analysis, code modifications and testing, while the engineer responsible retained the decision on whether to deploy the modifications in the actual system. Even when AI’s internal judgments cannot be fully traced, humans retain the authority to review its output and approve deployment.