Original date published: August 6, 2025
A Case for Human-AI Collaboration Research
A case for Human-AI collaboration research in the era of independently thinking software agents
Assuming that Situational Analysis and AI 2027 are valid predictions of the future, we may soon reach a point where we create a separate species of intelligence that far surpasses human knowledge and capabilities in every economically valuable task, and even in some forms of abstract reasoning. Both predictions foresee a period of human ignorance and doubt before, ultimately, AI systems surpass human existence. Within technology circles, several theories have been proposed to mitigate these risks, though some dismiss them as science fiction or fantasy. To some extent, I agree with these predictions, but I do not believe the timelines are accurate—they assume a linear and continuous flow of progress through the rest of the 2020s and early 2030s, culminating in AI domination. While I do not accept these timelines, I still take the underlying prophecies very seriously. I can confidently say that within my lifetime (born 2000), I expect to see at least 80% of these predictions come to pass.
Given that we have solved most of the technical challenges required to achieve AGI, and are beginning to see early signs of the possibility of ASI, it has become imperative to consider what will happen to humanity in such a world. Never before in known history have we had to grapple with the existence of another form of (and potentially superior) intelligence.
There are currently two dominant camps. The first are the accelerationists, who believe that AI capabilities will ultimately benefit all of humanity and that the economic growth from AI should allay any reasonable fears. Companies like OpenAI, Meta (MSL), and Nvidia fall into this camp. These groups are focused on building superintelligent software systems that are not just on par with the best of human intelligence, but are far superior to it. Undoubtedly, building such AI systems would create an incredible amount of value and wealth.
The second camp consists of the AI safetyists, who believe that advanced AI is very dangerous and could lead to the extermination of humanity if it is not aligned with human goals. Within this group, there are two main approaches: one seeks to halt AI development through regulation, human resistance, or even destruction; the other believes the best path is through technical alignment, commonly known as alignment research.
I believe there is a third approach. Current "agentic" systems rely on AI operating autonomously with minimal human supervision and involvement. Instead, we should be building systems that prioritize human-AI collaboration as a core part of their functionality. This is not only beneficial for obvious economic reasons, but also creates a path to achieve both acceleration and alignment as AI systems progress.
How do we build for collaboration? Current post-training techniques and evaluations focus on capability and accuracy; we should develop similar benchmarks for interpretability and cooperativeness. For effective collaboration with AI systems, we need models capable of long-term planning, with plans that are interpretable by human users. Users should also be able to interject and modify these plans to achieve shared goals.
I hope we can build a future where both biological and digital intelligences can flourish. I believe this may be one of the most consequential problems of my generation, and possibly the determining factor for the future of humanity.
References
Last updated at: August 6, 2025