On October 23, 2025, the Whiting School of Engineering at Johns Hopkins University welcomed Aaron Roth, a professor of computer and cognitive science at the University of Pennsylvania. Roth’s presentation, titled “Agreement and Alignment for Human-AI Collaboration,” focused on innovative approaches to enhance decision-making processes through the integration of artificial intelligence (AI) and human expertise. His discussion drew upon findings from three significant research papers presented at notable conferences, including the 2025 ACM Symposium on Theory of Computing.
Roth’s work delves into how AI can support professionals, particularly in critical fields such as healthcare. He illustrated this with the example of AI assisting doctors in diagnosing patients. In this scenario, AI generates predictions based on diverse factors, including past diagnoses and observable symptoms. The human doctor then assesses these predictions, using their clinical experience to either concur with or challenge the AI’s recommendations.
When disagreements arise, Roth proposed a structured approach for resolution. Doctors and the AI can engage in iterative rounds of discussion, refining their perspectives until they reach a consensus. This process hinges on the concept of a “common prior,” which suggests that both parties can start with shared underlying assumptions, despite possessing different pieces of evidence. Roth describes this interaction as an example of Perfect Bayesian Rationality, where both the AI and the doctor leverage their respective knowledge bases to reach a better-informed decision.
Despite the theoretical framework, Roth acknowledged practical challenges. Establishing a common prior can be complex in real-world situations, particularly when dealing with multifaceted issues like diagnostic codes. To address these hurdles, he introduced the concept of calibration, which ensures that the predictions made by both parties are consistently aligned. He likened calibration to evaluating the accuracy of a weather forecast, where success hinges on the ability to pass specific tests.
Roth also explored a scenario involving potential conflicts of interest. If an AI system is developed by a pharmaceutical company, the AI might prioritize its own products over more suitable options for patients. To mitigate this risk, Roth recommended that doctors consult multiple large language models (LLMs) from different drug companies. This competitive approach encourages LLM providers to enhance their models, fostering alignment and reducing bias in the recommendations made to healthcare professionals.
He concluded with a discussion on the nature of real probabilities, emphasizing the importance of understanding the underlying truths that govern real-world dynamics. While achieving perfect precision in probabilistic assessments is rare, Roth asserted that unbiased estimates based on limited conditions can still yield reliable insights. By collaborating effectively, doctors and AI systems could ultimately enhance treatment strategies, improve diagnostic accuracy, and drive better patient outcomes.
Roth’s insights underscore the potential of AI not just as a tool, but as a partner in complex decision-making processes across various sectors. As the integration of AI continues to evolve, understanding these dynamics will be crucial for harnessing its full potential.
