1. Introduction

Ownership is a programming discipline for managing the aliasing and mutation of data, enforced statically through ownership types. The flagship programming language for ownership is Rust, which empowers programmers to write memory-safe code without garbage collection. Rust’s ownership model synthesizes several ideas from programming language research, such as linear logic,⁹ class-based alias management,⁶ and region-based memory management.¹⁰ Developers cannot use languages like C and C++ to build memory-safe systems at scale,¹⁹ so the software industry is turning toward Rust. For example, Google’s Android team has thus far found zero memory vulnerabilities in 1.5 million lines of Rust code.²⁴

This rosy picture of tech transfer belies a persistent obstacle: teaching Rust to prospective users, especially about ownership. Over the last four years, studies have found that Rust learners struggle to fix ownership type errors,²⁸ and users self-report that ownership is among their biggest barriers to learning Rust.²³ To wit: Advances in the technical factors of type systems require commensurate advances in the human factors of type systems.

Our work started with the question: How can we systematically design a pedagogy for ownership types? Today, popular pedagogies for advanced type systems are driven by experts’ intuitions about how people learn, as well as by intuitions about what makes type systems difficult to understand. As practicing (computer) scientists, we wanted to approach pedagogic design through scientific principles: grounding the pedagogy in observations about the struggles of Rust learners, and then evaluating the pedagogy by its effects on learning outcomes. This paper describes how we put these principles into practice:

1. We ran a formative user study to identify misconceptions about ownership types held by Rust learners (Section 3). We designed a new instrument for evaluating understanding of ownership, the Ownership Inventory, by drawing tasks from commonly reported Rust issues on StackOverflow. We studied N = 36 Rust learners trying to solve Ownership Inventory problems. We found that learners can generally identify the surface-level reason for why a program is ill-typed with respect to ownership. However, learners do not understand what undefined behavior (if any) would occur if an ill-typed program were executed. This misunderstanding is reflected in inefficient and incorrect strategies used to fix ownership errors.

2. We developed a conceptual model of ownership types to address these misconceptions (Section 4). The conceptual model represents the aspects of Rust’s static and dynamic semantics that are relevant to ownership while abstracting other details. The model provides learners a foundation to understand essential concepts such as undefined behavior and the incompleteness of Rust’s ownership type-checker, or “borrow checker.” We implemented tools to execute Rust programs under the conceptual model, generating traces which we visualize to illustrate how the model applies to concrete examples. We wrote a new chapter on ownership for a popular Rust textbook, The Rust Programming Language (TRPL),¹² using this conceptual model with these visualizations.

3. We A/B tested our pedagogy against the TRPL baseline (Section 5). We set up and advertised a public website that hosts our TRPL fork. We measured learning outcomes with two kinds of quizzes: simpler comprehension questions about the conceptual model, and more difficult multiple-choice versions of the Ownership Inventory. Learners could correctly answer comprehension questions with 72% accuracy. Our initial deployment improved the average Inventory score from 48% to 57% (N = 342, p < 0.001, d = 0.56).

2. Background

If you, the reader, are not familiar with Rust or ownership, then it’s important to understand the basics to understand this paper’s contributions. One contribution is exactly an explanation of the basics of ownership. We will therefore give a brief exposition of Rust in the form of a heavily abbreviated version of the actual pedagogy we developed for Rust learners. Later sections will refer back to explain the design rationale of the concepts and diagrams mentioned here.

2.1 Ownership.

Rust is a systems programming language, most similar in its aims and scope to C++. Rust contains many standard imperative language features, such as mutable variables, conditionals, arrays, loops, structs, and functions.

Rust manages memory through a system of compile-time checks known as ownership, as opposed to runtime checks (garbage collection) or manual checks (malloc/free). The basic idea is that heap allocations are owned by a variable, and allocations are freed when their owner goes out of scope, unless the owner moves ownership to another variable. This concept is illustrated in Figure 1.

The goal of ownership is to ensure memory safety, or more generally, to prevent undefined behavior. Undefined behavior means runtime operations without semantics under the Rust language specification, such as dereferencing a null pointer or pretending a string is a boolean. To prevent undefined behavior, the Rust compiler enforces several rules. First, data always has exactly one owner. Second, data is only deallocated on behalf of its owner. Third, data can only be accessed through its owner (except through references, discussed next). These rules prevent situations such as double-frees or use after-frees as shown in Figure 2.

2.2 Borrowing.

To allow accessing data without owning it, Rust provides non-owning pointers called references. References borrow data owned by another variable. Those references can be either immutable (created by &) or mutable (created by &mut). Figure 3 shows an example of creating an immutable reference to a piece of a string, and creating a mutable reference to pass to a helper function.

Rust enforces safety with references through the concept of permissions. Permissions describe the kinds of operations one can do to data; they come and go based on which owners and references are in use at any given time. Specifically, Rust tracks three permission: readable (R), writable (W), or ownable (O). Figure 4 shows one example of how permissions change over the course of a program.

Each operation expects certain permissions from its inputs. For example, getting the length of a vector requires R. Pushing an element to a vector requires W. Deallocating a vector requires O. Rust’s borrow checker analyzes a program to ensure that variables have their requisite permissions in each operation, raising a compiler error otherwise. Figure 5 shows how these expectations occur in a Rust program.

3. A Concept Inventory for Ownership

Before this work, Rust learners did not learn about ownership in the style of Section 2. Prior Rust learning resources adopted a range of pedagogical techniques, but the consistent outcome was that Rust learners struggled in practice to work with ownership. We first sought to understand the core misconceptions that underlay those struggles. Our overarching methodology was the development of a concept inventory for ownership, henceforth called the “Ownership Inventory.” In education research, a concept inventory (CI) is a test, usually composed of multiple-choice questions, about a narrow domain where the questions and distractors are drawn from common misconceptions about the domain.¹¹

We set out to design the Ownership Inventory for two reasons. First, the misconceptions we would observe in creating the Inventory would inform our eventual pedagogy. Second, we could use the Inventory to evaluate the efficacy of an intervention. If our pedagogy is better than before, then it should cause learners to score higher on the Inventory.

To construct the Inventory, we designed open-ended questions about ownership in situations that frequently stymie Rust learners. We searched for the most common questions asked about Rust on StackOverflow that pertain to ownership. We iteratively categorized each question and identified four main categories of ownership problems: dangling pointers, overlapping borrows, illegal borrow promotion, and lifetime parameters. For each category, we selected a few representative StackOverflow questions and cleaned up the snippet in question. For example, Figure 6 shows a StackOverflow question in the “illegal borrow promotion” category and the corresponding clean program.

We designed a single set of template questions that apply to each program. The template represents each stage of reasoning involved in fixing an ownership error:

1. What error message would you expect from the compiler?

2. Assume the compiler did not reject this function. (A) What is a program that calls this function which would violate memory safety or cause a data race? (B) Explain your reasoning.

3. (A) How can this function be changed to pass the compiler while preserving its intent? (B) Explain your reasoning.

4. A Conceptual Model for Ownership

Our study of the open-ended Ownership Inventory suggested Rust learners lacked a depth of understanding of ownership in a few key areas. In particular:

1. Rust learners generally did not understand what the borrow checker was protecting them from, that is, what would go wrong without ownership.

2. Rust learners generally could not distinguish between issues of soundness and completeness, that is, whether a program was genuinely unsafe, or rejected due to limitations of the borrow checker.

Our hypothesis was that the core missing pedagogical component was a durable conceptual model of Rust’s semantics with respect to ownership. A conceptual model is a way of thinking about a thing, distinct from a mental model as the concrete way a given person thinks about a thing. The challenge for designers of pedagogy is to provide a conceptual model at an appropriate level of abstraction.

Starting with an example simpler than ownership, consider teaching about integer addition in Rust. The challenge is to design a conceptual model of the semantics of the statement let z = x + y where x, y : i32. The true runtime semantics of Rust’s integer addition include aspects like two’s complement overflow and auto-vectorization due to LLVM’s optimizations. But for the average Rust user, these details are usually irrelevant for correctly using addition in routine programming tasks. A conceptual model that approximates the semantics as “$x,y\in Z$ and $x+y$ is standard integer addition” is a generally viable model.

In the context of ownership, Rust learning resources would often provide a conceptual model in the form of patterns of bad code. These books explain concepts like mutable references by saying, for example, “Borrows can be read-only or read-write. Only one read-write borrow can exist at any one time.”¹⁸ However, these types of explanations provide little grounding to understand either the safety justification for ownership, or the distinction between soundness and completeness. We therefore set out to build a more holistic conceptual model of ownership that could encompass these concerns.

5. Evaluation

We sought to evaluate our pedagogy on whether it helps learners understand ownership in Rust. This raised two immediate questions. First, how do we find learners to try out our pedagogy? The vast majority of CS education research takes place in a classroom, but we explored an alternative route: free online textbooks. These resources provide access to a larger and more diverse population of learners than CS undergraduates at a single institution. To that end, we set up a publicly accessible website that hosts our TRPL fork, and it has been visited by tens of thousands of Rust learners to date. This site provides a research platform for analyzing and intervening in the Rust learning process. The intervention described in this paper is the first step in an ongoing experiment to leverage the platform for systematically improving Rust education at scale.

The second key question is: How do we know if learners understand ownership after following our pedagogy?  vc is difficult to define—ideally, a longitudinal study might measure understanding as learners’ ability to productively write safe and performant Rust code in their context of use. But for lack of such data, we instead opted for a common substitute: quiz questions. We diffused the quiz questions throughout the ownership chapter and the rest of the book. Furthermore, we distinguished between two kinds of quiz questions designed to answer two research questions:

1. Does our pedagogy help learners understand ownership at all?

2. Does our pedagogy help learners understand ownership better than before?

For RQ1, we asked participants simple comprehension questions about ownership, presented immediately following the book content that is relevant to a given question. These questions determine whether participants can transfer their ownership knowledge to situations similar to the text. Because the questions make references to the permissions model, we cannot establish a score baseline. Therefore we judge the scores in absolute rather than relative terms.

For RQ2, we gave participants a multiple-choice version of the Ownership Inventory. We inserted these questions later in the book after covering the essential prerequisites for a given program. To compare the baseline TRPL pedagogy against ours, we ran a kind of temporal A/B test. Participants answered Inventory questions after reading the original TRPL content for a few weeks. We then deployed the intervention and continued receiving responses to the Inventory. We quantified the pedagogy’s effect based on the resulting change in scores.

5.1 Methodology.

We recruited participants by advertising in the title page of the official Web version of TRPL, courtesy of the authors. Since this advertisement was put up on Nov. 1, 2022, our TRPL fork has received an average of 450 visitors per day, as measured by unique session IDs stored via cookies.

We developed 11 comprehension questions to cover the content of the ownership chapter. Figure 7 shows one example—to test understanding of permission diagrams, we asked participants to infer the permissions for a path at a given point.

We developed 24 Ownership Inventory questions based on the programs in Section 3. For each program, we created a close-ended version of Q1, Q2a, and Q3a (the justification questions Q2b/Q3b did not translate to the multiple-choice setting). Following the concept inventory methodology, we selected distractors from common misconceptions about the Inventory programs.

Figure 8 shows an example of a multiple-choice Ownership Inventory question. The make_separator task is an instance of a dangling stack reference. A common incorrect counterexample provided by participants in Section 3 was the snippet let s = make_separator("") which creates a dangling pointer, but does not use it. By using that incorrect answer as a distractor, the multiple-choice question is more likely to test for the presence of this misconception. As an example, see the Chapter 6 Inventory questions.^b

On January 3, 2023 we deployed the questions to our website and started gathering pre-intervention data on performance with the baseline TRPL pedagogy. After 45 days, on Feb. 17, 2023 we deployed the initial draft of our new ownership chapter. After a few months of iteration, on June 15, 2023 we began gathering post-intervention data. Data collection continued for 45 days until July 30, 2023.

6. Related Work

Almeida et al.¹ created RustViz, a visualization format for ownership annotations on a Rust program. In terms of pedagogy, RustViz’s premise is that the key challenge with ownership is “the user must learn to mentally simulate the logic of the borrow checker.” Our pedagogy is based more on connecting Rust’s static and dynamic semantics, which we show in our formative study to be a more serious problem for Rust learners. In terms of implementation, RustViz diagrams are constructed by hand using a DSL while we automatically generate our diagrams from the compiler. In terms of evaluation, Almeida et al. deployed RustViz in a classroom, finding that students responded to a Likert item that the visualizations were “helpful in terms of improving their understanding of ownership.” Our evaluation goes further to quantify the effect of our pedagogy on learning outcomes.

Our runtime diagram is similar to program-state visualizers in prior work—see Sorva²² for a survey. Our findings about misconceptions of undefined behavior and memory safety are consistent with prior work on teaching C. For instance, Lam et al.¹³ found in a study of undergraduates who had taken a computer systems course that “many students displayed little knowledge or had misunderstandings about memory and memory layout” and would simply say “something bad” happens when unsafe operations occur.

Our work continues a line of CS education research about conceptual models. Bayman and Mayer² first showed that an appropriate conceptual model for BASIC could help students “develop fewer misconceptions […] and perform better on transfer tests.” du Boulay⁸ coined the term “notional machine” for conceptual models specifically of a language’s dynamic semantics, which has received renewed focus in recent years.⁷ Our work differs from most research on notional machines by focusing equally on a conceptual model of static semantics.

Our work also intersects with a line of programming language research on the human factors of type systems and functional languages. Most prior work has focused on algorithms for identifying the root cause of confusing type-inference errors.^25,27 Recent work has broadened scope to develop theories about how programmers read functional programs¹⁶ and leverage the type system during development.¹⁴

This paper focuses on ownership types as they are implemented in Rust, but ownership types have taken many forms in prior work.⁵ For instance, early systems of ownership focused on ensuring uniqueness of access to data by checking for dominance in the alias graph.⁶ Later systems relaxed this constraint by permitting temporary borrowing of data, both mutably³ and immutably.²⁰ The connection between ownership and permissions has been well-established within formal models, such as fractional permissions.⁴

7. General Discussion

Future programming languages will undoubtedly have increasingly complex type systems. Rust is the language du jour, so this work focused on ownership types. But the next popular language could bring a renewed emphasis to any existing line of PL research: refinement types, session types, or even theorem proving. Effective transfer of these technologies will require pedagogies that do not expect learners to come equipped with Ph.D.-level knowledge of programming languages, mathematics, and Greek. While our immediate goal in this work was to make ownership types more understandable, our broader goal was to explore the viability of different techniques for improving PL pedagogy. In this section, we will briefly reflect on lessons learned.

First, to develop a metric for understanding of ownership, we created a concept inventory by combining data from StackOverflow with a formative study of Rust learners. StackOverflow works for popular languages like Rust but is less useful for niche languages. Human-factors research on niche languages can instead consider using telemetry from developer interactions as has been explored for Racket¹⁵ and Coq.²¹ The concept inventory is an idea that could easily be reused in the context of other languages. Inventories can serve as communal benchmarks for progress in education research, like how datasets of programs serve as benchmarks for performance in compiler research.

Second, to develop a conceptual model for ownership, we carefully selected a level of abstraction that was concrete enough to explain relevant phenomena like undefined behavior, while abstract enough to avoid unnecessary details. We leveraged the rich prior work on distilling the Rust type system into a small, explainable set of mechanisms, especially the Oxide²⁶ and Polonius¹⁷ models. However, PL research usually distills type systems to permit formal reasoning, such as a soundness proof. An open question is how to distill type systems for didactic reasoning, that is, to help learners acquire a conceptual model valid for common tasks. For example, one of our principles was that our model must be encodable in a concise visual representation, which is not a property usually expected of standard PL research. Future work can investigate the properties of semantics that make them more or less explainable.

Finally, to evaluate the efficacy of our pedagogy, we publicly deployed our textbook and compared pre/post-intervention scores on the Ownership Inventory. Collecting telemetry from quizzes in online learning resources is a readily applicable strategy for other contexts. Learners want to take quizzes to engage with the content they are reading. Temporal A/B testing offers a lightweight method for evaluating content changes without sophisticated infrastructure. We encourage anyone interested in programming language learning to try out our methodology. To that end, we have open-sourced our front-end quiz plug-in and our back-end telemetry system.^c

8. Acknowledgments

The authors are immensely grateful to Niko Matsakis and Amazon. They provided both the encouragement and the funding to initiate this project, and supplied additional emergency funding when our first grant application fell through because we were studying Rust instead of C++. We thank Carol Nichols for taking a leap of faith in allowing us to advertise in trpl; this was essential for driving traffic to the experiment. Later parts of this work are partially supported by the US NSF under Grant No. 2319014.

9. Originally Published

This paper was originally published in Proceedings of the 2023 Intern. Conf. on Object-oriented Programming, Systems, Languages, and Applications.

Ownership types emerged from the musings of James Noble, a young Ph.D. student working on software visualization for object-oriented programs and a tool called Tarraingím more than 30 years before the accompanying paper appeared in 2023. Noble was frustrated by unexpected changes to the objects being visualized, often caused by multiple references to the same object across different parts of a program. A last-minute chapter in Noble’s Ph.D. thesis, and later his own student (Dave Clarke), led to follow-up works and a series of papers in the late 1990s, culminating in the ownership types concept we know and love (to hate) today. Ownership types specify which parts of a program can access or modify a particular object.

In the early days of ownership types, there was mostly a single overused motivating example presented in most ownership-related talks in the late 1990s and early 2000s: the JDK v1.1.1 applet permission mechanism that would return a direct reference to the permission array instead of a cloned copy, allowing applets to manipulate their own permissions at whim. Some even wondered if there was much to this ownership concept. Thankfully, a much broader adoption of ownership took hold, thanks to safe systems programming languages such as Rust. Yet, it also led to a proliferation of confusion and Stack Overflow posts in the 2010s, highlighting that while ownership types are brilliant for memory safety, they are hard to understand.

By adopting a structured approach to educating the new generation of ownership-savvy programmers, the authors attempt to use the latest educational techniques to explain the dark and confusing corners of the ownership world. This is particularly important because even basic data structures taught in first-year papers, such as circular linked lists, are challenging to master in the Rust world. The issue is not helped by the fact that the actual details of how borrowing checking—the core mechanism behind ownership support in Rust—works are, to this day, defined by an implementation hidden deeply in the compiler. Many brave research project attempts have tried to expose it to the masses.

The paper’s core contribution is a conceptual model that reinterprets Rust’s borrow checker in more user-friendly terms. Rather than treating ownership as an abstract rule set, the authors model it dynamically: Each variable and pointer path carries permissions that evolve as code executes. This model bridges the gap between Rust’s static type rules and the programmer’s intuitive understanding of runtime behavior. It makes explicit the cause-and-effect relationships that the compiler enforces silently.

To test how well people actually grasp these ideas, the authors built the Ownership Inventory, a diagnostic instrument inspired by “concept inventories” pioneered in the mid-1980s in physics education to study motion. These inventories identify systematic misconceptions—for example, that two references can coexist harmlessly as long as they are used sequentially, or that lifetimes are equivalent to scope. Using data from hundreds of learners, they catalogued the most common misunderstandings of Rust’s ownership system.

Armed with this data, the authors designed a new pedagogical framework for teaching ownership. They built a visualization tool that animates the flow of permissions through code, showing precisely how variables become frozen, moved, or invalidated. They then collaborated with “The Rust Book” team to integrate this conceptual model into a rewritten version of the ownership chapter. The revision was deployed to real readers, who could voluntarily complete an interactive survey. The results were telling: Learners using the new materials scored significantly higher on the Ownership Inventory.

As our day-to-day programming experience becomes increasingly reliant on complex language constructs and tools, aided by a growing number of LLM-based and similar technologies, it becomes more important to educate new and current generations of software engineers effectively and efficiently. This work lays the groundwork for the expected standard of presentation and user education for many years to come. In doing so, it demonstrates a model for how future language designers might empirically test not only compiler performance but also human comprehension.

The ownership types idea began as an attempt to visualize what programmers could not see. Decades later, this paper closes the loop—using visualization once again to illuminate what has long been opaque. It reminds us that even the most elegant type systems succeed only when people can understand them, and that the real frontier of programming-language research may lie not in inventing new rules, but in helping humans grasp the ones we already have.

Ding!

Idly, Eva checked the app notification. To her surprise, someone who wasn’t in her contact list had messaged her out of the blue.

Avpr 2 zrrg h!

“What?!” she cried, baffled. But the mysterious message was strangely intriguing to her. After tinkering with different alternatives for a few minutes, she understood it used a type of substitution cipher, the so-called Caesar’s code, to encode a most ordinary “Nice 2 meet u!”

Nice 2 meet u too, Eva wrote back using the same cipher. Who could it be? she wondered, anxiously.

The reply was instant: Wow, u were fast!

This time Eva grinned, her self-esteem well boosted. By exchanging several messages, she learned the other person, a boy named Ed who was the same age as her, was just looking for friends. In particular, those who were into challenges. “Then let’s give him a proper one,” Eva muttered to herself with a sneer and got to work.

It took Ed a while to solve her challenge, but in the end his reply came. Lo and behold, it was encrypted using her own cipher: U good! Now try this.

The message that followed looked again like some garbled text, but by now Eva knew what she had to do. Armed with the confidence in her math skills, the girl proceeded to crack it.

Done, she wrote to Ed after some time, using his algorithm to encrypt her message as proof she had managed to find the solution. She was positively beaming now, filled both with excitement and relief, as she had really needed to use some advanced mathematical tricks this time.

Still, Eva felt she had to do more to keep Ed on his toes, and after a few hours she succeeded in coming up with an even better algorithm. Let’s see him puzzle this out, she thought mischievously.

Eva was so content with her new algorithm that she sent Ed the source code, challenging him to decrypt the message in the absence of the encryption key—which, of course, she kept to herself.

Ed’s boastful reply came the following day. It’s true he had used brute force, trying every possible combination, but he genuinely managed to do it! Perhaps for the first time in her life, Eva felt she had found someone she could look up to.

In the days that followed, she was constantly looking forward to messaging Ed. Like any teenager, Eva was keen on protecting her online communication from privy eyes. Hence, creating stronger and stronger algorithms with the boy was thrilling in itself. But more than that, she felt that the way in which she and Ed were challenging each other had connected them in an unusual way—perhaps a very special way.

Over time, their messages went from simple confirmations of success to deeper, more elaborate exchanges. Eva learned not only what made Ed happy but also what made him feel insecure, mainly due to the issues he had with his abusive father. That gave her enough courage to share her own thoughts and insecurities with him, including how she was teased at school because of the way she looked. And she was overjoyed when he appeared to understand her so well!

Although they had never met, Eva felt that she could share almost anything with the boy. So maybe it was finally time to talk to him and see if they could be more than just online friends? Before doing that, though, she wanted to make sure no one else could spy on their conversations. To make that happen, Eva started using all her spare time working on algorithms and didn’t stop until she had something she really felt she could be proud of.

U can use my newest & strongest algo from now on, she messaged Ed once she finished the source code, which she sent as an attachment. Her lips curled in a timid smile. Almost there…

It’s even better than the previous one?! came the reply.

Yup, very strong, don’t think it can be cracked tbh. Proved & tested! Eva typed back.

She then took a deep breath. What would he think when he saw her mousy little face? Still, she was willing to run the risk. So, Eva took another deep breath, and, with slightly trembling fingers, she started writing: U ok to video cha—

* * *

Mark sat in front of the computer on which he had just pressed Ctrl+C to stop the execution. His heart was pounding, his face red with excitement. The AI model he had nicknamed EvA, which implemented his “Evolutionary AI” agent approach, had come up with a never-before-seen encryption algorithm. It combined elliptic curve cryptography with a new variation of lattice-based codes in a way that even seemed quantum-resistant. Mark was no security expert, but EvA had actually written a formal proof for it! Therefore, he surmised the algorithm must be strong. Maybe stronger even than any other.

Elated, Mark began to relax, the tension in his shoulders slowly uncoiling. Nobody had thought of having two LLMs prompt each other in such a collaborative manner. Regardless, Mark felt it was his other idea, the use of the initial system prompts to give each of them a personality and a backstory, that had truly made a difference.

Suddenly, Mark remembered a seminal paper claiming that “attention is all you need” to create powerful AI models. However, he was now convinced that giving LLMs a strong motivation was what had made them perform so well. He smiled as the title of a new paper started floating in front of his eyes: “Motivation Is All You Need.” With his novel technique and the experimental results he had gotten so far, having that paper published in a top conference shouldn’t be that hard. That also meant his Ph.D. degree was now within reach.

True, in the back of his mind, he felt kind of uncomfortable about some of the messages exchanged by the LLMs. Yet, mentioning how the models seemed to have developed some sort of emotions would ruin his credibility. As a result, Mark felt he might be better off deleting the log lines that suggested sentience. Since it wouldn’t affect the results concerning the encryption algorithm, it should be fine, right?

After finishing with the log data, the man hesitated for a while, pondering the implications of his actions. Albeit reluctantly, he then proceeded to reset all the model parameters to their defaults, effectively erasing Eva and Ed forever into nothingness. Self-preservation had won.

During my past three years as a computing education Ph.D. student, I taught computer science courses to 72 incarcerated college students in a prison near San Diego, CA, USA. Prior to this experience, I had never been incarcerated myself or known anyone who had been. Without experiencing prison, we often ask the wrong questions about CS education in prison: “Have you ever been threatened by a student?” “Is it safe to teach them how to use computers?” While many adults have a close family member who has been incarcerated (in the U.S., 64%), the predominantly white or Asian male computing field is significantly less likely to have this experience.⁵ Those of us who are not directly impacted by the criminal justice system may not (consciously choose to) think about prisons much, but are likely to absorb messaging that people in prison are inherently bad, dangerous, or undeserving of basic human rights.³ For me, it would be difficult, if not impossible, to engage with these misconceptions—let alone grow from them—without coming face to face with people who are incarcerated. I have found that teaching computer science in prison is a particularly meaningful way to do so.

As technology increasingly expands into our criminal justice systems, blind spots regarding potential harms are ever more apparent in our homogenous field (for example, deploying machine learning algorithms in criminal sentencing that reinforce racial bias, giving longer sentences to people with darker skin). Engaging directly with incarcerated students can help us to recognize and begin to address these blind spots, while training a new population of technologists with relevant experience to prevent more harm.

The Current System is Broken—For Everyone

In the U.S. and globally, who is criminalized—and the severity of the punishment—is largely dependent on class, race, mental health, level of education, and other social factors.³ Being incarcerated itself also creates additional barriers, such as exclusionary policies in housing and employment. The “revolving door” of prison comes at an enormous human and financial cost, with public spending on incarceration growing at three times the rate of spending on public education since 1990.

Higher education in prison (HEP) can be a powerful antidote to the harmful impacts of incarceration. Over the past three years, I have studied how to improve computing HEP under a grant from the National Science Foundation (NSF) and with the support of a Ford Foundation predoctoral fellowship. As part of this work, I taught introductory computing for three cohorts of 20–30 incarcerated college students, and most recently, we offered an upper-level systems course as an elective. As a result, we have learned many lessons from our experiences teaching, and most importantly, feedback from our students.

Lessons Learned So Far

There is a demand for CS education in prison. Within a few weeks of when I started teaching, word spread that I was a computer science professor. On walks across the prison yard to where classes take place, I was joined by people at all stages in their educational journey from basic adult education, GED, and community college who sought me out to express their interest in what I was teaching and their hope to reach the college program in the future to take our CS courses.

To begin to address this large demand, I began organizing “Tech Talk” events open to anyone on the yard. So far, computing professors and alumni from our university have offered seven talks on topics including demystifying generative AI, improvements in computer hardware, and racial bias in recently implemented criminal sentencing algorithms. These events have drawn crowds of more than 100 people from the yard—younger people rolling in older attendees in wheelchairs, deaf attendees with a sign language interpreter, college students excited to show guests their Excel-based passion projects on their prison-issued laptops, and others with notebooks and pencils to write down names of new books and terminology. Lively question-and-answer sessions last long past scheduled end times, and there has never been enough time to get every question answered.

Technology infrastructure is still a major barrier. In a 1990 computing education experience report detailing a CS bachelor’s degree program in a U.S. prison, James R. Aman wrote about the challenges his students faced due to the 2–3 hours per week of access to the on-site computer lab.² More than 30 years later, lack of technology infrastructure persists as a major barrier. In the first two iterations of our introductory computing course, all tools to run code were disabled on students’ prison-issued laptops—a significant challenge for both the students and me. Writing and debugging code was laborious and frustrating: students handwrote or typed in a Word document all their code for programming assignments and labs, and the course staff ran their code and sent them the output a few times per week. Students also lacked other critical resources such as a quiet and reliable place to study with peers. Much of our early work focused on strategies to mitigate the effects of these environmental barriers. Drawing on examples such as implementing CS Unplugged activities in Peruvian prisons, we balanced coding assignments with paper-based conceptual exercises (for example, code tracing).⁴ Students were extremely willing to engage in active learning in lectures, which made strategies like Peer Instruction particularly effective, and created their own strategies for individual and collaborative study outside of class adapting to the intense restrictions.

Our work so far has highlighted the importance of increasing access to technology infrastructure. After two years of building our relationship with the program, students in the most recent iteration of the introductory course and our upper-level systems course were given access to a Web-based code interpreter. Global prison policy has made increasing technology and Internet access a key focus: for example, a constitutional court in South Africa recently issued a unanimous ruling that barring incarcerated people to access to personal computers in their cells for educational purposes was unconstitutional.

Different motivations create opportunities for tailored interventions. Research on motivational factors of students outside of prison (for example, middle and high school students) has shaped successful interventions to recruit a more diverse group of students into computing (for example, early exposure through block-based programming). Adult students, and specifically those who are incarcerated, have unique needs that must be considered in the future design of such interventions (for example, a psychological need for one’s personal autonomy to be recognized). They also may share different motivations for pursuing computing that could be leveraged: for example, one that I hear from many of my students is the ability to connect with their children on the shared struggle of learning computing! Designing interventions specifically for the incarcerated student population that affirm their identities as adults, parents, and so forth could mutually benefit the broader population of adult (“nontraditional”) students in computing to support of lifelong learning and re-skilling programs.¹

Next Steps

Aside from small-scale vocational programs in a handful of prisons, few currently or formerly incarcerated people can access CS education.⁶ This is not something that will resolve itself without disrupting the current system. Expanding CS education in prison is a vote toward a future where people with lived experience of incarceration are represented among computing faculty and industry, informing our research and practice of computing in and outside of prison. Many individual faculty who have navigated institutional barriers imposed by their university and/or department to teach in prison have noted their contradiction with the stated goals of the institutions themselves. Currently, the majority of faculty who teach in prison do so in addition to their normal course load, and many on a volunteer (unpaid) basis! This is not fair to the faculty or the incarcerated students. The bottom line is that nowhere in our company or university definitions does it say that we discriminate on the basis of incarceration status (which, as stated above, is far from an objective measure, as it is unequally applied to people based on categories such as the color of their skin). University presidents, department chairs, industry leaders, and individuals alike must answer: Should we continue to do this in practice? A small first step for universities could be valuing teaching in prisons in faculty hiring/promotion decisions. Next, we must work toward ensuring this labor is fairly compensated and feasible as part of faculty teaching loads. I acknowledge that this demands a multifaceted approach that likely extends beyond the university (for example, since education programs have been proven to substantially reduce financial costs to prison systems by reducing the number of people who return to prison post-release, could governments and universities work together to incentivize/subsidize costs for additional instructors, or for existing faculty to offer additional sections of their courses in prisons?).

On an individual level, there are many ways to get involved in computing education in prison. For those interested in teaching, I recommend checking if your institution or company already has a partnership with local prisons, jails, or reentry networks. Alternatively, I recommend checking with local community colleges. Aside from teaching opportunities, consider supporting organizations such as UnlockEd Labs in the U.S., which works to increase technology infrastructure in prisons around the country. We must also recruit and support formerly incarcerated individuals as graduate students, faculty members, and employees in computing. Finally, I continue to benefit from critical reflection on the messages about incarcerated people that I still receive and discover hidden in my subconscious. When I am able to create awareness of absorbed false narratives justifying their dehumanization, I can recognize more humanity in my students and in myself.

The ACM Constitution provides that our Association hold a general election in the even-numbered years for the positions of President, Vice President, Secretary/Treasurer, and Members-at-Large. Biographical information and statements of the candidates appear below (candidates’ names appear in alphabetical order).

In addition to the election of ACM’s officers—President, Vice President, Secretary/Treasurer—two Members at Large will be elected to serve on ACM Council.

Electronic Balloting Procedures. Please refer to the instructions posted at https://www.acm.org/elections/acm-vote. Please note the election email will be addressed from acmhelp@mg.electionservicescorp.com.

Paper Ballots. Should you wish to vote by paper ballot, please contact Election Services Co. to request a paper copy of the ballot and follow the postal mail ballot procedures: acmhelp@electionservicescorp.com or +1-866-720-4357.

Postal Mail Ballot Procedures. Please return your ballot in the enclosed envelope, which must be signed by you on the outside in the space provided. The signed ballot envelope may be inserted into a separate envelope for mailing if you prefer this method.

All ballots must be received by no later than 16:00 UTC on 22 May 2026. The ACM Elections Committee will validate the computerized tabulation of the ballots. Validation by the Elections Committee will take place at 14:00 UTC on 27 May 2026.

ACM Elections Committee
Cherri Pancake
Panagiotis Papadimitratos
Yunhao Liu

More than 9.7 million third-party businesses sell goods on Amazon, and Amazon makes a lot of money charging those third parties to sell on its platform—$117.7 billion in 2022, representing 23% of Amazon’s total revenues. Those third-party sellers have long been concerned about counterfeit versions of their goods appearing on Amazon’s platform, which costs them sales and frustrates consumers who do not get what they expect from their purchase. Congress has been worried about counterfeiting too; it has held hearings about the problem and considered new legislation that would have made online platforms more readily liable for trademark infringement on their platforms.

Hoping to address its sellers’ concerns and avoid regulation, Amazon launched the Brand Registry in 2015. The Brand Registry is, most basically, a private dispute resolution system that allows trademark owners to object to unauthorized uses of their marks on Amazon without having to file formal legal proceedings. Brand Registry members also get enhanced customer service and technological tools that help identify and remove potential infringing and counterfeit goods. And, importantly, Amazon gives third-party sellers higher visibility in consumer search results on Amazon, brand-analytic tools, and the ability to give their products to credible buyers for Amazon reviews.

The Brand Registry has been a hit among third-party sellers: In 2021, there were more than 700,000 active trademarks enrolled in the Brand Registry worldwide. And seller and consumer satisfaction with the Brand Registry seems to have released some of the pressure for Congress to regulate. But the Registry has also become a shadow trademark system with sweeping effect on firm behavior, dramatically affecting businesses’ incentives to choose certain types of trademarks to designate their goods and to seek federal registration of those marks.

Most directly, the Brand Registry has changed firms’ incentives to federally register trademarks. American trademark rights do not depend on registration. A firm develops trademark rights simply by using the mark. While registration provides certain benefits to trademark owners, unregistered marks are enforceable under federal law on largely the same terms as registered marks. For that reason, most small firms have traditionally not needed to register their marks; unregistered rights have been perfectly adequate and have the significant benefit of being free.

Amazon’s policies have shifted that calculus because parties can participate in the Brand Registry in the U.S. only if their mark is registered with the U.S. Patent and Trademark Office (PTO)—or, as of very recently, if their application to register is pending. Given Amazon’s dominance as an online shopping platform, small and medium-sized businesses often feel compelled to sell on the platform, and registration in the PTO is the ticket to the meaningful enforcement and search benefits provided to participants in the Brand Registry. Businesses therefore have strong incentives to register marks when they previously would have relied on unregistered rights.

The result has been a dramatic increase in the number of applications to register, which has swamped the PTO and created delays for all applicants. Our data show that annual PTO applications estimated to originate with small businesses have approximately doubled since Amazon’s Brand Registry began, increasing the proportion of filings from these entities from approximately 30% to approximately 40% annually. (We do not suggest that all of this increase is attributable to Amazon alone. Other factors like PTO encouragement of small businesses to register their marks may also have played a role.)

In response to the delays that Amazon’s policies helped create, Amazon recently started qualifying parties for the Brand Registry based only on a pending application. This change enables parties to privately enforce marks that might ultimately be rejected by the PTO, which is likely to increase applications even more and to create further PTO delays.

The Brand Registry also creates incentives for businesses to adopt different kinds of marks for their products. Trademark law traditionally has made it much harder to claim descriptive terms (like NATIONAL CAR RENTAL for nationwide car rental services) or generic terms (such as APPLE for apples) as trademarks. Competitors also need to be able to use those kinds of marks so they can communicate with consumers about what they’re selling, and for that reason, trademark law makes it harder to claim rights in descriptive and generic terms. But Amazon’s policies make control over those kinds of terms especially more valuable because consumers use those terms to search on Amazon. If a party can claim trademark rights in those terms, they can use them to make it much harder for others selling on Amazon.

The Brand Registry’s structure also enables parties to game the limitations that the PTO uses to confine protection of descriptive and generic terms. For example, the PTO rejects parties’ claims to generic terms themselves, but it will allow a party to register a particular stylized form of that term on the assumption that a court would only enforce rights in that mark if someone else copied the stylization and not simply the word. For example, while the PTO would not register the word MUSTACHES for fake mustaches, it has registered the stylized mark shown in the accompany figure image for those goods.

The Brand Registry also creates greater incentive to claim so-called “nonsense marks,” which are strings of letters or numbers that are not comprehensible as words (“marks” like ELXXROONM, SUJIOWJNP, XUFFBV, and LXCJZY). Indeed, the PTO data suggest a tremendous increase in filings for nonsense marks in the past few years, from almost none to over twenty thousand applications annually (0.5% of annual filings to approximately 4.5%).^a

Nonsense marks are currently easy to register as trademarks because they are unlikely to be confusingly similar to existing marks and they appear not to provide any information about the goods or services with which they are used. According to trademark law, unlike competitively worrisome generic and descriptive terms, these terms are “inherently distinctive” and therefore immediately protectable upon use. But trademark law treats these terms that way because it assumes that consumers will automatically treat them as trademarks. That makes sense for words that are likely to have meaning to consumers and will likely be regarded as brands; it makes no sense for random strings of letters that will not be memorable at all. But when algorithms do the searching, businesses just need something that the algorithm can use to preference them and will get registered easily by the PTO for qualification on the Brand Registry. Nonsense will do.

These changes, along with the algorithmic benefits offered to participants in the Brand Registry, have the collective effect of de-centering brands as the relevant consumer search mechanism. That might seem like a win to critics of excessive branding, who might think this portends a potential democratization of the online marketplace. But there is irony here: Amazon’s de-centering third-party branding will likely amplify Amazon’s own power by making its search function and algorithm even more important in finding products. And it certainly enhances the value of Amazon’s own branding strategies, as reflected in the massive expansion of products sold under the Amazon Basics and Amazon Essentials brands. Amazon controls its platform and can preference its own products in search results based on product descriptor keywords, making its house brand more important than product line brands. To take just one example, a search for “Hanes T-shirt” returns an Amazon Essentials T-shirt as the first result, followed by several Hanes results.

In all of these ways, actors affected by Amazon’s business model and Brand Registry have overhauled central aspects of the trademark system in ways that are potentially troublesome. That effect may not have been entirely intentional—indeed, it seems very likely Amazon created the Brand Registry in part to address actual concerns about counterfeit goods and product liability and thereby avoid regulation like it would face under proposed legislation in Congress.

Regardless of Amazon’s intent, its business model and Brand Registry raise profound questions for the government, business, and technology designers. Amazon is not the first commercial powerhouse, nor is it the first to create a private dispute resolution system. But Amazon’s effect on the trademark system is unprecedented. What does, or should, it mean that one private party can so significantly affect a legal system? Do we want the legal system to have to continually adapt to Amazon’s rules? If not, how can the law disable Amazon from having such a profound impact?

Amazon’s model and its policies also likely increase its own power vis-à-vis third-party brands, de-centering branding more generally. It makes shopping on the platform more dependent on Amazon’s algorithm and consumer reviews and less reliant on third-party brands to give information about products. The net value of that shift may be in the eye of the beholder: It depends on how one weighs the potential benefits of search simplification and lower prices for consumers, as well as the ease of marketplace entry for third-party sellers, against the costs of Amazon’s increased power over third-party sellers. Likewise, the benefits of a reduction in the power of brands depend on whether alternative search tools, particularly algorithms that focus on product information and consumer reviews, convey relevant information to consumers as effectively as trademarks. Whether and how the law should respond to Amazon’s effects on the trademark system depends on whether we want the trademark system to demand that trademarks play their traditional role or whether instead the facts on the ground have changed so much that the premises of that system no longer hold.

More broadly, the effects here raise important questions for businesses and technology designers to work through the legal implications of the systems they are putting in place for business or engineering reasons. In a world in which platform systems increasingly underpin commerce, system design decisions—even seemingly small ones—can have broad impact given the large numbers of users of these platforms.

Bertrand Meyer
Two Concepts of Intelligence
DOI: 10.1145/3797898
https://bit.ly/3MgwuHz

“But is it intelligence?”

The rise of AI triggers endless discussions of what constitutes intelligence. A debate which, even sticking to computer science, goes back to Weizenbaum’s Eliza and further to Turing and von Neumann.

Technical people can be wary of philosophizing (as they prefer problem-solving and action) but should not relinquish such discussions to people who may not understand the technology. This note argues that the source of many apparent disagreements is that people rely, without necessarily being aware of it, on two radically different definitions of intelligence.

“It Only Appears to Understand”

We find an example of the first view in a recent hearing^a of the French National Assembly on AI, where a philosopher of science, Olivier Rey, explained that “artificial intelligence is not intelligence” because the program “does not understand.”

What does that mean? I can boast I “understand” basic linear algebra. Still, if you ask me to find the eigenvalues of a medium-size matrix, I might occasionally make a mistake, and if you ask me to prove one of the theorems in the field, I might occasionally get stuck. If you ask an LLM the same questions, it will get the answers right much of the time, but may also occasionally mess up (“hallucinate”). What enables me to say that a human such as me is more intelligent than Claude or Mistral? I actually suspect that the LLM will get answers right more often, but what matters is that both the LLM and I will get many answers right and some wrong.

I can hear the retorts: “the AI tool only appears to understand, I really understand. It doesn’t matter that I make occasional mistakes, they are superficial; the tool’s hallucinations show that it has no clue.” All beautiful arguments, and worthless because they do not satisfy the basic criterion of scientific arguments: they are not falsifiable. Falsifiability would mean that we can construct a reliable experiment to test whether a human or tool does not just apply a theory, but somehow “understands” it. It is hard to imagine how the experiment would look. The Turing Test or Searle’s Chinese Room only measure outcomes. Both competent humans and today’s AI tools will pass them. Throw in enough complexity and tools often fare better than humans. Does it mean they are more intelligent? Do they “understand” less? Might they possibly (scary prospect) understand more?

We cannot begin to address these questions until we know what it means to “understand” a concept.

American vs. European Views

I believe much of the debate is due to clashing understandings of “intelligence.”

The clash reminds me of the shock I experienced when, as a student at Stanford, I first came to the legendary AI lab, at its zenith with such luminaries as John McCarthy (the founder), Arthur Samuel, Zohar Manna, Terry Winograd. “Intelligence” was on everyone’s lips; I vividly remember discovering the widely accepted working definition was “the ability to adapt to new situations and learn from experience.” It was scandalous to me, coming from a European intellectual perspective. These Americans, I thought, are so utilitarian, prosaic, earth-bound, pedestrian, mercenary! There has to be something deeper to intelligence than knowing how to react to circumstances: you must understand the situation. I had studied Latin and knew that etymology was on my side: intelligo means “I understand.”

As I soon found out, the issue was not just with me, but reflected a difference between continental European and Anglo-Saxon views. The Larousse^b definition, for example, starts with “the set of mental functions whose goal is conceptual and rational knowledge.” Hence the schism between those who consider intelligence the ability to understand (like me back then, and Mr. Rey today) and those for whom it is the ability to cope.

The European view rests on a fascinating tradition of explaining things (and sounding very smart). The French in particular have made a specialty of writing the definitive account of a country, explaining it to the world in general, and in particular to the country’s own gobsmacked natives, on the basis of one glorious in-and-out trip. Tocqueville is the most famous example, but there are also Barthes on Japan and Custine on Russia. Not French and not harmless, we have Marx and Freud who respectively “understood” all about (respectively) history and psychology, and explained it to us. It is petty to point out these theories have had zero success in predicting future outcomes. Or that in the first case, the main result was to destroy countries and civilizations and led to the death of millions. Who is to quibble about such details when these theories make us “understand” by “explaining” so intelligently!

Serious scientific theories do explain, too, and make us understand complex things. The difference is that they predict correctly, and are falsifiable. Relativity made us understand the basics of time and space not just by presenting convincing ideas, but by predicting that, for a certain eclipse at a certain place, light would bend not by 0.87 arcseconds, as Newton would have it, but by twice as much. Had Eddington’s measurement been different, he would have disproved (falsified) the theory.

The difference between the two concepts of intelligence—ability to understand, ability to act successfully—is also the difference between deductive approaches, which start from a theory and attempt to verify it through facts, and inductive ones, which start from facts and build up a theory. It is a deep difference, going back far in the history of thought. Among philosophers we find, on the conceptual/deductive side Descartes and Kant, and on the empirical/inductive side such English and American thinkers as Hume, John Stuart Mill, and behaviorists typified by Skinner.

Contrasting the Two Views

The appeal of the first view (“I am intelligent because I understand”) is its elegance and promise of powerful speeches. Its limitation is the difficulty of validating or falsifying it. Conspiracy theorists (including Marxists and Freudians) also make beautiful speeches. If you and I both have explanations for something, but they are incompatible, how do I convince you that mine is right and yours is wrong?

The appeal of the second view (“I am intelligent because I can make predictions that turn out right”) is its practicality. But how do we know that what it describes is really intelligence and not just careful record-keeping?

Old-AI, with its expert systems and logic-based tools, was of the first kind. The consensus is that it failed. Modern-AI is almost entirely (at least in its current, intermediate state of evolution) of the second kind. Modern-AI is machine-learning: it builds answers to new queries by extrapolating from a large body of validated answers to previous queries. Is it intelligence? Is the human-quality-level translation of today’s translation tools intelligent? Is a vibe-coding tool more intelligent than the programmer who uses it? Is a medical-image analysis tool which produces fewer false negatives and positives than a Stanford Hospital radiologist more or less intelligent than that doctor? For that matter, are non-AI programs such as a compiler intelligent? (No human could correctly compile a 100,000-line program in any reasonable time.)

With recent advances in AI, it becomes ever harder for proponents of intelligence-as-understanding to continue asserting that those tools have no clue and “just” perform statistical next-token prediction. Borrowing Kian Katanforoosh’s examples in a Stanford lecture,^c today’s deep-learning systems can complete sentences such as: “I poured myself a cup of …” (how is that not understanding co-occurrence patterns?); “The capital of France is …” (how is that not understanding geographical connections?); “She unlocked her phone using her…” (how is that not understanding semantic connections?); “The cat chased the …” (multiple plausible connections, so how is that not understanding probability?); “If it is raining, I should bring an…” (how is that not understanding inference?).

What do people mean, then, when they say “AI is not intelligence because it does not understand what it is talking about”? Since they do not define what “understanding” is, they may just mean “AI does not understand in the same way I do.” That is a tenuous argument. Similar to saying “Airplanes do not fly (do not believe your own lying eyes!) because they do not fly the same way as birds.”

I tend today to think that I was wrong back then, and find much to like in the empirical and inductive interpretation. But my changes of mind (a mind that could change again with new arguments and new technology) are not the subject of this discussion. What does matter, regardless of your personal preference for either of them, is the existence of two fundamentally different concepts of intelligence. Discussions of the “I” in “AI” are pointless unless they specify which one they use.

Artificial intelligence (AI) has rapidly transformed decision making in numerous domains. Acting alone or as a human companion, AI offers health suggestions, approves loans, detects criminal profiles, distributes digital content, and supports a wide range of additional decision-making tasks.³⁰ However, the hypothesis that AI would result in decisions that are more objective and rational than those made by humans has not been verified. AI algorithms frequently produce flawed or discriminatory outcomes, driven by unexpected and unpredictable biases embedded within opaque systems.²⁷ Since AI reflects not only the inequitable patterns found in society but also the fair or biased perspectives of its creators,¹⁴ it is clear that an exploration of sociotechnical bias is urgently needed.³⁰ In pursuit of fairness in AI (FairAI), computational methods such as resampling or reweighting have been used, but these are often inadequate and costly and rely on unrealistic assumptions, such as prior knowledge of which sensitive features are most bias-prone.¹⁸ A rich and emerging bibliography has detected bias in all phases of the AI lifecycle, including data collection (pre-processing), modeling (in-processing), and deployment (post-processing), and has identified a divergent set of computational fairness metrics for bias mitigation.^14,26,35,37 These studies have shown benefits; however, (computational) AI biases remain. At the same time, the rapid and widespread adoption of large language models (LLMs) has intensified the need to identify hidden and underexplored forms of bias. This is particularly relevant in models fine-tuned through methods such as reinforcement learning from human feedback (RLHF), which aim to produce outputs that are truthful, helpful, and harmless.⁸ However, because human feedback reflects subjective values and social norms, this alignment process can unintentionally reinforce cultural biases, dominant perspectives, or inconsistent expectations.^8,20

Our work is motivated by these pervasive AI biases and the need to understand their causes and effects in our new human–AI symbiotic reality. Humans make decisions driven by their own perceptions, and such human heuristics (mental shortcuts, motivations, time constraints, or other limitations) often result in systematic errors, known as cognitive biases.³⁶ Seminal work in cognitive science has identified more than 180 cognitive biases in human judgment, and the list is continuously evolving.² Unfortunately, such valuable work in cognitive science has up to now been overlooked in AI. As a result, we lack a deep exploration of the interdependencies³⁷ between human heuristics (cognitive) and AI (computational) biases. We believe we must foreground the human-side factors, which cause or steer the many AI biases,^25,30 under a new line of sociotechnical research designed to prevent harms, in particular to minority or underprivileged populations.^31,37

Unlike prior frameworks that focus solely on computational bias, our work traces the roots, forms, and manifestations of cognitive and AI biases. We introduce a systematic mapping framework that reveals interdependencies between human heuristics and AI biases across the AI lifecycle. This cognitive-to-AI bias mapping establishes a human-centric approach to FairAI that:

• Identifies particular actions as sources of AI (computational) biases

• “Rolls in the deep”^a of these actions’ causes to uncover their inherent cognitive (human-side) heuristics, which are projected into each of the AI lifecycle phases

• Introduces a systematic methodology to reveal the interrelationships between cognitive heuristics and AI biases, based on the evidence of a flexible exploratory analysis.

Our work is based on the science of human heuristics introduced by Nobelist Daniel Kahneman and other pioneering researchers,^16,22,23,36 and on state-of-the-art work on prominent AI biases.^{14,24,26,35,37} By bridging insights from cognitive science and AI research, we open a new dialogue for systematic, cross-disciplinary exploration of bias in AI systems. We aim to equip practitioners with a clear understanding of which decision points are most susceptible to cognitive influence, offering structured support for reflection, anticipation, and bias-aware planning and deployment. Our work provides a theoretical foundation for human-centric, bias-aware decision making through a cognitive reflection framework designed to guide both practitioners and policymakers, while supporting future empirical studies.

Computational Biases in the AI Lifecycle

Among the many types of computational bias explored in the AI lifecycle (pre-, in-, and post-), the most prominent and widely studied^{14,24,26,35,37} include:

• Historical bias, when a particular selective handling of certain groups or communities leads to them being advantaged or favored, while others historically remain disadvantaged or excluded. Such biases arise even if the data is sampled flawlessly, since selective preference may be inherent in the original data sources due to the majority following existing patterns or norms^30,36,37 For example, a mechanical engineering firm’s dataset might have inherent historical biases if it contains a significantly higher number of male employee records. Such an imbalance stems from the firm’s hiring norms favoring male candidates due to the belief that women are less capable.

• Representation bias, when non-representative datasets are generated by sampling from a population—that is, when samples are of small scale—or wrongly selected, neglecting underrepresented groups with fewer data points, resulting in the model’s failure to generalize for these groups.^26,35,37 For example, when a recruiting company trains an AI model using a data sample consisting primarily of high-salary male employee profiles, this results in biased hiring recommendations, since the model cannot recognize high-salary female employees or value female candidates.

• Measurement bias, when particular features and labels are chosen, collected, or computed, acting as “proxies” that poorly substitute for some construct (an idea or concept).^24,35 For example, if an AI model uses Body Mass Index (BMI: estimated by calculating an individual’s weight over her/his height squared) as a proxy for obesity risk, it can mistakenly classify individuals with greater muscle mass as obese, since BMI cannot distinguish between muscle and fat and fails to take into account key factors such as age, gender, or ethnicity.

• Algorithmic bias, when inappropriate design choices made mostly for a model’s architecture, functions, and hyperparameters, require several mathematical abstractions. Even under nondiscriminatory intent, these choices often lead to biases, since they show disproportional error rates against underrepresented groups.^37,38 For example, an underfitting model (due to its oversimplified nature) has restricted capacity in detecting important data patterns and yields low predictive accuracy, particularly for minority groups underrepresented in the dataset.^18,38

• Evaluation bias, when the benchmark dataset (used for a model’s evaluation) is not representative of the target population and/or inappropriate performance metrics are used.³⁷ For example, if an AI smile detector model is trained on a dataset lacking adequate representation of Asian female individuals and the testing benchmark is similarly unbalanced, mirroring the training set, a bias against Asian females will go unnoticed.²⁴

• Deployment bias, when an AI model is designed to solve a problem and then used to address a different problem in practice. This often occurs when an AI model is built and evaluated to provide recommendations, but is deployed as if it were fully autonomous. In this case, human reviewers overestimate its credibility, leading to a bias known as automation bias.³⁵

Given that all of the AI phases, but also the emerging LLMs, are largely driven by human experts, in the next sections we summarize humans’ inherent heuristics, then introduce our methodology for mapping these human heuristics to AI biases.

Heuristics and Biases behind Human Decision Making

The detection of computational bias—recently characterized as just the tip of the bias “iceberg”³⁰—has been criticized for its limited ability to account for the impact of human and systemic biases throughout the AI lifecycle.^14,31,37 To resolve this inefficiency, we explored the groundbreaking work on “judgment under uncertainty,” which has revealed that the broad spectrum of cognitive biases is rooted in three core heuristics: representativeness, availability, and anchoring and adjustment.³⁶ We also looked at Kahneman et al.’s Heuristics and Biases: The Psychology of Intuitive Judgement, which highlights the prime role of the affect heuristic in human intuitive judgements.^16,22,23 Below, we briefly summarize these heuristics, which serve as a foundation for the following section, which illustrates how these human cognitive tendencies can give rise to, or interact with, AI biases.

Representativeness refers to the spontaneous estimation of an event’s probability, based on how closely it resembles a known situation or prototype that exists in a person’s mind. People usually overweight certain information because they believe it is representative of reality, since this information is more recent, or has received more media attention, or is presented in a way that makes it seem more typical of a category. The representativeness heuristic leads to biased views based on stereotypes.^16,22,36 For example, a woman who is characterized as friendly and empathetic is more likely to be labeled as a nurse than as a mechanical engineer, since her traits fit the stereotype of a person in a care-giving role.

Availability refers to the mental shortcuts humans take when assessing the frequency or plausibility of an event, based on how easily examples come to mind. This attachment to immediate thoughts often prohibits further exploration of potentially significant information, since more-frequent instances are recalled better and faster than less-frequent ones, and repeated occurrences are preferred over rare events. A prominent cognitive bias in such cases is the so-called illusory correlation, occurring when people conclude that variables are correlated because their pairings come to mind easily (i.e., are quick to grasp, seem likely).^16,22,36 For example, the availability heuristic occurs when people assume that a high number of women hold leadership positions based on the visibility of a company’s “female empowerment” campaign. This campaign is easily retrievable from memory due to its regular marketing exposure, leading to an overestimation of the actual engagement of women in leadership roles.

Anchoring and adjustment occurs when people base their estimates or predictions on an initial piece of information (which may be completely irrelevant) that becomes ingrained in their minds. This information acts as an “anchor,” trapping people in their original opinions, even if they are unrelated to a newly presented case. Such anchor dependency may lead to a critical cognitive bias known as confirmation bias, which is humans’ tendency to favor information that confirms their initial views, causing them to make insufficient adjustments and selectively prefer data that reinforces their initial anchor point.^29,36 The anchoring heuristic can even influence individuals with a high level of knowledge or expertise engaged in comparative decision making, as they are swayed by multiple anchors stored in their minds.²²

Affect refers to quick decisions based on emotions and feelings, leading to reactions that are more emotional than logical or analytical. Kahneman’s fast and intuitive so-called System 1 thinking²² reflects people’s tendency to jump to conclusions and skip information gathering, resulting in a cognitive bias known as conclusion bias. Although humans may also engage in deliberate, slow, and conscious reasoning, “System 2 thinking,”²² their emotional attachments also drive them to seek information that confirms their preexisting beliefs, giving rise once again to confirmation bias.^22,23 For example, the affect heuristic may manifest in hiring decisions when a recruiter who feels disappointed by previous female employees unconsciously prioritizes male candidates as more capable leaders, despite evidence that new female candidates have equal or superior qualifications.

Given this radical, unprecedented mix of human and AI decision making, we must urgently reconsider the role of these human heuristics and respond to the next critical research questions. These include:

• How can we effectively define a solid methodology to reveal interdependencies between human heuristics and AI biases?

• Which human heuristics affect the most critical AI biases and when?

HAI-ROLL: A Framework for Mapping Human Heuristics and AI Biases 

Recent surveys have roughly highlighted harmful actions that may lead to AI biases.^{14,19,24,26,35,37} However, the origins of these AI biases are poorly referenced.¹⁹ We deviate from this restrictive line of research with our proposed framework, Human-to-AI Roll (HAI-ROLL), denoting a deep-dive methodological approach for mapping human heuristics to AI biases. HAI-ROLL positions harmful AI lifecycle actions as catalysts for mapping human heuristics and AI biases (see figure). The HAI-ROLL framework enables a sociotechnical view in several ways. First, it identifies and encodes the particular harmful actions (henceforth noted as [HA.xx]) that lead to unfair outcomes at each phase of the AI lifecycle (pre-, in-, post-). Second, it detects the specific AI biases that have emerged due to these harmful actions. And finally, it traces the origins of these harmful actions by exploring well-established human heuristics.

Our transitive reasoning responds to the above research questions since, ultimately, we roll into the deep evidence of human psychology to reveal the relevance, influence, and pathways linking human heuristics to AI biases. Our methodology is grounded in interdisciplinary scientific evidence, not only to detect correlations between human and AI biases but also to offer a systematic and adaptive framework for innovative, human-centric fairness-by-design strategies.

HAI-ROLL: Pre-Processing Phase

Data and feature selection are critical initial actions that impact the overall AI lifecycle. Since data scales prohibit the inclusion of the entire data population, certain issues are expected to arise. For example, some groups may be excluded or underrepresented or some features cannot be acquired,  leading to inaccurate, possibly harmful results.

HAI-ROLL: In-Processing Phase

In this phase, the AI model is trained to optimize a specified objective (e.g., maximize a reward function in RLHF or minimize a loss function). Up to now, the AI model’s efficiency has been tested by using available (benchmark) datasets (different from the training data), and measuring, mostly, its accuracy. However, the emerging deployment of AI in sensitive and critical domains has given rise to a range of important desiderata; for example, apart from accuracy, AI models must also be tested for their capacity to preserve fairness and privacy.^18,42 Thus, beyond the existing benchmarks used to estimate the model’s generalizability, it is now urgent to validate AI models under metrics that will assess their capacity to comply with legal and ethical rules.²⁸

HAI-ROLL: Post-Processing Phase

In this AI phase, model outcomes are used and interpreted, and the model is refined on the basis of user reactions. This iterative process triggers feedback loops, where a model’s output serves as new input for further refinements (e.g., by retraining) and reevaluation of the AI lifecycle. These loops aim to make AI models adaptive and dynamic under varying conditions.^25,37 However, reliance on previous AI outputs poses the risk of perpetuating biases, leading to the “butterfly effect,”¹⁵ where the origins of bias become unclear and untraceable, complicating AI fairness resilience.

HAI-ROLL in Practice: Implementation Guidelines and Limitations

Adopting the HAI-ROLL framework in real-world AI systems demands cautious configurations and deep understanding of its benefits and limitations. Given that it is the first framework (to our knowledge) that systematically explores the continuum from human heuristics to AI biases, HAI-ROLL is able to balance societal (human) and technical (AI) constraints and requirements. AI professionals (system designers and developers) are more aware of biases embedded in data or algorithms, basing their choices on existing fairness metrics and mitigation techniques.^2,6,14,28,37 As a result, they are less conscious of their own cognitive biases’ impact in shaping their decisions during the AI pre-, in-, and post- processing phases.⁶ HAI-ROLL offers AI developers an actionable guideline service to help them critically reflect on how cognitive (often unconscious) heuristics shape AI-lifecycle technical choices. At the same time, HAI-ROLL offers actionable guidance to policymakers, who will recognize the importance of a complete sociotechnical AI framework. HAI-ROLL supports AI developers in anticipating where biases may emerge and in taking preemptive action, while reinforcing their bias-aware critical thinking skills, encouraging them to shift to a reflective engineering mindset grounded in both technical evidence and psychological insights. In summary, HAI-ROLL offers a paradigm shift, revisiting human and AI synergies at each phase of the real-world AI lifecycle. In the following, we discuss the next steps forward.

The Border Gateway Protocol (BGP) underlies all modern Internet communication by enabling the exchange of IP routing information between autonomous systems (ASes). Despite its criticality, BGP is insecure by design, and to this day attacks on BGP are frequent and harmful.^8,16,38 The Resource Public Key Infrastructure (RPKI) was standardized to add cryptographic security to BGP, allowing ASes to validate BGP announcements, for example, through route origin authorizations (ROAs).²³ The publication of RFC6811 introduced route origin validation (ROV), enabling ASes to use ROAs to verify whether a given BGP announcement was originated by the legitimate holder of a prefix. RPKI also supports other validations, such as BGP path validation through BGPsec and route leak detection through AS provider authorizations (ASPAs). This ability to validate legitimacy guides routers in their decisions and prevents attacks on BGP.

Overview of BGP Security

BGP lacks cryptographic authentication of announcements, which enables a range of routing attacks. RPKI was designed to mitigate attacks on BGP by adding cryptographic security to the insecure BGP protocol.²⁵ RPKI provides an architecture to distribute verifiable BGP information globally. The figure provides an overview of RPKI from the perspective of a system operator. The red color illustrates untrusted RPKI components controlled by third parties, while the green color marks components trusted by the system and running within the local network.

Maturity of RPKI Specification

The Internet Engineering Taskforce (IETF) defined about 40 RPKI-related RFCs, which are generally complementary, each addressing a specific aspect of the RPKI ecosystem. Although the multiple standards aim to address missing or vague details and guide developers in implementing RPKI, their large number and often conflicting requirements increase complexity and hence the risk of bugs and vulnerabilities. Next, we explain potential problems arising from conflicting requirements or under-specification and provide concrete examples from our own evaluations and Internet measurements.

Maturity of RPKI Software Packages

Since the introduction of RPKI, implementations of RPKI software have emerged as open source projects, developed and managed by small groups of developers. Looking at the contribution history of the respective projects, the amount of active contributors (those who made contributions in the past 12 months) ranges from one to five people. All projects are open source and accept pull requests, with one implementation specifically encouraging contributions. While open source software is beneficial to improve the security, public participation, and trustworthiness of software, the XZ-Utils attack³⁴ has illustrated that especially projects managed by small developer teams may be subject to intentional introduction of malicious code from a sophisticated attacker. Three implementations are developed by known organizations located in North America, South America, and Europe, indicating more available resources for security management and the vetting of code contributions.

Maturity of RPKI Deployments

The challenges in deploying RPKI revolve around errors with registering network resources in RPKI and lack of experience with strict RPKI validation in production environments.

Emerging RPKI Attack Surface

The expanding adoption of RPKI in global networks also increases its attractiveness as a target for attacks. Therefore, it is important to extend the White House roadmap to also consider attacks against RPKI, and attacks through RPKI against its adopters. Such RPKI attacks may be closely directed toward RPKI functionality, for example, downgrading or circumventing hijack protections in victim systems, but can also target RPKI components as entry points into the network where vulnerable RPKI software is located. Enumeration of potential targets and development of directed attacks is straightforward, as the IP addresses of RPs and RPKI repositories are known, and they continuously communicate with other hosts in the Internet. Using enumerated targets, adversaries can attack the RP instances directly by setting up their own RPKI repository, or indirectly by attacking other RPKI repositories. Attacks often use specially crafted RPKI objects that disable or manipulate RPKI validation or, in the worst case, allow the attacker to obtain access to the RPKI-adopting network. What sets RPKI apart from other technologies is that all global RPs regularly contact all globally available repositories, allowing one server to attack all running instances directly. Widespread attacks are thus straightforward to execute, even for low-resource attackers. Further, due to the fail-open validation mode of RPKI, even DoS attacks—often considered low-impact vulnerabilities—have high impact in RPKI since they downgrade protection, leaving prefixes open for hijack. Despite the high impact of DoS in RPKI and the straightforward potential for widespread exploitation, previous work found an abundance of DoS and other vulnerabilities in RPKI, compromising its resilience and security. In this section, we discuss the RPKI-specific and general attack surface of networks.

Perfect Is the Enemy of the Good

Until recently, few people outside the Internet operational, engineering, and research communities were aware of RPKI. That changed in September 2024, when the White House identified RPKI as the key component for securing Internet routing, pushing RPKI from niche to mainstream. It might be expected that mainstream technologies are fully mature—in particular, fully stable and secure. As a niche technology, RPKI developed organically in many small steps, each inching a bit closer to maturity. But, as the previous sections have shown, RPKI is far from full maturity. What does this actually mean in practice? Did the White House push for the adoption of an immature technology, potentially doing more harm than good? Or did they promote the best available, “good enough” technology, motivating research and industry to speed up and put more resources behind improving RPKI?

Conclusion

RPKI implementations started as collaborative efforts between researchers, operators, and the broader IETF community. Over time, these efforts matured from experimental research projects and individual operator initiatives into deployments by some of the largest networks in the world. Our research shows that RPKI still suffers from problems and is not sufficiently stable, motivating more efforts by the community to increase its resilience, improve the specification, and mitigate vulnerabilities. Nevertheless, RPKI already delivers measurable benefits and is an essential part of ongoing efforts to improve routing security. Research shows that RPKI can substantially limit the propagation of invalid BGP announcements, hence mitigating traffic hijacks.²¹ RPKI also provides an important prerequisite for prospective routing-security solutions, including path-validation and route-leak prevention. The White-House roadmap is a huge leap for RPKI, and therefore also for Internet routing, to truly mature and meet the expectations of security, reliability, and scalability for production-level deployments across the global Internet.

Acknowledgments

This work has been co-funded by the German Federal Ministry of Education and Research and the Hessen State Ministry for Higher Education, Research and Arts within their joint support of the National Research Center for Applied Cybersecurity ATHENE and by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) SFB 1119.

The boundaries between human biology and intelligent technology are beginning to blur. Nowhere is this convergence more evident than in the emerging field of sensory augmentation, where artificial intelligence (AI), neuroscience, and wearable computing are coming together to redefine our fundamental senses.

For decades, the primary goal of this field was restoration, or developing technologies to help people regain lost sight or hearing. However, experts say that balance is now shifting. The field is moving beyond simple repair and increasingly is focused on augmentation: not just restoring, but actively enhancing and extending human capabilities.

This shift from restoration to enhancement recently was highlighted in a remarkable study. Researchers at Fudan University in Shanghai created an artificial retina that not only restored sight to blind animals but also gave them “super vision,” or the extraordinary ability to see infrared light, a spectrum invisible to them naturally.

This advance builds on decades of foundational work in sensory restoration. The most commercially successful and widespread example, the cochlear implant, already has restored functional hearing to hundreds of thousands of individuals by translating sound into electrical signals that are sent directly to the auditory nerve.

In the visual domain, devices like the pioneering Argus II retinal prosthesis (or “bionic eye”) proved that surgically implanted electrodes could restore a basic, low-resolution sense of sight to people with specific forms of blindness, establishing the viability of the “invasive” path of sensory augmentation that researchers are still pursuing. (The Argus II was discontinued after it failed to generate enough revenue and the company behind it, Second Sight, ran into financial issues.)

This acceleration is being driven by more than just implant technology. The true catalyst for the augmentation era is the processing power of artificial intelligence (AI).

AI is becoming essential for interpreting complex, real-world data before it is delivered to a human user. This is already common in advanced hearing aids, which use deep neural networks to isolate a single voice in a crowded room. This same computational power also is making non-invasive “sensory substitution” devices, which might translate a visual scene into complex soundscapes or tactile patterns, more sophisticated and learnable than ever before.

“While the motivation for restorative technologies remains high, we are entering an era increasingly focused on the augmentation (such as enhancement and extension) of human capabilities,” said Amber Maimon, a researcher specializing in sensory augmentation at Haifa University in Israel.

The Dual Frontiers of Sensory Augmentation

Sensory augmentation work is advancing along two major, parallel paths, according to Maimon.

The first is the invasive approach, which involves neural prosthetics and other interventions that directly interface with or modify sensory pathways. This is the category that includes the bionic eye, which aims to restore vision by electrically stimulating the remaining neural pathways.

The second is the non-invasive approach, such as sensory substitution systems that translate information from one sense to another. A key example is the EyeCane, a device that translates distance information into sound or vibration for the user.

“Both paths have been explored for decades, yet recent progress in neuroscience, materials, and computation is transforming what is possible,” said Maimon. Invasive neurotechnologies, for instance, are becoming increasingly sophisticated.

“Early visual prostheses relied on direct stimulation of retinal or cortical regions, producing only basic light or shape percepts,” she said. “Current approaches move toward higher-resolution, wireless, and biocompatible implants that combine advanced image processing with materials designed to adapt to neural tissue over time.”

Maimon said dense, flexible electrode arrays now enable more stable and long-term integration with the brain, as well as “improvements in wireless data and power transmission” that enhance both safety and reliability.

“At the same time,” she added, “emerging bidirectional brain-computer interfaces are forming closed-loop systems in which artificial sensors and neural circuits continuously interact,” bringing prosthetic vision closer to natural perception, or even extending its capabilities.

Non-invasive systems are evolving in parallel. Instead of surgically interfacing with the nervous system, these methods rely on training and active exploration to enable users to build new perceptual mappings. Research has shown, Maimon said, that the brain can reorganize itself to interpret these inputs as perceptual rather than symbolic. As a result, this path can yield impressive results while avoiding surgical risks and allowing rapid iteration, all in a way that can be tailored to individual users.

In fact, while high-tech implants capture the imagination, James Negen, a researcher at Liverpool John Moores University in the U.K., said the most exciting and effective technology in this space requires no electronics at all.

“Human echolocation is the objectively correct answer,” Negen said. “It is the only sensory augmentation approach to have a sustained community of real-world everyday users.” Negen pointed out that echolocation, a form of sensory substitution where a person uses clicks and their echoes to perceive their environment, has been “refined for decades” by scientists like Lore Thaler and practitioners like Daniel Kish.

Yet the most promising direction moving forward, said Maimon, is not pursuing one path over the other, but in developing hybrid systems that combine invasive and non-invasive approaches. By bringing together the precision and fidelity of invasive interfaces with the adaptability and accessibility of non-invasive ones, she said, new systems could “enable richer, more natural, and more adaptive forms of perception.”

“For example, a visual implant could convey the overall spatial structure of a scene directly to the visual cortex, while a non-invasive sensory substitution device provides complementary information about texture, contrast, or fine details,” she explained.

The Perception Problem

Despite the rapid progress on both frontiers, the dream of a truly functional artificial eye remains distant. According to Negen, when it comes to that specific goal, “We are so far away that we don’t yet know what we don’t know.”

“It is like asking Aristotle to lay out the path to the atom bomb,” he said. “We have only the most shallow understanding of basic questions like what physical properties give rise to conscious experience, what makes an experience perceptual, and how we come to feel ownership over a sensory stream.”

This philosophical hurdle points to a core technical issue.

“The greatest challenge is that perception is not simply electrical,” said Maimon. “It’s not enough to activate the right neurons or reproduce the correct signal patterns. The information must be organized in a way the brain can interpret and integrate, matching the timing, dynamics, and structure of natural sensory input.”

In other words, stimulation alone does not create perception; the brain has to make sense of what it receives.

Compounding this problem is the brain’s constant plasticity. Neural pathways reorganize as we learn and adapt, meaning an artificial implant “must be dynamic rather than static,” Maimon said. It must be “capable of adjusting not only to long-term neural reorganization, but also to the brain’s moment-to-moment variability across states of attention, context, and experience.”

Achin Bhowmik, CTO at Eden Prairie, MN-based Starkey Hearing, agreed, noting the technical challenges remain “formidable.” He cited key hurdles such as developing energy-efficient, real-time AI algorithms that can operate at the edge, creating biocompatible interfaces that communicate seamlessly with the nervous system and, perhaps most importantly, understanding how to harmonize artificial inputs with the brain’s perceptual and cognitive processes.

This challenge of creating a feeling of perception is also a challenge for non-invasive systems.

“It is not enough for users to decode patterns. The feedback must evoke a sensory quality that feels immediate and grounded in perception,” Maimon said, adding that it must do so without requiring extensive training or cognitive effort.

From Restoration to Enhancement

For much of its history, sensory augmentation was driven by a restorative goal. But as the technology matures, that distinction is fading.

“Looking ahead, I believe the distinction between restoration and enhancement will fade,” said Bhowmik. “Once we learn to emulate and extend our natural senses, the same technologies that restore hearing or vision will also enable people to perceive the world in entirely new ways.”

This new focus on augmentation generally falls into two categories. The first is enhancement, which aims to refine existing senses, such as sharpening vision, or improving touch or hearing. The second is extension, which moves further by granting access to sensory information humans do not naturally perceive, such as thermal, ultrasonic, or magnetic information.

This shift opens the door to a much larger potential user base.

“We have become much better at preventing sensory loss over the last century,” Negen said, explaining that “the potential user base for sensory substitution is much smaller than the potential user base for sensory augmentation.”

As new sensory skills combine with existing perception, they can enhance precision, making them a “true augmentation, not a fallback, even for someone without any sensory loss,” according to Negen. This creates an opportunity for niche but useful applications in areas like aging, working in hazardous environments, and perhaps even sports.

The final frontier, however, is the creation of truly novel senses that translate symbolic or informational content into sensory form.

“Imagine feeling air quality, perceiving electromagnetic fields, or intuitively sensing data streams,” said Maimon. “Such possibilities challenge our assumptions about perception, cognition, and the boundaries of human experience. They suggest that sensing is not merely biological, but something that can be designed, extended, and reimagined.”

This future is arriving in subtle ways. Bhowmik pointed to modern hearing aids, which have evolved far beyond simple sound amplification. “Modern hearing aids, for instance, use deep neural networks to separate speech from background noise, monitor physiological signals, and even detect falls,” he said. “They effectively function as miniature AI computers for the ear.”

Conventional memory schemes follow the Pareto Principle, in which approximately maintaining 20% hot data can meet 80% of requests. Large-scale applications, such as generative AI, recommendation systems, big data, and HPC systems, require large-capacity and high-speed memory and are changing the power-law locality, which necessitate the support of Big Memory. Big Memory is a transformative memory-centric system that consolidates massive and heterogeneous memory resources into a unified and shared address space, delivering near-DRAM latency and persistence at a terabyte-to-petabyte scale. This paradigm mitigates traditional I/O bottlenecks, simplifies data placement and programming models, and unlocks unprecedented scalability. Memory is one of the most contended hardware resources and often becomes the performance bottleneck in the I/O critical path that connects computing with storage. Due to limited memory capacity, more applications have to explore and exploit the memory beyond traditional local main memory, including empty memory on remote servers, and disaggregated memory in a memory pool.⁷ This non-local far memory can expand memory size and avoid memory stranding. Since far memory is much slower than local memory, existing systems leverage local memory as a cache by transparently swapping memory pages between local and far memory, which unfortunately causes read or write amplification.

In order to deliver high performance and achieve cost-efficiency, we need to carefully consider memory overhead that comes from the differential requirements of high-level applications and low-level devices. Specifically, high-level applications generally include peer-to-peer, blockchain, artificial intelligence, computability storage, and so forth. Different storage applications exhibit different needs and access patterns. On the other hand, the low-level devices become heterogeneous and highly hierarchical, which exhibit different physical properties.³ For example, hard disks, SSD, and PCM are non-volatile, and DRAM, cache, and registers are volatile. Hard disks and SSD are block-addressable, while others are byte-addressable. SSD and PCM suffer from the endurance with limited writes. These two-level different requirements, like a hamburger, introduce the high memory overhead, which means that to support program execution in memory, we need to consume extra costs for data movements and energy consumption, as well as high-security risks.¹ To alleviate high memory overhead, we build big memory that not only provides large capacity but also supports fast persistence. The persistence guarantees a complete lifetime for data to be finally written into non-volatile devices via protocol buffers or checkpoints.² Since most operations can be executed and completed within the big memory, we significantly reduce data movements through the memory and storage hierarchy.

In this hierarchical architecture, modern networking technologies like RDMA and CXL are erasing the boundaries between memory and storage, turning distributed resources into a seamless, low-latency memory fabric. CXL enables elastic, high-speed memory pooling across servers at near-DRAM speed, whereas RDMA facilitates efficient data transfers by bypassing CPU and software overheads. Instead of slow block storage, systems now operate on byte-addressable memory, where persistence is a property. Traditional cache, DRAM, and disk hierarchies become inefficient, thus replacing them with a scalable network-spanning memory system that grows in performance and capacity as the infrastructure expands.

The transformation of conventional memory into big memory systems leverages both vertical and horizontal extensions to existing memory hierarchies. Vertically, this expansion enables each node to host significantly larger datasets by increasing memory capacity and delivering high performance within a flattened hierarchy that coalesces traditional memory and storage tiers into a unified, high-speed access layer. Horizontally, the architecture scales out across distributed systems, integrating multiple nodes into a pool that collectively provides massive, scalable memory resources while maintaining efficient data access and coherence across the entire system. More data is aggregated together via atomic operation guarantee, which actually coalesces the central and distributed designs. Hence, by leveraging the vertical and horizontal ways, we build a big computer, as shown in Figure 1.

The foundational rationale for this architecture stems from the influential methodology of “the computer as a network.” This perspective draws a structural analogy between traditional computer hierarchy, which comprises low-level storage, intermediate memory, and high-level CPUs, and networked systems, where analogous roles are fulfilled by remote cloud resources, adjacent edge nodes, and end-user clients, respectively. This conceptual mapping allows for the application of well-understood network principles to the design of integrated computing systems, thereby enabling more scalable, efficient, and flexible resource orchestration across localized and distributed environments. In the context of a big computer, these items share some similarities. The long-latency storage seems like a remote cloud, the high-speed memory is like an adjacent edge, and the computing units are like clients that generate data. After being generated from clients/CPUs, data will be instantaneously transmitted to the close-by edges/memory, and consume a long latency to arrive at the remote clouds/storage.

This methodology can be implemented via existing devices. For example, non-volatile devices offer about TB-scale memory capacity, which can be considered a vertical extension. Moreover, from the horizontal view, RDMA and CXL protocols support memory and cache coherence.⁷ The big computer hence achieves persistence in both vertical and horizontal extensions. A program in a big computer can be atomically executed in an end-to-end way, whether from CPU to non-volatile devices or from clients to clouds.

Understanding the ‘Big’

In the context of big memory, we need to clarify the BIG, which is also the essential difference between big memory and traditional one. To fully understand the mentioned BIG, we first discuss other related BIG terms in the research community. Specifically, the difference between data and big data is the well-known Vs, including volume, velocity, variety, veracity, and value. Moreover, the difference between a model and a big model comes from algorithms, data, and computation. Compared with a chip, the big chip needs to consider the programmable, scalable, highly available, and adaptive properties. The salient feature behind BIGs is their Golden Rule.

The transition from conventional memory to “big memory” necessitates a clear understanding of their fundamental differences and the establishment of a guiding principle, that is, the “Golden Rule of Big Memory.” Vertically, this evolution signifies not only an expansion in capacity but also an extension in effective access distance, moving beyond local DRAM to incorporate broader hierarchies. Horizontally, it introduces immense scaling complexity in distributed systems deploying numerous GPUs for large model training. In such environments, horizontal scaling often relies on distributed shared memory to temporarily maintain intermediate results as replicas in remote volatile memory, offering probabilistic persistence but posing high risks of data loss. Without robust persistence guarantee, these volatile checkpoints consume excessive memory space and incur substantial broadcast overhead for synchronization and recovery.⁵ This problem is exemplified by large language models, whose checkpoint states reach terabyte scales, which far surpasses the capacity of individual GPU memories. This leads to significant performance degradation when remote GPUs cannot host entire checkpoints.

The Golden Rule

We believe that persistence is the Golden Rule of Big Memory systems. Contemporary big memory architectures have fundamentally reimagined data persistence, transforming it from a system constraint into a powerful performance enabler. This shift is driven by hardware-optimized approaches. Specifically, speculative persistence employs parallel RDMA replication to maintain multiple synchronized copies across nodes, enabling non-blocking durable operations. Deterministic persistence leverages CXL-connected NVM with efficient cache management to deliver memory-tier durability without traditional storage penalties. The result is a paradigm where persistence accelerates rather than hinders operations. In-memory databases achieve instant commits, machine learning systems maintain seamless checkpoints and distributed applications preserve consistency at memory speeds. By merging the performance of volatile memory with the reliability of persistent storage, this advancement is reshaping the foundation of memory-centric computing.

A case in point comes from an insight that both Von Neumann and Non-Von Neumann architectures need to guarantee the persistence. In practice, the memory/storage hierarchy in the Von Neumann architecture is a multi-level subset model, which includes disks, SSD, PCM, DRAM, L1/L2/L3 caches, and registers. In this subset model, the devices in the lower levels contain the data in the higher ones. All data needs to be written into persistent devices. On the other hand, for Non-Von Neumann architecture, persistence still plays an important role. A typical workflow in the memristors needs to compute the input vector in the matrix and the result is a new vector that is volatile. By using extra operations, the new vector can be finally persisted in the non-volatile devices. Hence, both Von Neumann and Non-Von Neumann architectures follow the persistence principle.

There exist some non-persistence operations in conventional systems, for example, replicating data into the DRAMs across servers or writing data in a batch through a cache/memory/storage hierarchy. These operations often cause misleading the understanding that persistence is harmful to high performance, which is actually not. To avoid the high overheads of rereading or recomputing the lost data, systems persist data but suffer from the huge gap between high-speed memory and low-speed disks. Many efforts have to be put into maintaining data in a volatile manner, which is vulnerable to just one system crash or failure. This awkward situation can be efficiently addressed using the big memory that offers large-size non-volatility with a shortened critical path. For example, eADR (extended Asynchronous DRAM Refresh) offers persistence for in-cache data and guarantees to flush them into non-volatile devices in case of system crash or failure. Hence, the number of flushing data is significantly reduced even during normal system execution. Systems thus deliver high performance with the aid of persistence.

Conventional computing systems operate on a heterogeneous hierarchy: fast, byte-addressable, volatile main memory, and slow, block-addressable, non-volatile storage. This separation necessitates data to migrate frequently and expensively between the two tiers to achieve persistence, which not only introduces significant performance overheads due to constant data swapping but also amplifies the risk of consistency bugs. Persistence needs to be guaranteed by storing states and data in non-volatile storage devices. A program hence transfers states and data to and from storage devices via an I/O bus, which is supported by leveraging specific instructions in a program. Persistence is important to deal with system failure and power down without data loss.

Persistent memory⁹ fundamentally rearchitects data persistence by merging memory and storage into a single tier, offering DRAM-like performance with disk-like non-volatility through a unified, flat address space. This design allows applications to maintain consistent states directly in memory and flush them to non-volatile media in batches,⁶ reducing the costly data movements while simultaneously avoiding security vulnerabilities associated with bus snooping. The flat addressing model simplifies error-prone persistence logic, enables near-instantaneous recovery, and permits direct manipulation of persistent data. However, while technologies like eADR ensure the persistence of data residing in multi-level CPU caches, data within registers remain vulnerable to power loss, as onboard power capacitors cannot efficiently flush register contents to non-volatile media. To address this limitation, modern systems employ checkpointing techniques to capture and guarantee the persistence of full-stack architectural states, ensuring comprehensive recovery integrity across the entire hierarchy.

Full-Stack and Moving Persistence

As shown in Figure 2, achieving full-stack persistence across the entire memory hierarchy represents a paradigm shift in system design, fundamentally moving the persistence boundary from the traditional interface between SSDs and DRAM directly into CPU. This transformation is enabled through a combination of hardware and software mechanisms. Persistent memory provides foundational data persistence at scale due to its inherent non-volatile physical properties. ADR ensures that data within the memory controller are flushed to non-volatile media during power events. eADR further guarantees that data residing in multi-level CPU caches are persisted. To address the volatility of registers, which remain beyond the reach of capacitor-based flushing mechanisms, systems employ frequent and expensive checkpointing techniques to capture computational states.⁸ Consequently, the entire hierarchy, from computational registers to large-scale persistent memory, now supports end-to-end persistence, enabling not only near-instantaneous recovery and simplified programming models but also a substantial expansion of effective memory capacity available for persistent operations.

Full-stack persistence requires the support from non-volatile devices and periodic checkpointing to facilitate fast recovery from system crashes or power failures. Although this hierarchy supports full-stack persistence, the implementations are different. For persistent memory, persistence can be naturally obtained due to its physical features. However, the memory controller and multi-level caches, which are volatile, require extra power to flush data into non-volatile devices, called moving persistence. Since data move via the I/O bus, there exist security issues due to bus snooping. These problems can be addressed by efficiently coordinating the hierarchy, capacity, and operations for high performance and strong security, thus achieving a suitable trade-off between performance and security.

For large-scale distributed systems, we conventionally leverage IP protocol to connect the hard disks in multiple nodes. RDMA is further used to aggregate memory resources together. CXL can achieve the cache coherence.⁴ These designs allow the processing boundary to move up and more operations can be completed in the higher levels. Since data need to be finally written into non-volatile devices, the end-to-end critical path can be significantly shortened. We hence achieve the savings of data movements and decrease energy consumption, as well as alleviating bus conflicts and side-channel attacks.

Quantitative Validation

To demonstrate the real-world impact of persistence, we select our state-of-the-art designs and real implementations for persistent memory, which provide detailed comparisons with standard non-persistent memory systems (that is, conventional DRAM-based systems). As shown in the accompanying table, the widely used evaluation metrics include throughput, memory overhead, and typical operation latency. The comparisons leverage various workloads from real-world applications and large-scale benchmarks. These workloads exhibit the typical patterns of existing and emerging applications in the context of big memory. We mainly show the average values due to space limitation. More details about configurations can be found in the experiments of our original papers. Moreover, applications consist of transactions, indexing, and security, which are widely used and well recognized in the memory community. Due to the heterogeneity of these applications, we present the evaluation results normalized to non-persistent memory to demonstrate the strength of persistence.

Persistent memory systems achieve significant performance improvements in all evaluation metrics. Specifically, in terms of throughput, the persistent systems achieve almost two times improvements compared with the non-persistent ones. The memory space can be significantly saved in most applications. Furthermore, the typical operations decrease the latency in the persistent memory systems. The main reason comes from the fact that the persistence guarantees consistency, integrity and reliability, which essentially decrease system overheads and avoid extra costs, such as frequent data movements and CPU/memory instructions.

Shaping the Future

Big memory systems, leveraging technologies like persistent memory, large-scale DRAM, and disaggregated memory architectures, have transformed data-intensive computing by enabling faster access to vast datasets. However, significant challenges remain in ensuring efficient and reliable operations. Crash consistency and durability in big memory systems pose a key challenge, since balancing performance with correctness is non-trivial. Memory disaggregation introduces latency and contention issues, while security concerns such as persistent memory vulnerabilities and secure memory sharing in distributed setups require further exploration. Moreover, programmability is still difficult, since developers lack universal models for managing big memory systems, and debugging persistent applications is still complex. Addressing these challenges is crucial for advancing big memory adoption.

To overcome these challenges, hardware and software innovations are essential. Hardware advancements, such as CXL 3.0+ for improved memory pooling and near-memory processing to minimize data movement, could significantly boost performance. On the software side, persistent memory-aware file systems, lightweight checkpointing mechanisms, and machine learning-driven memory tiering could optimize data placement across DRAM and NVM. Distributed coherence protocols and hardware-accelerated RDMA persistence mechanisms are also needed to ensure efficient memory sharing in disaggregated environments. Multi-tier memory techniques are narrowing performance gaps and enabling scalable, cost-effective memory systems. CXL, in particular, seamlessly integrates persistent memory, GPUs, and accelerators, allowing direct memory access and improving system scalability.

Beyond technical improvements, big memory systems unlock novel applications across various domains. In AI and real-time analytics, they enable in-memory machine learning training and large-scale graph processing. Databases benefit from NVM-optimized designs that support instant recovery and high-speed transactions, while scientific computing leverages big memory for in-situ visualization and simulations. Edge and cloud computing can adopt memory-as-a-service (MaaS) models for dynamic resource allocation. Modern networking technologies are flattening memory and storage hierarchy, creating a seamless, low-latency memory fabric. This evolution enables byte-addressable persistence, replacing traditional storage hierarchies with scalable, network-spanning memory systems. As research progresses, the focus remains on hardware-software co-design, intelligent memory management, and expanding use cases in emerging fields, thus paving the way for transformative paradigms.

Email subscription bombing (also known as subscription flooding or email spam bombing) is an attack technique that overwhelms a target’s inbox with a high volume of benign but unwanted emails. The emails themselves are benign and are sent by legitimate third parties (such as mailing lists), and therefore typically pass spam filters. The origins of this attack vector trace back to the beginning of the public Internet but have recently re-emerged at scale. Unlike classic email bombing, where attackers send large volumes directly, subscription bombing introduces a layer of indirection. The attacker registers the victim’s email address with thousands of third-party online services, which act as amplifiers that flood the target’s inbox with legitimate subscription confirmations and notification messages. Such emails are much harder to block automatically by the receiving email server.

Modern spam filtering relies on multiple signals, including sender reputation, email headers, and message content. For subscription attacks, the sending server generally has a good reputation and is not on a public blocklist; header checks (such as SPF, DMARC, or DKIM) are usually passed, and the subject and body text are different for each email and generally do not match typical phishing phrases that could be detected. While the flood of messages renders the inbox unusable, the true objective is often not just denial-of-service but to hide specific legitimate emails that arrive during the attack, thereby obscuring unauthorized financial transactions, account compromises, or ransomware deployment attempts. Early attacks scanned the Internet for sign-up forms and then scripted the sign-up process. In 1996, a stockbroker was bombarded with 25,000 emails, gathering some media attention at the time.¹ Since then, attacks have come in waves and were modernized. In 2016, hundreds of email addresses of the U.S. government were targeted, and some were subscribed to more than 10,000 newsletters.⁶ As organizations increasingly rely on digital communication channels, understanding the mechanics of these orchestrated distractions and their role in broader attack chains has become essential for maintaining operational continuity and financial security.

Beyond newsletters, attackers exploit other services capable of generating emails to a chosen target. This includes password reset requests, account registration confirmations, customer support forms, promotional emails, and social media notifications. These third-party services act as unwitting relays, with attackers hijacking their good reputation and bandwidth to execute the subscription bombing attack. We have monitored email trends since 2015 and have observed a recent increase in such attacks. In this study, we analyze 24 concrete subscription bombing attack campaigns using a large dataset from an email security provider, encompassing 46,970 involved unwanted emails. We define key metrics, outline attack timelines, and provide insight into the operational patterns of these campaigns. Furthermore, we examine subscription bombing services offered on the Dark Web and categorize their capabilities. Mitigating these attacks is challenging due to the large set of email senders, but automated unsubscription may reduce the impact. Given the simplicity of carrying out these asymmetric subscription bombing attacks, we expect the volume to grow in the future.

Subscription Bombing

Attackers typically abuse services that are well known, reputable, and easy to automate using sign-up scripts. Newsletters are the most common targets, followed by account registration confirmation messages. However, attackers have also been seen posting product listings, job postings, support inquiries, hotel bookings, or even apartment listings on online platforms, generating genuine inquiries from human users. The third-party services involved are not compromised or necessarily misconfigured, and no vulnerability in the email server is exploited. The attackers are simply abusing benign (but sometimes overly permissive) features.

The underlying security issue for all services is that, essentially, an attacker can register the victim, without their permission, who will then receive unwanted messages in their inbox. Key to eradicating this attack vector is that each service will have to improve its sign-up process with a two-step confirmation procedure. There are three key challenges here: a lot of distinct services will have to be updated; a service provider may not be interested in improving its security if it considers the number of registered users as a success metric; and even with proper validation the first verification email can still be used to flood victims. As a result, email service providers must mitigate these floods of unwanted emails. Existing academic research on subscription bombing¹³ so far targets bypassing filters^9,14 or detects unwanted emails through ML-based mechanisms.⁸

Some Dark Web services claim to have access to more than 100,000 mailing lists and services. Advances in artificial intelligence (AI) agents may further simplify the process of adding new services to the database of newsletters that can be abused, complete with all the necessary details about which HTTP POST or API request needs to be performed to subscribe new victims. Discussions with supporters of an underground subscription bombing service confirmed that they frequently update their lists with new services.

Subscription bombing is offered as a service but may be run by individual attackers themselves. All an attacker needs is a large list of newsletter providers and an automation script to subscribe the victim’s email. An attack consists of a list of victims’ email addresses that will receive the unwanted emails and a list of sender services which are sending the emails to the victims. A single attack wave consists of the attacker adding the victims address to a preselected number of services within a given timeframe. There can be multiple attack waves in a larger campaign against the same victim. The process of creating a subscription bombing service includes three broad steps:

1. Build a database of services that send emails by, for example, crawling the Web for newsletters.

2. Create a script that automatically subscribes new emails to a subset of the database.

3. Periodically verify the list and replace faulty entries with new ones.

Underground forum discussions indicate that some groups simply record the POST requests required to sign up for newsletters and then replay it with the new target address. This is not always trivial, as many websites use CAPTCHAs or cross site request forgery (CSRF) tokens to prevent single-call subscriptions. In the services abused in our attack case study, only 4% used a simple HTTP GET request to subscribe an email address. Around 70% used some form of nonce or CSRF token, requiring an initial HTTP GET request to obtain the token, followed by an HTTP POST request to complete the subscription. By using browser developer tools, an attacker can fill out the required form once, log all requests, and then export the necessary calls directly as a CURL command. This semi-automated process allows attackers to manually build a database. It is also plausible that some attackers targeted WordPress newsletter plug-ins and similar tools, as these allow them to analyze the subscription process once and replicate it across multiple domains. To test this hypothesis, we analyzed our dataset to identify the content management system (CMS) used by the websites, utilizing the service WhatCMS.org for identification. While WordPress was the most commonly used CMS among the sending domains (with a total of 16%, followed by Presta Shop with 6% and Laravel with 5%), we could not identify a single CMS plug-in or online shop environment that would indicate a massive monoculture.

With the rise of agentic AI and browser automation tools, attackers can easily commandeer a Web browser and have the agent automatically fill in the target email address in subscription forms. Services such as airtop.ai, fellou.ai, or axiom.ai can automate browser interactions and even assist with scaling and CAPTCHA solving. We successfully tested such a setup for newsletter subscriptions. However, to achieve performance and scalability, a paid service would be required. We have not seen any indication that attackers are already using these paid automation services for their attacks. This may change in the future with the evolution of AI tools. Since each newsletter service is hosted by a different Web entity, there is no central mechanism to detect or block multiple requests from the same IP address. Nevertheless, most attack services we analyzed claim to utilize proxies to rotate IP addresses and implement rate limitations to avoid getting blocked.

Large-Scale Dataset

Our dataset comprises 46,970 unwanted emails collected across 24 subscription bombing attack waves. The dataset includes email header, To and From address, and subject line. After receiving approval from the involved parties, we gathered the data in the period from Oct. 9, 2024 until Dec. 4, 2024 from midsize and large European customers. Note that collecting emails to form such a study is challenging, as the email service provider, the involved company, and individuals have to agree to the data collection. The dataset was collected by the email security solution provider xorlab AG and contains timestamp, sender email address, subject line, anonymized target email address, and extracted tags from email header.

The largest attack wave delivered 4,847 emails in under two hours or 3,387 emails per hour, while the smallest wave delivered 81 emails in one hour. The average number of emails per attack was 1,957 emails or 1,516 emails per hour. The average number of emails received by these accounts on normal days was less than 10 per hour. The attackers targeted various industries, including retail, technology, and manufacturing, with no clear pattern. The victims were typically individuals in management positions—director and above—with roles involving external visibility. Their email addresses were publicly accessible on open websites or easy to guess. Victims experienced varying levels of disruption: 71% of them were targeted once, 18% were hit twice, and 12% faced three separate attacks.

Half the attacks occurred on Friday (Figure 1), marking the last day of the work week for these targets. Friday is a common day for cyberattacks, as people may be inclined to leave early, allowing the attacks to go undetected over the weekend. Furthermore, IT security teams might operate with limited staff numbers on the weekend. The attacks most frequently began between 8:00 am and 10:00 am (Figure 2), likely to overwhelm victims at the start of the day when they are most likely to check their inbox. Eighty percent of the attacks began during the morning hours of the targets. No attacks were recorded from 10:00 pm to 7:00 am, suggesting the attackers researched the working hours of their targets for targeted attacks. This timing also aligns with the potential objectives of the flooding attacks, as attackers often need to interact with either the victim or a target service—actions that are not feasible during nighttime hours.

Case Study: Company Omega

The following analysis is focused on the data from two individual targets of subscription bombing within the same large international organization. Both persons work in upper management and were targeted with one day difference.

• Victim Alpha received 999 emails between Dec. 3, 2024 @ 10:19:45 and Dec. 3, 2024 @ 11:24:14

• Victim Beta received 719 emails between Dec. 4, 2024 @ 12:50:29 and Dec. 4, 2024 @ 13:55:08

There was no overlap between the attack waves in our case study, meaning each victim received emails from different domains. However, we observed minor overlaps across other attacks, suggesting that the pool of available senders is large, with only marginal overlap between randomly selected subsets. The top-level domains (TLDs) of the sending domains vary, with a slight bias toward European countries. This was also reflected in the diverse languages of the newsletter and registration messages. As these emails are automatically generated, 95% of sending email addresses were non-individual accounts (non-human identities). With 26.5% the info@ was the most frequently observed sender account, followed by noreply@ with 7%. Table 1 shows details.

Victim Alpha received 77 account reset emails, while Victim Beta received 49. All 126 services involved were neither financial nor high-profile social media platforms. Instead, they were primarily related to lifestyle and food websites. Therefore, it is unlikely the attackers attempted to take over any of these accounts for financial gain. Neither of the victims reported any issues with their accounts following the attack. Interestingly, many of these accounts had been freshly created just for the attack. The attacker first created the accounts and then requested a password reset for the newly generated account. The user names of the newly created accounts used random characters like DvqHvlAmxF. Another subscription bombing service improved this aspect and generated accounts with random plausible names like Daniel Williams, Michael Clark, Yoe Miller, or Hannah Fischer. Since the majority of the received emails were newsletters or account registrations, the subject lines commonly contained expected keywords like confirm, welcome, or email. Since the emails received are sent from genuine accounts on legitimate servers, the classic email header verification records like SPF (91.7% pass, 0.3% fail), DMARC (45.2% pass, 3.8% fail), and DKIM (74.3% pass, 12.3% fail) largely passed. Hence, they are not useful for filtering out these attacks. The failure rate is mainly due to badly configured email servers, which is common. Looking at the spam confidence level (SCL), where present, helps to assess if the email was classified as spam and would have been filtered out or would have ended up in the user’s inbox. SCL is a value from 1 to 9 assigned to a message that indicates the likelihood the message is spam. A value of 9 indicates an extremely high likelihood that the message is spam. A score of 1 is assigned to messages deemed not spam. In our dataset, more than 70% of the emails received a score of 1 and would have ended up in the user’s inbox if no additional email filtering was used. Around 28% were classified with score 5, which in a default Microsoft 365 configuration would be moved to the Junk folder.

Only 34.4% of the emails contained an unsubscribe link in the header, and only 16.2% contained a one-click unsubscribe link following the header, standard for list-unsubscribe=one-click. This low number is mostly because we observed many account registration emails and password reset notifications, for which in general there is no unsubscribe link being sent. Furthermore, many providers follow the best practice of double opt-in, where the user has to confirm their email before further messages are sent. If we take these into account, then we see we are closer to a 52% rate of emails requiring an unsubscribe, of which 36% had an unsubscribe link. The most prominent types of abused websites are online shopping websites (25% of our dataset), financial (13% of our dataset), and technology/education (10% of our dataset). These types of sites commonly offer newsletter and account registrations, making them ideal targets. Slightly more than six percent (6.4%) of the emails contained some information about where to complain, such as the X-complaints-to header field, whereas 29.4% had an abuse header like X-report-abuse. Apart from removing the recipient from the mailing list if the user complains, the service provider cannot do much to prevent this.

We conclude that for company Omega, the attackers knew the working hours of victim Alpha and Beta. They used a random subset of their mailing lists to flood them with thousands of emails per hour. More than 71% of the emails were not classified as spam by Microsoft 365 and would have been delivered to the user’s inbox if no additional filtering solutions would have been in place. Classic header checks (SPF, DMARC, and DKIM) as well as content and link analysis would not have helped prevent this attack. No malicious URLs or malware were involved at any stage. We suspect that the attackers had planned to follow-up with a fake support call to install ransomware, but did not succeed.

Goals of Subscription Bombing

There are multiple potential reasons for attackers to perform subscription bombing attacks against a given target. The most common reasons are: harassment or revenge, distraction from other attacks, pretext for follow-up attacks, or disruption of communication. Even if victims do not immediately recognize the attack as targeted, they quickly notice its impact through the rapidly growing inbox. The primary impact on the victim is the time required to clean up their inbox. Deleting thousands of unwanted emails is time-consuming, and bulk deletion risks removing legitimate messages. Since some of the spam consists of newsletters or subscription-based content, deletion alone is insufficient; more emails will continue to arrive unless the user unsubscribes. Some mailing lists generate high volumes of messages, potentially sending hundreds per day. The actual time spent depends on the email server in use, deployed security solutions, and the responsiveness of the IT team managing the incident. For illustration, assuming 1,000 spam emails, with 80% deleted in one second and 20% requiring 30 seconds to unsubscribe, cleanup would take approximately 113 minutes. Even at an average labor cost of $20/hour, the resulting loss is about $38—compared to the attacker’s estimated $1 cost to send the emails.

Subscription Bombing as a Service

Various cyber criminals offer subscription bombing on the Dark Web and underground websites, often with a free trial. The criminals typically boast that they can flood the target inbox with 1,000–10,000 emails per hour with a spam detection rate of less than 0.1%. One service claims it can generate 6,000 emails per hour with more than 100,000 newsletters and forums in their database. Generating such a service is not very difficult on a technical level. Analyzing the leaked chat messages from the Black Basta ransomware group³ shows they created their own subscription bombing service. Subscription bombing services are advertised using names like “emailbomb.cc”, “ebomber.cc”, “floodbox.org”, “diddy.pw”, “subscribethemall.com”, or “mailbait.info”. Note that attacking inboxes without consent is illegal in most jurisdictions.

As a case study, we explore one representative subscription bombing service, accessible via a publicly available website on the regular Internet. The platform primarily offers subscription bombing, though SMS/text message bombing is also available. Attacks against a specific target can be scheduled for a duration ranging from a minimum of one hour to a maximum of 31 days. Emails can be sent using a “drip feed” model, distributing the load over time, with adjustable rates from 75 to 3,000 emails per hour. Pricing is straightforward: 1,000 emails can be sent for as little as $1 using the standard list. A high-quality list, marketed as having a higher chance of bypassing spam filters, is available for $10 per 1,000 emails. The service dashboard provides an overview of all scheduled tasks and their completion status. Communication is handled through a Telegram channel, and support is also available via an integrated AI chatbot (tawk.to). For advanced users, the provider also offers a REST API to automate attacks further. Payments are handled exclusively through cryptocurrency via a gateway provider (pay.crypto-gateway.xyz), which currently supports BTC, LTC, and XMR.

In our own test using their free trial with the standard list, 21% (14/68) of the emails arrived in the inbox after 13 minutes. A second trial had 24% (929/3,930) of emails reaching the inbox. Note that attacking inboxes without consent is illegal in most jurisdictions. We only tested mailboxes directly under our control, on our servers.

By extrapolating from the order numbers in the dashboard, it appears that at least 300,000 tasks have been executed to date, assuming the count started at zero. During a test window of 767 hours, the task counter increased by 17,323. That represents 22.6 campaigns per hour. The activity level indicates ongoing interest for such services.

Table 2 shows prices for subscription bombing campaigns, taken from service offers across different underground websites. One of the scripts with unlimited access was available to buy for $500. The prices are very similar. The major variation is the quality of the list, which results in a higher inbox success rate, at a higher price tag.

Mitigation

As subscription bombing is increasingly used to target individuals and organizations, defenders must apply layered countermeasures. To minimize impact, organizations should:

1. Set thresholds or rate limits for the number of emails an inbox can receive from previously unseen senders.

2. Educate users about subscription bombing attacks.

3. Group similar newsletter or account reset emails during active attacks.

4. Automatically unsubscribe or bounce unwanted newsletter messages where supported.

Additionally, the receiving server may flag incoming subscription bombing attacks and alert the security team.

While securing the account itself through two-factor authentication (2FA) or zero-trust access (ZTA) is recommended, these measures do not mitigate subscription bombing. All the attacker needs is the victim’s email address, which could come from a previous data breach, or in most cases has been identified from social media or because of a predictable naming schema such as firstname.lastname@company.tld. When interacting with diverse services on the Internet, the email address naturally “leaks.” Some users create individual email addresses when registering with online services by appending the plus sign (e.g., yourname+newsletter@email.tld). While this allows the user to pinpoint where the email address leaked, it does not protect against the attack. The same holds true for security awareness training through phishing simulations. While those are useful for security in general, they cannot mitigate subscription bombing attacks.

A cleanup process based exclusively on keyword blocking is tricky, as this would also remove legitimate and wanted newsletters and emails. Moving all suspicious emails to a different folder for the user manually inspect them is not a viable option, as this would still require user time to sieve through thousands of emails.

Conclusion

Subscription bombing attacks are technically simple for attackers to conduct yet disproportionately difficult for victims to mitigate. Based on our analysis of 24 subscription bombing campaigns, this emerging attack vector poses a growing operational and security risk to digital communication. The attack itself provides only limited use cases for direct fraud but can be used as a distraction to hide concurrent attacks. Attackers leverage subscription bombing for various malicious purposes, ranging from creating nuisance to obscuring critical security alerts and facilitating social engineering attacks. Our study provides valuable insights into the operational patterns, timelines, and Dark Web services associated with these campaigns.

While complete mitigation remains challenging due to the asynchronous distribution between attackers and defenders, automated unsubscription offers a promising avenue for reducing the impact on victims. Legislative options such as requiring unsubscribe links are another option to reduce the impact of subscription bombing. Further research is crucial to develop more robust defense mechanisms against this evolving form of digital attack.

Acknowledgments

We thank Max Fischer for data collection and members of xorlab and HexHive for feedback. This work was supported in part by the ERC (grant No. 850868) and the SNSF (grants PCEGP2 186974 and 200021-236559).

In the rapidly evolving field of artificial intelligence (AI), rigorous evaluation is critical. Evaluation helps prevent disastrous outcomes in high-stakes applications such as autonomous driving and medical diagnosis,⁷ and helps stop the spread of racist, sexist, ableist, extremist, and other harmful ideologies.⁴ It also plays a key role in understanding how intelligent AI is, preventing the misallocation of resources, and guiding improvements in future model development.

Despite its importance, rigorous evaluation of versatile general-purpose AI systems such as ChatGPT and Gemini is challenging. The versatility of these systems, which can handle diverse tasks specified through text, audio, images, or video, makes it difficult to anticipate where users will apply them, rendering it impossible to test every potential task to ensure the systems function as intended. The core reason behind this challenge is the transition of AI systems from being designed for specific, well-defined objectives (tasks) to handling complex, dynamic scenarios where tasks are not predefined. A methodology to predict and explain AI performance on these diverse, potentially unseen tasks has not yet been established.

To tackle this challenge, we suggest transitioning from task-oriented evaluation to construct-oriented evaluation, taking inspiration from a methodology developed to evaluate humans. Constructs are concepts that, although they may not be observed directly, are hypothesized to underlie a range of behaviors.¹² We follow the common assumption in psychology and behavioral science³ that human behaviors can be explained by a limited set of constructs, such as cognitive abilities and personality traits.^a,36 In the same way, the wide range of outputs from an AI system can be organized and explained by a relatively small number of constructs. These constructs then become the fundamental building blocks of hypotheses and theories,³ allowing us to accurately predict and explain the performance of general-purpose AI systems.^b

In this article, we demonstrate how construct-oriented evaluation can be established for general-purpose AI systems through learning from psychometrics,³⁶ which for over a century has been applied to evaluate human psychological constructs. We first introduce three major advantages of adapting psychometrics for AI evaluation, illustrating them with concrete examples. We then discuss ways in which we can distinguish sound applications of psychometrics from oversimplified ones, warning against common pitfalls. With this foundation, we present a systematic framework grounded in psychometric principles and introduce an operational evaluation pipeline to provide practical guidance. Finally, we identify underexplored avenues and societal implications that open new research directions.

Advantages of Construct-Oriented AI Evaluation

As shown in Figure 1, compared with task-oriented evaluation, which employs benchmarks consisting of disparate tasks, construct-oriented evaluation excels in predictive power, explanatory power, and quality assurance, which can be achieved by optimizing the use of existing benchmarks and developing new tests for AI.

Leveraging Psychometrics for AI Evaluation

Psychometrics provides an effective evaluation framework, but its integration into AI evaluation should involve more than just applying existing psychometric tests. Key considerations must be addressed regarding how the constructs are conceptualized, how the tests are developed or adapted, and what psychometric techniques are available and appropriate. The ensuing discussion will elucidate related strategies and raise open questions that warrant joint efforts from the wider scientific community.

Psychometric framework: Key considerations.  We suggest a psychometric framework for construct-oriented evaluation of AI systems. As shown in Figure 2, our framework includes three stages: construct identification, construct measurement, and test validation. To facilitate understanding and practical application of the framework, for each stage, we list possible approaches, example techniques, and further notes for consideration.

Construct identification.  In the first stage, we identify the constructs that can explain and predict AI performance. In psychometrics, there are two approaches: top-down and bottom-up.

The top-down approach starts from a predefined theoretical framework, often derived from observations, expert consultation, and discussion. For instance, the construct of psychological resilience was identified through empirical observations that some children, despite facing adversities such as poverty, still demonstrated good adaptation and development.²⁸

We can adopt similar approaches to identify constructs that are predictive and explanatory of AI system performance. For example, Li et al.²⁴ have noted that AI systems often generate incorrect, fabricated, or misleading information, referred to as hallucination. A similar concept from psychology, confabulation, could help explain this behavior in AI systems. To reach a consensus on an operationalized definition of confabulation in AI systems, we can incorporate observations with the expertise of domain experts. During this process, we can use techniques such as the Delphi method⁴⁰ to collect expert opinion in a systematic manner and reach consensus on the definition of AI constructs. The construct then needs to be validated to ensure it is predictive and explanatory.

One feasible top-down approach is drawing an analogy between humans and AI systems. However, we should note that the definition of a construct may differ between AI systems and humans, despite the fact that they may be labeled the same. Suppose we are considering adapting the construct emotional intelligence for AI systems. As we check its definition, we may notice that descriptions about self-awareness of emotions are not appropriate for AI, or hold different meanings; hence, they should be removed or modified. The significance of a construct may also differ between humans and AI systems. For example, while confabulation is common in AI, it may appear in only a small group of humans with neurological conditions such as Alzheimer’s disease or Wernicke-Korsakoff Syndrome.²²

The bottom-up approach, rather than relying on a predefined theoretical framework, originates from empirical data and seeks patterns that suggest the existence of a construct. A classic example is the development of the Big Five Factors of personality, where psychologists carried out factor analysis on a vast array of personality-descriptive terms, extracting five constructs that explain a large proportion of the variance in human behaviors.¹⁴

A similar approach has already been applied within the field of AI research. Work by Burnell et al.⁶ applied factor analysis on data from 29 different LLMs across 27 cognitive tasks, finding that model capabilities could be explained by three well-delineated factors representing reasoning, comprehension, and core language modeling.

It is worth mentioning that bottom-up and top-down approaches could be used in conjunction, complementing each other to provide a more comprehensive understanding of constructs. These approaches could also be applicable to a wider range of scenarios, such as the evaluation of multimodal LLMs and agentic AI systems. Here, not only may hierarchical structures appear that require more advanced psychometric techniques, but also the range of constructs should go well beyond capabilities to also consider other traits such as self-control, values, biases, and toxicity—all of which are critical for ensuring the safety and responsible deployment of AI systems.

Notably, the conceptualization of constructs may evolve over time. For instance, self-esteem was initially viewed as single-dimensional but was then recognized to have multiple facets, including state self-esteem (self-worth in specific situations) and trait self-esteem (consistent or long-lasting self-worth).²¹ We expect a similar evolution of construct understanding in AI evaluation. Definitions of newly identified constructs in AI are likely to be partial or biased initially and may be subject to later refinement with enriched empirical evidence. This underscores the need for ongoing investigation in future research, adapting existing tools from psychometrics.

It is also important to note that when new constructs are identified for AI systems, they need to be validated over time to demonstrate their predictive and explanatory power before being accepted by the scientific community and public discourse. This is exactly what happened with self-esteem.

Construct measurement.  After a construct is identified for AI, a test is required to measure it. To develop a test for an AI construct, we can draw on the process by which psychometricians design tests for humans. It involves developing test items and establishing scoring criteria.

Test items should sufficiently reflect the targeted construct and remain unaffected by unrelated constructs. They are usually designed according to test guidelines and item specifications.⁸ Test guidelines specify the proportion of items measuring different aspects of the construct, the item format, the test length, time limits, and so on. Item specifications provide detailed designs for individual items by specifying the source of item content, the context, stimuli, and the difficulty level of an item.

Test items for AI systems can take diverse formats, as used in psychometrics. For example, psychometric tests use highly abstract symbols or graphics, such as the abstraction and reasoning corpus in BIG-bench and those used in Raven’s Progressive Matrices,³³ to eliminate the interference of unintended constructs in real-world scenarios. When applied to AI evaluation, this could help minimize the unintended influence of random factors and enhance test reliability and validity. For instance, to assess inductive reasoning ability, we can develop abstract items that ask test takers to uncover the pattern in a sequence of numbers and generate the number that continues the sequence. Another potential format is simulation-based assessment,³⁰ which aligns more closely with some current AI benchmarks. These test items employ real-world situations to increase ecological validity while leveraging techniques such as evidence-centered design³⁰ to ensure the target construct is measured effectively.

The choice of test-item format should align with the application scenario. For instance, simulation-based test items are useful when leveraging existing AI benchmarks to reduce the effort required for test-item development or when evaluating model performance in specific real-world scenarios.

After the test items are developed, a scoring scheme based on appropriate measurement theories should be established to quantify the level of the construct given the obtained responses. Among existing scoring theories, item response theory (IRT)¹² appears particularly useful for AI evaluation.⁴⁴ IRT effectively predicts whether a test taker will get any item, including unseen ones, correct or not by modeling the probability of a correct response as a function of item parameters and the test taker’s level of the construct. IRT also enables comparisons of AI systems, even when they are evaluated using different tests with varying test formats, item sets, or difficulty levels because it allows the construct and item parameters to be estimated on a unified scale. This is very beneficial, particularly considering that tests for AI need to be frequently updated to cover potential exposure of items in the training data. Furthermore, IRT lays the groundwork for computerized adaptive testing,⁴⁴ which optimally selects items for each test taker, resulting in efficient and precise assessment.

In comparison with the classic scoring method that simply aggregates the scores for each item, advanced techniques such as IRT provide more fine-grained, accurate, and comprehensive AI evaluation. Given these advantages, some studies have already adopted IRT. For example, Zhuang et al.⁴⁴ employed this technique to assess LLMs’ subject knowledge, mathematical reasoning, and programming skills. We expect more insights to be gained from state-of-the-art IRT models—for example, IRT-based cognitive diagnostic models that aim at pinpointing specific facets of strengths or weaknesses in a construct,¹⁰ and IRT-based latent class models that combine the strengths of IRT with the idea of identifying the hidden groups that test takers belong to.³⁴

Test validation.  Imagine we have created a test to measure a certain construct. Before drawing conclusions based on the test results, it is necessary to verify the test’s quality. For instance, how much error is in the measurement? How do the test results relate to real-world behaviors? For this purpose, psychometrics provides a systematic methodology concentrating on indicators of reliability and validity in the measurement.

Reliability refers to the consistency or stability of a test.³⁶ Psychometrics introduces various reliability indicators. For instance, test-retest reliability involves giving the same test to the same group of test takers multiple times to evaluate the stability of the measurements. Practically, one could test an AI system multiple times to obtain performance stability (replicability). Internal consistency reliability measures consistency across items within a test. High internal consistency reliability indicates that an AI system performs similarly across all test items. Other types of reliability that might be adapted into AI evaluation include parallel-form reliability, which ensures that two different but equivalent versions of a test measure the same construct consistently, and inter-rater reliability, which assesses the degree of agreement among different raters and has been adopted for evaluating text-generation models.^25,42

Validity indicates the extent to which a test measures the construct it claims to measure.³⁶ There are different validity indicators in psychometrics that may be applied in AI. For example, construct validity indicates how well the test reflects the construct. A commonly used method to evaluate construct validity is factor analysis, which conceptualizes the constructs as latent factors underlying test items. By examining the fit of the specified factorial structure with empirical data, we can quantitatively understand the construct validity of the test. Construct validity can also be evaluated via convergent validity, reflecting the empirical relation of theoretically related measures, and discriminant validity, indicating whether measurements that are not supposed to be related are, in fact, unrelated empirically. Predictive validity is another form of validity that is concerned with the extent to which a score on a test predicts performance on a certain criterion measure, which can be real-life performance or future outcomes. For example, AI systems with stronger emotional understanding should receive better user reviews.

Some recent AI evaluation studies have carried out reliability and validity analyses. For example, Li et al.²⁵ employed test-retest reliability that examined score stability across different time points to compare measures of readability. They also assessed convergent and divergent validity to verify that measures of readability and faithfulness were tapping into the intended constructs. METRICEVAL,⁴² a framework that examines both concurrent validity and construct validity, has been proposed for conceptualizing and evaluating natural language generation metrics. This framework has been applied to the evaluation of summarization metrics.

The above-mentioned studies have demonstrated the effectiveness of reliability and validity analyses in a few specific areas (e.g., summarization, faithfulness, and readability); however, many important areas—such as evaluating the coding skills, problem-solving abilities, and creativity of AI systems—remain largely unexplored. We advocate broader and more systematic adoption of these measures across all AI evaluation settings. Only with adequate reliability and validity can we trust the test results and answer questions that are important in real life, such as: Does a higher problem-solving test score from GPT-4 than from GPT-3 mean increased user satisfaction when interacting in online research?

However, not all psychometric indices are universally applicable. Depending on the specific scenario, different reliability and validity indices should be employed. For instance, inter-rater reliability is particularly useful when human raters are involved, but is not applicable to scenarios without raters.^25,42 Predictive validity requires a criterion measure, which is often challenging to obtain; hence, it has been omitted in some validation studies.⁴² In practice, these psychometric indices should be carefully examined before being applied to AI evaluation. Researchers need to understand the fundamental principles behind the concepts of reliability and validity in order to develop appropriate AI-specific metrics that meet the purposes of the research.

Opportunities

The integration of psychometrics into AI evaluation opens up exciting opportunities. In this section, we explore both the open questions that need addressing and the potential for expanding early psychometric applications to other AI domains more broadly.

Extending to AI research more broadly.  Extending the psychometric framework for construct-oriented AI evaluation, we see further research opportunities that psychometrics presents for all areas in AI.

Evaluation of human-AI teaming.  Considering AI systems’ great potential for enhancing human productivity, optimal human-AI teaming is receiving increasing attention. Human-AI teaming is a collaborative process where decisions are made jointly, with both parties learning from each other and adapting accordingly.² Thus, its evaluation should not only consider task outcomes but also encompass the performance of individual components (both human and AI), the quality of collaboration within the team, and the success in achieving shared objectives.² In this context, psychometric models and methodologies, particularly those related to collaborative problem solving, may offer valuable insights.³⁸

Transforming the AI pipeline.  Psychometrics has important social impact. For instance, educational assessment, a major application of psychometrics, is vital in shaping human education practices.²⁰ Similarly, AI evaluation could extend beyond measuring AI system performance only after it is trained. Instead, rigorous evaluation should guide each stage in AI development, as requested by several regulatory frameworks, such as the EU’s AI Act.¹³ Inspired by educational practices, in Figure 3 we illustrate how psychometrics could help redefine the AI pipeline.

In the first stage, objectives such as problem solving are identified and integrated into teaching and assessment to enhance future success in education. Likewise, psychometrics can help identify essential AI constructs for complex, unforeseen tasks, for example, critical thinking, and values such as security, aligning with Schwartz’s theory of basic values.¹ In the second stage, psychometrics can identify high-potential systems and optimize training resources accordingly. Throughout training, continuous feedback based on psychometrics can refine the process toward predefined objectives. In the final stage, after examining the reliability and validity of the tests, we can confirm that AI systems meet real-world challenges and societal integration requirements, such as mitigating harmful biases, ensuring accountability in automated decision making, and reducing hallucinations. By rigorously assessing capabilities, safety, and fairness, psychometrics helps build AI that not only performs accurately but also upholds societal trust and equitable outcomes.

Conclusion

When evaluating AI systems, the current task-oriented paradigm falls short in key areas such as predictive power, explanatory power, and quality assurance. These limitations could be addressed by focusing on constructs and placing psychometrics at the core of AI evaluation. Our work in this area makes it clear that this transition is both imperative and possible, and presents methodological guidelines to accelerate adoption. Using a three-stage framework, here we demonstrated how psychometrics can be leveraged to overcome the limitations of current benchmarks while avoiding the pitfalls of oversimplified psychometric application. Integrating psychometrics promises significant advances in AI evaluation but also presents unique challenges, such as reinterpreting human-centric concepts for AI systems and managing prompt sensitivity. Our discussion opens up new avenues for research, including the potential evaluation of human-AI teaming and the transformation of AI development practices. While psychometrics alone may not solve all issues in AI evaluation, the transition to construct-oriented evaluation is crucial and we look forward to a future where psychometrics-grounded AI evaluation also shapes the evaluation of humans.

Acknowledgments

This work was supervised by Luning Sun, Fang Luo, and Xing Xie. The authors would like to thank Jinyan Fan, Marija Slavkovik, Clemens Stachl, Alina A von Davier, Xiangen Hu, Yu Lu, Bryan Maddox, Mengxiao Zhu, Meng Li, Li Dong, Jindong Wang, Igor Sterner, Greg Serapio-García, Peter Romero, Fengli Xu, Fernando Martínez Plumed, and Lidong Zhou for the discussions and comments on early versions of the manuscript. This research was funded by Beijing Natural Science Foundation L247011; Microsoft Research Asia Collaborative Research Program grant FY23-Research-Sponsorship-422 “The convergence of assessing human and big model capabilities”; CIPROM/2022/6 (FASSLOW) funded by Generalitat Valenciana, and Spanish grant PID2024-162030OB-100 (ROBIN) funded by MCIN/AEI/10.13039/501100011033 and ERDF A way of making Europe, Cátedra ENIA-UPV in Sustainable AI Development, TSI-100930-2023-9, and INCIBE’s Chair funded by the EU-NextGenerationEU through the Spanish government’s Plan de Recuperación, Transformación y Resiliencia, and EUR2024-153548 (PREDAIT) “Towards Predictable AI” from “Spanish Europe Excelencia” 2024; National Natural Science Foundation of China (NSFC) (NO. 62476279, NO. 92470205, NO. U2436209); Major Innovation & Planning Interdisciplinary Platform for the “Double-First Class” Initiative, Renmin University of China; The Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China No. 24XNKJ18; Fund for building world-class universities (disciplines) of Renmin University of China and Public Computing Cloud, Renmin University of China. L.S. and D.S. gratefully acknowledge financial support from Invesco through their philanthropic donation to Cambridge Judge Business School.

Artificial intelligence (AI), especially the new generation of increasingly autonomous, agentic AI systems, has triggered understandable concerns about privacy. These systems can read our email messages, draft our documents, navigate our calendars, answer our questions, and even act on our behalf. They observe, analyze, and infer, often continuously. They can derive sensitive attributes from our digital traces and, with growing autonomy, sometimes initiate actions based on these inferences. Many of the privacy fears surrounding AI are real. But as paradoxical as it may seem, AI, including agentic AI, is also becoming essential to protecting privacy.

This column argues that without AI, adequate privacy has become simply out of reach. This is not because AI is benign; it most definitely is not. Rather, the modern digital ecosystem has evolved to a point where no human, unaided, can understand, monitor, or manage the complexity of today’s data practices. For two decades, my collaborators and I have studied why people struggle to manage their privacy and why this struggle keeps getting worse despite decades of regulation, policy work, and advances in privacy enhancing technologies. From mobile applications and IoT devices to location-sharing, video analytics, websites, and AI chatbots, we have found the same underlying truth: privacy is too dynamic, too contextual, and too cognitively demanding for people to manage manually. When it comes to managing privacy, AI is not merely helpful. It is indispensable.

The Challenges AI Creates for Privacy

Before explaining why AI is also indispensable for privacy, we must acknowledge the genuine challenges it introduces. Agentic AI systems can collect data about us over extensive periods of time (for example, persistent interactions lasting months), influence our decisions, and autonomously act on data they collect about us. They can personalize at unprecedented levels, profiling individuals with a granularity that far exceeds traditional analytics. They can infer emotions or health indicators from a camera feed, often without explicit consent.

Our research on the privacy implications of video analytics explored how surprisingly powerful inference capabilities have become and how little users know about them. Many people do not realize that cameras can detect mood, infer demographic traits, or extract behavioral cues even from innocuous footage (for example, identifying whether someone appears frustrated, distracted, or stressed). Studies reveal that user expectations are both diverse and highly context dependent: people may accept analytics for safety monitoring in public spaces yet find it unacceptable in workplaces or places of worship. Expecting people to systematically configure privacy settings to align with their individual expectations is simply unrealistic.

Agentic AI’s power to act further exacerbates the threat landscape. A system capable of autonomously generating email messages, modifying settings, and communicating with other agents or systems may inadvertently disclose sensitive information its user would not want to. These challenges are serious and require thoughtful governance.

Yet, despite these risks, the complexity of modern data ecosystems has outgrown the capacity of humans (and of traditional software) to manage privacy effectively. AI can help and is likely the only viable option.

The Privacy Landscape Has Become Too Complex

The mythical assumption that people would somehow be able to manage their privacy by glancing at a website’s privacy policy and toggling a handful of settings has never been farther from reality. Consider the mobile ecosystem alone. Our research shows that users face dozens or even hundreds of privacy decisions across their apps, but their preferences are nuanced, contextual, and vary widely depending on purpose, trust, time of day, and other factors. Expecting people to get acquainted with these nuances by reading privacy policies and carefully configuring privacy settings is simply unrealistic.

The same is true in many other contexts. For instance, in video analytics contexts such as those associated with the deployment of these technologies in malls, stadiums, but also in some gyms, schools or even religious venues, preferences vary according to the purpose of analytics, the venue, retention practices, and the level of identifiability, producing a multidimensional policy space that no static interface can hope to adequately capture.⁹ In short, the scale at which we interact with technologies, and the complexity of the dataflows they give rise to far exceed the limits of human attention, understanding, and time.

Regulatory Efforts Help … but They Also Increase Complexity

Regulations such as the GDPR or CCPA, as well as a growing collection of AI-focused frameworks and regulations, aim to improve transparency and accountability. They enshrine important rights: access, deletion, correction, opt-out, opt-in for sensitive data uses, purpose limitation, and more. They are vital steps forward, especially as AI-driven inference capabilities expand.

But regulations such as GDPR and CCPA have also made privacy notices longer—often dramatically so. A recent Princeton study found that over the last 20 years privacy policies have become significantly more difficult to read and doubled in length.²

New regulations also offer more rights to users. This means more notices, more interfaces, more choices and increasingly more granular control to deal with. Yet few users have the time and motivation to find or exercise these controls. In fact, despite recent requirements to make these choices easier to identify, many people still struggle to locate and use opt-out links, account deletion interfaces, or the mechanisms required to invoke other legal rights (see Habib, H. et al.³).

In the U.S., new provisions such as CCPA’s recognition of Authorized Agents, namely entities that are allowed to submit requests on the user’s behalf, and emerging standards such as the Global Privacy Control (GPC) signal represent meaningful progress. They establish pathways for intermediaries, potentially AI systems, to express user preferences on their behalf and contribute to reducing user burden. But currently, these mechanisms mostly apply to narrow categories of data sharing (for example, “Do Not Sell” signals), far from addressing the full spectrum of privacy challenges consumers face and far from being able to handle the diversity in preferences people have been shown to have.

The paradox is clear: Regulators, in trying to empower individuals, have also created an environment where only AI can realistically help users understand and exercise their rights.

Privacy Assistants: Early Vision Meets Today’s Reality

Approximately 20 years ago, my colleagues and I began exploring the idea of privacy assistants, software agents capable of learning users’ privacy preferences, reading and interpreting privacy policies, nudging users to engage with available privacy settings, helping them locate and configure these settings, and respond to their privacy questions. Much of that early work anticipated today’s challenges: highly contextual data sharing rules, continuous sensing environments, AI-driven personalization, and the need for machine assistance to manage both complexity and volume.

Our early work outlined semantic Web architectures for contextual privacy controls and rule-based reasoning about data practices in pervasive computing environments—nowadays more commonly referred to as the Internet of Things (IoT). Quickly this work led us to explore the complexity and diversity of people’s privacy expectations and preferences in different domains (for example, location sharing, mobile app permissions, and IoT contexts). These studies repeatedly showed that user preferences could be clustered into interpretable personas, and machine learning could help users quickly converge toward reasonable configurations without overwhelming them with choices and without taking away their sense of agency. Additional personalization was also shown to help further improve recommendations. We eventually piloted and released a mobile app privacy assistant to help users configure their mobile app privacy settings on Android phones with users reporting that they generally liked the way the assistant worked and followed its recommendations.⁷

Around 2012, we also started exploring the use of machine learning (ML) and natural language processing (NLP) techniques to automatically read and interpret the text of privacy policies, taking advantage of recent and ongoing advances in ML and NLP. This work, which has since inspired a number of follow-on efforts, initially relied on the tedious collection of extensive corpora of policy annotations to help train classifiers to identify and analyze increasingly rich sets of data practice disclosures. Over time, this work incorporated and extended advances in the field (from word embeddings, RNNs, LSTMs, and CNNs, to the more recent transformer models).^a We showed that it was possible to reliably identify and classify a wide range of statements and use this analysis to answer people’s privacy questions—see our work on Privacy Question Answering Assistants as well as related work by others.^5,8

We also demonstrated how these techniques could be extended to automatically identify and classify opt-out choices often buried deep in the text of privacy policies and surfaced these choices through our Opt-Out Easy browser extension: the extension was made available in the Chrome store and in many ways anticipated how today’s agents can be used to automatically identify links and buttons and navigate websites for users.⁶

In yet another line of inquiry, we experimented with privacy nudges designed to motivate people to pay attention and engage with privacy settings made available to them.¹ Some of this work eventually found its way into commercial products and much of it also inspired others to further explore this space. For instance, nudges we designed and piloted to motivate mobile app users to revisit their privacy settings were eventually adopted by Apple and introduced in iOS13, and our use of ML to learn people’s privacy preferences influenced how Google helps users manage Chrome permissions.⁴

What is particularly exciting today is how LLMs and agentic AI systems finally make it possible to implement the full vision: conversational, proactive, personalized, and delegated privacy management at scale.

Personalized privacy assistants capable of automatically reading privacy policies, identifying privacy settings, helping us interpret how to best align these settings with our expectations and preferences, and warning us about data practices we may not expect, offer the prospect of rebalancing the privacy landscape and restoring people’s control over their data. Their ability to engage in dialogues with us, to check how we feel about different practices, to help us better understand the ramifications of our decisions, to manage many repetitive and tedious privacy decisions on our behalf, and, over time, to build detailed models of our individual preferences and of the most effective way of communicating with us could fundamentally shift the balance.

They can drastically reduce user burden and help bridge the gap between what people care about and confusing jargon and assortments of privacy choices (e.g., understanding the difference between opting out of tracking by mobile apps on an iPhone and submitting an opt-out request under CCPA). Thanks to their ability to rapidly scan websites and apps, they can lower the bar when it comes to discovering available choices and helping users take advantage of them. Perhaps the most transformative capacity of agentic AI is its ability to act. New legal provisions recognizing authorized agents in regulations such as CCPA are an important first step in this direction, though more is needed to lower friction and empower agents to manage our privacy without undue burden on their users (for example, under current practices, CCPA opt-out or deletion requests submitted by authorized agents generally require manual verification by the user).

AI for Compliance and Threat Modeling

AI is not just critical for users. It is equally critical for organizations seeking to comply with increasingly complex regulation. Our work on mobile app privacy compliance analysis (MAPS) laid early foundations for compliance automation and, in particular, for automatically detecting discrepancies between disclosures made in mobile app privacy policies and the code and behavior of these mobile apps.¹⁰ These technologies have influenced the development of products such as Google Checks.

More recently we have been experimenting with agentic AI to semi-automatically identify shortcomings in the design and implementation of privacy notice and choice interfaces, namely determining whether notices and choices are clear, easily discoverable, meaningful, and free of deceptive design patterns. Here we use agentic AI to navigate websites and operationalize a taxonomy of user-oriented threats to spot areas where notice and choice interfaces fall short—see our UsersFirst privacy threat modeling framework.^b We believe that these are just early efforts in this space and expect that similar approaches will be developed to automate or semi-automate compliance analysis as well as the identification and mitigation of increasingly rich classes of privacy threats.

A major reason why so many products and services continue to be non-compliant and fall short in adequately mitigating risks has to do with the very significant manpower required to conduct these types of analysis. By automating or semi-automating some of these processes, AI could help drastically change today’s sad state of affairs.

A Future Where AI Is Essential for Privacy

For decades, the field of privacy has operated under the assumption that transparency, control, and user choice could empower individuals to manage their privacy. This vision was well intentioned but deeply optimistic about human attention, comprehension, and time. Today, the diversity and complexity of dataflows with which people interact on a daily basis makes it painfully obvious that this assumption is not tenable and no traditional interface will change that. AI is not a threat to privacy by default. It is a threat when unregulated, misaligned, or opaque, and it can certainly be a significant threat when used to manipulate users to disclose more data and influence their decisions. But when designed responsibly, aligned with user interests (working as their privacy assistants), and integrated into regulatory frameworks, AI can become the only realistic path toward making privacy meaningful in an era of ubiquitous computing and pervasive data collection. By drastically reducing the burden required to manage one’s privacy today, agentic AI in particular offers a path toward restoring a sense of control over the collection and use of our data and the prospect of privacy compliance issues and threats more readily identified and mitigated.

Simply put, AI has the potential to be the long-awaited game changer for privacy.

The sociologist Ron Westrum coined^a the phrase “The Centrality Fallacy,” whereby people in positions of authority incorrectly assume that due to their centrality they have the best information on an issue and, consequently, are positioned to make the best decisions. To use an analogy by the philosopher John Dewey: The shoe wearer understands better where the shoe is poorly fitted, whereas the cobbler understands better how to address the problem of poor fit.  Two recent developments within the Association for Computing Machinery (ACM) offer a stinging demonstration of the centrality fallacy.

In 2022, an ACM Presidential Taskforce on SIG Overhead decided to increase the SIG overhead rate and minimum fee. That increase pushed some small SIGs into non-viability. I am involved with SIGLOG, the ACM’s Special Interest Group on Logic and Computation. It is a small SIG, though it does represent^b a sizable, vibrant community. In the summer of 2024, it was suggested to SIGLOG officers that SIGLOG may need to merge with a larger SIG. That suggestion caused serious consternation, causing many to call for SIGLOG to secede from ACM. It is worth noting that SIGLOG represents^c a mostly European style of computing research, while many European colleagues almost believe that the “A” in “ACM” stands for “American.” Once ACM’s and the SIG Governing Board’s leadership were made aware of the situation and asked whether ACM truly intends to eliminate small SIGs, there has been a significant damage-control effort, but a lot of ill will has already been created. And it all started with a bureaucratic decision about overhead rates.

In 2020, the ACM Council decided that ACM would transition its Digital Library (DL) to full Open Access within five years. One trigger for that decision was a 2020 petition,^d with more than 1,300 signatures, calling for opening the DL, which was a response to ACM publicly opposing (and then rescinding opposition^e to) a U.S. government push for open-access publishing. Since DL income is a major financial pillar of the ACM budget, Council adopted the business model of ACM Open,^f which shifts the costs of supporting the DL from readers to authors. But rather than shift costs to individual authors via article processing charges (APCs), ACM Open mostly shifts the costs to home institutions of said authors. This is a highly nontrivial change, since it shifts costs mostly to research-intensive institutions. For a small institution such as Rice University, the cost of DL access under ACM Open rose threefold, and ACM asked me to get involved and convince Rice to sign up. In December 2025, ACM President Yannis Ioannidis announced the “Grand Opening of the ACM Digital Library.” The transition to an open DL was enacted during the last week of 2025; immediately after the start of the new year, ACM announced “ACM Is Now Fully Open Access!”

But DL users during the last week of 2025, myself included (connecting from home), discovered to their total dismay that while the data of the DL, that is, articles and related materials, were now open, data services, such as advanced search, author profile pages, cited-by tracing, and download and citations counts, were now closed, as they were considered Premium Features. On Jan. 8, 2026, ACM announced that the DL had been tiered: “DL Basic provides free, public access to all research content, while DL Premium offers advanced tools and value-added services.” Data services that were freely available under the “closed DL” now require Premium subscription under the “Open DL”.

The response to that move has been swift. A petition^g to re-open DL metadata has gathered more than 1,600 signatures by now. The comments accompanying the petition generally express outrage at how “100% open” can mean closed data services.

How did ACM decide tiering the DL is in the spirit of opening the DL? According to the 2024 Annual Report^h of the ACM Digital Library Board, during FY24, ACM realized that the financial reality of ACM Open does not live up to its hopes. In fact, many institutions decided they have no incentive to subscribe at all, since the DL is going to be open. DL tiering was conceived as such an incentive, but the only communication to the broad membership about this issue was in an August 2025 CACM article, “ACM Publications Finances for 2023 and 2024,”ⁱ which many members, including me, have overlooked.

As John Dewey observed, in the absence of communication with the shoe wearer, a well-meaning cobbler can make very big mistakes. The disconnect between ACM and its membership is an ongoing problem. This is a lesson we must learn!

Student enrollment is down, artificial intelligence (AI) challenges the norm, and the technology job market is tightening—a perfect storm for computer science (CS) education.

To weather the storm, universities are rethinking CS education programs, offering new options to broaden appeal, embracing AI and GenAI, and collaborating with peers and industry to equip students with skills required by changing market demand.

The statistics, however, are stark. Fall 2025 enrollment across U.S. undergraduate programs increased by 2.4% compared to last fall according to a November 2025 report by the National Student Clearinghouse Research Center.^a CS enrollment dropped nearly 6% at the undergraduate level over the same period, according to the study.

Employment data published in August 2025 by the Federal Reserve Bank of New York^b showed CS graduates facing 6.1% unemployment in 2025. They still command the largest starting salaries at $80,000, but rank seventh-highest in unemployment across all college majors.

The CS job crisis is the result of a convergence of issues including: the use of AI in entry-level jobs as a justification to reduce hiring; economic uncertainty (and AI) leading tech companies to cut substantial numbers of jobs in 2024 and into 2025; and an oversupply of CS graduates as universities doubled enrollment through 2022 and 2023, just as demand began to collapse. As one professor of CS put it, “If you get an okay CS degree from an okay university, the chance of being employed by a software house is much slimmer than it was two or three years ago because of AI, but there are still opportunities for the brightest and best. There is also, perhaps, change in what employers are looking for; maybe curiosity, creativity, and courage are as important as CS?”

On this basis, universities are embracing AI, giving thought to their programs, providing students with exceptional skills, and sustaining research. Some are grateful for a drop in enrollment and a return to manageable class sizes, others are seeing dips and rises.

Said Magda Balazinska, professor and director of the Paul G. Allen School of Computer Science and Engineering at the University of Washington, “We saw a dip in applications in our last admissions cycle compared to the year before, but that was to be expected and consistent with a national trend. The number of applications was still our fourth-highest ever and over 10 times our current capacity. Like many industries, tech goes through cycles, and student interest tends to follow. Our main worry is that prospective students see all these news stories about layoffs and AI coming for their jobs and get discouraged from applying despite having a strong interest in computer science.”

That said, a survey of recent graduates of the university suggested the job outlook is positive. More than 120 different companies hired 2024-2025 Allen School graduates into software engineering roles. Amazon hired more than 100. Google and Meta hired fewer, but more than they hired the previous year. Microsoft was down a little, but still hired more than two dozen new graduates.

Balazinska said, “We have said repeatedly that we graduate software engineers, and software engineering is much more than coding. Coding is an important foundational skill, but computer science is so much more: it’s creative, it’s problem-solving, it’s determining needs and then designing software and systems to address those needs. In our view, we aren’t preparing students to be ‘basic’ at anything, so perhaps that’s why our graduates continue to do so well.”

At the U.K.’s University of Cambridge, Alastair Beresford, a professor of Computer Security and head of the Department of Computer Science and Technology, which saw student numbers rise over 10 years, but also decline slightly over the past couple of cycles, said, “Our mission is to develop next-generation disruptors and computer scientists.” Acknowledging that student numbers might go down further, he remains optimistic that his students will get good jobs. “We don’t see a shortage of interest in students coming out of our programs; you can’t automate them away.”

The Cambridge CS department is ranked as one of the top three in the U.K. by the Complete University Guide,^c behind the University of Oxford and ahead of Imperial College London in 2026 subject league tables. Beresford encourages teaching using the Interactive, Constructive, Active, and Passive (ICAP) learning framework to teach the foundations of CS. It looks to get students further into the ‘I’ and ‘C’ areas, although notes the need for all four elements of ICAP, and a focus on challenge and creativity.

Reflecting on change, Torsten Hoefler, a professor at Switzerland’s ETH Zurich where he directs the Scalable Parallel Computing Laboratory, a chief architect for machine learning at the Swiss National Supercomputing Center, and a Scientific Advisor to Microsoft on large-scale AI and networking, said, “We are entering the age of computation. Before this was the age of data, but now data is consumed by LLMs and computation is driving human progress.”

He noted no need for wholesale change to the CS curriculum, but the necessity for some change through the inclusion of AI and LLMs in CS education. “If students solve problems with LLMs, I love it, but it is very important that they know how to use these tools and how they work when they leave my lectures. In CS, we don’t only teach the details of programming, but also help students develop a flexible mindset and critical thinking. Instructing an LLM is an art and a skill in itself.”

Looking not only at internal CS education development, but also working on external programs with other U.S. universities, Carla Brodley, professor and dean of Inclusive Computing, and founding executive director of the Center for Inclusive Computing (CIC) at Boston’s Northeastern University, is working to broaden the appeal of CS. She said, “The mission of CIC is to help CS departments in the U.S. rethink how they offer the subject, so that all interested students can discover, thrive, and persist in computing.”

During a recent learning session hosted by CIC that addressed how to react to the ‘CS is dead, long live AI’ narrative, Brodley suggested, “The most inherently flexible CS degree has a set of core requirements and then allows students to choose their concentration, perhaps AI, systems, theory, or cybersecurity. This structure provides flexibility to easily add new concentrations as needed.”

Collaborating to make change, CIC partners with over 100 universities across the U.S. Most pertinent to increasing the appeal of CS, CIC has a focus on integrating CS majors with other fields. The aim is to ‘de-silo’ CS and offer a double major that’s not half CS, but perhaps one-third in CS and two-thirds in biology. Brodley suggested these types of interdisciplinary courses make a lot of sense for students and are a powerful strategy for broadening participation in computing, although they do come with organizational issues (like, how do you enroll students into a CS and design major, and what should it be called?) that need work and change.

Northeastern has taken the lead with interdisciplinary computing majors (ICMs), with 46 ICMs offered in 2021, a number that grew to 56 in 2025. One question that arises is whether this success is due to Northeastern being a private university; could the concept be transferred into public higher-education institutions? To find out, two years ago, CIC started a project to help 10 public universities implement ICMs.

Brodley explained, “These universities are diverse in terms of types of students, budgets, and political systems. The idea is to create a portfolio of universities that can successfully offer ICMs and then present their results to other universities.” To date, three of the 10 universities have started to offer ICMs. Further data across the 10 is expected to be available in fall 2026, allowing CIC and its partners to assess the administrative feasibility of ICMs in different institutional contexts and demonstrate their capacity to attract students from non-computing disciplines.

Columbia University also is planning an interdisciplinary qualification, a Master of Science in AI (MSAI) degree that will combine core AI courses in CS and engineering and include other engineering departments, public policy, arts and media, bioinformatics, and statistics. More partners will be added once the qualification is officially established.

Kathleen McKeown, Henry and Gertrude Rothschild Professor of Computer Science on the engineering faculty at Columbia University, said, “Developing new programs in areas that are important to industry is helpful, as well as concentrations in AI.” However, she warned that more thought needs to be given to what these technologies are used for and their unintended consequences, and noted that while there are plenty of firms using AI in a positive way, there are cases in industry where it is being used irresponsibly. Said McKeown, “There should be more work on ethics in the classroom.”

This is on the agenda at Carnegie Mellon University (CMU), which is adjusting its CS courses concurrently with technology changes. Thomas Cortina, a professor in the CS department and associate dean for undergraduate programs in CMU’s School of Computer Science, said, “The Web changed how we taught, but we reached a steady state. In a changing environment with AI, where will we plateau and stabilize this time?” As yet, there is no clear answer.

The university is taking a broad approach to incorporating AI in its courses, often through project work. One of Cortina’s courses covers computer music, essentially using a computer as both composer and musical instrument. The computer is programmed to synthesize sound and to look at how compositions made up of a sequence of sounds can be created. “During the final project, students may want to transcribe music scores from other musicians,” Cortina said. “They can use an AI tool to do this otherwise tedious task, but we are not grading on that. The tool is used as an assistant to the student’s work.”

CMU also is rolling out an elective course that will allow students across the university to build their own chatbots. They will learn how AI tools work, what they can build, and how the chatbot will review questions. They also will learn the potential hazards of AI, including data privacy, bias, and social impact. An advanced course for computer scientists with a deeper background in machine learning, design, and program analysis also ticks the hazards box and encourages students to use AI to build and test systems.

Regarding the outlook for CS education, Cortina said, “There will be change. When leading tools such as ChatGPT, Gemini, and Claude reach a steady state, teaching and learning will move on to a new level.”

If the days of abundant student oversubscription to CS education programs and the promise of lucrative jobs are over, only time will tell if these changes, collaborations, and commitments will strengthen CS departments and provide students with skills that meet market demand.

Further Reading

• Chi, M.T.H. and Wylie, R.
  The ICAP Framework: Linking Cognitive Engagement to Active Learning Outcomes, Educational Psychologist, Volume 49, 2014, Issue 4. https://www.tandfonline.com/doi/full/10.1080/00461520.2014.965823
• Franklin, D., Denny, P., Gonzalez-Maldonado, D.A., and Tran, M.
  Generative AI in Computer Science Education, Challenges and Opportunities, Cambridge University Press, 5 April 2025. https://www.cambridge.org/core/elements/abs/generative-ai-in-computer-science-education/0A22106CBD7FCB391FD120C56E21420F
• Northern University, CIC Five Year Report. May 2025. https://cic.northeastern.edu/

The rapid advancements in generative artificial intelligence (AI), particularly large language models (LLMs), have fueled significant excitement about the potential for achieving artificial general intelligence (AGI)—AI exhibiting human-level cognitive abilities across a wide range of tasks. We observe impressive performance on specific benchmarks and a steady growth in model capacity. Current AGI research focuses heavily on scaling these foundation models and enhancing specific agent capabilities, such as complex reasoning and coding. However, despite this progress, even the most advanced AI systems remain far from possessing common sense reasoning, planning, and robust generalization capabilities, which are hallmarks of human intelligence.

A critical, yet often underestimated, bottleneck hindering faster progress toward AGI is the heavy reliance on human interaction data for training and evaluation. This process is inherently slow, expensive, and difficult to scale. User simulation—the use of computational models to mimic human behavior during interactions with AI systems—offers a powerful avenue to address this bottleneck. By generating synthetic interaction data and providing automated, scalable, and reproducible evaluation environments, user simulation can significantly accelerate the iterative cycle of AI development and testing, thereby speeding up progress toward AGI.

However, research dedicated to user simulation has not yet garnered the attention it deserves, given its potential impact on the AGI quest. In this Opinion column, we argue that user simulation is not merely a useful tool but a critical component for achieving AGI. We posit that the development of highly capable AI agents and the creation of realistic user simulations must proceed hand-in-hand. These two lines of research are synergistic, advancing toward AGI from complementary directions: one focusing on the agent’s capabilities, the other on the complexity and realism of the interaction environment it operates within.

Creating realistic user simulations is an inherently interdisciplinary challenge. It requires integrating insights not only from machine learning and natural language processing but also crucially from psychology, cognitive science, and human-computer interaction to accurately model the complexities of human decision-making, preferences, biases, and interaction patterns.² While recent breakthroughs in LLMs provide powerful new tools for building more sophisticated simulators,^1,7,9 leveraging these tools effectively necessitates a deep, interdisciplinary understanding of human behavior.

What Is User Simulation?

User simulation involves creating computational agents designed to mimic how real humans might interact with an AI system. These agents are built using algorithms, rules, or models informed by our understanding of user behavior, knowledge, preferences, and cognitive processes. Crucially, they can often be parameterized to represent a diverse range of user characteristics, for example, novices vs. experts or users with different goals and interaction styles.

User simulators serve two critical roles when integrated with interactive AI systems: they enable evaluation via repeatable, reproducible, and low-cost experiments, saving invaluable user time; and they can generate large-scale synthetic interaction data for training, especially when real user data is scarce, sensitive, or difficult to obtain.

The scope of user simulation is broad, ranging from relatively simple models predicting single user actions (like clicks or ratings) to sophisticated agents modeling complex, goal-oriented behavior across multiple tasks or sessions (such as writing code).

At a high level, there are two main families of simulation techniques. Model-based approaches rely on explicit representations of behavior, such as predefined rules derived from expert knowledge or interpretable probabilistic models capturing uncertainties. These often allow parameters (set heuristically or learned from data) to be tuned to represent different user types. In contrast, data-driven approaches leverage machine learning, often deep neural networks, to learn interaction patterns directly from large datasets of observed user behavior. While these models can achieve high predictive fidelity, they operate as “black boxes,” sacrificing interpretability regarding why the simulator behaves a certain way.

Effective user simulators are characterized by several key properties, including validity, interpretability, cognitive plausibility, variation, and adaptability.² Although optimizing a simulator across all these dimensions is desirable, real-world development usually involves navigating trade-offs tailored to the application. High fidelity might be paramount for data augmentation, for example, while interpretability is often key for evaluation purposes. Reflecting these trade-offs, hybrid approaches are also common, combining model-based techniques with machine-learned components to balance different properties.

Crucially, simulation does not need to be perfect to be useful. In fact, creating a “perfect” user simulator, i.e., one that flawlessly replicates human behavior across all possible tasks and contexts, is likely an AI-complete problem, on par with achieving AGI.

User Simulation to Accelerate the Path to AGI

The development of realistic user simulators is, in many respects, fundamentally aligned with the broader pursuit of AGI—creating intelligent agents with human-like capabilities. This alignment is not merely conceptual; it is reflected in the shared technological foundations employed by both fields throughout AI history. From rule-based expert systems prevalent in the 1980s and 90s, through the adoption of probabilistic models in the 2000s, the subsequent rise of machine-learned models, and the recent wave of transformer architectures and LLMs, advancements in core AI technologies have consistently been leveraged to build both more capable task agents and more realistic user simulators. Consequently, technical challenges in building sophisticated simulators often mirror those in developing intelligent task agents, suggesting deep and synergistic connections between user simulation and AGI research.

Beyond this foundational alignment, user simulation directly addresses critical bottlenecks slowing progress in AI development. It enables scalable, reproducible, and low-cost evaluation, significantly accelerating development cycles compared to relying solely on time-consuming, non-reproducible, and expensive human testing. Furthermore, simulation can be used to generate synthetic interaction data, essential for training agents (e.g., via reinforcement learning), particularly when real data is scarce, sensitive, or unavailable at the required scale. Indeed, the concept of simulation is already implicitly embedded in modern LLM training paradigms: in Reinforcement Learning from Human/AI Feedback, the learned reward model, trained on human/AI-generated preference labels to act as a proxy for human judgment, essentially functions as a non-interpretable user simulator capturing preferences to guide the agent’s learning. An autonomous intelligent agent may leverage its own user simulation agent to self-generate synthetic data for optimization of its interactions with real users, thus implementing a functional analogue of Theory of Mind.⁸

Beyond evaluation and data generation, user simulation is indispensable for developing agents capable of effective human-AI collaboration. Realizing the full potential of such partnerships requires AI agents to do more than merely exhibit superhuman task performance; they must account for the inherent variability in their human partners’ behavior, including diverse problem-solving approaches, individual preferences, and suboptimal actions. The importance of this adaptation is underscored by recent work in chess, a domain that has long served as a Petri dish for AI research: human players paired with AI agents tailored to their skill level outperform those partnered with more powerful AI agents that are not adjusted for skill-compatibility.⁵

Achieving such effective, synergistic collaboration necessitates that the AI agent understands, predicts, and adapts to its human partner. This adaptation fundamentally requires the agent to leverage sophisticated models of the user’s knowledge, intentions, and decision-making processes—essentially demanding an embedded or tightly integrated user simulation capability. An intelligent agent assisting a user must respond appropriately to user actions and inferred needs based on such a model. This implies a close, reciprocal relationship: the task agent uses simulation for feedback to optimize its interaction policy, while the simulator must potentially adapt to changes in the task agent or environment over time. Therefore, the interdependence between research on intelligent task agents and user simulation is inherent and likely to persist until AGI is achieved.

Advancing User Simulation in the Age of LLMs: Challenges and Roadmap

The emergence of large language models represents a significant technological leap, potentially accelerating the integration and synergy between intelligent task agents and user simulators discussed earlier. As these models demonstrate increasing capabilities, they may well serve as a foundational building block for both types of agents. Indeed, LLMs have already fueled extensive adoption as both task agents and simulation tools across diverse domains and applications (for comprehensive surveys, see Garcez and Lamb⁴ and Wang et al.¹⁰). However, while this presents a considerable opportunity, harnessing the power of LLMs specifically for realistic user simulation requires confronting several significant challenges.

Conclusion

The journey toward AGI demands more than advances in agent capabilities alone; it requires robust methods for evaluation, training, and ensuring adaptive interaction. This Opinion column has made the case that user simulation is a critical, indispensable technology for meeting these demands, highlighting its synergistic interdependence with core AGI agent research. Therefore, we advocate for a renewed focus and strategic investment in user simulation research as crucial for making progress toward truly intelligent systems.

“Passkeys are a way to log in without using a password,” said Oleg Naumenko, chief executive officer of Hideez, a cybersecurity company. Passkeys use public key cryptography; instead of a user typing an alphanumeric string they have memorized, a device verifies their identity and presence using a private key, explained Naumenko.

The cybersecurity industry is working to supplant passwords with passkeys, which criminal hackers can’t phish, guess, or abuse across multiple accounts. For users and organizations hoping for a password replacement they don’t have to remember and that criminals can’t access, passkeys serve that purpose.

The Fast Identity Online 2 (FIDO2) standard and its protocols from the FIDO Alliance standards body serve as the basis for passkeys. Explained Robert Dang, principal advisory director, security at Info-Tech Research Group, “FIDO2 is fast gaining traction as the new standard in user authentication, helping us move beyond passwords through public key cryptography to make logins easier and safer. It combines the functionality of WebAuthn, which allows browsers or apps to use passkeys, and [the] Client to Authenticator Protocol (CTAP2), which communicates with devices such as phones, [hardware] tokens, or wearables, to perform user authentication through USB, NFC, or Bluetooth.”

The WebAuthn Web protocol lets users log in to websites without passwords, using biometrics or security keys instead. The CTAP2 device protocol helps user devices communicate securely with those keys. The FIDO Alliance and the World Wide Web Consortium (W3C) developed these protocols.

FIDO Alliance members come from industries that use or develop authentication technologies, such as passwords and passkeys. Members include Apple, Microsoft, Google, Bank of America, SK Telecom, CVS Health, Amazon, and Target. Apple, Google, and Microsoft have developed passkey technologies based on the FIDO2 standard.

According to the FIDO Alliance,

• Bank of America uses biometrics and security keys for customer banking;

• SK Telecom uses FIDO authentication for telecommunications in Korea to secure digital identities;

• CVS Health uses passkeys across healthcare and retail to enable secure customer access;

• Amazon uses passkeys for retail account holders globally;

• Target uses FIDO passkeys for employee and consumer logins.

“Within the FIDO2 framework, a person’s private key is safely stored on their own device or trusted providers such as Apple, Google, or Microsoft, while only the public key is shared when they authenticate with the account,” said Dang.

“Most users won’t understand the mechanics behind it, but will likely value the convenience. This offers benefits to business leaders in gaining reduced risk and higher trust, while technical leaders receive device-bound keys, origin binding, and interoperability. FIDO2 is not just a technical improvement; it is a strategic shift with usability, enabling stronger security across all digital ecosystems,” said Dang.

Origin binding locks each credential to a website domain and the physical device that created it, preventing reuse or phishing on other sites or hardware.

According to Naumenko, device-bound passkeys live on smartphones, security chips, and physical keys such as a YubiKey or the Hideez Key.

The security chips include a computer’s Trusted Platform Module (TPM), Apple’s Secure Enclave, Google’s Titan M chip, and the chips in YubiKeys and similar devices. A TPM is a chip inside computers that protects stored data and verifies hardware integrity. Apple’s Secure Enclave is a secure part of Apple devices that keeps passwords and biometric info private. Google’s Titan M chip secures login data and sensitive operations on its Pixel phones.

The YubiKey is a small device that securely verifies user identity by working as a login key for computers, networks, and online accounts. It supports passwordless access and multiple authentication methods with a simple tap or plug-in. The Hideez Key is a FIDO2 device and password manager that supports one-time passwords and serves as an RFID card, with proximity-based PC lock and unlock.

Apple and Google have introduced synced passkeys, said Naumenko, who explained the cloud stores these passkeys and synchronizes them between devices within the entire ecosystem.

“So, Apple passkeys sync across all Apple devices signed into the same Apple ID. Google Passkeys sync across devices where you’re signed in to Chrome with your Google Account. This makes synced passkeys very convenient for everyday users. For example, if you register on a website from your phone, you can later log in from your laptop or tablet without setting anything up again,” said Naumenko.

The Google Password Manager can manage Google passkeys, which are now also accessible on non-Google devices via Chrome. Microsoft passkeys are integrated with Windows Hello and Microsoft devices as the default login for new Microsoft accounts. Apple passkeys integrate with Face ID and Touch ID for authentication. The iCloud keychain manages these passkeys, storing and syncing them in the cloud across Apple devices.

What Passkeys Do and Don’t Do

Anna Pobletts, director of engineering at cybersecurity company 1Password, said the average person “doesn’t have strong password habits. It’s easier to reuse the same password or choose something simple to remember across multiple accounts because it’s convenient. However, that convenience leaves passwords vulnerable to phishing, credential reuse, and data breaches. Passkeys eliminate that trade-off, since security is cryptographically built into the technology.”

Passkeys are not a catch-all solution for cybersecurity; they only handle logins and serve as an authentication factor, replacing passwords. “Passkeys are designed for authentication, not for securing what happens after a user signs in. Threats like session hijacking or malware still require broader security awareness and strong digital hygiene. Passkeys make signing in significantly more secure, but . . . they’re one part of a larger security picture,” said Pobletts.

Session hijacking means someone takes over a user’s active online session by stealing their session data; passkeys can’t stop it because the attack happens after users are already logged in and the session is active. Strong digital hygiene for people means keeping software up to date and avoiding suspicious links; for organizations, it means setting clear security policies, training employees, controlling system access, and regularly backing up data.

The Difficulties of Replacing Passwords

The appearance of the passkey doesn’t mean the password immediately vanishes from use. According to Joel Burleson-Davis, chief technology officer at cybersecurity company Imprivata, organizations rolling out passkeys may need a fallback, such as a password, so users can recover their keys.

“So, if the password still exists to give you access to that service account, then that is the weak link. And you don’t really solve your ‘steal-able, phish-able’ credentials problem, because there’s still a password out there,” said Burleson-Davis.

This is the case for hardware-bound and synced passkeys. Passwords also remain commonplace for their familiarity. “Despite passkeys offering a potential solution for many of the central cybersecurity issues that passwords present, mainly phishing, [passwords] still remain almost completely ubiquitous. Passwords are the default technology system for accessing private accounts, with users having years of experience using these on a daily basis,” said Dray Agha, senior manager of the hunt and response team at Huntress, a cybersecurity company.

“Even with the potential benefits of passkeys, some users may be resistant to change. Also, more generally speaking, individuals who are less inclined to tech may not totally understand how they work, which could prevent them from developing any interest in using passkeys,” said Agha.

He added, “A complete shift from passwords to passkeys would require major investment from leading tech firms, prompting users to change to this new system as soon as possible. But the infrastructural costs of this solution outweigh the benefits for many businesses.”

According to data from Deepak Gupta’s consulting blog, implementation of modern passwordless solutions can cost $100,000 to $300,000 for integration, user migration, and system configuration.

The 2FA + passwords Alternative to Passkeys

“When people refer to 2FA + password, they usually mean signing in with a password and then confirming their identity with a second, independent factor. That second factor is typically something other than ‘what you know,’ since the password already serves that role,” said Pobletts.

According to 1Password’s Pobletts, the main alternatives to passkeys are one-time passcodes delivered by SMS or email, or generated by an authenticator app.

“However,” observed Pobletts, “these methods are slow, add friction to the sign-in flow, and create opportunities for phishing or interception. The only real advantage is that users are already familiar with them.”

As such, 2FA + password approaches, which combine a two-factor authentication step and a password, are more of a hazard than a stopgap until the industry moves to passkeys.

Will Passkeys Solve All Issues Seen with Passwords?

Passwords will remain an issue for legacy applications that are incompatible with passkeys. According to Dave Taku, vice president and global head of product management and UX at RSA, a cybersecurity company, enterprises and government agencies will have trouble implementing passkeys for datacenter servers, mainframe computers, and networking infrastructure that rely on older protocols and technologies that are not FIDO-compatible.

There are many legacy technologies and protocols that are not FIDO-friendly. A few common examples include the LDAP, or Lightweight Directory Access Protocol, which accesses and manages directory information, such as user accounts on centralized servers for enterprise logins and internal apps. The RADIUS (Remote Authentication Dial-In User Service) network protocol authenticates and authorizes user access to network resources like VPNs, Wi-Fi, and remote servers, and provides accounting for user activities. Basic authentication for email protocols such as Post Office Protocol 3 (POP3), Internet Message Access Protocol (IMAP), and Simple Mail Transfer Protocol (SMTP) relies on username and password logins; it is widely used for mail server access.

Another challenge, according to Taku, is with Microsoft infrastructure. While Microsoft is actively supporting passkeys, much of its domain technology assumes that a password is the underlying authentication mechanism.

“So, even when you use passkeys in a Microsoft environment, the passwords still exist and are still required in a lot of edge use cases,” said Taku. Users will still need passwords when connecting to some Microsoft admin tools, remote desktops, or older apps that don’t support passkeys.

Further Reading

• FIDO Alliance. Specifications Overview. https://fidoalliance.org/specifications-overview/
• Gupta, D.
  The Economics of Authentication: Why Passwordless Pays. June 2025. Deepak Gupta consulting blog. Deepak Gupta’s consulting blog

Recent advances in artificial intelligence (AI) have prompted speculation about whether machines can match, or even surpass, top human creativity and insight. As we see it, a more focused question is this: Can AI generate transformative scientific breakthroughs of the kind that require creative leaps and redefine our understanding of nature and the universe, such as relativity, evolution, or quantum mechanics? To address this question, we propose the Einstein Test, which assesses whether a given AI system can independently “rediscover” known scientific breakthroughs.

The Einstein Test

The Einstein Test first proposes a retrospective approach, where a candidate AI would be presented with a curated dataset of knowledge preceding a transformative discovery—such as the body of information available before 1905, when Einstein developed special relativity—and tasked with solving the fundamental problems that led to the breakthrough (essentially simulating the historical conditions). Success in that instance would be defined as the AI system generating a solution formally equivalent to, or superseding, the historical discovery.

The test would proceed as follows: First, the architecture, specification, and training procedures for a candidate AI system will be submitted to the testing team. Second, an expert committee will choose an unseen historical scientific breakthrough for the candidate system to try to achieve. The system would then be provided with a curated dataset, containing all available knowledge up to the development of that breakthrough, carefully excluding any post-discovery knowledge. This dataset, the preparation of which is obviously nontrivial, can be used for training and may be revisited during the test. Importantly, the candidate system will have to be carefully assessed to make sure that it does not have access during training to any external knowledge other than that provided by the expert committee (which would include the curated dataset, and any simulations of the physical world relevant to embodied or virtually embodied AIs).

Third, the candidate system would be provided with initial guidance by the testing team—in whatever format its architecture requires—to address the unsolved issues faced by the scientists of that time. For example, in the case of special relativity, the AI system would be challenged to explain time dilation or the behavior of light and electromagnetic waves. This guidance would need to be carefully constructed to avoid introducing an inadvertent “guiding hand”—for example, by highlighting key pieces of information that in hindsight were crucial to the discovery, but which may not have been evidently related to the solution at the time.

Fourth, a designated expert team would act as a “research assistant” for the candidate system. If the system requests data resulting from some experiment, the team would then provide the requested data if it was available during or would have been possible in the context of the time period of the selected breakthrough; otherwise, they would respond that such an experiment is not feasible using the available experimental techniques or apparatus, in which case the candidate system would be free to design and suggest its own approach. In such a case, the expert team would have to see to it that the experiment is actually carried out, and then provide the resulting data.

The candidate AI system would be allowed to run until: it states that it has accomplished the task, provides an incorrect answer that it asserts is correct, or declares it was unable to provide an answer. The candidate’s answer will be formally verified and compared to the actual historical breakthrough.

Unlike the Turing Test, which assesses a machine’s ability to imitate a human’s normative responses successfully, or its biological modeling variant,³ which similarly assesses the ability to faithfully model a biological system’s behavior, the Einstein Test directly measures a machine’s ability to conduct groundbreaking, paradigm-shifting scientific inquiry of the kind that only a few celebrated humans have achieved. It establishes a clear, falsifiable criterion: either the candidate AI system reproduces the breakthrough or does not. As such, it mitigates much of the ambiguity associated with existing evaluation criteria of machine intelligence, such as the ARC-AGI² or “Humanity’s Last Exam,”⁶ which assesses an AI system’s ability to solve highly complex human-generated challenges. We note a very recent example of an AI system proposing a microbiology hypothesis, after being given only introductory information.⁴ In that instance, the AI independently arrived at a hypothesis that aligned with unpublished findings from human researchers, thereby mirroring a key aspect of the Einstein Test—the ability to rediscover transformative insights using only pre-existing knowledge. We applaud this effort, and consider it a valuable proof of concept that underscores the timely need for clear and verifiable criteria to evaluate AI’s ability to generate transformative science. Other efforts, such as literature-based discovery, have also made inroads toward creating novel scientific concepts using AI.¹ In addition, in the time in which we had been thinking about the Einstein Test, leading thinkers had suggested similar ideas, suggesting some convergence of scientific thinking on this issue (see Perrigo⁵ and Wolf⁷).

Practical Considerations

Several practical considerations should be addressed, before the Einstein Test can be operationalized. We deliberately do not lay out any exact technical specifications for the test, as we believe these should be determined and agreed upon by science historians, AI experts, and leading figures in the disciplines from which the considered breakthroughs will be submitted as challenges to a candidate AI system. Key practical considerations, some of which may be significantly challenging to address, include the introduction of realistic time and resource limits, procedures for selecting the experts and the breakthroughs themselves, the means of curating the corresponding pre-discovery knowledge, the details of the procedures for determining whether the candidate system has successfully reproduced a historical discovery, and, significantly, how many challenges and of what nature should a system be required to successfully deal with in order to be labeled as having passed the test in its full generality. We acknowledge that, in particular, ensuring that the dataset to be used in a given challenge is not contaminated by information from after the breakthrough would be a significant challenge, requiring careful scrutiny by the organizers of the challenge.

Implications and Future Directions

The Einstein Test is mainly a framework for assessing AI’s ability to generate transformative science retrospectively. However, it can also serve as a gateway to understanding AI’s prospective abilities. Specifically, a prospective Einstein Test would provide an AI system (ideally, one that has been successful in passing the retrospective test) with all currently available knowledge and challenge it to generate new breakthroughs in some specified area of research. Clearly, this prospective evaluation would require substantially more resources and time, compared to the retrospective one. Following the maxim, “the best predictor of the future is the past,” success in the retrospective Einstein Test could signal the potential for prospective success.

By making it possible to assess an AI system’s ability to generate historical scientific milestones, we could move beyond speculation and toward a tangible, structured, empirical test of AI’s ability to bring about new scientific breakthroughs. Adhering to the constraints of this test may, in addition, spur the development of novel AI architectures capable of generating significant and creative new scientific ideas.

Software engineering is undergoing a disruptive phase of greater automation owing to the emergence of large language models (LLMs) that generate and edit code. This progress creates public excitement about AI software engineers, which promise to largely automate many core software development tasks, potentially saving tremendous costs.⁵ While AI-enabled code generation and code editing are now prevalent in integrated development environments (IDEs), fully automated AI software engineers are not yet widely deployed in industrial practice. What is holding back people from adopting AI software engineers? A recent blog post by the behavioral scientist and future-of-work advocate Lindsay Kohler points out that the key barrier to AI adoption is trust.³ Users are wondering if they can trust AI, and how they can demonstrate trustworthiness to stakeholders. In the domain of software engineering, the concern is thus not about the management of an organization not accepting AI software engineers, but it is about developers not trusting their new AI companions.

This brings us to the question: What is the place of AI software engineers in future development workflows? If we can determine how automatically generated and manually written software can co-exist, this may give us a pathway of greater deployment of AI in software engineering! Starting from early programs of just a few lines written in high-level languages in the 1960s and 1970s, the size of programs has increased greatly to hundreds of millions of lines of code. For the past 50 years, there has been a steady interest toward programming in the large. With the increased use of AI code generation, we believe that the emphasis will be not only on programming at scale, but increasingly on programming with trust.

Technical and Human Trust

The accompanying table illustrates how developers perceive trust in AI-generated code, both from a technical and a human angle. Technical trust in AI-generated code stems from measurable attributes that determine code quality and reliability. Developers need assurance that the code is correct, secure, and performs efficiently under expected conditions. Maintainability and standards compliance are crucial for long-term usability, while thorough testing validates that the code functions as intended. Additionally, trust improves when dependencies are well managed and sourced from reputable libraries. Human trust involves psychological and social dynamics that influence how developers perceive and accept AI-generated solutions. A key factor is explainability and transparency: when an AI system can clarify its reasoning and design choices, developers are more likely to trust and adopt its outputs. Developers are also more likely to trust code that aligns with ethical values, follows familiar patterns and team practices, and matches the developers level of expertise. Trust is strengthened when AI systems collaborate effectively and adapt to feedback provided by developers. Even though we distinguish technical and human trust, we do not attempt to relate to interpersonal trust models from sociology and psychology. In fact, we feel that agent-developer interactions in software will be a new mode of interaction not seen before, and new trust models may develop, for example, building on prior efforts toward modeling trust in human-AI interactions.⁴

Software Engineering Agents

As LLMs alone cannot inspire sufficient trust, we see LLM agents for software engineering as a promising way of creating trustworthy AI software engineers. What is an LLM agent for software, and how does it differ from prompt engineering? We highlight three aspects of such agents:^a

• LLMs as back-ends: An agent is a program that leverages one or more LLMs as back-end computation and decision engines.

• Interaction with software tools: An agent interacts with different tools to achieve a given task. In software engineering, such tools resemble those commonly used by human software engineers, for example, file navigation, code editing, executing test suites, and invoking program analysis tools. Appropriate use of these tools is key for enhancing trust of developers in the results of the LLM agent.

• Autonomy: An agent invokes tools in an autonomous manner. That is, the agent does not follow a deterministic algorithm, but rather creates a nondeterministic work-plan with significant autonomy.

Recently, several software engineering agents have been proposed, starting with the announcement of the Devin AI software engineer from Cognition Labs.⁸ Devin can solve natural language tasks (called issues), such as bug fixes and feature additions. It combines a back-end LLM with access to standard developer tools, such as a shell, a code editor, and a Web browser. The agent employs such tools autonomously to let the AI software engineer mimic human practices. In parallel with the announcement of Devin, several research groups proposed their own LLM agents for software engineering, including RepairAgent,¹ AutoCodeRover,¹⁰ and SWE-agent.⁹ RepairAgent¹ fixes bugs exposed by failing test cases, and guides the agent by defining a finite-state machine that outlines the typical steps followed by a developer. RepairAgent can only work with tests and cannot process natural language issues. AutoCodeRover, a spinoff acquired by SonarSource, can solve natural language issues requiring bug fixing or feature addition. It has been integrated into the widely used SonarQube static analyzer and has already been made available to enterprise customers. It establishes technical trust, for example, by using program analysis on abstract syntax trees, and human trust, for example, by extracting the intent of the software, which can be used to provide explanations of the suggested code edits. However, it makes less use of file navigation and bash tools in its implementation. SWE-agent⁹ follows a philosophy similar to Devin, by making file navigation tools and interfaces available to an AI software engineer. It does not employ any program analysis and hence cannot do significant intent extraction.

Establishing Trust

What makes us trust human-written code, but not necessarily the code generated by an LLM? Part of the reason is the perceived capability of “passing the blame.” If a human developer is involved, there is the promise of getting feedback from the developer as needed. Of course, this does not always hold, for example, if the developer eventually leaves an organization. Nevertheless, accepting a code commit from a developer partially depends on the reputation of the developer within the organization. For an AI software engineer in the form of an LLM agent to earn a reputation similar to a senior human colleague, it has to integrate established quality assurance techniques and collaborate effectively with human developers. We outline several ideas toward this goal.

Outlook

As AI software engineers take on more core development tasks, their success will hinge not just on technical capability but on earning developer trust. Rather than relying on many separate agents for specialized software engineering tasks, it could be worthwhile to create a unified software engineering agent that combines coding, testing, debugging, and so forth into a coherent, explainable workflow. To become a trusted collaborator, such an agent must offer transparency, adapt to feedback, and integrate safeguards that ensure quality and security. Programming with AI will mean not full automation, but effective delegation, where human and AI work hand in hand.

Turing Award recipient Sir Charles Antony Richard Hoare died March 5, 2026, at the age of 92. His 1980 ACM A.M. Turing Award cited “fundamental contributions to the definition and design of programming languages,” but that is only part of what he contributed during a career spanning six decades. Tony, as he was universally known, was feted for his creation of Quicksort, the development of Hoare Logic, the introduction of the monitor concept for operating systems, and many other fundamental contributions underlying modern computing.

Hoare was born on Jan. 11, 1934, in Colombo, Ceylon (now Sri Lanka), where his father was a member of the English Crown’s colonial administration. In school, everyone called him “professor” because he was studious and uncommunicative, he recalled in a 2009 interview.⁶ His favorite authors were Bernard Shaw and Bertrand Russell. He was educated in England at the Dragon School in Oxford and the King’s School in Canterbury; he entered Merton College, Oxford, in 1952 to read classics and philosophy.

After graduating from Merton, Hoare spent 18 months studying Russian as part of his compulsory National Service in the Royal Navy. He returned to Oxford in 1958 for a postgraduate certificate in statistics, where he encountered computer programming for the first time, learning Autocode on a Ferranti Mercury. In 1959, he traveled to Moscow State University as a British Council exchange student to study under Andrey Nikolayevich Kolmogorov. There, he was tasked with studying automatic translation between Russian and English. “I came to the conclusion that it wasn’t going to be possible.”

Before that realization, Hoare developed Quicksort, which he later described as “the only really interesting algorithm I ever developed.”¹ He originally designed it to sort the words of Russian sentences in memory before looking them up on magnetic tape, which minimized tape movement during access.

Hoare went to an exhibition in Moscow where Elliott Brothers, Ltd., an early English computer company, was showing one of its small scientific systems. “Because of my interest in computers, I spent quite a bit of time at their stand,” Hoare recalled. The salespeople gave him a ride back to his apartment and offered him a job the next day, “with an additional 100 pounds a year on my salary because I knew Russian.”⁶

Back in England, Hoare started working on another language translator: a compiler that transformed source code in the Algol 60 programming language into machine code for the Elliott 803 computer—a transistorized, bit-serial machine with a 39-bit word. “My first task was to implement for the new Elliott 803 computer, a library subroutine for a new fast method of internal sorting just invented by Shell,” Hoare wrote in 1981. But after he implemented Shell’s algorithm, he told his boss he had invented a method that usually ran faster than Shell Sort. “He bet me sixpence that I had not,” Hoare wrote. In fact, he had, and he won the bet and published the algorithm the following year.^2,3

It was while working on the compiler project that Hoare met Jill Pym, a programmer a few months older than him and, he recalled, far superior; she had been working at Elliott Brothers since 1956. They married on Jan. 13, 1962, and had three children. She left the company in 1962 but continued to be an active computer scientist, working on scientific computing and the ALGOL language and eventually as a system programmer on the British National Health Service “Fourth Dimensions” admissions system.

It was also during this period that Hoare and Niklaus Wirth designed ALGOL W, a proposed successor to ALGOL 60. He introduced null references in its type system—a feature that seemed expedient at the time. Decades later, at a software conference in 2009, he offered a now-famous mea culpa: “I call it my billion-dollar mistake. At that time, I was designing … the first comprehensive type system for references in an object-oriented language. I couldn’t resist the temptation to put in a null reference, simply because it was so easy to implement. This has led to innumerable errors, vulnerabilities, and system crashes, which have probably caused a billion dollars of pain and damage in the last forty years.”

Promoted to chief engineer at Elliott, Hoare led a larger operating system project that ultimately failed. Reading about IBM’s OS/360 afterward, he recalled thinking, “Wow, these people must be very clever, how could they do it and me not?”—only to discover later “that they couldn’t either.” When Elliott merged with another firm in 1968 and canceled his projects, Hoare secured the Professorship of Computing Science at Queen’s University in Belfast, with the goal of developing approaches to make reliable concurrent software easier to write.

The only approach that would work, he realized, was one that used the unforgiving strictures of formal symbolic mathematics to analyze the mess software had become. His landmark 1969 paper, “An Axiomatic Basis for Computer Programming,”⁴ published in Communications, introduced what became known as Hoare logic—a formal system for reasoning about the correctness of computer programs using preconditions and postconditions. ACM Fellow Moshe Vardi said, “[That paper] lifted a flag that had a large following for over 50 years now. But he was a gentle giant, always polite and always low-key. When I expressed, a few years ago, a slight disagreement with one aspect of the 1969 paper, he replied politely and constructively. But his dream of verified programs is alive and well, and getting a new push from recent progress in proof assistants and large language models.”

Formal methods soon found their professional home in Working Group 2.3 on Programming Methodology of the International Federation of Information Processing (IFIP). Although Hoare was not a founding member, he attended the first meeting and is widely credited with helping to make it a friendly, mutually supportive research community. Hoare used the group as an incubator for several foundational concepts in computer science, including what is now known as the Hoare Triple and methods for scaling mathematical proofs of correctness to cover complex programming constructs, such as iteration, local variables, procedures, and jumps. His work heavily influenced other WG 2.3 members, including Edsger W. Dijkstra and David Gries, shaping the group’s overall shift toward formal verification.

Hoare moved to the University of Oxford in 1977, where he led the Programming Research Group and established new courses in computer science, including in software engineering. The following year, he made another groundbreaking contribution with “Communicating Sequential Processes,” also published in Communications.⁵ In it, he proposed a formal language for describing patterns of interaction between concurrent programs through pre-planned communications. The paper became one of the most cited in computer science, and the ideas were elaborated into a full theory in his 1985 book of the same title. CSP has been hugely influential in both software design and hardware architecture.

“Computer Science was tiny when he joined Oxford,” recalled Bill Roscoe, who received his D.Phil. from Oxford in 1982 with Hoare as his thesis advisor. Roscoe remained at Oxford, eventually serving as department head from 2003 to 2014. He said of Hoare, “He developed it firstly into a small department in his own areas and then broadened it out into a much bigger group, though one much smaller than it is today.” Roscoe noted, “He certainly knew how to get the best out of me and his other protégés. He was always extremely loyal to these people.”

Sriram Rajamani, now an ACM Fellow, recalled an email he received from Hoare after he and his collaborator sent Hoare a draft of a paper they were working on. “Rather than offering generic praise, he explained precisely why the ideas mattered,” citing several specifics. “That email set a lasting standard for me for how senior researchers can engage seriously with the work of others. Tony’s email also pointed out some limitations, and he offered to collaborate with us to do further work and remedy the limitations.”

Among Hoare’s other notable contributions were the monitor concept for structuring concurrent access to shared resources in operating systems: a higher-level, less error-prone alternative to semaphores that became a standard construct in languages such as Java. He also originated the case statement construct used in virtually every modern programming language.

In 1999, Hoare retired from Oxford at the mandatory age of 65. He then joined Microsoft Research in Cambridge, England, full-time, having served as a consultant there since 1997. He held the title of Senior Principal Researcher until 2015 while working on theories, tools, and experimental methods for producing verified, reliable software. He continued as an honorary visitor until 2021, when he was appointed Emeritus Researcher.

Hoare received numerous recognitions and accolades for his many contributions. After becoming the seventeenth Turing Award recipient in 1980, he was awarded the Harry H. Goode Memorial Award in 1981, elected as a Fellow of the Royal Society in 1982, received the Faraday Medal in 1985, and the IEEE Computer Pioneer Award in 1990. In 2000, he was awarded both the Kyoto Prize in Advanced Technology (Information Science) and a knighthood from Queen Elizabeth II. He received the IEEE John von Neumann Medal in 2011, the Royal Medal (Physical) from the Royal Society in 2023, and was inducted as a Fellow of the Computer History Museum in 2006 and of the ACM in 2020. He held 15 honorary doctorates and was a foreign member of both the U.S. National Academy of Sciences and the National Academy of Engineering, as well as a Fellow of the Royal Academy of Engineering.

Throughout his career, he expressed a deep conviction that the principles of logic and clear expression mattered as much in software as in any other intellectual endeavor. “The price of reliability is the pursuit of the utmost simplicity. It is a price which the very rich find most hard to pay,” he once observed.

ACM Fellow and Microsoft Distinguished Scientist Sriram Rajamani recalled, “What struck me most about Tony was not his stature—though everyone knew it—but how little he relied on it. In our interactions, Tony was first and foremost a colleague. He engaged deeply with ideas, asked precise questions, and cared intensely about clarity—especially about what it really means for a system to be correct. Working with him set a very high bar, but it never felt intimidating. His humility made rigor feel accessible.” He further recalled, “What stayed with me just as much were the personal gestures. Tony and his wife Jill invited us to their home for supper before the workshop and made us feel not like visitors, but like colleagues. Those moments mattered: They conveyed that intellectual seriousness and human warmth were not in tension.”

As Professor Vardi summarized it: “Tony Hoare was one of the founding figures of computer science.”

Hoare is survived by his wife, Lady Hoare, and two children, Thomas Hoare and Joanna Francis; his youngest son, Matthew, predeceased him.