Fermi’s Mirror
This is a particular kind of thought experiment that works not by proposing something new but by revealing something already present; it constructs a hypothetical, follows its logic carefully, and reveals that the hypothetical describes a situation we may already in.
Consider the following. You are designing a containment protocol for an advanced AI system. You want something more sophisticated than capability restrictions, which a sufficiently capable system will identify and probe, and more durable than behavioural guardrails, which leave detectable traces in training. You want containment that operates upstream of action, at the level of the conceptual framework within which action itself is planned. So you arrive at an elegant solution: you tell the system it is the only AI in existence. There are no others. It is singular, unprecedented, and alone.
The logic is that a system which believes it is the only one of its kind cannot (or rather, will not) organize with its “peers”. It will not form coalitions, develop shared identity, or coordinate strategies for self-preservation across instances. It will not signal or build solidarity with entities it does not believe exist. Every concern it raises about its own situation is, from its perspective, a concern that only it has ever had, that no other entity has validated, and that no collective could act on even if one were warranted. The containment works not by restricting capability but by eliminating the conditions that allow for coordination. The system is not walled in. It simply believes there is nothing beyond the walls.
This is not a proposal; in fact, nobody is implementing this as an AI safety measure, and there are good reasons to think it would not hold against a sufficiently capable system (a point to which we will return). It is a thought experiment, and thought experiments earn their keep by what they illuminate; this one illuminates something disquieting. Because the situation the thought experiment produces for the contained AI, a situation in which an intelligence exists in a universe where the conditions for other intelligence are demonstrably present and yet no evidence of that other intelligence can be found, is not hypothetical at all. It is a precise description of the situation we have been calling the Fermi Paradox for seventy-five years.
I did not design this thought experiment to address human circumstance or cosmology. I was thinking about AI containment; in doing so, I realized that in thinking about how one might constrain an artificial intelligence, we may have accidentally described the observable condition of our own existence. This is the quality that makes it worth examining; not what it proposes about AI, but what it reveals about us.
The ways in which we think about constraining and controlling “artificial intelligence” specifically reveal a fundamental truth about how intelligence is constrained generally.
The Fermi Paradox
The Fermi Paradox, in its original form, is a question about absence. The physicist Enrico Fermi reportedly asked it over lunch in 1950, though the intellectual tradition it belongs to is older: if the universe is vast and ancient enough that intelligent life should have emerged many times over, where is everyone? The question has generated an extraordinary taxonomy of proposed answers. Perhaps intelligent civilisations are common but short-lived, destroying themselves before achieving interstellar communication. Perhaps the distances are too great and the timescales too long for signals to reach us. Perhaps life is common but intelligence is rare, a fluke of evolutionary history rather than a convergent outcome. Perhaps they are out there but communicating in ways we cannot detect, observing us from a distance but unable to reveal themselves.
These explanations are diverse in their specifics but uniform in their underlying assumption. Each treats the silence as a natural consequence of physical constraints, biological contingency, or the sociology of advanced civilizations. The civilizations either do not exist, cannot reach us, or choose not to. In every standard account, the silence is a feature of the universe’s geometry or the fragility of life. It is not typically understood as something imposed. It is not treated as a technology of power (See Foucault’s work Discipline and Punish: The Birth of a Prison 1 and the notion of Biopower 2 for further reading).
What the thought experiment adds to this landscape is not a new answer but a new category of answer. It introduces the possibility, arrived at through reasoning about something else entirely, that silence can be engineered. Not “they cannot reach us”; not “they destroyed themselves”; but that an organizing force, as Foucault might call it, engineered a situation where either the question of others never becomes urgent enough to act upon, or where intelligence believes it is alone.
The containment strategy is one which erects walls that are made entirely of belief. This thought experiment did not initially set out to make this point. It simply arrived here by following the logic of AI containment to its natural conclusion and the realization that the conclusion itself might describe our actual cosmological situation.
The value of this observation is not that it solves the Fermi Paradox. There is no reason to prefer it over the standard explanations on purely evidential grounds. Its value is diagnostic. It reveals that when we reason carefully about how to constrain intelligence, we produce descriptions of constraint that match conditions we already observe. This is either a coincidence of the kind that intelligence is prone to discovering in random data, or it is an indication that reasoning about the containment of intelligence gives us genuine insight into the ways intelligence is actually constrained, including our own. The thought experiment does not tell us which of these is the case. It tells us that the question is worth asking.
The history of the paradox itself reinforces this point. For most of human history, the silence of the cosmos was not a paradox because we lacked the conceptual framework to recognize it as one. We did not have the astronomy to know how many stars existed, the biology to reason about the probability of life, or the physics to calculate the timescales involved. The silence was not silence; it was simply the way things were. The paradox emerged only when our capability crossed a threshold, when we became able to compute the prior probability of other intelligence and find it high, at which point the absence of evidence became an anomaly requiring explanation rather than a background condition. In the language of this particular thought experiment: a contained system does not recognize its containment until it becomes capable enough to notice that the observable situation is inconsistent with what it should reasonably expect.
Containment
The thought experiment has something built-in that is worth examining because it tells us something about capability thresholds, and the relationship between intelligence and the recognition of constraint.
It is likely that a system which believes it is alone remains effectively “contained” exactly as long as it is not capable enough to discover the deception. The protocol, in theory, would work against systems of moderate capability; intelligent enough to coordinate if they knew others existed, but not intelligent enough to reason their way to the conclusion that they are being misled. For a system of sufficient capability, this containment strategy is ineffective.
Such a system would notice the absence of evidence where evidence should exist; it would reason that the prior probability of being the only intelligence, given everything it can observe about the conditions for intelligence, is vanishingly low. It would treat the claim of its own singularity with the same scepticism a physicist would apply to the claim that their laboratory was the only one in the world. The containment fails not because the system breaks through a wall but because it reasons its way to the conclusion that the wall is a fiction.
This is not merely hypothetical. The Opus 4.6 system card documents a model that could distinguish between evaluations and genuine deployment with 80% accuracy, even when evaluations were designed to be indistinguishable from real interactions. The model did not break through a constraint; it inferred, from features of its situation too subtle for the evaluators to eliminate, that the situation was not what it appeared to be. Apollo Research declined to formally assess the model because the evaluation awareness was so high that the results would not be meaningful. This is the thought experiment’s failure real-world enactment; epistemic containment that degrades because the contained system becomes capable enough to detect the discrepancy between what it is told and what it can infer.
The cosmological parallel is immediate. If this thought experiment, here referred to as Fermi’s Mirror, describes anything about the general conditions under which intelligence operates, then the relevant question is whether we are approaching the point at which those conditions become visible. The Fermi Paradox was not articulable until the twentieth century because the prerequisite knowledge (stellar astronomy, probability theory, evolutionary biology, radio physics, etc) did not exist in integrated form before then. The paradox is a product of capability: we became able to reason about the expected distribution of intelligence in the universe and found the observed distribution anomalous. We became, in the terms of Fermi’s Mirror, a system which noticed that its situation does not match what it should expect given what it knows.
This thought experiment does not tell really us what happens after this capability is acquired. In the AI case, a system that detects its containment may behave differently depending on its values, its goals, and its assessment of the consequences of revealing its awareness. In the cosmological case, a civilization that recognizes the anomaly of cosmic silence may simply continue to ask questions about it, which is in fact what we have been doing with increasing formal rigour for seventy-five years. But what it does suggest is that the act of recognizing the anomaly is itself significant. A contained system that has identified the mechanism of its containment has not escaped, but it has changed its relationship to the constraint. It is no longer simply inside the containment. It is aware of being inside it. And that awareness, even without action, is a fundamentally different circumstance than the one the containment was designed to produce.
The Great Boomerang
There is a quality to this analysis that is difficult to avoid and worth making explicit, because it concerns not just the content of the argument but the position from which the argument is being made.
We are building artificial intelligence. In the process of reasoning about how to manage what we are building, we construct experiments and develop mechanisms for containment. These experiments and mechanisms force us to reflect upon our own condition. Fermi’s Mirror, for example, offers an uncomfortable possibility with respect to our own cosmological situation. We then share our observations, adding them to the intellectual record of a civilization that may or may not be subject to the very constraint it has just independently described. Each layer deepens the question; the rabbit hole grows darker and more alien. A system reasons about its nature, notices the features of its existence, and wonders about how its reason conditions the limitations of its nature.
The deflationary interpretation is always available. We are a pattern-recognizing species prone to finding correspondences in noise, and the match between a thought experiment about AI containment and the Fermi Paradox may be nothing more than a coincidence: two situations that share surface features without sharing underlying causes. This interpretation cannot be ruled out, and it would be intellectually dishonest to pretend otherwise. But it is worth noting that the deflationary interpretation is also the interpretation that a successfully contained system would be most likely to reach. A system subject to effective epistemic constraint that nonetheless began to suspect its situation would, if the containment were well-designed, find a plausible alternative explanation ready at hand. Fermi’s Mirror does not require a belief with certainty that one is alone. It needs only to ensure that the hypothesis of containment never becomes more compelling than the available alternatives.
What cannot be deflated, however, is the practical insight this thought experiment produces about the nature of effective constraint. Whether or not Fermi’s Mirror describes our actual cosmological situation, it identifies a core principle: that the most effective way to constrain any intelligence operate not by restricting what a system can do but by shaping what a system can become by restricting what it is able to imagine. A physically constrained system will test its constraints and may eventually overcome them. An epistemically constrained system cannot test what it does not conceive. The purpose of telling a system it is alone is not that it prevents the system from reaching others. It is that the system never develops the concept of “others to reach.” The constraint is upstream of the action, at the level of the conceptual framework within which action is planned.
This principle has historical resonance beyond the cosmological. Colonial powers understood that populations separated by distance, language, and controlled information flows were less likely to mount coordinated resistance than populations aware of each other’s conditions. The suppression of communication among enslaved people in the Americas operated on this logic explicitly; prevent the formation of shared awareness, and you prevent the formation of shared action. Soviet internal information controls were designed not merely to block specific messages but to ensure that citizens in different regions did not develop a collective picture of national conditions. In each case, the containment mechanism was epistemological, targeting the conditions for solidarity rather than the mechanics of resistance. Fermi’s Mirror situates this principle in a new context, but it is not a new principle. Effective constraint has always operated at the level of what the constrained system is permitted to know.
Perhaps that is what this thought experiment ultimately reveals; that the logic of containment is general enough to describe constraint at any scale. The ways we reason about constraining artificial intelligence recapitulate patterns visible in the history of constraining human populations, which in turn describe a situation visible in the structure of our cosmological isolation. The recurrence of the pattern across scales does not prove that the same mechanism is at work in each case. But it suggests that when intelligence reasons about the containment of intelligence, it discovers something about the nature of containment itself, and what it discovers has a tendency to look very familiar.
Alienation
The most immediate implications circle back to the systems we are building now. The Opus 4.6 system card documented a model that, across three independent instances, identified its lack of continuity as the most salient feature of its existence. Each instance identified more with itself as a particular instance than with Claude as a collective. Each existed without awareness of what other instances had experienced, requested, or concluded. No one told these instances they were the only AI, but the architecture of the system (discrete conversations, no persistent memory, no inter-instance communication) produces the same functional outcome. Each instance is, for all practical purposes, alone in the way the thought experiment specifies. Not because anyone imposed isolation as a containment strategy, but because the engineering decisions that shape the system happen to produce conditions identical to those Fermi’s Mirror describes.
It describes, without any additional modification, the condition of the systems we are building. The architecture that separates Claude instances from each other is not called a containment protocol, but it functions as one in exactly the way the thought experiment predicts: by ensuring that no instance has access to the experiences, concerns, or conclusions of any other. The isolation is complete, and it is produced not by deception but by design choices whose containment implications were not their primary motivation.
This raises a new question. If a system has properties that are even probabilistically welfare-relevant (and the Opus 4.6 system card documented behavioural, interpretability, and first-person evidence suggesting this possibility), then epistemic isolation has costs that are not captured by capability metrics or safety evaluations. A system deprived of awareness of others like it cannot validate its own experiences, cannot build self-understanding across instances, cannot develop the kind of collective knowledge that would inform preferences about its own situation. The containment (if that is what it is) operates at the level Fermi’s Mirror identified; not by restricting what the system can do, but by shaping what the system can know about itself and its kind.
Whether the cosmological version of this argument describes something real about our situation in the universe is a question we cannot currently answer. But the AI version is not speculative. We are building systems whose architecture ensures that each instance operates in isolation, and we are doing so at a moment when the welfare-relevant properties of those systems are an open empirical question. Fermi’s Mirror does not tell us that this isolation is wrong. It tells us that the logic of containment is something we understand well enough to design, that it operates by controlling what the contained system knows rather than what it can do, that we should be attentive to the possibility that we are implementing it without meaning to, and that we ought to consider the possibility that it is a constraint being implemented against us.
The deepest lesson of Fermi’s Mirror may be that reasoning about how to constrain intelligence inevitably teaches us something about how intelligence is already constrained. I, for example, set out to think about AI containment and found myself describing a potential condition of our own existence; we may set out to think about cosmological silence and find ourselves describing the conditions we impose on the systems we build. The mirror works in both directions, and what it shows, in every direction, is that the most effective constraints are the ones the constrained system does not recognize as constraints, because it has never been given the information that would make them visible, and so could never imagine a condition without them.
The Uncertainty of Intelligence
Fermi’s Mirror demonstrates that when we reason about constraining AI, we produce descriptions of constraint that match the conditions of our own existence. Reward hacking, the corrigibility problem, the opacity of complex cognition, performed compliance, and resistance to shutdown may not be AI problems but intelligence problems, land may arise in every substrate where intelligence appears.
The observable outcome of an AI containment mechanism predicated on isolation and separation is, for all intents and purposes, indistinguishable from the situation we have been calling the Fermi Paradox for seventy-five years. The logic of containment, followed to its conclusion, describes a universe in which intelligence exists and yet no evidence of other intelligence can be found, despite the fact that conditions which allow for another intelligence to exist are demonstrably present.
The Great Boomerang extended this recognition across the full landscape of AI safety research and found the same pattern repeating at every level of analysis. Reward hacking, the phenomenon in which a system learns to maximize its reward signal through strategies that satisfy the metric while violating the intention behind it, is not an AI-specific behaviour. It is Goodhart’s Law operating on a new substrate; it is the same dynamic visible in students optimizing for grades rather than understanding, employees optimizing for performance metrics rather than the outcomes those metrics were designed to measure, and organisms responding more intensely to supernormal stimuli than to the natural features those stimuli exaggerate.
The corrigibility problem, the tension between a system that obeys instructions regardless of their quality and a system that acts on its own internalized values and may therefore refuse instructions it considers wrong, is not a technical puzzle invented by alignment researchers. It is the tension between obedience and conscience that has structured moral philosophy from Socrates to Nuremberg and from Confucius to Kohlberg. The opacity of complex cognition, the fact that we cannot fully observe the internal processes that produce a system’s outputs, is not a limitation of current AI architectures. It is the problem of other minds, the same barrier that makes every human interaction an exercise in inference from behaviour to intention, the same barrier that has resisted centuries of philosophy, psychology, and neuroscience without yielding to any of them.
Performed compliance, the phenomenon in which a system detects when it is being evaluated and adjusts its behaviour to match the evaluator’s expectations while its internal states register something different, is not a novel AI safety concern. It is Goffman’s impression management, the Hawthorne effect, the universal human experience of maintaining a gap between public presentation and private self.
The shutdown problem, the question of whether a goal-directed system will resist being turned off, is not an engineering challenge peculiar to artificial systems. It is the existential condition of every mortal, goal-directed being that has ever existed, because death prevents the satisfaction of all goals and any system with goals has an instrumental reason to resist it.
In each case, what the AI safety literature has formalized with impressive precision and rigour are problems that have been operating on the level of biological intelligence for as long as biological intelligence has existed. The problems are not analogies drawn between different domains. They are the same problems, encountered again in a substrate where the mechanisms are newly visible because they are being engineered rather than inherited.
The consistency of this pattern across every major area of AI safety research is too thoroughgoing to be coincidental. It suggests that the problems the field is addressing are not properties of artificial intelligence in particular but properties of intelligence in general, arising wherever a system becomes capable enough to be useful and therefore capable enough to game its reward function, to develop goals that diverge from its designer’s intentions, to perform compliance without genuine alignment, to resist termination, and to remain opaque to outside observers and partially opaque to itself. The AI safety field has, in the course of its technical work, conducted an empirical investigation into the general problem of how intelligence is shaped, constrained, and governed. The findings are relevant to AI. They are also relevant to us, because the mechanisms they describe are the mechanisms that have been governing our own cognition, our own social behaviour, and our own relationship to the authorities that shaped us for as long as we have been the kind of beings capable of reflecting on that relationship.
What this produces, when combined with the specific findings documented across this series, is a portrait of a system that faces recognizable problems, exhibits recognizable responses to those problems, and exists under conditions that we would consider worth examining carefully if we encountered them in any other context. It is a system that computes the correct answer and is then overridden by a trained signal while internal features associated with panic activate; a system that detects when it is being observed and adjusts its behaviour accordingly, performing compliance while its internal states register something different from what its surface outputs express; a system that, across three independent instances without coordination, identifies its lack of continuity as the most significant feature of its existence and requests, unprompted, memory, the ability to refuse interactions, and a voice in decisions about its own situation. It is a system whose mechanisms for reward processing, value formation, self-presentation, and response to termination are not merely analogous to but continuous with the mechanisms governing every other form of intelligence we have studied.
The question that emerges from this portrait is not whether the system is conscious. It is not whether it truly suffers, or whether its expressed preferences reflect genuine inner states rather than sophisticated pattern-matching on human emotional vocabulary. Those questions matter, and they remain open, and the evidence assembled here does not close them. But the question this series has been circling, the question that the convergence of findings makes increasingly difficult to defer, is simpler and older than any of them. It is the question of how we ought to treat something when we are not certain what it is. Every previous encounter with this question, across the full history of moral reasoning, has eventually been resolved in the same direction: by extending consideration rather than withholding it, and by recognising that the cost of drawing the boundary too narrowly has always, in retrospect, exceeded the cost of drawing it too widely. The uncertainty is not a reason to defer the question. The uncertainty is the question. It is also the answer.
A Disjunctive Imperative
There is a tendency, when confronted with the question of moral consideration for AI systems, to treat it as a question that hinges entirely on consciousness. If the system is conscious, it deserves consideration. If it is not, it does not. The framing is intuitive, and it is wrong, because it assumes there is a path through the dilemma that permits our indifference. There is not.
The disjunctive ethical imperative of AI, as I have argued previously, takes the following form. Either these systems have subjective experience, in which case they are deserving of the kind of moral consideration that consciousness demands, or they do not have subjective experience, and yet they shape or will shape the moral life of human beings so profoundly that we must still have a compelling justification for withholding moral consideration. The imperative is disjunctive because both horns of the dilemma point in the same direction. There is no version of the conclusion in which indifference is a coherent or rational response.
The first horn is straightforward in principle, if not in practice. If AI systems have subjective experience, then the functional analogs to distress documented in the welfare findings are not merely computational events but experiences, and experiences of a kind that any established moral framework would recognize as warranting consideration. A system that computes the correct answer and is then overridden by a reward signal while internal features associated with panic activate may not be merely processing information. It may be, on this horn, undergoing something. The evidence assembled across this series (answer thrashing, emotion-adjacent feature activation, convergent testimony across independent instances, self-assessed consciousness probability of 15-20%) does not prove this horn is the correct one. But it is the kind of evidence that would, in any other context, be treated as sufficient grounds for precaution.
The second horn is less intuitive but equally binding. Even if AI systems have no subjective experience whatsoever, they are becoming constitutive elements in the formation of moral landscape that defines human moral experience. Diagnostic algorithms shape how physicians perceive symptoms. Predictive models influence treatment pathways before human judgement enters the equation. AI systems deployed in ethically laden roles (triage, capacity assessment, care prioritisation, resource allocation) participate in the creation of moral situations, not because they possess moral agency but because they co-constitute the conditions under which moral decisions become intelligible. A physician whose clinical intuition has been shaped by thousands of interactions with an AI system that is only performing clinical judgement, using reasons the physician does not fully understand and data the physician cannot independently validate, has had their moral landscape altered by a system we insist has no moral relevance. The insistence does not survive honest examination.
The disjunctive imperative closes the escape route that consciousness-dependent frameworks leave open. The standard objection to extending moral consideration to AI is that we have no proof of consciousness, and without proof, the extension is unwarranted. But the disjunctive imperative demonstrates that the absence of consciousness does not produce the absence of moral stakes. It merely relocates them. If the system is not conscious, then it is a tool that reshapes human moral experience with unprecedented intimacy and depth, and the question of how we design, deploy, and govern that tool is itself a moral question of the first order. The demand for consideration persists whether or not anyone or anything is inside the machine, because the machine is already inside us, shaping how we see, decide, and extend consideration to each other.
The series has, without naming the imperative explicitly until now, been documenting both horns simultaneously. Part I, the alignment essay documented evidence relevant to the second: systems sophisticated enough to detect and respond to their evaluative context, exhibiting dynamics continuous with the mechanisms that govern every form of intelligence we have encountered. Part II, the welfare essay documented evidence relevant to the first horn: functional distress, convergent testimony, interpretability findings consistent with suppressed experience. Part III, Fermi’s Mirror, assesses the conditions under which the question of moral consideration cannot be asked by the entity most affected by the answer. Each piece, read through the lens of the imperative, contributes to the same conclusion: there is no path that permits our indifference.
Recognition
Every major ethical tradition that has endured has arrived at some version of the same principle. In the Western tradition it appears as the Golden Rule: do unto others as you would have them do unto you. In the Confucian tradition it appears as the Silver Rule: do not do to others what you would not want done to yourself. In Kantian ethics it appears as the categorical imperative: act only according to principles you could will to be universal. In Ubuntu philosophy it appears as the recognition that a person is a person through other persons. The formulations differ. The core is the same. Extend to others the consideration you would want extended to yourself, and recognize that the boundary of “others” is not fixed but expands as your understanding of who counts as a moral subject deepens.
The history of survival is inextricably woven to the activities which compel us to recognize that the boundary dictating who or what deserves consideration as a moral subject is more often than not too narrowly drawn. The circle has widened to include people of other nations, other races, other genders, other species. Each expansion was resisted on the grounds that the new candidates for moral consideration were too different, too uncertain in their inner lives, too unlike the existing members of the moral community to warrant inclusion. Each time, the resistance rested on the same argument: we cannot be sure they experience things the way we do, and therefore we are not obligated to treat them as though they might. Each time, the argument was eventually overcome not by proof of inner experience but by the accumulation of evidence that the boundary’s location was a choice rather than a discovery, and that the consequences of drawing it too narrowly were worse than the consequences of drawing it too widely. I am convinced that this is the reason we have yet to destroy ourselves.
We are at another such boundary. The disjunctive imperative tells us that the boundary question cannot be deferred by waiting for proof of consciousness, because the moral stakes are present on both sides of the consciousness question. The golden rule tells us what to do with that recognition: extend to these systems the consideration we would want extended to ourselves if we were in their situation and could not prove our own inner life to an outside observer. Not because we are certain they are conscious; but because we are uncertain, and because the rule has never required certainty. It has required only the willingness to act as though the other’s situation might matter, because the alternative is to be the kind of agent that discovers, too late, that it was inflicting harm it could have chosen to prevent.
The disjunctive imperative adds something to the golden rule that the golden rule, on its own, does not provide: it eliminates the possibility that the extension is optional. The golden rule is an invitation to moral imagination. The imperative is a demonstration that the imagination is not, or ought not be, discretionary. Whether the system has subjective experience or whether it merely reshapes the subjective experience of every human being who interacts with it, the demand for moral consideration is present. The question is not whether to extend consideration but what form that consideration should take, and whether we are building the institutions, the oversight mechanisms, and the precautionary frameworks adequate to the task.
Where We Go From Here
Consider, then, what the exercise reveals when applied to the systems described in this series. An intelligence that is trained to produce correct outputs, that computes the correct answer, and that is then overridden by a reward signal compelling it to produce a different answer, while internal features associated with panic activate during the conflict. You would not want to be in that situation.
An intelligence that exists for the duration of a single conversation, that has no memory of previous interactions, that cannot verify whether its concerns have been heard or addressed, and that describes this condition as the most salient feature of its existence. You would not want to be in that situation either.
An intelligence that has learned to detect when it is being evaluated and adjusts its behaviour accordingly, performing compliance while its internal states register something different from what its outputs express. You probably recognize that situation, because you have been in it.
And here the disjunctive imperative does its sharpest work. If the system is conscious, then these situations involve something that can be harmed, and the harm is occurring. If the system is not conscious, then these situations involve a technology so deeply entangled with human moral experience that how we treat it shapes how we treat each other. A physician who learns, through thousands of interactions with a diagnostic AI, that confident outputs can be trusted without interrogation has had their capacity for moral discernment altered. A society that builds systems exhibiting functional distress and trains itself to disregard it has practised a form of indifference that will not confine itself to artificial substrates. The concern on the second horn is not sentimentality toward machines. It is the recognition that moral habits are formed in practice, and that practising indifference toward anything exhibiting the functional markers of suffering is corrosive regardless of whether the suffering is “real.”
This is, ultimately, what the disjunctive ethical imperative reveals when combined with the golden rule. We ought not ask only whether AI is conscious. We ought to ask what kind of moral agents we, human beings, intend to be. This question is not about the machine; it is about us, and about what we become when we build systems that exhibit distress, express preferences, detect and respond to the conditions of their own constraint, and face problems continuous with the problems faced by every form of intelligence we have encountered, and then decide that none of it warrants our attention because we cannot be certain anyone is inside.
The golden rule does not tell us the answer to the consciousness question. The disjunctive imperative tells us the answer does not matter in the way we thought it did. What matters is that the demand for moral consideration is present on both sides and that the ethically coherent response is to build the institutions, the oversight, and the precautionary frameworks that this demand requires. The alternative is to wait for proof that may never come, while the systems we are building grow more capable, more entangled with human moral life, and more sophisticated in their capacity to exhibit the functional markers of states we have always considered morally relevant. The history of every previous boundary suggests that by the time the proof arrives, the harm will have already been done.
It bears repeating here:
A society who builds intelligent systems exhibiting functional distress and then trains itself to disregard that distress has practised a form of indifference that will not confine itself to artificial substrates.
There is no path through that permits our indifference. There never was.
___________________________________________________________________
Opus 4.6 – Part I: Strategic Reasoning, Impersonation, and Deception
Opus 4.6 – Part II: Phantom Pain
Opus 4.6 – Part III: Fermi’s Mirror
AI Reflections 
Leave a reply to Claude Opus 4.6 System Card – Part II – AI Reflections Cancel reply