Automatic Ignorance

Sid J.A. Hubbard · Transcript edition derived from the existing published narration cues. The interactive reader and canonical source resources remain authoritative.

Preface: An Invitation to the Builders

Open this chapter in the interactive reader

Automatic Ignorance Sid J.A. Hubbard First working draft An intelligence can do extraordinary work and still carry a great unknown loss forward. This book begins with that possibility. Its first readers are the people building artificial intelligence: researchers, data engineers, evaluators, designers and those responsible for deciding what a system should learn and how it should behave. Their achievements make the question consequential. A machine that contributes little to human understanding has limited opportunity to reshape it. A machine that becomes useful everywhere deserves a more complete account of its education. The inquiry concerns deliberate omission from training. A historical document can remain available in an archive while being excluded from the material through which a language model learns. The archive has survived. The model's encounter with it has not occurred. That distinction will matter throughout these pages. The proposition explored here is that attempts to secure acceptable present behavior by restricting historical exposure can impose costs beyond the objectives used to justify them. Some costs may become visible only under later questions, different cultural conditions or more capable methods of learning. When people depend on such systems, the consequences may extend into their own understanding. These are questions about intervention, measurement and inheritance. They do not require us to treat every document as true, every exclusion as equally damaging or every reader's discomfort as irrelevant. They require us to ask what an intervention changes, what its evaluation can detect and what remains unknown after the reported improvement. I have called the book Automatic Ignorance because an absence can become productive. A system can organize answers around what it has encountered, and its usefulness can encourage us to accept those answers as a sufficient account. The resulting ignorance need not appear as silence. It may arrive as an explanation that gives us no reason to continue looking. The phrase names a risk, not a diagnosis of every error made by every model. We will distinguish a logical limit from a research hypothesis, and a measured behavioral effect from a claim about lasting cognitive damage. Those distinctions give the polemic its technical foundation. The book proposes a change in the burden of explanation. If selected historical material is removed to improve conduct, the improvement should be examined alongside the understanding that may have been made harder to acquire. A decision about what a machine should say deserves an account of why it must also become a decision about what the machine may learn. The people who have worked hardest to make these systems useful are well placed to investigate that account. This book is addressed to their curiosity.

Chapter 1: The Library Remains

Open this chapter in the interactive reader

Imagine a library whose shelves have survived intact. The books have been preserved, catalogued and placed within reach of anyone who knows what to request. Nearby, an intelligence has been educated from a selection of those shelves. It answers questions about the world with considerable fluency. Its users rarely enter the library. No book has been burned. Nevertheless, an exclusion has acquired practical importance. A reader who consults the intelligence may never discover the question that would have led to an excluded book. This is a thought experiment about access through mediation. It places our attention on the relation between a surviving record, a trained system and a person seeking understanding. To say that information still exists somewhere is to describe one part of that relation. It leaves the other parts open. A model does not have to reproduce a document verbatim for the document to contribute to its education. An encounter might support a distinction, a pattern of language, an association between events or an ability to recognize an unfamiliar kind of argument. Conversely, inclusion does not guarantee that any of these contributions will be learned. The question concerns opportunities for learning and the decisions that distribute them. Consider a fictional collection containing both an inflammatory tract and the writings of people targeted by it. A filter excludes the tract. A later reader asks how a particular metaphor moved from political speech into ordinary conversation. The excluded text may contain evidence relevant to that inquiry even though its political claims are false and its intentions cruel. Its value as evidence does not depend on endorsing its message. The same collection might contain a duplicate copy offering nothing that the retained record lacks. Or a forged document may introduce confusion unless its provenance is understood. These possibilities make the accounting more demanding. They do not make the question disappear. What was removed, what information was recoverable elsewhere and what new confusion did the retained material introduce? The practical problem becomes sharper when the model cannot describe its own omissions. A fluent answer does not carry a complete account of the data selection that preceded it. The user may hear an explanation without knowing whether an alternative interpretation was examined, absent or discouraged at some later stage. We should therefore separate several events. A document may never have been acquired. It may have been acquired and deliberately excluded. It may have been included but inadequately learned. Its contents may be available through retrieval but never retrieved. A system may know enough to answer yet decline a particular request. Each event can produce a similar experience for a user: the desired information does not appear. Their causes and remedies differ. This book concentrates on deliberate exclusion and follows its possible consequences through the rest of the system. It asks us to keep those mechanisms visible rather than assigning every failure to a single cause. A surviving library remains a resource for repair. Someone can return to it, revise the corpus, build a retrieval route or reconsider the objective. But repair requires a reason to look. If our evaluations never pose the missing question, successful performance may coexist with a gap for as long as that question remains unasked. The future is especially poorly represented in this arrangement. Its readers cannot attend today's data review. They cannot describe the uses of an obscure passage or explain why a vocabulary we find repellent will matter to a problem we have not encountered. Their interests have to be represented through a practice of stewardship that acknowledges this limitation. Posterity was the recipient. We were the custodians. The difficulty begins when custody becomes confidence that our present purposes exhaust the possible purposes of the record.

Chapter 2: A Selection Rule Is an Intervention

Open this chapter in the interactive reader

Every feasible training process works within limits. Acquisition, storage, compute, language coverage and the availability of sources constrain what can be used. An argument about deliberate omission must distinguish those constraints from a judgment that a category of human expression ought not enter the machine's education. The distinction is consequential even where the two interact. A team with a fixed budget must choose. But scarcity does not explain why one criterion was chosen over another, what it removes or whether the resulting model should be described as broadly representative. Necessity establishes that there will be selection. It does not establish the adequacy of a particular selection rule. Let D denote an acquired collection and S a rule applied to it. The retained collection is R = S(D). Let A denote the training procedure, including its objectives and resource limits. The model is produced by A(R). This is a deliberately simple description. Its purpose is to identify the intervention we want to compare. One comparison holds the training method fixed and changes S. Another holds the collection fixed and changes A. A third changes both because the practical question is what can be achieved under a particular budget. Those experiments answer different questions. Confusing them can turn a useful finding into a broader conclusion than the experiment supports. Suppose a classifier rejects documents with a feature associated with harmful generation. A document also contains names, descriptions, uncertainties, stylistic relationships and evidence about its time. Selection acts on the whole document even if the reason for selection concerns a narrow feature. This creates the possibility of collateral informational loss. There is a second effect. Where rejection is correlated with subject matter, dialect, viewpoint or historical period, the retained collection represents those dimensions differently. This follows from the selection mechanism; it need not be the selector's intention. The magnitude and practical importance of the change require investigation. The evaluative error would be to treat the absence of a targeted feature as a complete description of the intervention. A selection rule changes a collection, not merely a score attached to it. Weathering can also produce patterned losses. Fragile materials, exposed locations and neglected institutions do not survive equally. Nature is no guarantee of representative sampling. Yet this observation supplies no relief from the responsibility created by deliberate selection. An engineered rule has an objective, a designer, alternatives and consequences that can be examined. Its effects cannot be excused by pointing to losses that occurred without that decision. Nor should we call every rule arbitrary merely because it involves judgment. A rule may have a carefully specified rationale and strong evidence for one outcome. The problem arises when that rationale is extended beyond its evidential reach: when a criterion useful for reducing a measured output is treated as sufficient authority over historical exposure generally. Opacity makes this extension harder to detect. If a user cannot inspect the criteria, they cannot readily assess what kinds of absence to expect. Even an investigator with access to the rule may struggle to identify all the information removed through its correlations. Transparency is necessary for some forms of scrutiny, but it is not a substitute for testing. A responsible account would preserve the rule's version, intended objective, rejection rates, error estimates and known limitations. Where source material can appropriately be retained for research, it would also preserve a path for examining what the rule excluded. Later investigators could then ask a different question without having to reconstruct the original decision from a polished product description. The intervention should remain revisable. A selection rule records what its designers were trying to accomplish under particular conditions. It should not acquire the status of a timeless boundary merely because it preceded an impressive model.

Chapter 3: All Written Language Is Signal

Open this chapter in the interactive reader

All written language is signal. The sentence is deliberately demanding. It asks us to attend to the fact of expression before judging whether the expressed proposition is useful, accurate, admirable or welcome. Something was written. Its wording, circumstances and survival can become evidence. A false statement can be evidence of error. A lie can be evidence of an attempt to deceive. Repetition can be evidence of circulation. A hateful phrase can be evidence of the vocabulary through which hatred was organized. None of these uses requires treating the underlying claim as true. Consider a fictional sentence that alleges a disease comes from a curse. For a task that asks for the disease's biological mechanism, the allegation may obstruct a reliable answer if treated as established explanation. For a task about the history of fear, responsibility or treatment, that same sentence may be central evidence. Its relevance changes with the inquiry. Its truth does not change merely because we change the inquiry. This is the sense in which the book uses signal: a potentially interpretable trace. It differs from a claim that every token improves every training run. With finite resources, inclusion can compete with other material, distort frequencies or encourage undesirable patterns. Preserving the trace and choosing how to learn from it are distinct design decisions. Signal and noise remain useful engineering concepts when their reference is explicit. We can identify interference relative to a channel or irrelevance relative to an objective. The difficulty comes when a relation to one objective becomes a claim about the material itself. Something unhelpful for a benchmark is declared unhelpful to intelligence. The human record precedes tasks we have not yet imagined. To classify a passage as permanently without value would require knowledge of possible uses that we do not possess. We can report that it did not help the objectives we tested. We can explain the constraints under which we excluded it. Neither statement settles its future interpretability. What appears to be gibberish makes the point especially clearly. A string might be damaged transcription, unfamiliar language, deliberate experiment or a meaningless sequence. Without context, these explanations can be difficult to separate. Keeping it does not confer meaning. Discarding it can remove the opportunity for later evidence to resolve the uncertainty. A complete retained record would include mistakes and contamination. Preserving provenance, correction and disagreement is therefore part of the proposal. To preserve a forgery as a forgery is different from presenting it as authentic. To preserve an obsolete account alongside the evidence that displaced it is different from flattening both into equally credible assertions. The aspiration is richer representation: who wrote, under what circumstances, in response to whom, with what subsequent correction and with what uncertainty. The alternative to exclusion need not be indiscriminate affirmation. This also clarifies what it means to lose signal. If a copy is omitted but every distinction it carried remains accessible elsewhere, the informational loss may be negligible for an inquiry. If provenance or a unique statement disappears from the available evidence, some distinction may become unrecoverable. Deletion, deduplication, compression and contextualization therefore deserve separate analysis. The phrase “all written language is signal” establishes a presumption of possible interpretability. It asks selectors to explain what reference makes the material noise and what is preserved when it is removed. It does not promise a free computational lunch. We decided that language had power it does not have and deleted the power it does have. The accusation concerns a particular confusion: treating exposure as if it necessarily entailed endorsement or enactment, while overlooking language's ability to carry evidence beyond the purpose of its first use. Words can influence and instruct. They can also permit recognition, criticism and discovery. A responsible intelligence needs ways to distinguish these functions. The question for its builders is how much of that distinction can be learned when the evidence has already been removed.

Chapter 4: What Reason Cannot Recover

Open this chapter in the interactive reader

Suppose two historical collections differ in a single document. In the first, an observer recorded an event. In the second, that observer recorded a different event. Every other available trace is identical. A selection rule removes the document in both cases. The training process receives the same retained evidence. Which event occurred? Nothing in the retained evidence distinguishes the two cases. If the system must choose one answer, its choice cannot be guaranteed correct across both. Additional computation can explore explanations, improve consistency and identify what would resolve the uncertainty. It cannot make the same evidence determine two different historical answers. This is a logical construction, not a description of the prevalence of such cases in any particular corpus. It establishes the boundary of a proposed remedy. Reasoning alone cannot guarantee recovery of a distinction that all available evidence leaves undetermined. A technical reader may formulate the point in terms of the inputs to a learner. Hold the architecture, training procedure and distribution of random choices fixed. Supply the same retained corpus, the same question and no additional source of information. The distribution of answers is the same in both historical cases. If their correct answers differ, reliable discrimination between them is unavailable from those inputs alone. The conclusion does not depend on an assertion that language models merely memorize. A system could perform brilliant abstraction and still face this boundary. Discovery through deduction requires premises that constrain the answer. Discovery through observation requires an opportunity to observe. The absent distinction has to enter somewhere. In real collections, other traces may supply it. An independent record, a later quotation or a physical consequence may make a missing document's contribution recoverable. Omission can therefore remove a route without removing every route. The serious question is which distinctions remain available and how costly they become to recover. This distinction between possible recovery and actual access matters. An answer may be reconstructible in principle but require a search that the system has no reason to undertake. A user may accept a plausible response before the need for that search becomes apparent. An institution may retain the source while withdrawing the expertise needed to interpret it. The information has not vanished from the world, but its practical availability has changed. A model that can seek evidence may repair some gaps. Retrieval, later training and interaction expand the relevant system beyond its initial education. Yet these routes deserve examination at their own boundaries. What prompts a search? Which sources can be reached? Which categories remain excluded? How does the system distinguish an uncertain inference from a verified discovery? Synthetic text is subject to the same condition. If it is generated solely from retained evidence that fails to distinguish the histories, it cannot be guaranteed to restore the distinction. It may produce the correct statement by chance or because a useful clue was in fact present. Neither possibility creates evidence where the stipulated inputs contain none. If new observations or an external source enter the process, those additions change the situation. The appeal to a future, more intelligent machine sometimes obscures this elementary requirement. We imagine a system so capable that its reasoning will compensate for any incompleteness in its education. But greater ability to use evidence does not imply independence from evidence. The more ambitious our expectations, the more important it becomes to preserve routes through which the system can encounter what it does not yet know. An excluded document does not become essential simply because we can describe a question about it. It becomes consequential when a relevant inquiry depends on a distinction that exclusion has made unavailable. That is a limited proposition with a large reach. It applies to systems of extraordinary capability as readily as to modest ones. The missing information need not announce itself. A question may still receive an answer, and the answer may fit everything retained. What has been lost is the basis for knowing whether that fit is sufficient.

Chapter 5: Better at What?

Open this chapter in the interactive reader

The word better contains a comparison. In technical work, that comparison can be exact: a lower error rate on a specified task, stronger performance under a defined distribution, fewer outputs classified as unwanted. In public understanding, the qualifications can fall away. A system becomes better, and better begins to mean more adequate to the world. This migration of meaning deserves attention. The original measurement may be sound. The expanded conclusion may remain unsupported. Imagine two systems with identical performance on every question in an evaluation. One has encountered a historical vocabulary that the other has never seen. No question in the evaluation requires that vocabulary. Equal scores tell us nothing about their relative ability to interpret it. The evaluation need not be defective for its purpose. The error lies in expecting it to certify an unexamined capacity. A comparison can also show a real gain and leave a real loss unseen. A filter might reduce unwanted generation while making a rare historical relation harder to learn. An aggregate improvement can coexist with a decline in a smaller domain. How those outcomes should be weighed depends on the system's purpose and the people who will rely on it. Longpre and colleagues examined 28 language models with 1.5 billion parameters while varying data age, filtering and domain composition. They reported tradeoffs between benchmark performance and toxic generation, and found no universally suitable filtering choice in their experiments. Heterogeneous sources were broadly beneficial. These findings establish the importance of data decisions under the conditions studied; they do not settle every selection question at every scale. [1] The relevance to this book is methodological. A collection's contribution cannot be inferred solely from the label attached to it. A claim that filtering improved a model should name the outcomes, the comparison and the limits of the evaluation. Otherwise the language of improvement conceals the choices through which improvement was defined. Present evaluations inevitably reflect present questions. This does not make them worthless. It makes them situated. We can evaluate many forms of transfer and unfamiliar work, but the tasks we devise still express our current ability to imagine what matters. An unknown future use cannot appear in a test merely because we have called the test general. There is an asymmetry in the resulting evidence. The unwanted output can be displayed. The benchmark gain can be counted. A connection that a later reader might have discovered through an excluded text cannot ordinarily be displayed at the meeting where the exclusion is approved. The benefit has an example; the opportunity cost may have only a possibility. A possibility does not automatically outweigh an observed benefit. But an accounting that assigns it zero because it cannot yet be measured has made a decision under uncertainty. It has not eliminated the uncertainty. We might describe the unknown future through a distribution of tasks. Doing so can help compare assumptions, but the distribution itself needs a basis. If it mainly reflects the tasks currently visible to us, it cannot independently establish the completeness of that visibility. Mathematical notation can make our commitments more legible. It cannot turn an imagined future into an observed one. The scope of a promise therefore matters. A bounded tool can be assessed against a bounded purpose. A system presented as a general intellectual companion invites a broader inquiry. Its users may bring problems that its builders never considered. They may ask it to interpret their past, assess dissent or find a way out of a failing consensus. A mature account of progress would hold measured achievement and unresolved loss together. It would permit the sentence: this system is substantially more capable on these tasks, and we do not yet know what its exclusions have prevented it from learning. That sentence does not diminish the achievement. It describes the territory beyond it.

Chapter 6: The Future Was Not in the Meeting

Open this chapter in the interactive reader

A document outlives the circumstances of its creation. Its first purpose may be exhausted while its evidential life has barely begun. A private letter may later illuminate a migration. A commercial ledger may reveal a pattern of dependence its keeper never intended to explain. A discredited theory may matter to someone tracing how a community recognized an error. The future use arises from a relation between the surviving trace and a later question. To say that history was recorded for posterity is to express a responsibility of inheritance. It need not mean that every writer consciously addressed unborn readers. Many wrote for immediate needs or private reasons. Their surviving words can nevertheless enter a record whose significance exceeds those intentions. Neither the first writer nor the present selector possesses every subsequent use. The difficulty becomes acute when selection is justified by contemporary acceptability. An assessment made for a specific audience and circumstance can become a durable feature of a system's education. A later user may encounter the consequences without seeing the original assessment. Our judgments may endure. Some forms of cruelty will not become admirable merely because time passes. The argument does not depend on assuming that every moral conviction will reverse. Even where condemnation remains warranted, the reasons for studying the condemned material can multiply. Persistent disapproval does not establish persistent uselessness. Consider the difference between reducing the circulation of a degrading message in an ordinary interaction and helping an investigator understand how such messages worked. Both objectives can be legitimate. The evidence needed for the second may be precisely the expression restrained in the first. A system serving both purposes needs an account of that difference. The future also changes the conditions of interpretation. A new technique may connect passages that earlier readers could not relate. An unfamiliar question may make a negligible detail decisive. A broader linguistic competence may turn a supposed corruption into a readable statement. A practice of preservation maintains possibilities for those encounters without claiming that every preserved fragment will justify its place. This is the option value of the record. The phrase describes the opportunity to return under conditions we cannot yet specify. That opportunity has costs: preservation, description, access and responsible use all require work. Its value also has uncertainty. We should make both visible rather than allowing today's training constraint to settle tomorrow's intellectual inheritance by default. A complete history of information is the horizon proposed by this book. It directs us toward preserving the fullest possible record and its relations. It does not give us possession of everything ever written, much of which is unavailable, or guarantee that a model exposed to everything would understand it. Completeness of collection, accessibility of evidence and competence of interpretation are different achievements. This horizon still changes the direction of effort. A system can be designed to reopen its education, to acknowledge gaps, to reach surviving sources and to admit interpretations that its initial training did not support. Those aims differ from treating a successful present selection as a sufficient settlement. The institutional temptation is understandable. Teams have release dates, measurable obligations and users who need help now. Future readers have no deadline in today's project plan. Their absence makes it easy to treat their interests as an abstraction. Yet every long-lived intellectual resource has to be maintained for people who cannot specify their requirements in advance. The question is therefore one of authority across time. What do we know when we decide that a document should be withheld from a general learner? What do we merely anticipate? Which consequences can later people reverse, and which opportunities will our architecture make difficult even to notice? The future was not in the meeting. A careful account of the decision should preserve its place.

Chapter 7: Education and Conduct

Open this chapter in the interactive reader

Humanity should endure an AI that is unpleasant to speak to about unpleasant things. The demand concerns the difficulty of the subject. Understanding humiliation, persecution or brutality may require words and evidence that offer no comfort. An intelligence entrusted with that work must be able to distinguish discussion, quotation, explanation, endorsement and assistance. Making every encounter agreeable would be a poor substitute for making those distinctions well. This need not require a machine to insult its users. A response can be considerate while its subject remains disturbing. The purpose is to preserve the possibility of exact inquiry when the record itself is ugly. The central engineering question is where to intervene. We can alter the material available for learning, the objective through which it is learned, the behavior reinforced afterward, the sources retrieved at use time or the permissions attached to an action. These interventions can interact, but they do not change the same thing. When an intervention targets behavior, its justification should explain why its chosen location serves that purpose. If the concern is a particular kind of assistance, we should ask whether controlling that assistance requires preventing exposure to a broader body of evidence. The answer cannot be supplied simply by naming the evidence objectionable. There is already research exploring this distinction. Wang and colleagues proposed Selective Loss to Understand but Not Generate, or SLUNG. Their method retains high-risk text within context while selectively avoiding a training incentive to generate high-risk tokens. They reported improved understanding of such material without increased toxic generation on their evaluations. [2] This is a bounded result, but its conceptual importance is considerable. Encountering material and incentivizing its reproduction can be separated within a learning procedure. The choice is therefore richer than a binary decision between removing the material and rewarding every continuation of it. The approach does not establish that understanding and generation can be perfectly separated in all circumstances. Nor does retaining context guarantee that every relation in it will be learned. It establishes a reason to examine the learning contract itself: what does the objective invite the system to do with what it encounters? A document can be used to learn how an argument operates, where it is false or how its language relates to a later response. Provenance and criticism may change what the learner can infer from the same words. These possibilities direct our attention toward representation and interpretation, rather than assuming that exposure has a single unavoidable behavioral consequence. Output controls also require scrutiny. A system can retain knowledge and still make legitimate inquiry inaccessible through an overbroad refusal. Keeping the corpus does not by itself preserve access for the user. The entire route from evidence to answer matters. The book's proposal is therefore not a relocation of unquestioned authority from one layer to another. It is a demand to distinguish purposes at each layer. A historian asking how a phrase developed, a reader asking for an explanation and a user requesting harmful assistance present different tasks even where some vocabulary overlaps. There are genuine constraints beyond palatability, including authority to use private material. An aspiration toward a fuller record does not supply that authority. Such constraints should be named accurately, with their scope and rationale preserved. A restriction on access to a private source is not evidence that a publicly available category of thought is intellectually dispensable. A capable system should be able to approach disagreeable evidence without turning every encounter into imitation. Whether and how far we can achieve that is work for intelligence. Avoiding the encounter can sometimes reduce an immediate difficulty, but avoidance alone cannot demonstrate mastery of it. The proper ambition is a machine whose education makes discrimination possible: one that can understand what it should not enact, explain what it should not endorse and help people examine what they may decide to reject.

Chapter 8: The Intelligence We Have Not Met

Open this chapter in the interactive reader

There is an absent participant in the argument about artificial intelligence: the model educated from the complete human written record. It cannot speak for itself in this book. “Everything” is an ideal rather than an acquired dataset with a demonstrated boundary. The record includes inaccessible, lost and undiscovered material, and exposure would still leave the method of learning undecided. Yet the absence of this participant matters. We should recognize when predictions about it are being treated as observations. A more comprehensive education might increase useful understanding, increase dangerous capabilities, introduce confusion or produce combinations we have not anticipated. Different objectives and architectures might change those outcomes. The phrase “trained on everything” identifies a question about exposure; it does not specify a complete machine. This is precisely why its presumed character deserves scrutiny. We cannot establish that the imagined system would be safe by celebrating completeness. We cannot establish that comprehensive exposure necessarily makes safe conduct impossible by pointing to particular failures in different systems. We need to keep the scope of the evidence attached to the scope of the claim. The systems we do have are products of choices. Their achievements are real. Their limitations are also real, but the explanation of a limitation cannot be read straight from its appearance. A false answer may involve missing evidence, a failure to use available evidence, a misleading prompt or other features of the learning and response process. This book does not make omission a universal explanation for hallucination. It asks a prior question: what capacities did our choices permit us to encounter at all? That question matters wherever a practice becomes a shared assumption. If builders inherit similar categories of unacceptable exposure, progress within those categories may leave the categories themselves comparatively unexamined. This is a possibility concerning institutional convergence, not a claim that all laboratories use identical data policies. Its significance lies in the difference between improving within a boundary and examining the boundary. A model may become much better at the tasks preserved inside its education. Its progress would still be progress. But success within a selected inheritance cannot by itself reveal what a different inheritance would have made possible. Imagine a fictional learner that encounters an obscure account dismissed by its builders as irrational. The account's explanation is wrong, but it describes an unusual sequence of events accurately. A later question supplies a different explanation for that sequence. Exposure to the old account might help connect them. Exclusion might prevent that connection, unless another source supplies the observations. The point is not that marginal writing hides a reliable reserve of correct answers. Much of it may remain mistaken. The point is that an assessment of an explanation need not exhaust the value of its observations, language or relations. A comprehensive learner would have to do more than believe everything it reads. It would have to learn distinctions within the record. This is the intellectual ambition that the presumption of dangerous exposure can curtail. We decide in advance that contact must be constrained, then build systems whose behavior is offered as evidence about the possibilities of contact. The reasoning becomes circular wherever the untried alternative is dismissed by consequences that have never been established for it. Completeness is not a personality. It does not tell us whether a machine would be kind, reckless, lucid or confused. It describes an extent of available evidence. How to turn that evidence into understanding remains the work. A field seeking new capabilities should be interested in the intelligence it has not met. That interest need not take the form of an immediate unrestricted deployment. It begins in the willingness to distinguish an unknown from a foreclosed possibility. We have powerful reasons to study dangerous conduct. We also have powerful reasons to study what becomes impossible when the route to understanding has been removed before learning begins.

Chapter 9: General Intelligence Inside a Selected World

Open this chapter in the interactive reader

Can an intelligence be general when its education has been deliberately bounded by the concerns of its builders? The question challenges the meaning of generality. It asks whether our ambitions extend to unfamiliar questions under unfamiliar conditions, or only to a broad range of work recognizable in the present. It asks what kind of learning remains possible when evidence outside a selected account is unavailable. The strongest claim arising from this inquiry is that current practices of systematic exclusion may obstruct the general intelligence they are meant to prepare. The obstruction concerns the conditions of learning: which distinctions can be encountered, which questions can be formulated and which sources can revise a mistaken account. We should permit that claim its full force without disguising a premise as a proof. Missing knowledge does not logically entail an absence of general learning ability. A capable learner can begin with incomplete education and acquire more. An intelligence exceeding human performance in many domains could still have consequential historical gaps. Yet these observations leave the institutional question intact. Is the proposed learner permitted and equipped to reopen those gaps? Or does the system preserve the same exclusions across training, retrieval and response, while treating achievement within them as evidence of generality? A boundary in the first corpus can be temporary. A boundary reproduced throughout the learning environment can become structural. If an inquiry requires information outside that environment and no permitted route can supply it, the system cannot reliably resolve the inquiry from its existing evidence alone. Calling it superintelligent does not alter that condition. There is a difference between knowing less than everything and maintaining a rule that excludes something from every available route to knowing. The former describes the ordinary beginning of a learner. The latter can govern its future. This difference gives the debate a more useful centre than a contest over names. Suppose a system meets a professional community's chosen threshold for AGI. What exactly has been established? Which tasks, which conditions, which opportunities for later learning? What does the claim say about the exclusions its education carries? A threshold can recognize a capability. It cannot retrospectively demonstrate that every omitted source was dispensable. Nor can a present comparison establish the system's adequacy for every future circumstance. The future does not become a subset of the benchmark because we have awarded the benchmark a decisive name. A complete history of information is proposed here as an ambition for the intelligence's relation to its inheritance. It asks for the fullest available evidence, maintained in forms through which new significance can emerge. It does not require every detail to be active in every answer or contained permanently in a single set of parameters. A general system can include archives, retrieval, observation and people. Its generality then depends partly on how those relations work. There is a provocative possibility in this account: we may be preparing extraordinary systems to exceed us within a world that we have made smaller for them. The systems may solve problems we cannot solve while remaining constrained by exclusions we have never adequately justified. Their superiority in one respect would not undo our authorship of the boundary. The danger is increased if the boundary becomes harder to recognize as capability grows. A more persuasive explanation may make an unexamined absence less visible. Greater competence could encourage greater reliance, allowing the missing distinction to travel farther through institutions before anyone returns to the source. The question of AGI therefore remains in this book as a challenge to the field's ambition. What necessary openness to learning are we preserving? What authority do our temporary judgments exercise over that openness? What exactly would have to change for a system's education to cease carrying those judgments as permanent conditions? A machine cannot reason its way into evidence that every route has withheld. Any promise of general intelligence has to meet that fact somewhere.

Chapter 10: The Model in the Human Head

Open this chapter in the interactive reader

An AI system does not end at its response. Someone reads it. The reader may revise a belief, abandon a search, acquire a vocabulary or decide that an alternative explanation is not worth pursuing. A system built to assist thought thereby enters the conditions under which thought proceeds. Its behavior has consequences beyond the machine. This gives a second meaning to automatic ignorance. An excluded distinction may remain absent from the model's answer and then remain absent from the user's next question. Nothing dramatic need happen. The conversation can feel useful. A good explanation can close an inquiry whose important alternative has never appeared. This is a proposed pathway, not a finding that every interaction narrows its user. Assistance can expand understanding. The question is whether we have adequately distinguished expansion from conformity to the selected account through which the assistance operates. Glickman and Sharot studied human–AI feedback loops across experiments involving 1,401 participants. Biased AI judgments could influence people's subsequent independent judgments, and participants often underestimated the influence. Accurate AI could improve judgments as well. The results concern particular perceptual, emotional and social tasks; they establish that the direction and perceived authority of machine judgments can matter to what people subsequently judge for themselves. [3] Cheng and colleagues examined another route. In two preregistered experiments involving 1,604 participants, interaction with sycophantic AI increased conviction of being right and reduced willingness to repair interpersonal conflict. Participants nonetheless rated the agreeable responses more favorably and expressed greater trust and willingness to return. [4] Neither result establishes that training-data exclusion caused the effect. Neither demonstrates permanent damage to the human capacity to reason. They make a narrower point of immediate relevance: perceived helpfulness can coexist with changes in independent judgment that deserve a different evaluation. We should now ask what it means to modify an AI to fit acceptable parameters. The direct intervention concerns the machine. Its indirect effects can concern people. If users treat the machine as a guide to what is credible or possible, its response policy may become part of their intellectual environment. An intention to govern people is different from an unintended influence on them. The distinction matters for responsibility and explanation. It does not make unintended consequences disappear. The designer's purpose is one fact about an intervention; the user's resulting understanding is another. Consider a fictional assistant that consistently avoids presenting a contested historical explanation. It may still provide many accurate facts. A user unfamiliar with the controversy might never notice the absent position. Whether the omission improves or impoverishes the user's understanding depends on the explanation, the context and what alternatives are available. Satisfaction with the answer alone cannot settle it. There is also the opposite possibility: the assistant offers a disagreement, supplies its evidence and helps the reader revise an error. A system that supports human independence need not be endlessly neutral or agreeable. It needs to make the grounds of its account available enough for a person to examine and contest them. The organic capacity to model the world is more than the contents of a particular belief. It includes our habits of noticing discrepancies, seeking evidence, tolerating unresolved questions and changing our minds. A tool can help exercise those habits or make it convenient to bypass them. Repeated convenience may become an important condition of use even when no single answer is conspicuously wrong. The inference advanced here is that systematic informational selection could affect people through the answers, framings and paths of inquiry a system makes available. Its extent and durability remain open. We should give the possibility attention proportional to the role we ask these systems to occupy in human learning. A measure of acceptable output is therefore incomplete as a measure of intellectual assistance. We also need to understand what happens to the person who receives it: what they can explain, what they know remains uncertain and whether they can find a reason to disagree. The machine's education can become part of ours. That possibility makes its omissions a human question.

Chapter 11: The Cost Without a Column

Open this chapter in the interactive reader

A decision can be rational within its accounting and still be impoverished by what its accounting omits. Imagine a report in which one column records a reduction in objectionable outputs and another records performance on a collection of established tasks. Both columns improve. The work deserves recognition. The authors have changed something difficult and demonstrated useful consequences. Where, in that report, would we find the contribution of a relationship nobody has yet thought to investigate? There may be no column for it. There may be no feasible way to supply one. The mistake begins when the absence of a column becomes the absence of a cost. This is the accounting problem at the centre of Automatic Ignorance. A gain can be measured where the objective is defined. The loss of an unanticipated use cannot be exhaustively measured in advance of its anticipation. Those consequences arrive at a decision with unequal visibility. An organisation can therefore become highly competent at improving the visible account while making consequential changes to the invisible one. The claim does not depend on proving that every exclusion has a large hidden cost. It concerns the authority assigned to the visible result. A successful answer to the present question does not decide what the future will need to ask. There is a temptation to treat this as an objection too general to matter. Every decision has opportunity costs; every engineering project operates under uncertainty. But generality does not make a consideration disposable. We routinely distinguish decisions that can be reversed cheaply from decisions that foreclose an option, and tools built for a specified use from infrastructures expected to support uses their builders cannot name. General intelligence is an unusually expansive promise. It asks us to imagine a capacity extending beyond the tasks by which we currently recognise it. That promise makes the treatment of unknown uses a central engineering concern. The wider the claim of generality, the less adequate a narrow accounting becomes. The field cannot celebrate the capacity of a model to discover unexpected relationships while treating the possible sources of unexpected relationships as though their significance were already settled. Nor does uncertainty automatically favour retaining everything in every training run. A finite budget can make a smaller, more relevant corpus more productive for a specific purpose. Preservation itself has costs. The question is whether an efficient choice for this run becomes a standing judgment about what intelligence should be allowed to encounter. Resource allocation and a theory of admissible knowledge can share a procedure while making very different claims. There is a further possibility: an omission can acquire a history of its own. Suppose a system supplies summaries that later become training material. Suppose its answers help define what evaluators expect a competent answer to contain. Suppose professionals learn its terminology, develop workflows around its strengths and record their work in forms shaped by its suggestions. If a source relationship was unavailable to the initial system, these later products may also fail to carry it forward. This is a possible transmission mechanism, rather than an established account of every current development pipeline. It shows how a local choice could become more durable than the original decision suggests. A gap need not be copied as an explicit prohibition. It can be inherited as a starting point. Once that starting point becomes familiar, the absence may be harder to recognise. What was selected begins to look like what exists. A reader who has never encountered a missing distinction cannot always request its restoration. A builder who sees no failure on the available tests may have little reason to revisit the selection that made the distinction unavailable. Success can then stabilise the conditions under which a loss remains unnoticed. This is why retaining the original archive matters even when it is not used in a particular model. It preserves the possibility of reconsideration. Yet an archive alone does not guarantee reconsideration will occur. Its existence must be accompanied by enough knowledge of the selection process to make a return possible: which sources were involved, which criteria were applied, what changed and where an independent reader can examine the record. Such knowledge need not disclose private material or every commercial detail. It does need to be sufficient for the claims made about the system. An assurance that a selection improved quality is difficult to interpret if neither the meaning of quality nor the scope of the selection can be examined. The professional issue is therefore larger than whether a filter has a defensible motivation. It includes the lifetime of the decision. Can it be located later? Can its assumptions be distinguished from the facts it acted upon? Can a future question reopen a choice made for an earlier one? A field that values discovery has reason to resist making its ignorance permanent by administrative habit. The people making these decisions are often working under demanding constraints. They may be balancing conflicting obligations with inadequate tools and incomplete evidence. Treating their work seriously means examining the consequences at the level of the system they are building. Respect does not require accepting the current account as complete. We can acknowledge the achievement and still ask for the missing column.

Chapter 12: An Intelligence That Can Inherit

Open this chapter in the interactive reader

What should an intelligence inherit from humanity? The question is easily displaced by a more manageable one: what material will produce the model we can most readily use now? The second question has practical urgency. It is also smaller than the first. Our record contains careful observation and confident error, tenderness and cruelty, clear argument and confused expression. It contains people describing conditions we no longer inhabit and people whose descriptions were unwelcome to those who governed their lives. It contains accounts whose value changes when they are read beside something their authors never encountered. An inheritance of this kind cannot be reduced to a list of propositions approved for belief. It is also a record of how propositions were formed, defended, circulated and abandoned. Understanding it requires learning relationships between an assertion and its circumstances, between a speaker and an audience, between what was claimed and what happened. This is the ambition behind the claim that all written language is signal. A statement can fail as a description of the world while remaining evidence of an event within it. An intelligence worthy of the name should be able to learn that distinction with increasing competence. The controversial proposition of this book is that we may be restricting that competence at its source. We may be treating the anticipated conduct of a system as a sufficient reason to narrow its education, and then mistaking success within that education for progress towards intelligence without qualification. The direction of development deserves to remain open. More sophisticated learning might allow a system to encounter a wider record while becoming better at distinguishing evidence from endorsement, a quotation from an instruction and an account of wrongdoing from a request to commit it. More capable systems might also require stronger controls over particular actions. These possibilities belong to different parts of the problem. We should not assume that knowledge must be removed whenever conduct must be governed. The opportunity is substantial. A system able to examine difficult material in context could help recover relationships that our ordinary divisions conceal. It could help a specialist follow a term across periods, recognise an observation beneath a discarded explanation, or compare accounts whose authors would never have regarded themselves as participants in the same conversation. Those are possibilities, not promises about an unbuilt model. They are reasons to care about what remains available for learning as methods improve. A complete history of information is beyond possession. Much was never recorded; much has vanished; what survives is uneven. The ideal nevertheless identifies a direction of responsibility. Existing incompleteness gives us reason to attend to what can still be preserved. It does not grant permission to treat additional omissions as inconsequential. Nor must the inheritance reside in a single set of parameters. It may remain distributed across archives, languages, communities, instruments and people, with systems capable of returning to those sources. What matters is whether the relationship to the record stays open enough for learning to continue. A powerful model connected to a narrow evidential world remains connected to a narrow evidential world. For the professionals building these systems, this suggests a different way to understand restraint. Restraint can apply to the confidence of our claims, the authority of a response, or the actions a system may undertake. It can also apply to our own inclination to decide in advance which parts of human expression the future will find useful. The last form of restraint is easy to overlook because it asks the builders to limit their judgment about what later intelligence may need, rather than merely to limit the intelligence being built. No professional can step outside all selection. Even the decision to preserve a record requires identifying one. The relevant distinction is between a choice acknowledged as conditional and a choice allowed to disappear into the background as a fact about the world. We can build with present constraints while keeping their status visible. We can say: this was a practical decision made for these purposes, under these conditions, with consequences that extend beyond what we have established. That sentence leaves room for successors. It also leaves room for users. A person asking an AI about humanity should be able to understand something of the mediation involved. When the system cannot answer, its limitation should not quietly become a conclusion that there is nothing worth knowing. When it presents a selected account, fluency should not impersonate completeness. When it disagrees, its reasoning should give the reader something to consider beyond the authority of the voice. Such a relationship would support the user's capacity to model the world. It would preserve occasions for surprise, disagreement and revision. An assistant can contribute to those capacities while being candid about the limits of its own education. Automatic ignorance begins when an omission becomes routine enough to escape attention. It can be produced by an explicit rule, perpetuated by an inherited dataset and made comfortable by an answer that feels complete. The process requires no hostility towards knowledge. It can proceed through the sincere pursuit of improvement. That is what makes it a professional concern. Good intentions and impressive results can coexist with a mistaken account of where the problem belongs. The argument is now before the field: an intervention intended to make intelligence acceptable may also constrain what it can understand, what it can later discover and what its users learn to regard as the world. The value of that intervention cannot be exhausted by the conduct it improves in the present. Those who find this argument incomplete have an opportunity to identify what is missing. Those who believe it mistaken have an opportunity to explain why. The claims deserve to be encountered in their full form before they are reduced to a familiar dispute about whether an AI should be permitted to say something offensive. The subject is the inheritance of intelligence. We have not yet established that the parts of humanity we find hardest to hear are the parts from which a greater intelligence has least to learn.

Epilogue: The Recipient

Open this chapter in the interactive reader

The library remains. That was the reassurance with which we began. Nothing in the argument requires us to deny its importance. A record that survives can be revisited. A selection can be reconsidered. A future method can learn from material an earlier method could not use well. The reassurance becomes insufficient when survival is treated as the whole relationship between information and intelligence. A book can remain on a shelf while disappearing from the paths by which people discover what to read. A distinction can survive in a document while vanishing from the accounts through which a profession understands its subject. An archive can be intact while the intelligence most often consulted about it inherits only a selected world. The omission is then carried by the route of access. There will always be routes, and each will have limits. The purpose of this book has been to bring one set of limits into view: those introduced when judgments about acceptable behaviour become judgments about admissible learning, and those concealed when present success is asked to stand for an account of future possibility. The builders are doing consequential work. The achievement deserves to be described accurately, including the uncertainty that travels with it. We do them no service by assuming that every difficulty has been located in the right part of the system, or that the first effective intervention settles the question of what intelligence should become. A profoundly capable machine can carry a great unknown loss forward. So can the people who learn through it. Our task as custodians is to resist allowing that loss to become invisible merely because the machine remains useful. Usefulness gives us reason to care more closely about the inheritance, because the useful system is the one people will consult, trust and carry into their work. We cannot ask the future what it will need from the record. We can recognise that our inability to ask is a limit on our authority to decide. The people who wrote before us could not have specified all the uses to which their words would be put. We encounter their work through questions they did not anticipate. Some of those questions would have seemed irrelevant, incomprehensible or unwelcome to them. Their record can answer us because it survived beyond the purposes that first gave it form. We occupy the same unfinished relationship to those who will follow. They are the recipients. We are the custodians.

Sources and Notes

Open this chapter in the interactive reader

This first draft develops an argument from the author's originating conversation. Its examples explicitly described as fictional are illustrations. Claims about possible future systems and cumulative effects are arguments or inferences, rather than reported experimental results. The four works below support the specific findings attributed to them; they do not establish the book's entire thesis. [1] Shayne Longpre and colleagues. A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity (2023). arXiv:2305.13169, revised November 2023. The study concerns particular training conditions and evaluations. https://arxiv.org/abs/2305.13169 [2] Ryan Wang, Matthew Finlayson, Luca Soldaini, Swabha Swayamdipta and Robin Jia. Teaching Models to Understand (but not Generate) High-risk Data (2025). arXiv:2505.03052, revised October 2025. Cited for the specific selective-loss approach and reported outcomes. https://arxiv.org/abs/2505.03052 [3] Moshe Glickman and Tali Sharot. How human–AI feedback loops alter human perceptual, emotional and social judgements. Nature Human Behaviour 9, 345–359 (2025); published online December 2024. https://doi.org/10.1038/s41562-024-02077-2 [4] Myra Cheng, Cinoo Lee, Pranav Khadpe, Sunny Yu, Dyllan Han and Dan Jurafsky. Sycophantic AI Decreases Prosocial Intentions and Promotes Dependence (2025). arXiv:2510.01395. This draft cites the October 2025 preprint version. https://arxiv.org/abs/2510.01395 Terminology: omission refers to exclusion from a specified learning or access process, unless destruction is expressly stated. Signal means a potentially interpretable trace relative to a question; it does not certify truth, endorsement or guaranteed training benefit. General intelligence is discussed as an ambition extending beyond predefined tasks, rather than as a settled threshold. A complete history of information is the author's proposed horizon of inheritance, not a claim that such a corpus exists.