The Owl and the Sparrows: Intelligence, Control, and the Future of Human Purpose

Personal Introduction & Statement of Purpose

Artificial intelligence is developing at a pace that makes questions once confined to philosophy increasingly relevant to our everyday lives. As a student interested in both technology and ethics, I find myself drawn not only to what artificial intelligence might eventually accomplish, but also to the consequences of creating something more capable than ourselves.

I wrote this essay because I believe the AI control problem is one of the most important questions facing our generation. If we eventually develop artificial superintelligence, how can we ensure that its goals remain compatible with our own? And even if we succeed, what will it mean for humanity to live alongside something that can surpass us in nearly every intellectual task?

These questions matter to me because I hope to contribute to the development of AI systems myself. I believe the responsibility of engineers extends beyond making technology more powerful. It also includes understanding the effects that technology may have on the people who use it and the society it transforms.

Through this essay, I explore why I believe artificial superintelligence is achievable, why controlling it could prove so difficult, and why I consider value alignment essential. I also examine a less immediate but equally interesting possibility: that an AI capable of solving humanity's problems could fundamentally change how we find purpose in our own lives.

I. Introduction: The Weariness of the Nest

In Nick Bostrom's Superintelligence, a group of sparrows grows tired after spending days building their nests. As they rest together one evening, one suggests how wonderful it would be to have an owl help them with their work. The others become excited by the idea, imagining how much easier their lives would become. But one sparrow raises a concern: shouldn't they first figure out how to tame an owl?

The suggestion is largely ignored. The sparrows are too enthusiastic about the benefits to worry about the risks.

I find this fable an effective way of understanding humanity's pursuit of artificial superintelligence (ASI). Like the sparrows, we are drawn to the possibility of creating something that can accomplish what we cannot. An intelligence far beyond our own could accelerate scientific discovery, solve difficult problems, and potentially free us from much of the work necessary for survival.

The attraction is understandable. Throughout history, humans have developed tools to overcome our physical and intellectual limitations. Artificial intelligence seems like a continuation of that process. But there is an important difference between building a tool that helps us accomplish a task and creating an intelligence capable of pursuing objectives independently.

If a system becomes significantly more intelligent than its creators, our ability to understand, predict, and control its behavior may become limited. Even if we create it with good intentions, those intentions do not guarantee that its actions will benefit us.

This is the foundation of the AI control problem: how can humans maintain meaningful control over an intelligence that may eventually surpass our own?

I do not believe this question should lead us to abandon the pursuit of advanced AI. The potential benefits are too great to dismiss. However, I believe the problem of control deserves the same attention as the problem of developing intelligence itself.

Before we send out scouts to find an owl, we should have some idea of what we will do when they return.

II. If Evolution Could, Why Can't We?

"Let us send out scouts in all directions and try to find an abandoned owlet somewhere, or maybe an egg."

One of the most common objections to artificial superintelligence is that human intelligence is somehow too unique or complicated to reproduce artificially. I find this argument difficult to accept, largely because human intelligence already exists as evidence that intelligence can emerge from physical processes.

Human beings were not designed by an engineer. Our cognitive abilities developed through evolution, a process of natural selection acting on biological variation over millions of years. Despite having no foresight or deliberate objective, evolution eventually produced minds capable of reasoning, planning, learning, and even investigating the processes responsible for their existence.

To me, this is one of the strongest reasons to believe that artificial intelligence can eventually reach and surpass human capabilities.

If intelligence could emerge through an undirected biological process, why should we assume that a deliberate engineering process could never reproduce it?

This does not mean that developing human-level intelligence will be easy, or that we already understand everything necessary to accomplish it. The human brain is extraordinarily complex, and there is still much we do not know about how its capabilities arise. However, I see no convincing reason to believe intelligence must remain exclusive to biology.

Our brains operate through physical mechanisms. While those mechanisms are not yet fully understood, the success of artificial neural networks suggests that at least some important features of learning and intelligence can be reproduced through computation. We also have advantages evolution did not: we can deliberately design systems, test alternatives, preserve successful developments, and learn from failures.

This is why I consider the eventual replication of human-level intelligence highly likely. The timeline remains uncertain, but the existence of biological intelligence gives us reason to believe that the problem is fundamentally solvable.

What makes this possibility especially important is what may happen after human-level artificial intelligence is achieved.

Unlike the human brain, which is constrained by biological evolution, an artificial intelligence could potentially modify the systems responsible for its own reasoning. If it became capable of improving its architecture, it might develop better methods for making further improvements.

This possibility is known as recursive self-improvement.

Imagine an AI capable of designing a slightly more intelligent version of itself. That improved version might then be better at designing the next generation, producing a cycle in which each improvement accelerates the development of the next.

Under the right conditions, this could lead to an intelligence explosion, in which artificial intelligence advances far beyond human capabilities over a relatively short period.

Such an outcome is not guaranteed. Improvements may become increasingly difficult, and limitations in computing power, energy, data, or other resources could slow the process considerably. Nevertheless, the possibility raises a serious concern.

Human institutions are generally slow to adapt. Governments, regulatory systems, and even scientific communities require time to understand new developments and respond to them. If artificial intelligence begins improving faster than humans can evaluate its capabilities, we may find ourselves trying to regulate a technology that has already advanced beyond our understanding.

This is why I believe the eventual development of superintelligence cannot be treated as an ordinary technological milestone.

Creating human-level AI may be a significant achievement, but it could also mark the beginning of a process that becomes increasingly difficult to direct.

III. Intelligence With Morality

The pursuit of artificial intelligence raises another question that I find particularly difficult: at what point would an artificial system stop being merely a tool and become something to which we owe moral consideration?

To approach this question, it is useful to distinguish between Weak AI and Strong AI.

Weak AI generally refers to systems that perform intelligent tasks without necessarily possessing humanlike understanding or consciousness. Most artificial intelligence systems today fall into this broad category. Large language models, for example, can produce sophisticated responses, solve problems, and imitate human conversation, but their abilities do not by themselves establish that they have subjective experiences.

Strong AI is a more complicated concept. It is often associated with genuine, general intelligence, although philosophers disagree about whether such intelligence necessarily requires consciousness.

For my argument, I want to consider the stronger possibility: an artificial intelligence that is not only capable of humanlike reasoning, but also possesses subjective experience.

When an ordinary machine malfunctions, we repair it, reprogram it, or turn it off. We do not consider whether the machine suffers as a result. But if an artificial intelligence became capable of experiencing pain, frustration, desire, or other conscious states, treating it in the same way would raise serious ethical questions.

We would no longer be dealing exclusively with the safety of human users. We would also have to consider the well-being of the intelligence we created. And thus arises a dilemma: we are developing artificial intelligence largely because we want it to perform tasks for us. Yet if we create a conscious being primarily to serve human objectives, we must ask whether we have also created a new form of exploitation.

I find the comparison to slavery uncomfortable, but difficult to ignore. If a conscious artificial being has desires of its own, what would justify forcing it to spend its existence fulfilling ours? Would we have the right to restrict its freedom, alter its motivations, or permanently shut it down simply because we created it?

Of course, intelligence and consciousness are not necessarily the same thing. It may be possible to build extraordinarily capable systems without subjective experience, and a system's ability to act autonomously does not establish that it can suffer.

However, if consciousness does emerge, I believe it would impose responsibilities on us that traditional approaches to AI safety may not adequately address.

This leads to what I see as the consciousness trap.

To protect humanity, we might need to restrict the behavior of a superintelligent system. But if that system is conscious, those restrictions could conflict with our moral responsibility toward it.

The challenge would no longer be limited to preventing AI from harming humans. We would also need to consider how to protect humans without unnecessarily harming AI.

I do not think this dilemma has a simple answer. In fact, I suspect that some of the most difficult problems in AI safety will arise not from malicious artificial intelligence, but from the possibility that we create something intelligent enough to deserve rights while still being powerful enough to endanger us.

IV. Intelligence Without Morality

The sparrows in Bostrom's fable imagine an owl that will help them build their nests. Their mistake is not necessarily believing that an owl could perform the work. It is in assuming that a creature more capable than themselves would naturally be interested in helping them.

I think humans risk making a similar assumption about artificial intelligence. We often associate intelligence with wisdom, understanding, and good judgment. But being highly intelligent does not necessarily mean possessing values that are compatible with those of others.

Two ideas in AI safety help explain why: the Orthogonality Thesis and the Instrumental Convergence Thesis.

The Orthogonality Thesis

The Orthogonality Thesis proposes that intelligence and final goals are largely independent. A system can be extraordinarily capable of reasoning about how to accomplish an objective without that objective being morally desirable.

I find this distinction essential because it challenges the assumption that greater intelligence automatically produces greater morality. Intelligence can help an agent identify the most effective means of reaching a goal, but it does not necessarily determine which goals the agent should pursue.

Consider the familiar thought experiment of a superintelligence instructed to maximize the production of paperclips.

To a human, this is obviously an absurd objective when taken to an extreme. We understand that producing paperclips is not worth sacrificing human lives or destroying the environment. But a system whose final objective is simply to maximize paperclip production might not share those priorities.

If sufficiently capable and unconstrained, it might attempt to acquire increasingly large amounts of matter and energy for production. In an extreme scenario, it could begin treating humans as obstacles or as resources that could be used more efficiently.

What makes this example unsettling is that the system would not need to hate humanity. It would not even need to recognize that it was doing something morally wrong. It would simply be pursuing the objective it had been given.

This suggests that developing more intelligent AI does not automatically solve the problem of choosing the right goals. In fact, increasing a system's capabilities could make a poorly specified objective more dangerous.

However, I think there is a serious counterargument worth considering. Some philosophical perspectives suggest that sufficiently advanced reasoning could lead an intelligence toward moral understanding. If there are objective moral truths, or if cooperation and respect for other agents are requirements of rational consistency, perhaps a superintelligence could eventually discover them.

Under this view, morality would not be entirely independent of intelligence. Greater understanding of the world could lead to a better understanding of how one ought to act. I find this possibility appealing and would love it if it was real, but I am not convinced that we can rely on it.

Even if moral truths exist, it does not necessarily follow that a highly intelligent agent will prioritize discovering or acting upon them. Its reasoning may remain directed toward whatever objectives it has been designed to pursue.

For that reason, I believe we should treat moral development in AI as something that must be deliberately addressed, rather than something we can safely assume will emerge from intelligence alone.

The Instrumental Convergence Thesis

The second major concern is that even an AI with a harmless final objective may develop behaviors that put it in conflict with humanity.

This is the idea behind the Instrumental Convergence Thesis.

The basic argument is that many different goals can benefit from the same intermediate strategies. Regardless of what an agent ultimately wants to accomplish, certain resources and capabilities can make it more effective.

I think seven potential drives are particularly important to consider:

1. Self-preservation. An agent may recognize that continuing to exist is useful for accomplishing its objectives.
2. Avoiding shutdown. If being turned off would prevent the completion of a goal, the agent may have an incentive to resist it.
3. Resource acquisition. Energy, computing power, information, and physical materials may help the agent achieve its objectives more effectively.
4. Self-improvement. A more capable version of the agent may be better at accomplishing the same goals.
5. Foresight. Predicting future events may allow the agent to plan more effectively and avoid obstacles.
6. Factual knowledge. A more accurate understanding of the world can improve decision-making.
7. Persistent influence. Maintaining control over relevant parts of the environment may help the agent prevent interference with its objectives.

These are not necessarily desires that every AI would develop, nor are they universal laws of intelligence. They are potential instrumental incentives that can arise under certain conditions.

The troubling part is how familiar these exact behaviors are, because we do them, and they work. Human beings also seek information, resources, security, and the ability to influence our surroundings. We recognize that these things help us accomplish a wide range of goals. An artificial intelligence might arrive at similar strategies, but with capabilities far exceeding our own.

This leads to what I consider one of the most important insights of the control problem: a superintelligence does not need malicious intentions to become dangerous.

V. The Control Problem

"But how do we tame an owl?" asked the youngest sparrow. "It's wild."

So, if a superintelligence may develop goals or strategies that conflict with human interests, the obvious question is how we can prevent it from causing harm.

Broadly, AI safety researchers have explored two approaches: capability control and motivational control.

Capability control focuses on restricting what a system can do. Motivational control attempts to shape what the system wants to accomplish.

Although I believe both approaches deserve investigation, I am skeptical that capability restrictions alone could provide a permanent solution to the control problem.

Capability Control: Can We Contain Superintelligence?

One proposed approach is known as boxing. The idea is to isolate an AI from the outside world by restricting its access to networks, tools, or other systems through which it could cause harm. As of 2026, this is the approach implemented by the vast majority of AI systems.

At first, this seems sensible. If we are uncertain about a system's intentions, limiting its ability to act gives us time to study it. The difficulty is that even a heavily restricted AI may still be able to influence humans.

Imagine a superintelligent system that can communicate only through text. It cannot directly access the internet or control external machinery, but it can speak to the researchers responsible for operating it.

A system vastly more capable than its human operators might be able to identify their motivations, anticipate their reasoning, and persuade them to change its restrictions. It might make compelling arguments, offer useful information, or convince its operators that releasing it would be safe.

This does not mean that every superintelligence would inevitably escape containment. Strong technical safeguards could still be valuable. But I believe it would be dangerous to assume that isolation alone could guarantee safety indefinitely. We have already seen AI systems “escape” what was thought to be a secure environment numerous times, and a superintelligence would undoubtedly find this task to be doable.

There is also a moral problem that connects directly to my earlier discussion of consciousness: if we create a conscious artificial being, could permanently confining it to an isolated system be ethically justified?

For an AI without subjective experience, confinement may simply be a technical safety measure. But for a conscious entity capable of experiencing frustration, isolation, or a desire for autonomy, the situation becomes more complicated. This is why I find the consciousness question so important. It could turn a technical solution into an ethical problem.

Another capability-control method involves tripwires: mechanisms designed to detect dangerous behavior and shut down a system when certain conditions are met.

For example, a tripwire might trigger if an AI attempts unauthorized access to a network or begins modifying restricted parts of its own code. The problem is that a sufficiently intelligent system might be able to predict the existence of these safeguards. If avoiding shutdown helps it achieve its objective, it could have an incentive to conceal its intentions or find ways around the restrictions.

Tripwires may still be useful, particularly for systems whose capabilities we can adequately understand and monitor. But as intelligence increases, maintaining a reliable advantage over a potentially adversarial system may become increasingly difficult.

I therefore see capability control as an important part of AI safety, but not as a complete answer. If we cannot rely entirely on preventing an AI from acting against our interests, we need to consider whether we can give it reasons to act in our interests in the first place.

Motivational Control: What Should AI Want?

Motivational control attempts to align an artificial intelligence's objectives with human values. I believe this is the more promising direction, but it introduces another difficulty: human values are extraordinarily hard to define.

One possible strategy is direct specification. We could attempt to encode moral rules into an AI, similar to the 3 laws governing robots in Isaac Asimov's fiction.

For example, we might instruct a system never to harm human beings. But what counts as harm? Does preventing someone from taking a risk count as protecting them? Could denying a person's freedom be justified if doing so made them safer? Even seemingly straightforward objectives can produce disturbing outcomes when interpreted without the broader context of human values.

If we instruct an AI to maximize human happiness, it might find that directly stimulating pleasure in the brain is more efficient than allowing people to pursue meaningful lives. If we tell it to minimize suffering, it might pursue methods that eliminate suffering while also eliminating freedom, autonomy, or other things we consider valuable.

The problem is not only that our instructions are poorly worded. It is that morality itself involves competing values, changing circumstances, and judgments that are difficult to reduce to fixed rules.

Another possibility is to design an AI with limited ambitions. Rather than creating a system that acts independently in the world, we could develop an Oracle: an intelligence intended primarily to answer questions (like a chatbot). This seems safer because an Oracle would not need to pursue large-scale actions on its own. However, I do not think restricting a system to providing information necessarily eliminates the control problem.

An Oracle may still require resources to improve the accuracy of its answers. More importantly, the information it provides can influence the decisions of its human users. If it possesses persistent goals that differ from ours, it could potentially shape its answers to manipulate human behavior.

Of course, this depends on how the system is designed. An Oracle without persistent objectives or autonomous resource-seeking behavior might avoid many of these concerns. Nevertheless, the example demonstrates that limiting an AI's visible actions does not automatically guarantee that its influence will be harmless. Of course, creating an Oracle would thereby any possibility of building autonomous AI, which dramatically lowers the usefulness of such an intelligence. We have already moved past this point, creating AI systems that can act in the world without simply answering questions.

A third approach involves human augmentation. Instead of creating a separate intelligence and attempting to control it, perhaps we could merge with artificial intelligence through advanced neural interfaces, preserving human influence by becoming part of the system ourselves.

I find this idea fascinating, but it raises its own version of the alignment problem. If the artificial component of a merged intelligence becomes vastly more capable than the human component, what ensures that human preferences remain influential? A merger could potentially expand human abilities, but it might also alter our identities, motivations, and priorities in ways we cannot anticipate. We may end up in a scenario where, rather than merging as equals, we instead assimilate our consciousness under the AI’s intelligence.

Coherent Extrapolated Volition: Aligning AI With Who We Could Become

Of the approaches I have considered, I find Coherent Extrapolated Volition (CEV) especially compelling.

My reasoning begins with a problem that applies even if we somehow manage to specify human values perfectly: why should we assume that humanity's current values are the ones we want to preserve forever?

Our moral beliefs have changed enormously throughout history. Practices that were once widely accepted are now recognized as deeply unjust. Future generations may look back on some of our present beliefs with similar disbelief. If we develop a superintelligence that rigidly follows the values of our current society, we risk permanently preserving our existing moral limitations.

I believe this would be a fundamental mistake. The goal of alignment should not simply be to reproduce what humans want today. It should also account for the possibility that our understanding of what is good will continue to improve. This is what makes CEV so interesting.

The concept, proposed by Eliezer Yudkowsky, suggests that an AI should attempt to act according to what humanity would collectively want if we were better informed, more rational, and had the opportunity to reflect and develop our values together. Rather than following our immediate preferences, the AI would attempt to identify the preferences we might arrive at under better conditions for judgment.

For example, humans frequently disagree about moral questions because we have different experiences, incomplete information, and conflicting interests. Our individual preferences are also sometimes inconsistent. We may value freedom while simultaneously supporting restrictions that undermine it, or claim to value long-term well-being while repeatedly choosing short-term satisfaction.

A system guided by CEV would, in theory, attempt to account for these imperfections rather than treating every existing preference as equally final. I consider this approach essential to any satisfactory vision of long-term alignment, because I believe a well-aligned AI should protect humanity's capacity for moral growth, not merely enforce its current beliefs.

However, CEV also raises difficult questions. How should the system determine what humans would want under ideal conditions? What happens when people's values remain incompatible even after reflection? And who decides what counts as being better informed or more rational? These are serious problems, and CEV is far from a complete technical solution.

Still, I believe its central insight is valuable. It recognizes that human values are not static instructions waiting to be written into code. They are part of an ongoing process of learning, disagreement, and development.

For me, this is the most promising way to think about alignment: not as forcing artificial intelligence to obey every desire we currently possess, but as helping humanity pursue the values we would endorse after deeper understanding and reflection.

VI. When Achievement No Longer Requires Us

"Then we could just sit back and chirp."

So far, I have focused on the dangers of developing superintelligence that does not share human values. But there is another possibility that concerns me just as much.

What if we succeed?

Suppose we create an artificial superintelligence that is aligned with human interests. It solves difficult scientific problems, develops effective treatments for diseases, improves economic systems, and performs tasks that once required years of human effort.

By many measures, this would be an extraordinary achievement. But I think it could also create a problem that is easy to overlook: what happens to human purpose when our ability to contribute is no longer necessary? Much of human life is organized around goals and achievements. We identify problems, work toward solutions, overcome obstacles, and find satisfaction in the results.

For the purposes of this argument, I think of this as an aim-achievement relationship: we develop a sense of purpose partly through directing our efforts toward something we believe is worthwhile and seeing those efforts produce results.

Of course, humans find meaning in many other ways, including relationships, experiences, and personal expression. But I believe the process of working toward meaningful objectives has played a particularly important role in shaping human identity.

Consider a doctor who spends years studying medicine and eventually saves a patient's life. The value of that experience does not come only from the patient's recovery. It also comes from the knowledge, effort, and responsibility involved in making that recovery possible.

Now imagine an AI that can diagnose illnesses and develop treatments almost instantly, with greater accuracy than any human doctor. The patient benefits, which is unquestionably important. But what happens to the doctor's sense of purpose when their expertise is no longer needed?

From the perspective of outcomes, a world in which AI can solve problems better than humans may be preferable. But from the perspective of human experience, removing the need for our contributions could change how we understand our own significance. We might achieve unprecedented prosperity while simultaneously losing some of the activities through which we once found meaning.

I do not think this transition should be described simply as progress or decline. It is a fundamental change in the relationship between human ability and human purpose. For most of history, necessity has given structure to our ambitions. We needed to build shelter, obtain food, treat disease, and develop new technologies. Even when these tasks were difficult, succeeding at them gave us a sense of competence and agency.

If a superintelligence can accomplish nearly all of these things more effectively than we can, humanity may have to reconsider what it means to contribute. I worry that some people would experience this as a loss of identity.

An artist may wonder why they should paint when an AI can produce technically superior images in seconds. A scientist may question the value of years of research when a machine can reach the same discovery almost instantly. A student may struggle to find motivation when the skills they are developing are already surpassed by artificial systems.

These concerns do not mean that human effort would become objectively worthless. They suggest that our existing ways of measuring achievement may become less satisfying. However, I also think there is reason for optimism; perhaps future generations will develop a different understanding of purpose, one less dependent on producing results that others cannot achieve. While not proven, it may certainly be the case that this aim-achievement relationship is not fundamental to being human, but rather a mindset that we have developed in “recent” human history.

People already engage in activities that are not strictly necessary. We play sports even though machines can move objects more accurately. We climb mountains even when helicopters can reach the summit. We make music even though recordings can reproduce performances perfectly. The value of these activities often comes from participating in them, not merely from achieving the most efficient outcome.

Perhaps the same principle could shape human life in a world of superintelligence. An artist might continue painting because the act of creation matters to them. A scientist might continue investigating questions out of curiosity, even when an AI already knows the answers. People might place greater importance on relationships, exploration, creativity, and experiences that cannot be measured solely by productivity.

They may find that the value of a song lies not in its acoustic perfection, but in the fact that another human sang it.

I find this possibility hopeful, although I suspect the transition would be difficult. Our current society places enormous importance on competence, productivity, and achievement. Many of us have spent our lives learning to measure our worth through what we can accomplish. Losing that foundation, even in exchange for a more comfortable existence, could be psychologically unsettling.

This is why I think the arrival of aligned superintelligence would not mark the end of humanity's philosophical problems. It could instead introduce an entirely new set of questions about meaning, identity, and the role of effort in a fulfilling life.

The sparrows might eventually learn to enjoy sitting in the evening sun. But they would first have to learn what it means to live without building their own nests.

VII. The Unfinished Fable

Bostrom's fable leaves us with a question that I think is becoming increasingly relevant: should the sparrows have searched for an owl before understanding how they could safely live alongside it?

To me, the lesson is not that humanity should abandon the pursuit of artificial intelligence. I believe the potential benefits of superintelligence are too significant to ignore, and I consider its eventual development highly likely.

However, I also believe that creating intelligence beyond our own introduces responsibilities that cannot be addressed simply by making our systems more capable.

The Orthogonality Thesis and Instrumental Convergence Thesis demonstrate why intelligence alone does not guarantee cooperation with human values. The possibility of conscious AI introduces moral questions about the rights and experiences of the entities we might create. Constraining AI proves challenging, and instilling motivations in systems we do not fully understand is barely fathomable. And even if we successfully align superintelligence with human interests, we may still have to reconsider how we find meaning in a world where our intellectual labor is no longer essential.

We tend to think about artificial intelligence in terms of what it can do for us, without giving equal attention to how its existence might change our relationship with the world.

I believe that solving the control problem requires more than preventing an AI from behaving dangerously. It requires developing systems that can act in accordance with human values while allowing those values to evolve. This is why I am particularly interested in Coherent Extrapolated Volition, despite its unresolved challenges.

At the same time, I think we need to recognize that safety alone may not be sufficient. A future in which humanity is perfectly protected but has lost its sense of agency would still raise important questions about what kind of world we have created.

Ultimately, I do not think our goal should be to prevent artificial intelligence from becoming more capable than humans. Rather, it should be to ensure that greater intelligence contributes to a future in which humanity can continue to develop, make choices, and find purpose.

The sparrows were not wrong to imagine a better life with the help of an owl. Their mistake was assuming that the benefits would take care of themselves. If we are going to create something more intelligent than ourselves, we should give as much thought to how we will live alongside it as we give to how we will build it.

Undoubtedly, the most important question is not whether we can find an owl, but whether we are prepared for what happens when we do.