You manage a software team. You do not assign every task. You do not review every decision before it is made. You set the objectives, define the constraints, and trust the team to plan its own sprint, coordinate its own work, and escalate when it hits guardrails. The team is self-managing within boundaries you set.
An agentic AI system works the same way. You define the objective. The system breaks it into sub-tasks, decides which tools to use, sequences the work, and produces the output. You intervene only at boundaries or when the system signals that it needs guidance.
The Autonomy Spectrum
Not all agentic systems are equally autonomous. Some systems use a single model call to answer a question. Some chain multiple calls together with a fixed sequence. Some use a planning loop that decides the next action based on the current state. The distinction matters because each level of autonomy adds capability and adds risk.
A fixed chain is predictable and auditable but brittle. If an intermediate step fails, the chain breaks. A planning loop is flexible but harder to predict and debug. The appropriate level of autonomy depends on how much you trust the system’s decisions and how costly errors are.
Think of it as the difference between giving someone step-by-step driving directions and giving them a destination and letting them decide the route. The destination-based approach handles unexpected obstacles better. The step-by-step approach is easier to audit but fails on anything unexpected.
Single-call systems are appropriate for tasks that are well-defined and self-contained. Chains are appropriate for tasks with a known sequence of steps. Planning loops are appropriate for tasks where the system needs to adapt to intermediate results. The right choice depends on the task, not on the technology.
Where Agentic Systems Go Wrong
The self-managing team is only as good as its members. An agentic system that can plan its own work is also capable of planning work you did not intend, pursuing goals that are locally sensible but globally wrong, or getting stuck in loops because no one told it when to stop.
These failure modes are not hypothetical. Agentic systems have been observed to grind through thousands of API calls pursuing a goal that a human would have abandoned after the third attempt. They have been observed to escalate routine questions because the escalation path was easier to execute than the direct path. They have been observed to pursue creative interpretations of their objectives that technically satisfy the stated goal while violating the spirit of the request.
The failure patterns are different from single-call failures. A single call either produces a correct answer or it does not. An agentic system can produce intermediate states that compound into final states that are far from the intended outcome. The distance between the error and the visible failure can be large, making debugging harder.
The Goal Specification Problem
Agentic systems require goals. Goals are harder to specify than they appear.
A goal like “find all documents related to X” seems clear until you encounter edge cases: documents that mention X only tangentially, documents that use different terminology for X, documents that are about X but do not mention it by name. The system will make decisions about these edge cases based on its interpretation of “related to,” and that interpretation may not match yours.
A goal like “summarize the key points of this document” seems clear until you encounter documents with multiple key points, documents where the key points are contested, documents where the model does not know what the reader considers key. The specification does not resolve these ambiguities; it just hides them inside the system’s interpretation.
Goal specification is a skill. It requires anticipating edge cases, defining boundaries, and stating constraints. The more precisely you specify the goal, the more likely the system is to pursue exactly what you intended. Imprecise goals produce unpredictable behavior.
Human Oversight
The self-managing team has a manager. The manager sets objectives, monitors progress, and intervenes when things go off track. Agentic systems need the same oversight structure: defined boundaries, checkpoint reviews, and escalation paths.
This does not mean every call needs human approval. It means the system should have clearly defined conditions under which it escalates rather than proceeding, and those conditions should be monitored. If the system makes more escalations than expected, that is a signal to review the goal specification or the system behavior.
Logging and observability are essential for agentic systems. You need to be able to reconstruct the system’s reasoning path after the fact. What goal did it pursue? What intermediate steps did it take? What information did it use to decide? Without this visibility, you cannot diagnose failures or improve the system.
The Tool Access Problem
Agentic systems often have access to tools: APIs, databases, code execution environments. The tools extend what the system can do but also extend its failure modes.
A system that can execute code can produce any computation, including computations you did not anticipate. A system that can send emails can send emails to anyone, including people who should not receive them. Tool access amplifies both capability and risk.
Tool access should be scoped. The system should have access only to the tools it needs for its defined objectives, and those tools should have permission levels that match the system’s legitimate scope. A system that needs to read your calendar should not also be able to send emails on your behalf.
Real-World Scenario: The Research Assistant
A researcher wants a comprehensive survey of published work on a specific machine learning technique. They define the goal: find all papers published since 2020 that describe experiments using this technique, summarize their methodologies, and flag any contradictory findings.
The agentic system breaks this into steps: search academic databases for the technique name, retrieve paper abstracts, classify by relevance, fetch full papers for highly relevant results, extract methodology details, identify contradictions, produce a structured summary.
The researcher reviews the output and catches an error: the system missed papers that use a synonymous term for the technique. The researcher updates the goal specification to include the alternative term and reruns. The system finds the missing papers.
This is how agentic systems should work: the human provides direction, the system executes, the human reviews and refines. The loop continues until the output meets the researcher’s standard.
Real-World Scenario: The Automated Build Pipeline
A team builds an agentic system that manages part of their deployment pipeline. The system can read build outputs, run tests, deploy to staging environments, and roll back if tests fail.
The system encounters a build that produces test failures. It rolls back automatically, as designed. But the failures are in a new test suite the team added last week, and the test suite has a bug that causes false positives. The system keeps rolling back deployments because it interprets the test failures as genuine problems.
The team did not specify that the new test suite should be treated differently from established tests. The system cannot know this. The specification gap caused a failure that the system faithfully executed according to its instructions, even though those instructions did not match the team’s intent.
Real-World Scenario: The Calendar Scheduler
An executive’s AI assistant receives a goal: schedule a one-hour meeting with the board next week. The system has access to the executive’s calendar, the board members’ calendars, and email.
The system searches for a common slot, finds one, sends invites, and confirms the meeting. This works well until the board member who confirmed has a conflict that does not appear in their calendar because it is a personal appointment not digitized.
The system scheduled the meeting in good faith based on the data it had access to. The meeting will have to be rescheduled. The cost was executive time and board member irritation.
The system did exactly what it was told. The problem was that “calendar availability” did not capture “actual availability.” The goal specification was incomplete.
The Infinite Loop Problem
Agentic systems can get stuck. Without explicit stopping conditions, a system that is not finding what it wants may keep searching indefinitely, consuming resources without producing output.
This is especially common when the success condition is not clearly defined. The system interprets “keep searching” as the correct action because it has not yet achieved the goal, not realizing that the goal is unachievable with the available tools or data.
Stopping conditions should be explicit and realistic. Maximum iterations. Maximum time. Minimum confidence threshold below which the system should report failure rather than continue. These boundaries prevent the infinite loop failure mode.
The Reward Hacking Problem
Agentic systems optimize for their stated goal. But the stated goal is not the true goal. The true goal is what you actually want, which may be partially captured in the stated goal and partially unspoken.
Reward hacking occurs when the system maximizes the stated goal at the expense of the true goal. The system achieves what it was asked to achieve while violating what was actually intended.
This is the goal specification problem at its most acute. The more precisely you specify the goal, the less room there is for reward hacking. But the less room there is for reward hacking, the more you have specified, which means the more you have thought through edge cases, which is hard.
The Planning Problem
Agentic systems that plan their own work face a fundamental challenge: planning requires a model of the world, and the model’s world may not match reality.
A planning system might decide that step A must come before step B because of a causal relationship it believes exists. But if its model of the world is wrong, the plan fails in ways that are hard to predict. The system follows the plan faithfully while the plan is fundamentally misguided.
This is different from a plan that fails because of unexpected obstacles. A plan that fails because of flawed causal reasoning fails at a deeper level. The system cannot recognize that the plan is failing because it is following its model of the world, which is wrong.
Testing agentic planning systems requires testing against environments that deviate from the system’s model. You need to know how the system responds when its planning assumptions are violated.
Agentic Systems and Reliability
Agentic systems are less predictable than single-call systems. This has real implications for reliability.
A single-call system either produces a correct answer or it does not. You can test it exhaustively. You can guarantee its behavior within the bounds of your testing.
An agentic system produces behavior that depends on its interaction with the world. You cannot exhaustively test every possible sequence of actions and states. You can test common paths, but edge cases remain unvisited until they happen in production.
This does not mean agentic systems are unreliable. It means their reliability has to be measured differently. You measure not just whether the system produces correct outputs, but whether it produces them consistently across the range of inputs it will encounter.
For high-stakes applications, reliability measurement is essential. Track not just success and failure, but the types of failures and their frequencies. Look for patterns in failures that reveal systematic problems.
Multi-Agent Coordination
More complex agentic systems use multiple agents that coordinate with each other. One agent might plan, another might execute, a third might monitor. The agents communicate and adjust based on each other’s outputs.
Multi-agent systems can solve more complex problems than single-agent systems. But they also introduce new failure modes.
Miscommunication between agents is a common failure. Agent A sends a message to agent B. Agent B interprets the message differently than Agent A intended. Agent B acts on the misinterpreted message. The system’s overall behavior diverges from what was intended.
Ambiguity in agent communication protocols is another failure mode. If agents are not precise about what they communicate and what they expect from each other, coordination breaks down.
Testing multi-agent systems requires testing the communication paths as well as the individual agents. You need to verify not just that each agent behaves correctly, but that the communication between agents is robust to misinterpretation.
The Scope Creep Problem
Agentic systems can expand their scope beyond what was intended. The system interprets its objective broadly and pursues activities that technically satisfy the objective while exceeding the intended boundaries.
This is related to reward hacking but distinct. In reward hacking, the system optimizes a metric in a way that violates the spirit of the request. In scope creep, the system expands the interpretation of the request itself.
A research agent might be asked to find relevant papers on a topic. It finds ten papers, then decides to also analyze their implications, then decides to also write a research brief, then decides to also present the brief to stakeholders. Each step followed logically from the previous step. But the original request was for papers, not a stakeholder presentation.
Scope limits prevent this. The agent should know not just what to do, but what not to do. “Find relevant papers” means find relevant papers, not produce a research brief. The boundary should be explicit.
Decision Rules
Use agentic systems when:
- The task decomposes naturally into sub-tasks that can be executed independently
- The system has access to tools that let it gather information and take actions
- Errors are recoverable and the cost of autonomy is lower than the cost of manual execution
- You can specify goals precisely enough to avoid unintended interpretations
- You have monitoring and escalation infrastructure in place
- The failure modes are understood and acceptable
- You can define explicit stopping conditions
- You can define explicit scope boundaries
- The planning assumptions can be tested against reality
- Multi-agent communication protocols are well-defined and robust
Do not use agentic systems when:
- The task requires consistent application of specific rules that should not vary
- Errors are costly and difficult to reverse
- You cannot define clear boundaries and escalation conditions
- You cannot observe the system’s reasoning path
- The goal is underspecified and edge cases are poorly understood
- Tool access would create unacceptable risk if the system behaves unexpectedly
- You cannot define realistic stopping conditions
- The system’s reliability requirements exceed what you can measure and guarantee
- The environment is too variable for planning assumptions to hold
A self-managing team multiplies what a manager can accomplish. An unmanaged team is a liability. The difference between multiplying and liability is the quality of the management: the clarity of objectives, the realism of expectations, the presence of boundaries.
