🧠Maybe I Just Do Not Get It!

I have been working with AI for a while now.

Not as a tourist. Not as a weekend hacker. Deep in it. Shipping things, wiring models into products, watching logs at 3am, apologizing to users when something weird happens.

And still, every time I see one of those posts that says:

"Autonomous agents will run your business for you."

I feel this quiet, uncomfortable voice in my head:

"Maybe I am the one who does not get it."

Everyone seems so confident about this idea that we can just:

• Plug a model into a tool layer
• Wrap it in a few prompts
• Let it call APIs
• And watch it "run operations" or "manage your company"

And here I am, on the side, asking questions that sound almost boring:

• Are we actually giving this thing real power over important decisions?
• What exactly is our control surface?
• What happens when it fails in a way we did not anticipate?
• Do we really think prompt = governance?

Part of me worries that I am simply being too cautious. Too old school. Too stuck in "engineering brain" while the world evolves into something more fluid and probabilistic.

But part of me also thinks:

Maybe someone has to say out loud that prompts are not control.

So here is my attempt.

Half confession, half technical rant, fully self doubting.

The uncomfortable feeling of being the skeptic in an optimistic room

Let me describe a pattern that keeps repeating.

I join a meeting or a call about "AI adoption" or "agents for ops" or "automating workflows." Someone shares a slide that looks roughly like this:

User request -> LLM -> Agent framework -> Tools -> Profit

The narrative is usually:

"We will have agents that autonomously handle customer tickets, manage payments, adjust pricing, write code, do growth experiments, generate content, and so on. Humans just supervise."

While people nod, I feel like the awkward person in the corner thinking:

"This is still a stochastic model that completes text based on training data and prompt context, right? Did I miss the memo where it became an actual dependable decision maker?"

And then the self doubt kicks in.

Maybe:

• I am underestimating how good these models have become
• I am too attached to the old way of building systems with explicit logic and clear invariants
• I am biased because I have seen too many subtle failures in practice
• I am projecting my own fear of losing control over systems that I am supposed to understand

On a bad day, the internal monologue is literally:

"Everyone else seems comfortable delegating to this. Maybe I am just not visionary enough."

But then I look at the actual properties of these models and toolchains and the rational part of my brain quietly insists:

"No, wait. This is not about lacking vision. This is about basic control theory and risk."

Prompting feels like control, but it is not

There is a deep psychological trick happening here.

When you write a prompt, it feels like you are writing a policy.

Something like:

"You are an operations assistant. You always follow company policy.

You never issue a refund above 200 EUR without human approval.

You always prioritize customer safety and data privacy."

That looks like a rule set. It looks almost like a mini spec.

But underneath, what you really did is feed natural language text as conditioning into a statistical model that has:

• No strict guarantee that those words will be followed
• No built in concept of "policy" or "violation"
• No deterministic execution path
• No awareness of the "blast radius" of a single wrong action

You gave it text. The model gives you back more text.

The feeling of control is coming from you, not from the system.

It is your brain that reads that prompt and says:

"Ah, yes, I told it what to do. So now it will do that."

The model does not "know" that those instructions are sacred.

It has patterns in its weights that say: "When the input looks like this, texts like that often follow."

The distance between those two things is exactly where a lot of risk lives.

What real control looks like in software systems

If I forget the AI hype for a moment and think like a boring backend engineer, "control" has always meant things like:

• Permissions

  • Which identity can do what, where, and how often

• Boundaries

  • Network segments, firewalls, read only versus read write access, rate limits

• Auditability

  • Who did what, when, and using which parameters

• Reversibility

  • Can we undo this operation? Can we restore from backup?

• Constraints and invariants

  • Account balance must never be negative in this system
  • Orders must always have a valid user id and product id
  • This service cannot push more than X updates per second

And then, over all of this, we layer:

• Monitoring
• Alerts
• Fallback paths
• Kill switches
• Change management

It is tedious and unsexy, but it is what makes serious systems not collapse.

Now compare that with "control" when we talk about AI agents:

• "We wrapped the model with a prompt that tells it to be safe."
• "We added a message saying: if unsure, ask a human."
• "We configured a few tools and let the agent decide which to call."

There is a huge gap between these two worlds.

I keep asking myself:

"Am I overreacting by demanding similar rigor for AI automation?

Or are we collectively underreacting because the interface is so friendly and the output is so fluent?"

Why prompt based control is fragile

Let me list the main reasons I struggle to trust prompt = control as a serious safety mechanism.

Non determinism

Call the same model with the same prompt and temperature 0.7 ten times and you get:

• Slightly different reasoning chains
• Occasionally very different answers
• Sometimes rare but catastrophic failure modes

This is fine in a chat setting. It is far less fine if:

• The output approves or denies a refund
• The output decides whether to escalate a compliance issue
• The output sends an email to an important customer

If your "policy" is just in the prompt, the model can randomly deviate from it when some token path goes weird.

Context dilution and instruction conflicts

In complex agent setups, the model context looks something like:

• System messages ("You are X")
• Task instructions
• Tool specs
• History, previous steps, errors
• User messages
• Tool responses

Your carefully written safety instructions can:

• Get buried far up in the context
• Be overshadowed by later messages
• Conflict with tool descriptions
• Interact strangely with user input

You cannot be sure which instruction wins inside the model internal weighting. You are left hoping that the most important part is loud enough.

Distribution shift and weird edge cases

The model was trained on static data. Then it is thrown into:

• An evolving product
• Changing user behavior
• Novel business processes
• Adversarial inputs from clever users

Whatever behavior you saw in your internal tests is not a formal guarantee. It is just evidence that under some sample of conditions, it behaved "well enough."

It might take only one weird edge case to cause a big problem.

Lack of grounded state and formal rules

Older systems have explicit state machines and rules. You can formalize, prove, or at least reason about them.

AI agents usually do not have:

• A formal internal model of the environment
• A provable decision process
• Compositional guarantees

This means if you want real control, you need to build it around the models, not inside the prompt.

Which brings me to the part where I keep asking:

"Why are so many people comfortable skipping that part?"

The three A's: Automation, autonomy, authority

I find it useful to separate three concepts that often get blended into one in marketing material.

Automation

This is what we have done for decades:

• Cron jobs
• Scripts
• Pipelines
• Daemons

Automation means: "This specific routine step is handled by a machine."

Poorly designed automation can still cause trouble, but at least the logic is explicit.

Autonomy

Autonomy is stronger:

• The system can decide which steps to take
• It can react to the environment
• It can pick from multiple possible actions
• It can generate plans that you did not hardcode

This is where LLM based agents live. They choose tools, infer goals, adapt to context.

Authority

And then there is authority:

• The system has the power to directly impact important resources
• It can move money, change production systems, talk to customers, sign something that looks like a commitment

Authority is where risk explodes.

You can have autonomous agents with:

• No authority at all (pure recommendations)
• Limited authority (sandboxed actions)
• Full authority (unbounded write access to real world systems)

My fear is not autonomy itself.

My fear is autonomy plus authority with only prompt based "control" and minimal protective scaffolding.

That feels like a fragile tower.

Why "let the AI run it" is so seductive

To be fair, I understand the attraction. It is not just hype or stupidity. There are real pressures that make people want this to be true.

Some of them:

• Economic pressure

  • Reduce headcount
  • Do more with smaller teams
  • Compete with others who claim they have agent driven operations

• Cognitive overload

  • Businesses are complex
  • It is tempting to offload routine decision making to something that can read all the documents and logs

• Interface illusion

  • Talking to an LLM feels like talking to a very capable person
  • It sounds confident
  • It apologizes when it fails
  • We project far more understanding into it than it actually has

• Narrative momentum

  • Investors, founders, vendors, content creators all benefit from the story that "this is the future"
  • Nobody gets likes by saying "We built a constrained automation with careful permissions and small risk boundaries"

And so I watch this wave of "AI will run your company" rising, and my own position starts to feel almost embarrassing:

"Sorry, I still think we should treat this like an unreliable but useful colleague, not an all knowing operations overlord."

Am I being too negative? Or are others being too optimistic? I genuinely do not know.

What a more honest control architecture might look like

Let me try to articulate the model that my brain keeps coming back to. Perhaps this is my bias. Perhaps it is just boring good practice.

I imagine AI systems in terms of layers.

Models as suggestion engines

At the core, the LLM or vision model or other component is:

• A suggestion engine
• A planning assistant
• A pattern recognizer

It produces options, not truth. It drafts, proposes, clusters, explains.

In this framing, the default is:

"Everything the model says needs to be checked or constrained by something else."

Policy and constraints outside the model

Policies like "never refund above X without human approval" should not live only in prompt text.

They should live in actual logic that wraps the model.

MAX_AUTO_REFUND = 200  # euros

def handle_refund_suggestion(user_id, suggestion_amount):
    if suggestion_amount <= MAX_AUTO_REFUND:
        issue_refund(user_id, suggestion_amount)
        log_event("refund_auto_approved", user_id=user_id, amount=suggestion_amount)
    else:
        create_manual_review_ticket(user_id, suggestion_amount)
        log_event("refund_needs_review", user_id=user_id, amount=suggestion_amount)

The model can still say "I think we should refund 300 EUR."

But authority is delegated through hard limits, not polite reminders in a prompt.

Tooling as a narrow interface to the world

Agents should not see a raw database or a root shell.

They should see:

• Narrow tools that do one thing
• With explicit safe defaults
• With input validation and sanitization
• With rate limits and quotas
• With clear logging

def send_email(to, subject, body, *, template_id=None):
    # Validation
    assert isinstance(to, str)
    assert "@" in to
    assert len(subject) < 200
    assert len(body) < 5000

    # Logging
    log_event("email_requested", to=to, subject=subject, template_id=template_id)

    # Send through provider
    provider.send(to=to, subject=subject, body=body)

The agent can choose to call send_email, but cannot bypass:

• Validation
• Logging
• Provider boundaries

Human checkpoints and degrees of autonomy

The idea of "agent runs everything" feels wrong to me.

A more grounded model:

• Level 0: Advisory only
  • The model suggests, humans decide
• Level 1: Low risk autonomy
  • Model can execute actions with small impact, easily reversible
• Level 2: Medium risk autonomy
  • Model can act but behind stricter limits and with additional monitoring
• Level 3: High risk decisions
  • Model can only propose. Mandatory human review

You can even encode this in configuration:

tasks:
  - id: reply_to_low_risk_tickets
    autonomy_level: 2
    max_impact: "single_customer_message"

  - id: adjust_pricing
    autonomy_level: 0
    requires_human_approval: true

  - id: issue_refund
    autonomy_level: 1
    max_amount: 200

And then enforce this at the orchestration layer, not just in a paragraph of English.

Observability and traceability

If a system is "running your business" in any sense, you want to know:

• What it decided
• Why it decided that
• Which tools it called
• What inputs it saw
• How often it was wrong

That means:

• Structured logs
• Traces per request
• Metrics
• Failure classification
• Ability to replay a scenario and see exactly what happened

Without this, you are blind.

The voice that keeps whispering "Maybe you are overdoing it"

Let me be honest.

When I design systems with:

• Orchestrators
• Multiple agents
• Guards
• Checks
• Extensive logging

I sometimes feel like the paranoid one in a world of brave explorers.

Every confident demo of an "autonomous" system triggers a small internal comparison:

• They are moving faster
• They ship bold things
• They talk about "hands free operations"
• They have nice UIs and slick narratives

And then I look at my own mental model:

"Treat the AI like an unreliable colleague. Give it power only inside tightly defined boundaries. Observe it at all times."

That can feel conservative, almost limiting.

The self doubt is real.

Sometimes I really think:

"Maybe I should relax and just trust the agents more."

Then I remember practical incidents:

• Models hallucinating facts that never existed
• Wrong tool calls due to slightly ambiguous tool descriptions
• Subtle prompt drift when context windows get large
• Surprising interactions between two agents that were not tested together
• Simple failure cases that completely break the illusion of "autonomy"

And the cautious part of me wins again.

Could I be wrong about all this?

To be fair, yes. I could be wrong in several ways.

Possibilities that I keep in mind:

• Models might reach reliability levels I do not currently anticipate

  • With better training, better fine tuning, better safety layers
  • Maybe we will actually get to a place where prompt specified policies are followed with extremely high consistency

• The average business might tolerate more risk than I think

  • Maybe for many processes, "good enough, most of the time" is perfectly fine
  • Maybe the cost savings outweigh occasional screw ups

• New agent frameworks might enforce more structure internally

  • Instead of raw prompt based decisions, they might encode policies and constraints in more robust ways, even if the interface still looks like "agents"

• I might be stuck with an outdated mental model

  • My training is in building explicit systems
  • Perhaps my discomfort is just a mismatch between that training and a new probabilistic world

I try to keep this humility active. I do not want to be the person yelling "cars are dangerous, horses are better" forever.

But even if I am wrong about the magnitude of the risk, I still believe:

"Any time you mix autonomy and authority, you need real control structures, not only nice English."

And I have not yet seen a convincing argument that prompts, by themselves, are a sufficient control mechanism for important decisions.

How I am personally responding to this tension

Instead of picking a side in the "agents are the future" versus "agents are useless" debate, I am trying to sit in the middle:

• I accept that LLMs are incredibly powerful tools for reasoning, drafting, and planning
• I reject the idea that this power makes them inherently trustworthy decision makers
• I still want to leverage them deeply in systems that matter
• And I want to design those systems with explicit, boring, old fashioned control surfaces

That is why I care about things like:

• Orchestrators that define flows explicitly
• YAML or other declarative formats that separate "what should happen" from "how the model reasons"
• Service nodes that encapsulate capabilities behind strict boundaries
• Observability that lets you replay what happened and why

It is also why, when people tell me:

"We have an autonomous agent that runs X."

my first questions are:

• What is the maximum damage it can cause in one execution?
• How do you know when it did the wrong thing?
• How do you stop it quickly?
• Where are the hard rules, outside of the prompt?

If the answers are vague, my self doubt temporarily fades and my confidence in my skepticism grows.

At the same time, I am building tools in this space myself. That adds a weird layer:

• I do not want to sound like I am attacking the very category I am working in
• But I also do not want to oversell autonomy and under discuss control

So when I mention my own work, I try to be very direct:

"I am working on an orchestration layer for AI agents, with explicit flows, branching, service nodes, and traceability.

The whole point is not to 'let the AI run everything' but to give humans a clear frame to decide what the AI is allowed to do, when, and with which safeguards."

If you are curious, that work lives in the OrKa Reasoning project.

It is part of my attempt to reconcile "use AI deeply" with "do not surrender control through vibes."

OrKa Reasoning

If the idea of explicit control over AI agents resonates with you, you can explore the OrKa Reasoning stack here:

• OrKa Reasoning repository:
  
  https://github.com/marcosomma/orka-reasoning

• Quick start and docs:
  
  https://github.com/marcosomma/orka-reasoning/blob/master/docs/getting-started.md

OrKa Reasoning is my attempt to give developers a YAML first orchestration layer where:

• Flows and branching are explicit
• Memory, tools, and services are wrapped in clear nodes
• You can inspect traces and understand why a decision was made

In other words, it is an experiment in building the control surfaces I wish existed by default in modern AI stacks.

An invitation to other quietly skeptical builders

If you feel a similar discomfort, I want to say this clearly:

"You are not alone."

You can:

• Respect what these models can do
• Push their limits in creative ways
• Build serious systems around them

and still say:

"No, I am not giving them full, unbounded control over important decisions.

Control belongs to humans, encoded in systems that are understandable and auditable."

You can be:

• Excited and cautious
• Curious and skeptical
• Ambitious and unwilling to pretend that prompt text equals governance

I still doubt myself almost every time I voice these concerns.

I still worry about sounding like the boring adult in a room full of enthusiastic kids.

But I would rather live with that discomfort than pretend that we solved control by writing "please be safe" at the top of a prompt.

If anything in this ramble resonates with you, I would love to hear:

• How are you designing control into your AI systems?
• Where do you draw the line for autonomy versus authority?
• Have you found practices that actually reduce your anxiety about letting agents act?

Drop a comment, poke holes in my thinking, or tell me why I am wrong.

I am very open to the possibility that I am the one who "does not get it."

But if we are going to let AI agents touch real businesses, real money, and real people, I would really like us to get control right first.