Pawel Brodzinski on Leadership in Technology

Whatever it takes to lead a team, build a product, and run a business

Category: ai

The Ultimate Question: What Does the Endgame Look Like?
The last couple of years have been riddled with speculations about how AI will change the world. Software development and the broader IT industry are among the most affected contexts. Things are changing. The future is uncertain.

In such a landscape, it’s easy to subscribe to any speculation, like the infamous doom and gloom Citrini prediction. Before we fall for that, though, let’s look at the historical data.

It’s Q2 2026. If the AI predictions had been correct, then:
- We have AGI. As Sam Altman predicted.
- Thus, most developers don’t code anymore. That’s Matt Garman from two years ago.
- Why would they? After all, no one learns to code these days, because “everybody is now a programmer.” Courtesy of Jensen Huang.
- If you want concrete data, since last autumn, AI writes 90% of the code. Yup, that’s Dario Amodei.
- No one drops a tear over developers, though, because we already see how AI is in the midst of wiping out half of entry-level white-collar jobs. That’s Dario Amodei, too.
Yeah, I get your skepticism. That’s not reality I see around either. Fear not, however. Software engineers will go extinct this year. This time for real, says Dario Amodei. This time, we can trust him. For sure. Probably. Maybe.

Each time such an alarmistic prediction emerges, I ask one question: If X is true, what does the endgame look like?

What Is the Endgame?

I borrowed the idea of endgame from gaming, duh! Some gaming genres are built around a character progression. However, when a player character reaches the maximum level, the original game engine ceases to work. There’s no more level to grind. No more progression to make.

Thus, the endgame content was born. These are parts of a game designed specifically for max-level characters to keep the players interested. Typically, these are increasingly challenging. This time, the goal is not progress, but mastery. It’s like a game in a game.

The endgame content responds to the question of a hypothetical newbie player: “What happens if I play this game and keep progressing with my character?”

The question is interesting because we can envision the progression and intuitively realize that it can’t last indefinitely. At some point, an external constraint would impose itself, and our linear approximation of the trend (leveling up in this case) would break.

Thus, the question: What does the endgame look like?

The Endgame Question Is More Than Relevant in Business

If we look at market trends, the dynamics are surprisingly analogous. It’s not a game, so we don’t control the trends, but they’re there, sure enough. And they can’t last indefinitely. There’s always an external constraint that will impose itself.

The market share can’t go higher than 100%. The exponential growth can’t last more than a few years. Businesses need to make a profit eventually. And so on.

Now, if we ask the right questions, we don’t need to wait for the change to happen to see how the landscape will evolve. Better yet, we might see other facets of the change. Think of it as ripple effects. Then, suddenly, the landscape is richer, and we may come to very different conclusions from those we’d make if we looked at a trend in isolation.

A good example is what’s been dubbed a SaaSpocalypse—a recent devaluation of many SaaS businesses. What some perceived as the new trend predicting the end of SaaS, I consider merely a regression to the mean.

If this trend continued, the purchase price of these “old-school” product companies would be a bargain. They have healthy financials. Some have just recorded the best year ever. Unlike some of the tech scene darlings, they’re making actual profits. Plenty of them. Fundamentally, little has changed for these companies short- and mid-term.

It’s then relatively easy to see the endgame. The trend won’t continue too far, as eventually it would mean buying a dollar for fifty cents.

The Interconnected Trends and Second-Order Consequences

The endgame question is even more interesting whenever there’s no obvious limiting condition (like “you can’t have more than 100% of market share”). A good example is how AI affects coding.

We see increasing AI use in code generation. It’s not anywhere close to 90%, sure, but no one challenges that we’re doing more of that. Also, it’s obvious that AI agents can generate tons of code. And then some. No sweat.

The trend, then, suggests that we will have more and more of AI-generated code. Let’s then draw the trend line to the future and ask: What does the endgame look like?

Given how increasingly useful AI tools are, there’s no stopping the trend. At this pace, we will soon generate more code than we can reasonably review as we go. Once we stop the just-in-time code review, we will lose comprehension of what’s at the code level in our products.

The endgame is either a lights-out codebase or a risk of being outpaced by competitors. That’s an interesting dilemma. So far, research suggests that AI models are incapable of maintaining code in the long run. Yet, the business risk coming from potential competitors is real, too.

These are second- or third-order consequences of code-generation capabilities we have thanks to AI tools. And these are precisely the considerations that any product business should take into account these days.

These are far more interesting than boasting about how much code is AI-generated. As a customer, I couldn’t care less whether you generate 30% of your code. Or 90%. Or none at all. I do care whether the product solves my problem now and whether it will be technically sustainable in a year from now.

And you don’t hear Satya Nadellas and Mark Zuckerbergs of this world discussing their concerns about the maintainability of their products.

The Dynamics of the Endgame Question

The reason why the endgame question is so powerful is that it skips the current condition and jumps directly to the future state:
- What will be new or different once this new thing becomes the norm?
- When does the trend become unsustainable?
- How do correlated trends behave?
Think of it as a model. We look at one thing and have historical data on how it has behaved so far. Now, the simplest possible thing is to extend the trend line indefinitely into the future.

Except, as we already established, things do not work like this. In no reality does OpenAI have 8 billion paying ChatGPT users. So, before we predict the future, we consider external constraints.

Once we make it explicit, it becomes obvious that a naive version of the future will not happen. Even if we assume the most optimistic scenarios, the trend line will have to change its shape.

Well, that’s different now, thank you. But we’re not done yet. The most interesting things happen at the intersection. We can ask ourselves which other trends are correlated with whatever we focus on.

Like, if there’s more of this, there should also be more of that. Or vice versa, if there’s more of this, there should be less of that. As with our example, if we generate more and more code, there will necessarily be less technical comprehension. The stronger one is, the weaker the other will become.

Since we already have a clearer picture of the landscape, it’s not that hard to predict how an inversely correlated thing will change. And to what degree. Suddenly, we are equipped to ask questions about second-order consequences.

That’s where the endgame question shines. Instead of boasting about which big tech generates more code or predicting when developers go extinct, we may consider possible futures.

Human in the Loop and Coding

To run a quick example I touched on earlier, let’s consider AI and coding. Dario Amodei is wrong about how fast his AI models will take over coding. But it’s not because of the lack of capabilities of said models. I mean that too, but he knows more about these capabilities than you or me, and maybe he has all the right to believe it’s a technical problem that’s going to be fixed eventually.

He’s wrong because he considers code generation in a surprisingly isolated sandbox. If we were to believe Amodei’s predictions, we would have to assume that human-in-the-loop will be gone from software engineering.

I mean, physically, we can keep humans there, but they will have no real role. They’d be overloaded and incapable of good judgment. In fact, it’s already happening. Speculatively, though likely, in recent wars, humans-in-the-loop had the final call with decisions about strikes. Yet, you can’t expect good judgment if someone is expected to make 80 life-or-death decisions per hour.

There might still be a human body in the loop. The judgment, though? With enough cognitive load, it’s gone.

Just compare these two predictions. The first is a naive one, and considers a thing in isolation. The second attempts to understand what would change and how if the current trends stay with us. These two look very different.

The Endgame Question for Coding

So let’s look at what answers the endgame question yields in the coding example. In the past decades, we’ve been creating a growing amount of code. And yet, code review as a practice has also been increasingly popular.

AI has introduced a foreign element to our system. Now we can easily generate as much code as we want. Increasingly, we do. That changes the current dynamics of software development trends.

But wait, so far, the “code review” trend has been all good. The practice has been growing in popularity, despite the fact that, as a whole, we were developing more code.

Hell, one way of looking at it is that all code has been reviewed, since the developer creating it was doing a sort of review as part of the creative process.

The only problem is that code review is a cognitive task that requires attention. And we have a limited pool of it. If we suddenly needed to review 10x as much code, we don’t have enough engineers to handle that.

Even if we try to keep up, which I call an “optimistic” scenario here, we eventually hit the ceiling. There’s no more available attention to pay.

A side note: we could argue that we actually raise the limit by freeing developers from writing code, so they have more time to review it. That’s fair. However, we also claim we don’t need no new developers (so we don’t train them) and lay them off (so they change industries). Effectively, we’re working the limit line in both directions. In either case, even if it goes somewhat up, we’ll cross it soon enough.

With that, we’ll create a gap between the amount of code we create and the loads we are capable of reviewing. And that gap will only keep growing. Fast.

That, in turn, is the exact reason the “optimistic” scenario will not happen. Playing a losing game is no fun. Even less so if that’s an increasingly losing game. The only sensible expectation is that we will stop playing the game altogether.

The new reality doesn’t mean stopping the reviews entirely. But we’ll need to pick our battles. And we’ll need to be increasingly picky about picking them. We’ll choose only the most critical parts of the code and maintain active knowledge of them.

Second-Order Consequences of the Coding Endgame

Things get even more interesting when we consider ripple effects. Before AI, basically, all the code was read. I mean, a human wrote it, so part of the process was looking at the thing. The “code read” curve was identical to the “code created” one.

However, as we stop writing code ourselves and expect the code review rate to nosedive, we’ll look at a completely different reality. “Code read” line will detach from “code created” and will follow “code reviewed.”

Now, that’s interesting. There is more code, but save for very few carefully chosen code bases, we neither read nor understand it. It’s as if there were islands of comprehension in a black-box ocean. That is, unless we fundamentally change something. So, are we ready to run critical systems on software we can’t comprehend? Because that’s what the endgame looks like.

And that’s but one example why the endgame question is such a neat trick. The moment we start asking it, we start seeing scenarios that go way beyond the hype. It’s not just “Claude Code is so awesome; it can do the coding for me.” It’s “Would I trust a vibe-coded e-commerce with my credit card number?” Or even “How would I feel if Visa or MasterCard ran on software no human comprehends?”

The Ultimate Question

Now, I know I rode the example of AI in coding in this post. The applicability of the endgame question is way broader, though. It literally pops up anytime someone makes a kind of bold prediction, well, about anything. You know, the type of “AI is capable of erasing half of white-collar jobs, so AI labs will get unfathomably rich,” or something along the same lines.

What does the endgame look like? Well, we make half of the knowledge workers unemployed, and who’s paying the AI bills, again?

Or take this: “AI will take over content generation as it can create 100x times as much as humans can, no sweat.” What does the endgame look like? We don’t have 100x as much attention, so the vast majority of the generated content will not be consumed at all. We may have the effect of bad money driving out good, but we won’t fundamentally have use of more content.

“Thanks to AI capabilities, we’ll see a surge of new products. Anyone will be able to run a product now.” What does the endgame look like? Again, the attention constraint (or the demography) suggests we won’t have 100x as many customers. So, if anything, we’ll just increase the failure rate. While running a startup is already unappealing, it will become even less of a winning proposition, which will actively drive people away from that path.

“AI will automate applying for jobs.” What does the endgame look like? Both sides get automated to handle an increasing load. Eventually, it’s one AI agent negotiating with another to figure out whether a human is a good fit for an organization. The system is bound to be misaligned and thus gamed. What follows is that we’ll either accept hiring candidates who are increasingly unfit for the role (but who played the game better) or reinvent the hiring system altogether.

So before we jump on another bit of “CEO said a thing” journalism, it’s worth asking: if that’s true, what does the endgame look like?

As hilarious as it would be, given the topic, this post has not been AI-generated. 웃 https://okhuman.com/wLBTwg
April 13, 2026
Would You Pay to Have Your Resume Read?
As a job applicant, would you pay to make sure someone reads your application?

Here’s a sad reality for many people applying for a job:
- Their competitors (i.e., other candidates) use AI tools to mass apply.
- As a result, hiring companies are flooded with applications, and sifting through all of them is impractical.
- What follows is that hiring companies defer to other AI tools to filter out the vast majority of applications (often as much as 95%+).
- The recruitment game becomes one of prompting one AI agent to pass through the filters of another AI agent.
Realities of Job Seekers A.D. 2025

Imagine that there is a job that you really want to get. It doesn’t even matter why. It may be because you know that the company is great, or the job profile matches your dreams perfectly, or you perceive the experience you’d get there as unique, or whatever. You just want in.

But hey, since all those other people are using AI tools to spam the hiring company’s application form, your submission will disappear in that flood.

It’s even worse than that. If you hand-craft your application to show your genuine care for the job, it’s almost certain that you’ll be rejected. After all, your original story will be written to a hiring manager (a human), but it’s never going to get there in the first place. It will be rejected by an automated AI tool (a bot) precisely because it’s non-conformist.

Such a resume doesn’t match the most common patterns. There aren’t many similar examples in the AI model’s training data. It’s not common enough.

If you want your application to get past the AI filter, you kinda have to play the game everyone else does. Optimize for what a bot wants. And it’s impractical to do it by hand. Just hire another AI agent to do it for you.

Except that you’ve defeated the purpose that way. First, you aren’t more likely to get through. Second, even in the case that you do, the hiring manager will see another similar, bland-but-professional resume. You will not stand out.

Most importantly, you will not carry over your care about that job.

Recruitment in the AI Era Is Irrevocably Broken

The story above neatly pictures how broken the recruitment has become. What’s more, there’s no going back.

You can pretend it’s 2020 and send your manually-crafted CV, but you’re going to lose to people auto-submitting thousands of AI-generated resumes. Oh, and said resumes will be automatically tweaked to better match a job description, with no human effort whatsoever.

A resume doesn’t work as a token of information exchanged between two humans (a hiring manager and a candidate) anymore.

The career of a resume is over. At least the one that we know. If anything, a CV becomes a token exchanged between two AI agents, neither of which is programmed by the actual candidate.

No matter how hard we try, there’s no coming back. We can’t make resumes unbroken again. Even if we aspirationally tried to restore the original meaning of a CV, there will always be a rogue player who will exploit that trust by mass-applying with generated stuff. And since that will give them a short-term advantage, others will follow suit.

Winning the Game by Not Playing It Altogether

It’s ironic how both sides of this equation—recruiters and candidates alike—are losing in the new setup. Candidates have it harder to show their care about specific jobs. Companies give up on the best matches because they employ a bot to reject 95% of applicants. And yet, no one can change the rules anymore.

So, is conforming to the new state of things the only option?

Image from the WarGames movie

In the classic movie WarGames, the AI, which is trying to “win” the nuclear war, eventually learns that it always ends in mutual assured destruction. The only winning move, thus, is not to play at all.

It’s the same with recruitment. If the current system forces us to mass-produce thousands and thousands of resumes that no one will ever read, we’re just adding noise to the system. The winning move? Not to play.

But wait, if you want to change jobs, how are you supposed not to play the game? If you never apply, you never get that dream job of yours. Or a better one than you have now.

Trust Networks as Antidote to AI Slop

In recruitment, as much as in any other area, we will defer to trust networks to circumvent the noise. The more toxic AI slop is in the feed, the less we trust the feed altogether, and the more we rely on human-to-human connections.

One side of relying on trust networks is that companies increasingly go for employee referrals rather than traditional open recruitment processes. That doesn’t solve the other part of the equation, though. What if I am a candidate and want that specific job?

Do the same. Build a connection with someone at that company. We live in an interconnected world, and there are still places where a genuine message will stand out. They may attend local meetups, be active on LinkedIn, maybe publish a blog or a Substack, or engage in some other professional activities. If you care, you will figure that out. Get to know people first, and only then apply.

Does it seem like a lot of effort? That’s precisely the point. It shows how much you care.

Very recently, we made our first hire in almost two years. We didn’t even open a recruitment process. There was this guy who stayed in contact after we talked a few years back. And then, eventually, it was a good time for him and a good time for us. A win-win.

The point is: he made the effort to reconnect. He made it easy for us to remember.

This could only happen because we’ve built the human connection beforehand. We were two parts of the same trust network.

Would You Pay To Put Your Resume at a Hiring Manager’s Desk?

I admit, relying on trust networks is a lot of effort. And it takes time. Both would make the approach impractical at times. So what if there were a shortcut?

That brings me back to my original question. As a candidate applying for a job, would you pay to skip the AI line? Would you pay to ensure that your application is read by a human?

Note, your resume would still go through regular scrutiny. It’s just you’d know a human would do it, not a black-box AI agent.

There’s an interesting balance here. Make it too cheap, say $0.02, and it changes nothing. People would still be mass-applying all the same, so no one would take that seriously. Make it too expensive, say $200, and it’s probably not a good return on investment for a candidate. After all, no one would hire such a candidate or even rate them any better. A hiring manager would just read and assess the resume as if it passed the AI filters.

What’s in it for a candidate? It’s an open avenue to show genuine care. Since the applicant knows they’re not going through AI, they are free to optimize their application for a human reader. Hell, they actually are encouraged to go the extra mile with their application.

What’s in it for a hiring company? I reckon it wouldn’t make sense for a candidate to pay for mass applying, so they’d do that only for jobs they actually care about. So the hiring company gets a token of care along with a resume. Recruiters can still assess skills the way they do, but before committing any effort in interviews, they clearly know which candidates consider the position a great match.

So, would you pay to guarantee your resume is reviewed by a hiring manager? If so, how much?

Here’s a little experiment that’s in the spirit of the post. This link here is a token of human effort behind the post.
웃https://okhuman.com/CuC1uw
December 5, 2025
Trust Networks as Antidote to AI Slop

This week, AWS went down, along with a quarter of the internet. It’s funny how much we rely on cloud infrastructure even for services that should natively work offline.

That is, “funny” as long as you’re not a customer of said services trying to do something important to you. I know how frustrating it was when Grammarly stopped correcting my writing during the outage, even if it’s anything but a critical service to me.

While AWS engineers were busy trying to get the services back online, the internet was busy mocking Amazon. Elon Musk’s tweet got turbo-popular, quickly getting several million pageviews and sparking buzz from Reddit to serious pundits.

Admittedly, it was spot on. No wonder it spread like wildfire. I got it as a meme, like an hour later, from a colleague. It would fit well with some of my snarky comments about AI, wouldn’t it?

However, before joining the mocking crowd, I tried to look up the source.

Don’t Trust Random Tweets

Finding the article used as a screenshot was easy enough. It was a CNBC piece on Matt Garman. Except the title didn’t say anything about how much AI-generated code AWS pushes to production.

Fair enough. Media are known to A/B test their titles to see which gets the most clicks. So I read the article, hoping to find a relevant reference. Nope. Nothing. Nil.

The article, as the title clearly suggests, is about something completely different.

I tried to google up the exact phrase. It returned only a Redit/X trail of the original “You don’t say” retort. Googling exact quotes from the CNBC article did return several links that republished the piece, but all used the original title, not the one from the smartass comment. It didn’t seem CNBC had been A/B testing the headline.

By that point, I was like, compare these two pictures. Find five differences (the bottom one is the legitimate screenshot).

Top picture from the tweet Elon Musk shared. Bottom from the actual CNBC article.

So yes, jokes on you, jokers.

Except no one cares, really. Everyone laughed, and few, if anyone, cared to check the source. Few, if anyone, cared to utter “sorry.”

Trustworthiness as the New Currency

I received Musk’s tweet as a meme from my colleagues. It went through at least two of them before landing in my Slack channel. They passed it with good intent. I mean, why would you double-check a screenshot from an article?

It’s a friggin’ screenshot, after all.

Except it’s not.

This story showcases the challenge we’re facing in the AI era. We have to raise our guard regarding what we trust. We increasingly have to assume that whatever we receive is not genuine.

It may be a meme, and we’ll have a laugh and move on. Whatever. It won’t hurt Matt Garman’s bonus. It won’t have a dent in Elon Musk’s trustworthiness (even if there were such a thing).

It may be a resume, though. A business offer. A networking invitation, recommendation, technical article, website, etc. It’s just so easy to generate any of these.

What’s more, a randomly chosen bit on the internet is already more likely to be AI-generated than created by a human. Statistically speaking, there’s a flip-of-a-coin chance that this article has been generated by an LLM.

It wasn’t, no worries. Trust me.

Well, if you know me, I probably didn’t need to ask you for a leap of faith in the originality of my writing. The reason is trustworthiness. That’s the currency we exchange here. You trust I wouldn’t throw AI slop at you.

If you landed here from a random place on the internet, well, you can’t know. That is, unless you got here via a share from someone whom you trust (at least a bit) and you extend the courtesy.

Trust in Business Dealings

The same pattern works in any professional situation. And, sadly, it is as much affected by the AI-generated flood as blogs/newsletters/articles.

When a company receives an application for an open position, it can’t know whether a candidate even applied for the job. It might have been an AI agent working on behalf of someone mass-applying to thousands of companies.

While we’re still beating a dead horse of resume-based recruitment, it’s beyond recovery. Hiring wasn’t healthy to start with, but with AI, we utterly broke it.

A way out? If someone you know (or someone known by someone you know) applies, you kinda trust it’s genuine. You will trust not only the act of applying but, most likely, extend it to the candidate’s self-assessment.

Trust is a universal hack to work around the flood of AI slop.

Outreach in a professional context? Same story. Cold outreach was broken before LLMs, but now we almost have to assume that it’s all AI agents hunting for gullible. But if someone you know made the connection, you’d listen.

Networking? Same thing. You can’t know whether a comment, post, or networking request was written by a human or a bot. OK, sometimes it’s almost obvious, but there’s a huge gray zone. In someone you trust does the intro, though? A different game.

The pattern is the same. Trust is like an antidote to all those things broken by AI slop.

Don’t We Care About Quality?

Let me get back to the stuff we read online for a moment. One argument that pops up in this context is that all we should care about is quality. It’s either good enough or not. If it is, why should we care who or what wrote it?

Fair enough. As long as consuming a bit of content is all we care about.

If I consider interacting with content in any way, it’s a different game.

With AI capabilities, we can generate almost infinitely more writing, art, music, etc. than what humans create. Some of it will be good enough, sure. I mean, ultimately, most of what humans create is mediocre, too. The bar is not that high.

There’s only one problem. We might have more stuff to consume, but we don’t have any more attention than we had.

Now, the big question. Would you rather interact with a human or a bot? If the former, then you may want to optimize the choice of what you consume accordingly.

Engageability of our creations will be an increasingly important factor. And it won’t be only a function of what kind of call to action a consumer feels after reading a piece, but also whether they trust there’s a human being on the other side.

It’s trust, again.

Trust Networks as the New Operating System

Relying solely on what we personally trust would be impractical. There are only so many people I have met and learned to trust to a reasonable degree.

Limiting my options to hiring only among them, reading only what they create, doing business only with them, etc., would be plain stupid. So how do we balance our necessarily limited trust circle with the realities of untrustworthiness boosted by AI capabilities?

Elementary. Trust networks.

If I trust Jose, and Jose trusts Martin, then I extend my trust to Martin. If our connection works and I learn that Martin trusts James, then I trust James, too. And then I extend that to James’ acquaintances, as well. And yes, that’s an actual trust chain that worked for me.

By the same token, if you trust me with my writing, you can assume that I don’t link shit in my posts. Sure, I won’t guarantee that I have never ever linked anything AI-generated. Yet I check the links and definitely don’t share AI slop intentionally.

If such a thing happened, it would have been like Musk’s “you don’t say” meme I received—passed by my colleagues with good intent.

The degree to which such a trust network spans depends on how reliably a node has worked so far. A strong connection would reinforce its subnetwork, while a failing (no longer trustworthy) node would weaken its connections.

Strong nodes would allow further connections, while weak ones would atrophy. It is essentially a case of a fitness landscape.

New Solutions Will Rely on Trust Networks

The changes we’ve made to our landscape with AI are irreversible. In one discussion I’ve had, someone suggested a no-AI subinternet.

It’s not feasible. Even if there were a way to reliably validate an internet user as a human (there isn’t), nothing would stop evil actors from copypasting AI slop semi-manually anyway.

In other words, we will have to navigate this information dumpster for the time being. To do that, we will rely on our trust networks.

Whatever new recruitment solution eventually emerges, it will employ extended trust networks. That’s what small business owners in a physical world already do. They reach out to their staff and acquaintances and ask whether they know anyone suitable for an open position.

Content creation and consumption are already evolving toward increasingly closed connections (paywalled content, Substacks, etc.), where we consciously choose what we read and from whom. Oh, and of course, the publishing platforms actively push recommendation engines.

Business connections? Same story. We will evolve to care even more about warm intros and in-person meetings.

Eventually, large parts of the internet will be an irradiated area where bots create for bots, while we will be building shelters of trustworthiness, where genuine human connection will be the currency.

Like hunters-gatherers. Like we did for millennia.

October 22, 2025
We Will Not Trust Autonomous AI Agents Anytime Soon
OpenAI and Stripe announced what they call the Agentic Commerce Protocol (ACP for short). The idea behind it is to enable AI agents to make purchases autonomously.

It’s not hard to guess that the response from smartass merchants would come almost immediately.

As much fun as we can make of those attempts to make a quick buck, the whole situation is way more interesting if we look beyond the technical and security aspects.

Shallow Perception of Autonomous AI Agents

What drew popular interest to the Stripe & OpenAI announcement was an intended outcome and its edge cases. “The AI agent will now be able to make purchases on our behalf.”
- What if it makes a bad purchase?
- How would it react to black hat players trying to trick it?
- What guardrails will we have when we deploy it?
All these questions are intriguing, but I think we can generalize them to a game of cat and mouse. Rogue players will prey on models’ deficiencies (either design flaws or naive implementations) while AI companies will patch the issues. Inevitably, the good folks will be playing the catch-up game here.

I’m not overly optimistic about the accumulated outcome of those games. So far, we haven’t yet seen a model whose guardrails haven’t been overcome in days (or hours).

However, unless one is a black hat hacker or plans to release their credit-card-wielding AI bots out in the wild soon, these concerns are only mildly interesting. That is, unless we look at it from an organizational culture point of view.

“Autonomous” Is the Clue in Autonomous AI Agents

When we see the phrase “Autonomous AI Agent,” we tend to focus on the AI part or the agent part. But the actual culprit is autonomy.

Autonomy in the context of organizational culture is a theme in my writing and teaching. I go as far as to argue that distributing autonomy throughout all organizational levels is a crucial management transformation of the 21st century.

And yet we can’t consider autonomy as a standalone concept. I often refer to a model of codependencies that we need to introduce to increase autonomy levels in an organization.

The least we need to have in place before we introduce autonomy are:
- Transparency. We can’t let people make decisions without relevant data to inform them, or decisions will be plain wrong.
- Technical excellence. Independence in acting requires the capabilities to perform these acts competently.
- Alignment. Unless we align everyone’s efforts, more autonomy only means more pull toward opposing directions.
- Explicit boundaries. We need to understand the limits within which we can act autonomously. Otherwise, we’d be both overwhelmed with possibilities and petrified by potential consequences.
- Care. Without intrinsic, genuine care about the outcomes of our decisions and actions, it’s just flailing around (and without purpose, let me add).
Remove either, and autonomy won’t deliver the outcomes you expect. Interestingly, when we consider autonomy from the vantage point of AI agents rather than organizational culture, the view is not that different.

Limitations of AI Agents

We can look at how autonomous agents would fare against our list of autonomy prerequisites.

Transparency

Transparency is a concept external to an agent, be it a team member or an AI bot. The question is about how much transparency the system around the agent can provide. In the case of AI, one part is available data, and the other part is context engineering. The latter is crucial for an AI agent to understand how to prioritize its actions.

With some prompt-engineering-fu, taking care of this part shouldn’t be much of a problem.

Technical Excellence

We overwhelmingly focus on AI’s technical excellence. The discourse is about AI capabilities, and we invest effort into improving the reliability of technical solutions. While we shouldn’t expect hallucinations and weird errors to go away entirely, we don’t strive for perfection. In the vast majority of applications, good enough is, well, enough.

Alignment

Alignment is where things become tricky. With AI, it falls to context engineering. In theory, we give an AI agent enough context of what we want and what we value, and it acts accordingly. If only.

The problem with alignment is that it relies on abstract concepts and a lot of implicit and/or tacit knowledge. When we say we want company revenues to grow twice, we implicitly understand that we don’t plan to break the law to get there.

That is, unless you’re Volkswagen. Or Wells Fargo. Or… Anyway, you get the point. We play within a broad body of knowledge of social norms, laws, and rules. No boss routinely adds “And, oh by the way, don’t break a law while you’re on it!” when they assign a task to their subordinates.

AI agents would need all those details spoon-fed to them as the context. That’s an impossible task by itself. We simply don’t consciously realize all the norms we follow. Thus, we can’t code them.

And even if we could, AI will still fail the alignment test. The models in their current state, by design, don’t have a world model. They can’t.

Alignment, in turn, is all about having a world model and a lens through which we filter it. It’s all about determining whether new situations, opportunities, and options fit the abstract desired outcome.

Thus, that’s where AI models, as they currently stand, will consistently fall short.

Explicit Boundaries

Explicit boundaries are all about AI guardrails. It will be a never-ending game of cat and mouse between people deploying their autonomous AI agents and villains trying to break bots’ safety measures and trick them into doing something stupid.

It will be both about overcoming guardrails and exploiting imprecisions in the context given to the agents. There won’t be a shortage of scam stories, but that part is at least manageable for AI vendors.

Care

If there’s an autonomy prerequisite that AI agents are truly ill-suited to, it’s care.

AI doesn’t have a concept of what care, agency, accountability, or responsibility are. Literally, it couldn’t care less whether an outcome of its actions is advantageous or not, helpful or harmful, expected or random.

If I act carelessly at work, I won’t have that job much longer. AI? Nah. Whatever. Even the famous story about the Anthropic model blackmailing an engineer to avoid being turned off is not an actual signal of the model caring for itself. These are just echoes of what people would do if they were to be “turned off”.

AI Autonomy Deficit

We can make an AI agent act autonomously. By the same token, we can tell people in an organization to do whatever the hell they want. However, if we do that in isolation, we shouldn’t expect any sensible outcome. In neither of the cases.

If we consider how far we can extend autonomy to an AI agent from a sociotechnical perspective, we don’t look at an overly rosy picture.

There are fundamental limitations in how far we can ensure an AI agent’s alignment. And we can’t make them care. As a result, we can’t expect them to act reasonably on our behalf in a broad context.

It absolutely doesn’t limit specific and narrow applications where autonomy will be limited by design. Ideally, those limitations will not be internal AI-agent guardrails but externally controlled constraints.

Think of handing an AI agent your credit card to buy office supplies, but setting a very modest limit on the card, so that the model doesn’t go rogue and buy a new printer instead of a toner cartridge.

It almost feels like handing our kids pocket money. It’s small enough that if they spend it in, well, not necessarily the wisest way, it’s still OK.

Pocket-money-level commercial AI agents don’t really sound like the revolution we’ve been promised.

Trust as Proxy Measure of Autonomy

We can consider the combination of transparency, technical excellence, alignment, explicit boundaries, and care as prerequisites for autonomy.

They are, however, equally indispensable elements of trust. We could then consider trust as our measuring stick. The more we trust any given solution, the more autonomously we’ll allow it to act.

I don’t expect people to trust commercial AI agents to great extent any time soon. It’s not because an AI agent buying groceries is an intrinsically bad idea, especially for those of us who don’t fancy that part of our lives.

It’s because we don’t necessarily trust such solutions. Issues with alignment and care explain both why this is the case and why those problems won’t go away anytime soon.

Meanwhile, do expect some hilarious stories about AI agents being tricked into doing patently stupid things, and some people losing significant money over that.
October 16, 2025