Tag: software craftsmanship

  • We 3.5x’d Our Pull Requests with AI: Now We Catch Fewer Bugs

    We 3.5x’d Our Pull Requests with AI: Now We Catch Fewer Bugs

    I made a few claims in the past stating that the average size of pull requests went up by an order of magnitude. I largely based it on hallway conversations with developers at Lunar Logic.

    AI pull request size 10x

    It turns out, the actual data is not that hard to check. Aaaand I was wrong.

    AI Effect in PR Size

    We pulled data from two very similar projects in terms of complexity, effort, and team size. Both codebases started greenfield and covered a few hundred pull requests in the analyzed period. There was even an overlap across the engineering team. The key difference? How we used AI.

    The first project, let’s call it Helpful, happened in the pre-Claude era. While we already used AI to support development, it was predominantly autocomplete with occasional trips to ChatGPT to suggest solutions to pesky problems. All code was managed by developers in real time.

    The second gig, I’ll call it Grateful, was full-on Claude. The basic assumption was that none of the code was written by hand. Engineering responsibilities were in context management, prompting, and review.

    The bottom line? In an AI-heavy project, average PR size increased by a factor of 3.5.

    Yes, I was wrong, but only about the relative scale of the change.

    Tiny PRs Have Disappeared

    It would be easy to say that AI made us tackle bigger tasks. That’s not what data suggests, though. At least not when we look at the distribution of the PRs.
    Note: We sum the added and removed lines to calculate the PR size.

    pull request size ai

    The bulk of PRs in both cases are small to medium ones. A few hundred lines of code tops (our cutoff line was 500). It was the bread and butter of engineering work. It still is.

    Sure, in Project Helpful (the “no AI” one), these PRs were skewed toward smaller sizes, while in Project Grateful, the center of gravity was 100-200 lines of code heavier. Still, for an engineer familiar with the codebase, that’s not a challenge.

    So, how come the average went up that much?

    The answer is on the fringes. The smallest pull requests—the proverbial one-liners—all but disappeared. That’s the single most significant change. Tiny PRs were 1 in 5. Now they are 1 in 20.

    The whole class of work items that was easiest to review is at risk of extinction. Let’s park this thought. I’ll be back to it soon.

    Large PRs Are on the Rise

    What we lost from the tiniest bits of work, we make up for with the largest.

    • There are almost 3 times as many large PRs.
    • The 90th percentile size increased 3x, from 600 lines of code to 1799 lines of code.
    • The outliers inflated even more—the largest PR in Project Grateful was 30k+ lines of code, almost 7 times bigger than its equivalent in Project Helpful.
    hundred thousand lines of code pr made by claude code

    Big just got bigger. And we get more of it, too.

    Still, these chunks of code do not dominate the work. Definitely not just yet. However, there are enough of them to start paying attention.

    Coincidentally, this is a class of items that is the most challenging for a reviewer. By now, you can probably guess where it is heading.

    The Effects of Processing 3 Times as Many Lines of Code

    No matter how I slice the data, it seems that we now deal with tasks that are roughly three times as big as they used to be.

    • The average PR size went up from 232 to 817 LoC—a 3.5x increase.
    • The median PR size went up from 66 to 210 LoC—a 3.2x increase.
    • The percentage of big and large PRs went up from 13% to 33%—a 2.5x increase.

    Long story short, our brains process three times as much information per task as they used to. Common sense suggests that the review can’t be as thorough as it was when done in smaller bits.

    defect density versus lines of code in pull request

    Research seems to concur. Well-recognized Smart Bear/Cisco study advises keeping pull request size below 200 lines. Above that size, reviewers start overlooking the issues.

    “Reviewers are most effective at reviewing small amounts of code. Anything below 200 lines produces a relatively high rate of defects, often several times the average. After that the results trail off considerably; no review larger than 250 lines produced more than 37 defects per 1000 lines of code.”

    Executive summary: Heavy use of AI makes individual chunks of work larger, and thus, overall quality drops.

    Quality Drop Is Not Inevitable But Highly Likely

    If we consider the changes, none of them seems inevitable. I mean, we can tell Claude Code to work in smaller chunks so it’s more convenient to review. Heck, we can make it use the annotation technique advised by the Smart Bear study. As a result, we should sustain most of the quality standards.

    There’s only one issue. We won’t do any of these.

    It would require engineers to artificially throttle their coding agents. It would mean more back-and-forth between humans and their tools. It would work against our “laziness” instincts.

    If an agent handles a big task, why should we split it into smaller ones and review them gradually one by one, again? Isn’t it more effective to have the whole thing run at once rather than stopping it each time it approaches 200 lines of code changes? (By the way, it isn’t, but that’s another discussion.)

    Finally, we get the perceived efficiency gains right here, right now, while the cost of lower quality is deferred to the future. Sadly, sticking to the engineering practices that kept the quality high seems highly unlikely.

    Glimpse Into the Future

    We could ask my favorite question: What does the endgame look like?

    • As the capabilities of the models allow them to handle larger and larger coding tasks, the typical pull request size will go up.
    • As a result, reviewers will overlook more and more defects.
    • The fact that reviewers don’t dive deeply into the code conceptually will only exacerbate the quality issue.
    • Thus, we will increasingly develop software riddled with defects.
    • Said defects will add rework for coding agents and humans alike.
    • The pace of delivery of value-adding items will necessarily slow down, as more effort goes into rework (and rework of rework).

    We will go so much faster, only to go as slow as we did in the past. Or slower still. That is, assuming that we stick to the idea of the human reviewer in the loop. And that’s not granted.

    For decades, we tried to learn to work in small batches. The hard way, let me add. Now, with AI, we’re making a U-turn as if none of it mattered. I have bad news. It still does. It was never a software-specific thing. In fact, we stole it from manufacturing in the first place.

    We will relearn small batches. Sooner than we think.

  • Care-Driven Development: The Art of Giving a Shit

    Care-Driven Development: The Art of Giving a Shit

    We have plenty of more or less formalized approaches to development that have become popular:

    I could go on with this list, yet you get the point. We create formalized approaches to programming to help us focus on specific aspects of the process, be it code architecture, workflow, business context, etc.

    A bold idea: How about Care-Driven Development?

    Craft and Care in Development

    I know, it sounds off. If you look at the list above, it’s pretty much technical. It’s about objects and classes, or tests. At worst, it’s about specific work items (features) and how they respond to business needs.

    But care? This fluffy thing definitely doesn’t belong. Or does it?

    An assumption: there’s no such thing as perfect code without a context.

    We’d require a different level of security and reliability from software that sends a man to the moon than from just another business app built for just another corporation. We’d expect a different level of quality from a prototype that tries to gauge interest in a wild-ass idea than from an app that hundreds of thousands of customers rely on every day.

    If we apply dirty hacks in a mission-critical system, it means that we don’t care. We don’t care if it might break; we just want that work item off our to-do list, as it is clearly not fun.

    By the same token, when we needlessly overengineer a spike because we always deliver SOLID code, no matter what, it’s just as careless. After all, we don’t care enough about the context to keep the effort (and thus, costs) low.

    If you try to build a mass-market, affordable car for emerging markets, you don’t aim for the engineering level of an E-class Mercedes. It would, after all, defeat the very purpose of affordability.

    Why Are We Building That?

    The role of care doesn’t end with the technical considerations, though. I argued before that an absolutely pivotal concern should be: Why are we building this in the first place?

    “There is nothing so useless as doing efficiently that which should not be done at all.”

    Peter Drucker

    It actually doesn’t matter how much engineering prowess we invest into the process if we’re building a product or feature that customers neither need nor want. It is the ultimate waste.

    And, as discussions between developers clearly show, the common attitude is to consider development largely in isolation, as in: since it is in the backlog, it has to add value. There’s little to no reflection that sometimes it would have been better altogether if developers had literally done nothing instead of building stuff.

    In this context, care means that, as a developer, I want to build what actually matters. Or at least what I believe may matter, as ultimately there is no way of knowing upfront which feature will work and which won’t.

    After all, most of the time, validation means invalidation. There’s no way to know up front, so we are doomed to build many things that ultimately won’t work.

    Role of Care in Development

    So what do I suggest as this fluffy idea of Care-Driven Development?

    In the shortest: Giving a shit about the outcomes of our work.

    The keyword here is “outcome.” It’s not only about whether the code is built and how it is built. It’s also about how it connects with the broader context, which goes all the way down to whether it provides any value to the ultimate customers.

    Yes, it means caring about understanding product ownership enough to be able to tell a value-adding outcome from a non-value-adding one.

    Yes, it means caring about design and UX to know how to build a thing in a more appealing/usable/accessible way.

    Yet, it means caring about how the product delivers value and what drives traction, retention, and customer satisfaction.

    Yes, it means caring about the bottom-line impact for an organization we’re a part of, both in terms of costs and revenues.

    No, it doesn’t mean that I expect every developer to become a fantastic Frankenstein of all possible skillsets. Most of the time, we do have specialists in all those areas around us. And all it takes to learn about the outcomes is to ask away.

    With a bit of luck, they do care as well, and they’d be more than happy to share.

    Admittedly, in some organizations, especially larger ones, developers are very much disconnected from the actual value delivery. Yet, the fact that it’s harder to get some answers doesn’t mean they are any less valuable. In fact, that’s where care matters even more.

    The Subtle Art of Giving a Shit

    Here’s one thing to consider. As a developer, why are you doing what you’re doing?

    Does it even matter whether a job, which, admittedly, is damn well-paid, provides something valuable to others? Or could you be developing swaths of code that would instantly be discarded, and it wouldn’t make a difference?

    If the latter is true, and you’ve made it this far, then sorry for wasting your time. Also, it’s kinda sad, but hey, every industry has its fair share of folks who treat it as just a job.

    However, if the outcome (not just output) of your work matters to you, then, well, you do care.

    Now, what if you optimized your work for the best possible outcome, as measured by a wide array of parameters, from customer satisfaction to the bottom-line impact on your company?

    It might mean less focus on coding a task at hand, but more on understanding the whys behind it. Or spending time on gauging feedback from users instead of knowing-it-all. Definitely, some technical trade-offs will end up different. To a degree, the work will look different.

    Because you would care.

    Care as a Core Value

    I understand that doing Care-Driven Development in isolation may be a daunting task. Not unlike trying TDD in a big ball of mud of a code base, where no other developer cares (pun intended). And yet, we try such things all the time.

    Alternatively, we find organizations more aligned with our desired work approach. I agree, there’s a lot of cynicism in many software companies, but there are more than enough of those that revolve around genuine value creation.

    And yes, it’s easy for me to say “giving a shit pays off” since I lead a company where care is a shared value. In fact, if I were to point to a reason why we haven’t become irrelevant in a recent downturn, care would be on top of my list.

    care transparency autonomy safety trust respect fairness quality
    Lunar Logic shared values

    But think of it this way. If you were an aerospace industry enthusiast, would you rather work for Southwest or Ryanair? Hell, ask yourself the same question even if you couldn’t care less about aerospace.

    Ultimately, both are budget airlines. One is a usual suspect when you read a management book, and they need an example of excellent customer care. The other is only half-jokingly labeled as a cargo airline. Yes, with you being the cargo.

    The core difference? Care.

    Sure, there is more to their respective cultures, yet, when you think about it, so many critical aspects either directly stem from or are correlated with care.

    Care-Driven Development

    In the spirit of simple definitions, Care-Driven Development is a way of developing software driven by an ultimate care for the outcomes.

    • It encourages getting an understanding of the broad impact of developed code.
    • It drives technical decisions.
    • It necessarily asks for validating the outcome of development work.

    It’s the art of giving a shit about how the output of our work affects others. No more, no less.

  • Code Better or Code Less?

    An interesting discussion (that might have happened):

    I would rather students apply their effort to writing better code than to writing better comments.

    ~ Bob Martin

    But…

    I would rather students apply their efforts to writing less code than writing “better” code.

    ~ Bob Marshall

    Because…

    There is nothing so useless as doing efficiently that which should not be done at all.

    ~ Peter Drucker

    Having read this, one realization is that better code often means less code. I don’t think about lines of code exactly, or something similarly stupid, but in terms of meaningful code. However, argument for less code isn’t about making code as compact as possible, avoid redundancy, etc.

    The argument is about not writing code at all whenever reasonable or possible.

    Should we focus on deciding what should and what should not built instead of polishing our software development craft then?

    Yes and no.

    Yeah, I know. Exactly the kind of answer you expected, isn’t it? Anyway, you can’t answer this question meaningfully without a context.

    Better code

    One perspective is the one of a developer. The developer in almost every medium-to-big organization, and in quite a lot of small ones too, is pretty much disconnected with product management/product ownership part of a project. It means that they have very little-to-no knowledge what actually should be built.

    Of course being a developer I can, and should, share my concerns about usefulness of building specific features but it’s unlikely I have enough information to judge such situations correctly in many cases. By the way, even when I’m right and this darn feature shouldn’t be built odds are that it’ll be built anyway because a client says so. Sounds stupid? Sure, it does! Does it make the client change their minds? Not very often.

    If you’ve ever worked on one of those big contracts where everything is (allegedly) specified upfront and no one on a client’s side is willing to change anything because of internal politics, you exactly know what I’m talking about. If you haven’t, well, damn you, you lucky bastard.

    So it might be a great idea not to build a feature but developers either don’t have enough knowledge to be aware of the fact or aren’t allowed to skip the feature anyway. In this case a better thing to do is to focus on building better code, not less code, because one can hardly say what meaningful less is.

    Less code

    The other perspective is the one of product management folks, however this specific role is called in your org. For them, their first and main focus should be on building less code. Yes, product owners, product managers, etc. Yes, less code. And yes, I do know they don’t write code. It still should be their main goal.

    You see, this is the place where meaningful decisions about not building features can be made. Product folks should know what adds value and what doesn’t. What’s more, they are usually better suited to start such discussions with clients, whenever needed. After all, it is so common that clients want, and pay for, unnecessary features and useless code.

    Organizational-wise you get more value, or less waste, focusing on building less code. Given that you’re free to work on both: better code and less code across the organization, it would likely be wiser to choose the latter. At the same time efficiency of your efforts depends much on the part of the organization you work with and, locally, it may be a much better choice to focus quality of code and not quantity of code as an issue to tackle.

    So if I could choose what kind of superhero posters are in rooms of my people I’d go with Peter Drucker for product folks and Bob Martin for developers.