Sherman On Software Latest

  • You’ve Built A Bad Implementation Of Other Software, Now What?

    You’ve Built A Bad Implementation Of Other Software, Now What?

    The immediate response to 20 Things You Shouldn’t Build At A Midsize SaaS was, what do you do if you HAVE built some of these things?  Great question!

    First, don’t panic.  We’ve all been there.  I have built some of these things myself – multiple CSV writers and parsers, a DateTime library, and an ORM.  We are all human, and we all make mistakes from time to time.

    Whether you built, or are responsible for, a bad implementation of other software, what do you do now?  It depends on which of 3 buckets the implementation falls into.

    It works and you don’t need to touch it

    CSV parsers and writers get written a lot because they are very simple.  Yes there are edge cases when it comes to escaping strings and unicode, but mostly once you have it working, it’s “fine”.

    If you don’t need to touch it, don’t!  Let sleeping dogs lie.  Ideally put a comment in the code that the next dev should replace it with a standard library instead of expanding the code.

    It works ok, but it’s incomplete

    This is the biggest bucket.  Implementations of DSLs, ORMs, and Frameworks usually fall into this group – the software works ok, but it is missing key features and robustness.  Observability is usually extremely lacking.

    This makes the software low priority tech debt.  There’s opportunity cost from continuing to use it, a cost to replacing it, but no direct development costs.

    Generally speaking, this is the kind of thing that gets ignored until you need a missing key feature.  Then the debate between continuing with the local implementation and replacing it, resumes.

    The best way to approach these situations is to look at the interfaces and requirements for the replacement software, and converge the local implementation over time.  This combines regular maintenance with replacement work.   As a bonus, it lowers the cognitive load on developers by making the local implementation work more like standard offerings.

    It requires continuous maintenance

    Some kinds of software require continuous maintenance.  When you write your own, you have to do all the maintenance yourself.

    Scripting languages are a wonderful source of Halting Problem issues.  Your customers will be excellent at creating scripts that will never exit and will consume resources until the server restarts.  The more successful you are, the worse the problem will become.

    I saw a SaaS with 2 full time developers working to close all of the ways that customers were accidentally creating infinite loops with their homegrown scripting language.

    I worked at a company using objective-c which had extended Apple’s compiler.  This was back in the days of installed, on-prem, software.  Every time Apple had a release, we had to warn customers not to upgrade until we could re-extend the compiler, recompile, and ship the code.

    When you encounter implementations that require continuous maintenance, you have to start working to remove them immediately.  The longer they remain, the more entrenched they become.  The sunk cost fallacy will work against you in strange and insidious ways.

    Iterative removal is the only real option in these cases.

    No matter which bucket the badly implemented software falls into, the first step is always to recognize the problem.  If there’s already an implementation, you shouldn’t write your own without a good reason.

  • The Never Rewrite Podcast, Episode Eighty-Nine, How Sonos Lost $500 Million Attempting a Rewrite

    The Never Rewrite Podcast, Episode Eighty-Nine, How Sonos Lost $500 Million Attempting a Rewrite

    How did Sonos lose $500 Million and trash their brand; with a disastrous app rewrite of course! Isaac and I discuss the Sonos disaster and how well it fits into our model of a rewrite. Was Sonos unlucky, or was this disaster nearly guaranteed?

    Watch on YouTube or listen to it at Spotify, Apple Podcasts, or your favorite podcast app, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!

  • 20 Things You Shouldn’t Build At A Midsize SaaS

    20 Things You Shouldn’t Build At A Midsize SaaS

    I have seen developers build a lot of unnecessary and counterproductive pieces of software over the years.  Generally, developers at small to midsize SaaS companies shouldn’t build any software that doesn’t directly help them deliver a service to their customers.

    Whether it was the zero interest rate period, bad management, or hubris, developers spent a lot of company money on projects that never made sense given their employer’s goals and size.  I have seen custom implementations of every type of software on this list.  None of it worked better than open source, and none offered a competitive advantage.

    If you find yourself developing or managing any of these twenty types of projects, stop and seriously consider what you are doing.

    1. Scripting languages
    2. Compiler extensions
    3. Transpilers
    4. Database extensions
    5. Databases
    6. DSLs
    7. ORMs
    8. Queues
    9. Background work schedulers
    10. GraphQL
    11. Stateful REST
    12. Frontend Frameworks
    13. Backend Frameworks
    14. Servers
    15. Dependency Injectors
    16. CSV writers or parsers
    17. Cryptography Implementations
    18. Logging Libraries
    19. DateTime libraries
    20. Anything from “First principles”

    There are always exceptions, if building this software has some competitive advantage, go ahead.  In general, anyone suggesting these projects is biting off more than they can chew and doesn’t fully understand the problem they are trying to solve.

    Most often things start out as a quick hack – “I’ll just concatenate these strings with a comma, it will be faster than finding a full CSV library.”  Soon you’re implementing custom separators and string escaping.

    If your company has done their own implementations don’t despair, iterate towards a better library!

  • Learning When To Stop Developing A Project

    Learning When To Stop Developing A Project

    Robert Moses used lies and trickery to ensure that his projects were completed.  He loved to start projects and let pride, politics, and sunk costs pull them to completion.  Software development is notorious for grinding on and delivering projects years late that don’t solve the original problem.

    SaaS project deadlines are artificial, usually the only thing that can stop a project is completion.  Even projects with no developers will shamble on, zombie-like, eating a bit of everyone’s brain as they stumble through the code.

    When you find yourself confronted with a long lived, poorly defined project, start asking questions:

    1. Was there a time element to the project, and has it passed?
    2. Have the assumptions behind the project changed?
    3. Have the company’s goals changed?

    Was there a time element to the project, and has it passed?

    Calendars are cyclical; it’s technically never too late or too early to get ready for your industry’s high volume period.  But if you see a project to scale for Black Friday, in January, there’s a good chance that you don’t need to finish it.

    Your company got through Black Friday without the project, why do you need it now?

    Have the assumptions behind the project changed?

    I have been involved with many “if we switch from technology X to Y, we can save a lot of money” type projects.  Less than half produced significant cost savings.  In most of the cases we knew that the savings wouldn’t materialize early in the project.  The projects kept going anyway.

    It is much easier to rationalize “the cost savings may not be there, but technology Y is better” than to work out if the new tech justifies the project on technical merits.

    Have the company’s goals changed?

    Long running, nebulous, projects run the risk of having the company’s goals change.  I “increase performance” and “scale the system”.  But when the customer profile changes, I am often scaling the wrong part of the system.

    Learning To Question Is The First Step

    Before you can stop a project, you have to question the project.  Question the timing, the assumptions, and the company’s goals.

    If the answers are no, it might be time to stop development.

  • The Never Rewrite Podcast, Episode Eighty-Eight, Creating a Culture of Continuous Improvement with Guest Erik Enge

    The Never Rewrite Podcast, Episode Eighty-Eight, Creating a Culture of Continuous Improvement with Guest Erik Enge

    This week, Isaac and I are joined by Erik Enge, COO of Ghost, to talk about creating a culture of continuous improvement within organizations. Erik talks us through learning loops; how to apply the scientific method in business, setting measurable goals, and reflecting on the outcomes. We work through some real-world examples to illustrate how continuous improvement and learning loops can lead to better decision-making and improved outcomes.

    Watch on YouTube or listen to it at Spotify, Apple Podcasts, or your favorite podcast app, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!

  • Robert Moses On Completing Projects

    Robert Moses On Completing Projects

    “Once you sink that first stake, they’ll never make you pull it”
    – Robert Moses, The Power Broker by Robert A Caro, page 207

    Robert Moses built most of the highways, parkways, bridges, tunnels, and parks in and around New York city.  His most effective tool was to lie about the costs so that he could start the project.  Once the project was started he knew that the approver’s refusal to admit a mistake and the sunk cost fallacy would allow the project to continue to completion.

    Moses’s method is a clear moral hazard – start a project and the approver has to help complete it, or admit that they made a mistake by giving the initial approval.

    Does the project cost more than expected?  Keep going.

    Will the project take longer than expected?  Keep going.

    The project no longer makes sense?  Learned that the project won’t solve the problem?  That the solution isn’t cost effective?  Keep going.

    Depending on which article you read software 45-90% of software projects are late, cost 15-50% more than expected, and have a 20% chance of not delivering the expected value.

    I once joked about a company that “started projects on time” but never had the resources to finish anything.  Unfinished projects drain resources and make it harder to finish any other project.

    Because as Robert Moses knew, once you start, projects keep going.

  • Do You Need Permission To Write Quality Code?

    Do You Need Permission To Write Quality Code?

    A developer recently confided in me, “I wanted to write good code on this project, but when I asked my manager he said we needed to do whatever it took to hit our delivery date.”  He was explaining why he shoved his changes into the existing, giant, untestable, functions instead of refactoring and writing tests.

    He asked his manager for permission to write quality code, when he didn’t get it he wrote shit code, and he missed the delivery date.

    Asking permission to write quality code is the same faulty thinking that leads managers to skip testing, the idea that low quality is faster.  Low quality code is just low quality, developers won’t create it faster than high quality code.

    I wrote lousy code in college because I had no idea what I was doing.  I wrote lousy code for my first year as a professional programmer because I still had no idea what I was doing.  After being on the job for about 18 months I knew I was writing lousy code but I did it anyway because that’s what I was used to doing.

    After about 18 months I started copying the more effective developers and writing higher quality code.  My velocity increased with quality.  By the end of my second year as a professional programmer I was writing high quality code because that was the fastest way for me to deliver results.

    When you ask for permission, you are asking for someone else to take responsibility for the decision.  If developers ever ask for permission it is a sign that they don’t believe in quality, or worse, that they don’t believe that their manager believes.

    If you’re a manager and ever get asked for permission to “do it right”, ask yourself, “what has gone wrong, and how do I fix it?”

  • The Never Rewrite Podcast, Episode Eighty-Seven, 2025 Update – State of AI Programming

    The Never Rewrite Podcast, Episode Eighty-Seven, 2025 Update – State of AI Programming

    This week, Isaac and I discuss advancements of AI in coding and whether it is ready to replace junior developers. We cover the evolving role of developers and the implications of AI tools on productivity. Does AI represent the end of developers, or the end of how developers have worked for the past 50 years?

    Watch on YouTube or listen to it at Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!

  • Abandoned Wells And Other Dangers In Your Codebase

    Abandoned Wells And Other Dangers In Your Codebase

    Giant, long lived, codebases are full of code that should never be used for anything new.  Not only shouldn’t the code be used for anything new, it shouldn’t be used at all; but if it ain’t broke, don’t fix it.

    The brownfield codebase gets littered with mineshafts, wells, and other hazards from products that were discontinued or never made it into production.  Will the shaft collapse if you start digging?  Maybe.  Are those abandoned wells the right size to swallow new developers?  Absolutely.

    How do you move forward when your codebase has become a hazardous environment?

    When a developer hangs the UI for your biggest customers because a reporting function is slow; is that the UI developer’s fault for not testing well enough, the report developer’s fault for not adding a warning about the function’s performance, or a sign of a dangerous codebase?

    Once your codebase becomes hazardous it is everyone’s responsibility to be cautious, and everyone’s responsibility to warn of danger.  Develop as a team and watch each other’s backs, otherwise you might end up stuck in a well.

  • The Anna Karenina Principle of Scaling Software Systems

    The Anna Karenina Principle of Scaling Software Systems

    The Anna Karenina Principle says “All happy families are alike; each unhappy family is unhappy in its own way.”  The same is true for software scalability.  All scalable systems are alike; each unscalable system is unscalable in its own way.

    Scalable Systems Scale Linearly

    Scaling software systems means doing more work with more resources.  As software scales various state management issues will require ever more resources.  In the beginning processing 2x more requests will require less than 2x more resources.  Over time, the ratio will become 1-to-1 and continue to slide.  2x more requests will require 4x more resources.

    So long as the ratio is linear, the system is scalable.  When the ratio becomes exponential, for example 2x more requests require x^2 more resources, the system is no longer scalable.

    Scalable Systems Are Fault Tolerant

    A system with 5 nines will process 99.999% of all events correctly.  Failures are 1 in a million.  A system doing 1 million events per second will have a failure every second.

    Scalable systems have defined ways of handling failure – they will return an error and make the caller handle it, they will do retries, they may even fail silently.  What they won’t do is crash or error out.

    Scalable Systems Are Idempotent

    Exactly once delivery is nearly impossible.  As a system scales, it becomes inevitable that some events will be processed multiple times.

    Scalable systems are idempotent and indifferent to how many times an event is processed.  It doesn’t matter how many times an event enters the system, once, twice, 100 times, it is all the same.

    Non-idempotent systems are much easier to build, but they will fail in all kinds of different ways as customers and networks send in duplicate events.

    Scalable Systems Have Minimal Human Interaction

    Humans are expensive and do not scale.  The more human intervention a system requires, the less scalable it is.

    Scalable systems do not require humans to deploy code, scale up or down, or watch the logs.

    Conclusion

    Remember, all scalable systems are alike:

    1. Linear Scaling
    2. Fault Tolerant
    3. Idempotent
    4. Require Minimal Human Interaction

    Systems that are not scalable are not scalable in their own unique way.

    If you don’t know why your system isn’t scalable, these 4 points are a great place to start looking.