Sherman On Software Latest

  • Calling The Baby Ugly Won’t Short Circuit The Emperor’s New Clothes 

    Calling The Baby Ugly Won’t Short Circuit The Emperor’s New Clothes 

    You can’t point out that the Emperor Has No Clothes until the emperor puts on his invisible garments.  Pointing out the problems before it is too late, is Calling The Baby Ugly.  Incomplete software can be “ugly” right now and still be great by the time it is released.

    That potential, it might still be fine, makes it difficult to short circuit the story as it unfolds.  Software literally emerges from thin air as developers work.  Pointing out that the pants only have one leg, the response can be that the second leg is coming in phase 2.

    By itself, pointing out the flaws will not short circuit the process.  Flawed architecture, wrong technology, and hostile UX don’t matter until it is too late.  Nothing is real until it is time to put on the clothes.  Calling the baby ugly makes you Cassandra, doomed to be correct and ignored.

    The key to the Emperor Has No Clothes is that the truth emerges when the software meets reality.  The Grand Reveal is a mistake and totally unnecessary.  The baby may be ugly, but the Emperor can only have no clothes when he puts on a whole new wardrobe for the first time.

    Iterative delivery short circuits the process because software is constantly going into production.  Don’t bet on the Emperor’s New Clothes, send him out with a new jacket and find out if people see it, or the regular shirt underneath.

  • The Never Rewrite Podcast, Episode Fifty-Nine, The Myth of the Boy Scout Rule

    The Never Rewrite Podcast, Episode Fifty-Nine, The Myth of the Boy Scout Rule

    Isaac and Jeffrey discuss the myth of the Boy Scout rule, which is the idea of leaving code better than you found it. Isaac shares his experience of encountering problems when trying to follow this rule in a codebase with no test coverage. He explains how fixing one issue led to unintended consequences and a cascade of errors. They emphasize the importance of being cautious when making changes in old and tightly coupled code and the need for thorough testing. They conclude that while refactoring is important, it should be done intentionally and not mixed with other changes.

    Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!

  • Common Cause Vs Special Cause in Software Variation

    Common Cause Vs Special Cause in Software Variation

    In Yes, Software Execution Has Variation, I laid out a dozen places where successful software will have variation:

    There are at least a dozen places for variation in a single, successful, RESTful POST:

    1. Time to establish a connection between client and server
    2. Time to send the data between client and server
    3. Time for the event to go up the OSI stack from physical to application layer
    4. Time for the application to process the event.  This is impacted by CPU, Memory, Thread models, etc.
    5. Time for the event to go back down the OSI stack
    6. Time needed to connect to the database
    7. Time database needs to perform the action
    8. Network time to and from the database
    9. Time for the event to go back up the OSI stack
    10. Time for the application to process the database’s response and prepare a response to the client
    11. Time for the event to go back down the OSI stack
    12. Time to send the response to the client

      Let’s say that your SaaS has an SLA that all calls should return to the customer within 300ms.  Looking at the system metrics, you see that your endpoint meets the SLA 95% of the time.  What to make of the remaining 5%?

      Common Cause vs Special Cause Variation

      Common Causes are issues due to the nature of the system and will continue until the system is changed.  In software, Special Causes are bugs and hiccups and will appear and disappear at random.

      For our RESTful POST that needs to return in under 300ms, a source of Common Cause could be the physical distance between the client and server.  If you are running on AWS in US-EAST-1, it is physically impossible to meet the SLA for customers in Asia.  Round trip to places like Seoul and Tokyo is at least 450ms!

      Requests from Asia will fall outside the SLA 100% of the time.  The only solutions are to change the SLA or stand up an instance of your system closer to your Asian customers.  You must change the system.

      An example of a Special Cause could be an overloaded database.  Some requests will be fine, others will break the SLA.  The issue will go away once load decreases.  The answer may be to change the system by increasing the size of the databases.  Or keep the system the same and change the software to make the database insert asynchronous.  The software change decouples the SLA from database performance.

      Deming’s Path Of Frustration

      Fixing all the Special Causes won’t solve all the problems.

      Knowing that 5% of your requests break the SLA doesn’t tell you anything about Common vs Special Cause.  Developers can fix some of the problems with software, but some can only be fixed by changing the system architecture.

      Until you analyze and determine the source of your variation, you’ll be stuck on the Path Of Frustration.  Pouring ever more resources into ever smaller gains.

    1. Yes, Software Execution Has Variation

      Yes, Software Execution Has Variation

      Software development has strongly resisted learning about quality from physical manufacturing.  After all, our product isn’t physical and isn’t bound by the same constraints as the physical world.  One of Deming’s core messages is that quality will increase as variation decreases.  This applies to software too!

      Even a simple CRUD app has tons of variation.

      There are at least a dozen places for variation in a single, successful, RESTful POST:

      1. Time to establish a connection between client and server
      2. Time to send the data between client and server
      3. Time for the event to go up the OSI stack from physical to application layer
      4. Time for the application to process the event.  This is impacted by CPU, Memory, Thread models, etc.
      5. Time for the event to go back down the OSI stack
      6. Time needed to connect to the database
      7. Time database needs to perform the action
      8. Network time to and from the database
      9. Time for the event to go back up the OSI stack
      10. Time for the application to process the database’s response and prepare a response to the client
      11. Time for the event to go back down the OSI stack
      12. Time to send the response to the client

      Most of the time, things like traversing the OSI stack, CPU and memory management take minimal time and have minimal variation.  One major source of variation is when the JVM or CLR enters a garbage collection cycle and halts execution for several seconds.

      Even when every request succeeds, some requests can take 100x or even 10,000x longer than the median.  Does that variation matter?  Sometimes.

      In rare cases, like High Frequency Trading, this variation can be the difference between earning millions and going bankrupt.  Most of the time the variation is a sign of system health and doesn’t matter much in isolation.  But if the variance gets bigger, week after week, release after release, it is a sign that your system has long term health problems.

      Just because the code does the same thing every time, doesn’t mean that it executes the same.  Variation applies to software and we can learn a lot from physical manufacturing.

    2. Writing A Run Book Can Be Your First Iterative Step

      Writing A Run Book Can Be Your First Iterative Step

      Writing a Run Book can be your first iterative step towards mitigating recurring problems.  Recurring problems can cause massive productivity problems, but don’t get fixed because the immediate issue is elsewhere.

      For example, background worker systems rarely fail on their own, instead some unique situation will cause the workers to get stuck, the controller to get confused, or the queue to be poisoned.  Each time, there are really two issues.  The bespoke issue that broke the background processes and the recovery of the background worker system.

      Since each new failure is unique, there is a tendency to treat the background system recovery as a unique problem too.  This increases recovery time and prevents you from learning from past mistakes.  Because the bug isn’t in the background system itself, there is often no motivation to spend time on the code.  Fix the bug, restart the system, and move on with your day.

      Enter the run book.

      Write down the steps needed to mitigate the problem.  This is for humans, so it can be an open ended description of what to look for, it won’t be very programmatic.

      Once you have it, keep iterating.  Add code snippets, descriptions, and flow logic.

      As you iterate, you will notice that some parts of the process can be scripted, or even automated.

      Iteration after iteration, more and more of the run book will become code, which makes it easier to code up the remaining pieces.

      Will you be able to iterate the recurring problem out of existence?  Maybe, maybe not.  But with a run book and a plan, you will make progress and not be waiting for the next outage to wreck your day.

    3. The Never Rewrite Podcast, Episode Fifty-Eight, When You Can’t Iterate

      The Never Rewrite Podcast, Episode Fifty-Eight, When You Can’t Iterate

      Isaac and Jeffrey discuss situations where software development teams are pushed to have fewer, larger releases. They highlight the challenges of big release-driven development and the need for careful planning and more QA before each release. They also explore strategies to make the problem less severe, such as minimizing the final push, using an API-first strategy, and feature flagging.

      Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!

    4. Patterns of Data Loading – Topics and Broadcast

      Patterns of Data Loading – Topics and Broadcast

      Continuing the discussion of the data loading and messaging, this post is going to cover the tradeoffs to consider when working with Topic and Broadcast based systems.

      Directly Consuming Topics Isn’t Really A Thing

      Topics, Pub/Sub, and other one-to-many messaging systems are an extremely interesting and important abstraction, but they don’t overlap with a discussion of data loading.  The message broker will either use a queue for each client, which is covered here (link), or it is a broadcast, without guarantees. 

      Broadcast 

      Broadcast based systems come without guarantees.  The broadcast server sends the message with no idea about how many listeners want the message, or how many that want it, get it.  There is also no guarantee that messages will arrive in order. 

      These systems are designed to minimize the latency and push all of the queuing onto the receiver.  This is when you need to understand your processor’s OSI Layer, implementation.  It’s the rare time that the amount of memory on the network card might be important!

      But, for the purposes of message processing and data loading, all of those details will be abstracted away.

      To the application, things look very similar to running off a queue.  The application has an ordered list of messages, and it needs to process them as quickly and consistently as possible.

      Broadcast doesn’t have message visibility or the risk of double processing.  Instead the pressure comes from internal queue and buffer management.  The messages are on the processor, if they aren’t processed fast enough you will run out of memory.

      Conclusion

      Topic and Broadcast based messaging systems are extremely powerful systems.  When it comes to data loading however, they tend to be a subset of queue processing.

      The need for consistent performance will push your design to the lower right quadrant. 

    5. The Never Rewrite Podcast, Episode Fifty-Seven, White Label CRM Updates with Dustin Rea

      The Never Rewrite Podcast, Episode Fifty-Seven, White Label CRM Updates with Dustin Rea

      We check in with Dustin Rea, head of Red Hook Agency, to discuss where White Label CRM is with their ‘turnaround story’ – how they’re handling ‘bug hell’, scaling issues, and what their trajectory is now.

      Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!

    6. The Other 94%, Why Getting Better At Your Craft Isn’t Economical

      The Other 94%, Why Getting Better At Your Craft Isn’t Economical

      I saw an ad on LinkedIn with the hook “Frustrated by getting 200% better at your craft and only getting a 6% raise?”  The ad seemed to be selling grievances, and a course on becoming a freelance developer.

      What the ad wasn’t doing was disagreeing with the idea that getting 200% better at your primary skill doesn’t make you more valuable or productive.  A 6% raise for improving 200% implies that 94% of your job isn’t your primary skill.  Or that your employer can’t use the marginal value of you getting better.  Becoming a better programmer, by itself, isn’t very valuable to whomever pays your salary.

      The ad was trying to do a sleight of hand: Improve your pay by becoming a freelance developer!  By learning skills that don’t improve your craft!  Being a successful freelancer means learning business, marketing, and sales skills.  The same skills the ad starts off by bashing.

      If someone is paying you to be a developer, chances are that becoming a better developer will have diminishing returns.

      You can be angry, or you can learn about the other 94% of what makes a developer valuable.

      Learn your industry, your company’s business model, and business strategy.  Learn how to be a better teammate, to lead, to inspire, and help others to grow.

    7. The Never Rewrite Podcast, Episode Fifty-Six, AI-Driven Testing

      The Never Rewrite Podcast, Episode Fifty-Six, AI-Driven Testing

      Isaac and Jeffrey discuss the potential of AI-driven testing and code generation. They reference an article by Codium AI, which explores the use of AI to generate tests and increase code coverage. While the generated tests may not be perfect, they can serve as scaffolding for legacy code and help identify areas for improvement. The conversation also touches on the idea of using AI to provide business context and legal compliance guidance during code refactoring. Overall, the discussion highlights the potential benefits of AI in improving code quality and reducing risk.

      Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!