Sherman On Software

Category: iterative delivery

  • Yes, Software Execution Has Variation

    Yes, Software Execution Has Variation

    Software development has strongly resisted learning about quality from physical manufacturing.  After all, our product isn’t physical and isn’t bound by the same constraints as the physical world.  One of Deming’s core messages is that quality will increase as variation decreases.  This applies to software too!

    Even a simple CRUD app has tons of variation.

    There are at least a dozen places for variation in a single, successful, RESTful POST:

    1. Time to establish a connection between client and server
    2. Time to send the data between client and server
    3. Time for the event to go up the OSI stack from physical to application layer
    4. Time for the application to process the event.  This is impacted by CPU, Memory, Thread models, etc.
    5. Time for the event to go back down the OSI stack
    6. Time needed to connect to the database
    7. Time database needs to perform the action
    8. Network time to and from the database
    9. Time for the event to go back up the OSI stack
    10. Time for the application to process the database’s response and prepare a response to the client
    11. Time for the event to go back down the OSI stack
    12. Time to send the response to the client

    Most of the time, things like traversing the OSI stack, CPU and memory management take minimal time and have minimal variation.  One major source of variation is when the JVM or CLR enters a garbage collection cycle and halts execution for several seconds.

    Even when every request succeeds, some requests can take 100x or even 10,000x longer than the median.  Does that variation matter?  Sometimes.

    In rare cases, like High Frequency Trading, this variation can be the difference between earning millions and going bankrupt.  Most of the time the variation is a sign of system health and doesn’t matter much in isolation.  But if the variance gets bigger, week after week, release after release, it is a sign that your system has long term health problems.

    Just because the code does the same thing every time, doesn’t mean that it executes the same.  Variation applies to software and we can learn a lot from physical manufacturing.

  • Writing A Run Book Can Be Your First Iterative Step

    Writing A Run Book Can Be Your First Iterative Step

    Writing a Run Book can be your first iterative step towards mitigating recurring problems.  Recurring problems can cause massive productivity problems, but don’t get fixed because the immediate issue is elsewhere.

    For example, background worker systems rarely fail on their own, instead some unique situation will cause the workers to get stuck, the controller to get confused, or the queue to be poisoned.  Each time, there are really two issues.  The bespoke issue that broke the background processes and the recovery of the background worker system.

    Since each new failure is unique, there is a tendency to treat the background system recovery as a unique problem too.  This increases recovery time and prevents you from learning from past mistakes.  Because the bug isn’t in the background system itself, there is often no motivation to spend time on the code.  Fix the bug, restart the system, and move on with your day.

    Enter the run book.

    Write down the steps needed to mitigate the problem.  This is for humans, so it can be an open ended description of what to look for, it won’t be very programmatic.

    Once you have it, keep iterating.  Add code snippets, descriptions, and flow logic.

    As you iterate, you will notice that some parts of the process can be scripted, or even automated.

    Iteration after iteration, more and more of the run book will become code, which makes it easier to code up the remaining pieces.

    Will you be able to iterate the recurring problem out of existence?  Maybe, maybe not.  But with a run book and a plan, you will make progress and not be waiting for the next outage to wreck your day.

  • Patterns of Data Loading – Topics and Broadcast

    Patterns of Data Loading – Topics and Broadcast

    Continuing the discussion of the data loading and messaging, this post is going to cover the tradeoffs to consider when working with Topic and Broadcast based systems.

    Directly Consuming Topics Isn’t Really A Thing

    Topics, Pub/Sub, and other one-to-many messaging systems are an extremely interesting and important abstraction, but they don’t overlap with a discussion of data loading.  The message broker will either use a queue for each client, which is covered here (link), or it is a broadcast, without guarantees. 

    Broadcast 

    Broadcast based systems come without guarantees.  The broadcast server sends the message with no idea about how many listeners want the message, or how many that want it, get it.  There is also no guarantee that messages will arrive in order. 

    These systems are designed to minimize the latency and push all of the queuing onto the receiver.  This is when you need to understand your processor’s OSI Layer, implementation.  It’s the rare time that the amount of memory on the network card might be important!

    But, for the purposes of message processing and data loading, all of those details will be abstracted away.

    To the application, things look very similar to running off a queue.  The application has an ordered list of messages, and it needs to process them as quickly and consistently as possible.

    Broadcast doesn’t have message visibility or the risk of double processing.  Instead the pressure comes from internal queue and buffer management.  The messages are on the processor, if they aren’t processed fast enough you will run out of memory.

    Conclusion

    Topic and Broadcast based messaging systems are extremely powerful systems.  When it comes to data loading however, they tend to be a subset of queue processing.

    The need for consistent performance will push your design to the lower right quadrant. 

  • For Better Iteration, Imagine Heaps, Not Queues

    For Better Iteration, Imagine Heaps, Not Queues

    A new project arrives, you gather the team, and start planning.  You talk through the requirements, write up the work, and put it in order.  Then you iterate!  Repeatedly work on the first task in the queue and keep going until the queue is empty.

    Iterating off of a queue sounds iterative, it’s right there in the description.  But it misses the mark because it ignores everything learned from implementing each piece of work.

    Instead of imagining a work queue, envision a heap. With a heap, you are only guaranteed that the top element is most important.  That’s it, the top, and the rest.  Critically, when you take the top piece of work, you have to sort the heap.  A heap forces you to consider what you’ve learned and apply it to determine what comes next.

    An iterative process requires constant re-evaluation.  There’s the current most important step, and a heap of work.  Iterating off a perfect work queue is a sirin song, an illusion and a trap.  To iterate, learn, and apply your learning, after every step.

  • Explanation – Please Disregard

    Explanation – Please Disregard

    This is a real piece of mail from a real dental insurance company.  I blacked out the PII and company name, and added the highlighting.

    I understand how this can happen.  

    I’m sure they started with the Short-Term disability benefits template and then spent all their time getting the dental benefit information to print correctly.  

    It wasn’t until it was too late that they actually paid attention to the address page.  That’s when they discovered that the address page is hard coded to say SHORT – TERM DISABILITY BENEFITS.

    Tech, product and legal got together and decided that “Please disregard” was the best that they could do.

    Please disregard – Integration came too late in the project.

    Please disregard – Our software isn’t flexible and our deadlines are tight.

    Please disregard – This was the best we could do.

    Explanations: Management doesn’t care about quality.

  • Two Clock Problem Of Rewrites

    Two Clock Problem Of Rewrites

    Rewrites often start off with the assertion that the current code is buggy and can’t be fixed.  The replacement code will fix all of the problems because the technology will be better, the devs have improved, or for any other reason.

    The rewrite project will get off to a great start, but before production, it will run into The Two Clock Problem.

    What is the Two Clock Problem

    The Two Clock Problem occurs when you have two clocks that run at different speeds.  One clock must be wrong.  With only two clocks, you can’t tell which one is wrong.  Even worse, the second clock can also be wrong.

    Rewrites Have Two Clocks

    The first clock is the original system.  The second clock is the rewritten system.  They produce different answers.  The original system was deemed unreliable, so different answers should be a good thing.  Except that different doesn’t mean correct, it means different.  Both systems could be wrong.  Sometimes, the original system is correct.

    Two Clocks and No Confidence

    The two clock problem kills rewrites because there is no way to gain confidence in the new system.  Customers will see that the data has changed, but without an explanation there won’t be trust or confidence.

    Finding the explanations requires finding and fixing all the differences.  Fixing bugs, differing definitions, and edge case logic until the two systems return the same result.

    Finally, when your two clocks agree, then you are ready to release the rewrite.

    Except by that point you don’t need the rewrite anymore.

  • Frosting A Turd And Iterating To A Cupcake

    Frosting A Turd And Iterating To A Cupcake

    Recently a colleague described one of my plans as “Frosting a Turd and Iterating to a Cupcake.”

    It was meant as a pejorative, but I was excited because he finally got it!

    We were discussing a report.  We agreed that the current version was indeed a turd!

    My colleague wanted a full rewrite; I was pushing for TheeSeeShipping, an iterative approach.

    Should we build a new report with a new UI, or reskin (frost) the original report and iterate?

    Should we tell customers that something new is coming, or show our progress and solicit their feedback?

    When you have knowingly served turd to your customers, what are the odds that you can pick the right frosting and cupcake flavor?

    How many turds are you willing to serve while you figure it out?

    Frost the turd every time!

  • Convincing People Not To Touch Hot Stoves Is Easy

    Convincing People Not To Touch Hot Stoves Is Easy

    The hard part is convincing people that what they’re reaching for is a hot stove.

    Having developers work for 6 months and then do a single release is appealing because development proceeds in the straightest line over the shortest distance.  When I push for small releases and iteration, the main pushback is that the incremental releases add work.  The tradeoff appears to be one release after 6 months, or 7 monthly releases.

    Seven months is more than six months, so iteration is more work!

    The problem is that the single release after 6 months of work is “hot stove”.  It’s a plan that needs the requirements to be perfect at the start.  Nothing new can be learned during the project.  Customers and the market have to remain exactly the same.  It’s a plan that will leave you burned.

    It isn’t six months versus seven months.  It is 6 months, plus 3-6 months of rework versus seven months.

    Explaining the risks is ineffective.  People will agree that if the plan is wrong, that if we learn new things, or if the market changes then of course a 6 month plan will end up delivering the wrong thing.  Of course you will then need another 3 months to correct.  That’s obvious.

    After agreeing about the risk, they will then declare that those risks don’t apply to THIS project.  This project isn’t at all like those “hot stoves”.  This project plan is correct and everything has been considered.

    Describing the danger doesn’t work if the problem doesn’t resonate with the listener.  Instead of urging people not to touch hot stoves, work on helping them identify the hot stove.

    • Instead of talking about the risks of a single release, talk about the value of iteration.
    • Don’t talk about the likelihood that the new workflows aren’t perfect, talk about the value of early customer feedback.
    • Don’t talk about the risk that the 3rd party system doesn’t really work like their spec, talk about the value of early integration.
    • Don’t talk about holding back valuable features for a gigantic release, talk about delivering value to customers.

    Once people see the risk and the tradeoffs, you won’t have to convince them not to burn themselves.  After all, everyone understands that you shouldn’t touch hot stoves.

  • AI First API Strategy

    AI First API Strategy

    What does BYO-AI mean for your company’s API strategy?

    This article goes into the obvious, and subtle, changes that need to happen to support customers bringing their own AI.

    All Features Need To Be Fully API Accessible

    AI can only interact with publicly facing endpoints.  Any feature that isn’t publicly facing, or is only accessible through the UI, isn’t accessible to your customer’s AI.

    Embracing your customer’s use of AI means making all of your features publicly available.  Now is time to reevaluate past decisions about excluding features for performance, scalability, security, pricing, and other reasons.

    Customers come to you with problems and AI opens up entirely new avenues for solutions; but only if you let AI into your API.

    All Features Need To Be Documented

    AI is going to learn about your API by reading the docs, reading the example code, and then reading StackOverflow.

    If you want AI to do a good job writing code against your API, you need to do a good job telling it how.  Undocumented endpoints, missing parameters, and bad code examples make that harder.

    Outdated docs and libraries need to be scrubbed from the internet.  It is very much like hiring a reputation company to ensure that what AI sees and knows about your API is what you want it to see and know.

    Technical writers and other DevRel positions are notoriously understaffed.  This is a great time to hire talented writers and get them writing for AI instead of human readers.

    Features and Docs are the Easy Part

    Fleshing out your API and producing up to date, readable docs are things you should have been doing all along anyway.  AI doesn’t change that, it merely raises the cost of poor software hygiene.

    Once you get the table stakes out of the way, things get harder.  Or at least take more effort.

    Assume That AI Has No Idea What It Is Doing

    AI is the ultimate expression of the developer attitude, “I don’t want to understand the business, I just want to code.”  AI can’t understand the context and the customer can’t understand code.  This model works out because this is sort of how most software is written anyway.

    These are the conceptual shifts you need to make with your API to support AI development.

    Make Your API Simple and Verifiable

    Users are going to manually confirm API actions through the UI so your API needs to be simple and easy to verify.  Endpoints that take multiple, open ended collections are great for performance and pushing a lot of data.  

    If they can partially succeed, customers will find them a nightmare as they try to understand the results of the AI’s code.

    Lean on strong validation, all or nothing changes, and understandable error responses.

    Things Are Going To Be Rerun.  A Lot.

    Everything will end up being rerun.  Customers won’t understand data models and won’t understand the risks with rerunning things.  

    Embrace it by making everything as idempotent as possible.  

    Make non-idempotent actions all-or-nothing.  AI is going to get close, a lot.  The last thing your customers expect is for all that flailing to wreck their data.  Bias the endpoints to either fully succeed or to take no action.  Don’t do partial updates and then bail on the request.  Change your processing to do full validation, and make changes only when you know you can make the full change.

    Idempotent and all-or-nothing can be shockingly hard changes to implement.  Fortunately, you can iteratively improve your API!

    Error Messages Need To Be Non-Developer Readable

    Put effort into making your errors human readable and reasonable.  AI is better at grokking text than dense error responses.

    Even better, when something fails validation include a link to the documentation where you’ve explained the correct structure.

    Shifting to An AI First API Takes Time

    Shifting to an AI first API isn’t complicated, but it will take time and effort.

    Start with the fundamentals – make all of your features available through your API, and get your documentation up to date.  Then, if need be, work to make you API AI friendly.

    Customers are going to bring their own AI to your API and do amazing new things that you would never have imagined.  Embrace the change.

  • The Grand Reveal Violates The SaaS Business Model

    The Grand Reveal Violates The SaaS Business Model

    Apple is legendary for the Grand Reveal.  Steve Jobs would say “Just one more thing” and then reveal an entirely new category of device or software.  It was amazing theatrics, shot Apple into the stratosphere, and cemented The Grand Reveal as a technology marketing tactic.

    But for a SaaS The Grand Reveal violates a key principal: customers always have the latest software.  Holding back months of development and new features in order to create an exciting reveal breaks that promise.

    Anytime you’re considering holding back a feature to create buzz ask yourself, does this delay help my existing customers?  You have customers that have already trusted you by buying subscriptions in exchange for the latest software.  Is it fair to hold back, to devalue the service, in order to market to new customers?

    Of course not!

    As a SaaS you’ve promised customers that they will always have the latest software.  Holding back breaks your promise.  Never hold back from your existing customers to market to new ones.  

    Always be releasing!