Sherman On Software Latest

  • State, Persistence, And Cold Restarts

    State, Persistence, And Cold Restarts

    In a SaaS, state exists in many places, some of them outside of your control.  If you don’t know where to look, you will never truly understand your system.

    State You Can Control

    As systems grow in complexity, state creeps into an ever growing number of places:

    • Databases
    • Caches
    • Running Software
    • Config Files
    • Shell scripts
    • Flat files

    Do your caches have state?  If they went down and it impacted anything other than performance, then they have state.

    Do your shell scripts have state?  If they reference specific customers then they certainly do.

    The list goes on.

    Some of your state has backups, like databases, source code, and some config.  Your only backups for running software and caches are more instances of running software and caches.

    State You Can’t Control

    State also creeps into things that aren’t software, and you can’t control:

    • Tribal knowledge
    • Manual Procedures
    • Anything in someone’s head that isn’t documented

    People are a critical, and critically overlooked, part of any complex system.  The most statefull people aren’t the programmers.  It’s the operations and support people that interact with customers that have the most state and least backup.

    Disaster Recovery and Cold Start

    A cold restart is the hardest kind of disaster recovery.  You have all new server instances, whatever is in source control, and whatever is in your backups.

    Things that live in people’s heads are even harder to recover.  Which repos are important, which services are supposed to run, what does the system even look like?

    Depending on your setup you may be ok, you may never recover.

    I was at GuaranteedRate when it acquired Discover’s Home Loan operations.  As part of the acquisition we got their mortgage software and their developers, but no state.  I watched the team of developers that created the system spend a year trying to do a cold restart.  They failed.

    Consider State Now, Before It Gets Away From You

    State grows organically with systems and keeping it under control requires effort.  Beyond the fear of disaster recovery, knowing where state exists in your system is key to maintenance and growth over time.

    Once you start looking for state in your systems, you’ll find it everywhere.

  • The Technical Problems You Should Solve At A Midsize SaaS

    The Technical Problems You Should Solve At A Midsize SaaS

    20 Things You Shouldn’t Build At A Midsize SaaS was about technical problems developers at a midsize SaaS shouldn’t try to solve.  Is midsize SaaS development all glue work?  Have all the problems been solved?

    Of course not.  Midsize SaaS is the garden of Refactoring, Scaling, and Performance.

    To make it sound fancy: The deep work for developers at a midsize SaaS is designing solutions for emergent architectural problems.

    A Midsize SaaS Has Different Problems Than A Startup SaaS

    Startups have unproven theories about what customers want.  They need to get features out as quickly as possible to test theory against reality.  Worrying about multiple data centers, global latency, or the performance of features customers can’t see, is a waste of time.

    At a startup you should write good code, find product market fit, and don’t worry about how the system will perform when you have 10,000 paying customers.

    Once you have thousands of paying customers, that’s when architectural gardening kicks in.

    How To Support What Customers Want

    The startup phase will leave you with a valuable product and an almost random set of assumptions.  You get to puzzle out the assumption, the reality, and choose solutions.

    If your systems are in the United States, and all of your customers are in the United States, you will have different architecture needs than if your customers are globally distributed.  Linear and exponential scaling produce different problems and require different solutions.

    You need to identify which problems you have, and iterate towards standard solutions.  Standard solutions are critical because it makes your competitive advantage, the differences that are valuable, shine through.  You can’t find the valuable unique differences when everything in your system is bespoke. 

    Conclusion

    The deep work at a midsize SaaS is identifying emerging problems and iterating towards solved solutions.  Pathfinding from wherever the startup phase has left you towards known destinations.  Moving towards known standard solutions makes it easier to find and improve valuable differentiators.  Building unique versions of everything makes everything harder without adding value to your customers.

  • The Never Rewrite Podcast, Episode Ninety-One, Pushing Pain to Users

    The Never Rewrite Podcast, Episode Ninety-One, Pushing Pain to Users

    his week, Isaac Askew and I work through the implications of externalizing costs in software development. What are the consequences of pushing pain to users? How do externalities impact customer experience and support? Is prioritizing features over user satisfaction ever a good tradeoff?

    Find out the answers to these and other burning questions in this week’s episode of NeverRewrite!

    Watch on YouTube or listen to it at Spotify, Apple Podcasts, or your favorite podcast app, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!

  • How To Say No As A Staff Engineer

    How To Say No As A Staff Engineer

    I say “no” to tech plans.  A lot of tech plans.  Most of the time, if you ask my opinion about a tech plan, I’m going to tell you not to do whatever you’re planning.  If product owners are involved the odds of me saying no go way up.  And yet, few people dread asking me about their plans.

    Staff Engineers can’t just say “no” and leave it at that.  The power of no comes with responsibility.  When I say “no”, I have to explain why, offer alternatives, and make myself available to work through emergent issues.  When product owners are involved it is my responsibility, as a Staff Engineer, to help separate the product goals, which are fine, from the solution, which got a “no”.

    Let’s break it down.

    Explain Why

    My expertise is in Scaling and Performance.  I say no when the solution won’t scale, won’t perform well, or has problematic edge cases.  Often, something that works fine for the average customer won’t work well for the largest 10% of customers.

    When I say no to a design that won’t scale, I speak to real numbers.  “That’s fine for a table with 1,000 rows.  We have customers with a million rows in that table.  They’ll be waiting minutes for a response from the backend.”

    Offer Alternatives

    Decades of experience doesn’t make me right, but it does give me a lot of experience to draw on.  There were specific problems in the design that caused me to say no.  Those problems are solvable.  The solutions may not be appealing, but they exist.

    If you have data centers all over the world, a central database full of global data is going to be a problem.  There are lots of alternatives depending on why you wanted global data in the first place.  Alternatives are a conversation, not prescription.

    Be Available For Emergent Issues

    Now that I’m part of the solution, I’m responsible for helping with emergent issues.  As the project continues, new constraints will emerge and new requirements will be discovered.  Some of the emergent issues are going to impact Scale and Performance.

    Things change and the old recommendations may become invalid.  Staff Engineering recommendations have to be living documents.

    Saying No Should Be More Work!

    It would be much easier to say “Looks good to me!”  If I pretend to be on board I’m no longer responsible.  I have sabotaged the asker, but they’ll be too busy fighting endless fires to realize that I could have stopped everything from going wrong.

    For Staff Engineers, saying no to a proposal comes with an implied offer to help.

  • Iteration Is Not A Speed, It’s A Journey

    Iteration Is Not A Speed, It’s A Journey

    Iterative development is faster, but iteration isn’t a speed, it’s a journey.  Apple iterates the iPhone annually.  So do car companies.  

    All iteration requires is to try and learn from the previous iteration.  Iterative development isn’t faster because iteration is fast, it’s faster because learning is part of the process.

    Sometimes the iterations in software are fast.  Sometimes they are glacially slow.

    As long as you are learning, and feeding the learning back into the next iteration, your software will improve.  You rarely only get one chance to release a piece of software.  Do your best, release, and learn.  Iteration isn’t a speed, it’s a journey towards better.

  • The Never Rewrite Podcast, Episode Ninety, The Hero and the Lifeguard

    The Never Rewrite Podcast, Episode Ninety, The Hero and the Lifeguard

    Isaac Askew and I explore the dynamics between two types of helpful developers: the ‘hero’ who swoops in to solve problems and the ‘lifeguard’ who encourages self-sufficiency. We dive deep into the tension that arises when these two personalities clash, and how to keep the peace between them. If you’ve ever struggled with helping too much or too little, this is a great episode for you!

    Watch on YouTube or listen to it at Spotify, Apple Podcasts, or your favorite podcast app, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!

  • Things I Learned By Helping To Program A Clavinet Clone

    Things I Learned By Helping To Program A Clavinet Clone

    Over the past few months I have helped my brother-in-law create a Clavinet Clone for musical keyboards.  Those of you who know me are undoubtedly laughing at the image of me creating a musical instrument because I have no rhythm, no timing, no ear, and generally musical ability of any kind.

    But, I do have over 20 years of programming experience, and I haven’t worked with someone who knows “just enough to be dangerous” in a long long time.  So I was surprised by some of the lesions.

    Source Control Is Not Intuitive

    Older source control systems like CVS were pretty simple, but didn’t support distributed development.  Git, on the other hand, is complicated.  Each time we did a handoff, I would make a branch, develop, and merge back to main.

    When I handed it back, my brother-in-law would copy the file, append “-new” and continue.  Every handover resulted in new copies of the files.  Sometimes entire directories would be copied and get “-old” added to the name.

    Branching, merging, and visual diffs may be second nature to full time developers, everyone else is going to need lesions.

    IDEs Will Spoil You

    The code is in Kontakt scripting language, which is an internally designed and developed scripting language.  I would do my development in Sublime Text, then switch over to the UI to compile and test.

    This got old after the 3rd syntax error.  A 15 second code-to-compile loop wasn’t bad back in 1992; now I just wanted syntax highlighting.

    Custom Scripting Languages Make Everything Harder

    Custom scripting languages make everything harder.  They make it harder on the implementer because they have to build and maintain a custom scripting language.  They make it harder on users because they have to learn a new language.

    A language with minimal functionality, limited documentation, and few examples.  The biggest time waster is the lack of a testing framework.

    With all that said, the Kontakt scripting language may not have been a mistake.  The same code and controls work on physical and virtual keyboards.  Getting code to work consistently on multiple hardware platforms is no small feat.

    If You Can’t Have Tests, Commit Continuously

    Continuous committing and diffs are a poor way to handle regressions, but they are better than nothing.  If you’re ever in this situation, make your commits as small and targeted as possible.  Regressions will still be frustratingly common and tedious, but at least you will be able to step back through your changes.

    Learning Things Sounds Great!

    You can buy the best sounding Clavinet Clone here!

    Full disclosure: I helped program it, it belongs to my brother-in-law, and I do not get a commission.

  • You’ve Built A Bad Implementation Of Other Software, Now What?

    You’ve Built A Bad Implementation Of Other Software, Now What?

    The immediate response to 20 Things You Shouldn’t Build At A Midsize SaaS was, what do you do if you HAVE built some of these things?  Great question!

    First, don’t panic.  We’ve all been there.  I have built some of these things myself – multiple CSV writers and parsers, a DateTime library, and an ORM.  We are all human, and we all make mistakes from time to time.

    Whether you built, or are responsible for, a bad implementation of other software, what do you do now?  It depends on which of 3 buckets the implementation falls into.

    It works and you don’t need to touch it

    CSV parsers and writers get written a lot because they are very simple.  Yes there are edge cases when it comes to escaping strings and unicode, but mostly once you have it working, it’s “fine”.

    If you don’t need to touch it, don’t!  Let sleeping dogs lie.  Ideally put a comment in the code that the next dev should replace it with a standard library instead of expanding the code.

    It works ok, but it’s incomplete

    This is the biggest bucket.  Implementations of DSLs, ORMs, and Frameworks usually fall into this group – the software works ok, but it is missing key features and robustness.  Observability is usually extremely lacking.

    This makes the software low priority tech debt.  There’s opportunity cost from continuing to use it, a cost to replacing it, but no direct development costs.

    Generally speaking, this is the kind of thing that gets ignored until you need a missing key feature.  Then the debate between continuing with the local implementation and replacing it, resumes.

    The best way to approach these situations is to look at the interfaces and requirements for the replacement software, and converge the local implementation over time.  This combines regular maintenance with replacement work.   As a bonus, it lowers the cognitive load on developers by making the local implementation work more like standard offerings.

    It requires continuous maintenance

    Some kinds of software require continuous maintenance.  When you write your own, you have to do all the maintenance yourself.

    Scripting languages are a wonderful source of Halting Problem issues.  Your customers will be excellent at creating scripts that will never exit and will consume resources until the server restarts.  The more successful you are, the worse the problem will become.

    I saw a SaaS with 2 full time developers working to close all of the ways that customers were accidentally creating infinite loops with their homegrown scripting language.

    I worked at a company using objective-c which had extended Apple’s compiler.  This was back in the days of installed, on-prem, software.  Every time Apple had a release, we had to warn customers not to upgrade until we could re-extend the compiler, recompile, and ship the code.

    When you encounter implementations that require continuous maintenance, you have to start working to remove them immediately.  The longer they remain, the more entrenched they become.  The sunk cost fallacy will work against you in strange and insidious ways.

    Iterative removal is the only real option in these cases.

    No matter which bucket the badly implemented software falls into, the first step is always to recognize the problem.  If there’s already an implementation, you shouldn’t write your own without a good reason.

  • The Never Rewrite Podcast, Episode Eighty-Nine, How Sonos Lost $500 Million Attempting a Rewrite

    The Never Rewrite Podcast, Episode Eighty-Nine, How Sonos Lost $500 Million Attempting a Rewrite

    How did Sonos lose $500 Million and trash their brand; with a disastrous app rewrite of course! Isaac and I discuss the Sonos disaster and how well it fits into our model of a rewrite. Was Sonos unlucky, or was this disaster nearly guaranteed?

    Watch on YouTube or listen to it at Spotify, Apple Podcasts, or your favorite podcast app, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!

  • 20 Things You Shouldn’t Build At A Midsize SaaS

    20 Things You Shouldn’t Build At A Midsize SaaS

    I have seen developers build a lot of unnecessary and counterproductive pieces of software over the years.  Generally, developers at small to midsize SaaS companies shouldn’t build any software that doesn’t directly help them deliver a service to their customers.

    Whether it was the zero interest rate period, bad management, or hubris, developers spent a lot of company money on projects that never made sense given their employer’s goals and size.  I have seen custom implementations of every type of software on this list.  None of it worked better than open source, and none offered a competitive advantage.

    If you find yourself developing or managing any of these twenty types of projects, stop and seriously consider what you are doing.

    1. Scripting languages
    2. Compiler extensions
    3. Transpilers
    4. Database extensions
    5. Databases
    6. DSLs
    7. ORMs
    8. Queues
    9. Background work schedulers
    10. GraphQL
    11. Stateful REST
    12. Frontend Frameworks
    13. Backend Frameworks
    14. Servers
    15. Dependency Injectors
    16. CSV writers or parsers
    17. Cryptography Implementations
    18. Logging Libraries
    19. DateTime libraries
    20. Anything from “First principles”

    There are always exceptions, if building this software has some competitive advantage, go ahead.  In general, anyone suggesting these projects is biting off more than they can chew and doesn’t fully understand the problem they are trying to solve.

    Most often things start out as a quick hack – “I’ll just concatenate these strings with a comma, it will be faster than finding a full CSV library.”  Soon you’re implementing custom separators and string escaping.

    If your company has done their own implementations don’t despair, iterate towards a better library!