Sherman On Software

Sherman On Software Latest

  • Fixing All The Bugs Won’t Solve All The Problems – Deming’s Path Of Frustration

    Fixing All The Bugs Won’t Solve All The Problems – Deming’s Path Of Frustration

    A program of improvement sets off with enthusiasm, exhortations, revival meetings, posts, pledges.  Quality becomes a religion.  Quality as measured by results of inspection at final audit shows at first dramatic improvement, better and better by the month.  Everyone expects the path of improvement to continue along the dotted line.

    Instead, success grinds to a halt.  At best, the curve levels off.  It may even turn upward.  Despondence sets in.  The management naturally become worried.  They plead, beg, beseech, implore, pray, entreat, supplicate heads of the organizations involved in production and assembly, with taunts, harassment, threads, based on the awful truth that if there be not substantial improvement, and soon, we shall be out of business.

    W. Edwards Deming, Out of the crisis, Page 276

    In software, as in manufacturing, some problems occur due to bugs or “special causes”, and some are “common cause” due to the nature of the system’s design and implementation.  Fixing bugs is removing special causes.  Removing bugs greatly improves software quality, but it won’t impact “common cause” issues.

    Some “common cause” software performance issues I have encountered:

    • The software is “in the cloud”, but really it is in one data center in the US.  As a result the software is slow and laggy for customers in Europe and Asia.
    • The software runs slowly because the hardware is underprovisioned.
    • The software runs slowly because large amounts of unnecessary data are being sent to the users.
    • The software runs slowly because of inefficient data access patterns.

    Even with no bugs, “common cause” issues can result in low quality software.  

    The way off of Deming’s Path Of Frustration is to attack system design and implementation issues with the same fervor used to fight bugs.

  • The Never Rewrite Podcast, Episode Forty-Eight, Falsehoods Programmers Believe About Projects

    The Never Rewrite Podcast, Episode Forty-Eight, Falsehoods Programmers Believe About Projects

    In this episode, Isaac and Jeffrey discuss falsehoods that programmers believe about projects. They explore assumptions such as projects having defined beginnings and endings, the role of priorities in project work, and the importance of clear communication and feedback. They also touch on the challenges of working on successful projects that transition into ongoing maintenance and the consequences of failed projects that continue to exist. The conversation highlights the need for programmers to question these assumptions and adapt their approach to project management.

    This episode is a supplement to my post about Falsehoods Programmers Believe About Projects.

    Takeaways:

    • Projects often don’t have defined beginnings or endings, and assumptions about their start and end dates can lead to confusion and miscommunication.
    • Relying solely on personal priorities to determine when to start working on a project can lead to conflicts and misunderstandings with others who have different priorities.
    • Clear communication and feedback are crucial in project work to ensure everyone is on the same page and to avoid missed signals and mixed signals. – Programmers should be proactive in sharing their expertise and insights on projects, even if they haven’t been explicitly asked, to prevent unnecessary rework and improve overall project outcomes.
    • Successful projects often transition into ongoing maintenance and feature development, while failed projects that are not fully turned off can continue to exist and incur costs.
    • Questioning assumptions and adapting project management approaches based on the specific context and circumstances of each project is essential for success.

    Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!

  • Falsehoods Programmers Believe About Projects

    Falsehoods Programmers Believe About Projects

    Years ago, Patrick McKenzie, wrote an article titled Falsehoods Programmers Believe About Names.  The article inspired a short lived burst of other programming falsehoods.  This is a very late entry in the genre, covering incorrect assumptions about software projects.

    All of these assumptions are wrong.  Try to make fewer of these assumptions when working on projects.  Since you are always working on projects at work, always be questioning your assumptions about projects.

    1. Projects have defined beginnings.
    2. Maybe not formal beginnings, but there is a point when you are supposed to start work.
    3. Your manager knows when you should start working.
    4. You can use your priorities to determine when you should start working.
    5. You should be working on the project because you were asked.
    6. You should not be working on the project because you weren’t asked.
    7. Projects have defined endings.
    8. Successful projects have endings.
    9. Failed projects have endings.
    10. The project will solve the problem.
    11. The project will solve a problem.
    12. The project won’t make the problem worse.
    13. Everyone on the project agrees on what problems the project is supposed to solve.
    14. Everyone agrees about what solving the problems means.
    15. Solving the problem will make the project a success.
    16. Not solving the problem will make the project a failure. 
    17. There is a relationship between the project’s success and the status of the problem.
    18. The software you are asked to write will solve the problem.
    19. The software you are asked to write will make the project a success.
    20. Writing the software you are asked to write means you are doing a good job.

    At best, projects are best guesses by well intended people.  At their worst, projects can become meaningless busywork that is completely unrelated to any problems or desires at a company.  The fewer false assumptions you buy into the more effective you will be. 

  • The Never Rewrite Podcast, Episode Forty-Seven, Maintenance Work as a Flight to Safety – A Manager’s Perspective

    The Never Rewrite Podcast, Episode Forty-Seven, Maintenance Work as a Flight to Safety – A Manager’s Perspective

    Isaac and Jeffrey discuss the flight to maintenance work as a safety option from the manager’s perspective. They explore different perspectives on maintenance work and the value of taking on such work. They emphasize the importance of understanding the reality of the system and approaching maintenance work as an opportunity to solve real problems. They also discuss the role of accountability and changing the culture around maintenance work. The episode concludes with a discussion on the impact of not addressing maintenance work and the benefits of going slow to go fast.

    Takeaways:

    • Maintenance work should not be viewed as a punishment, but as an opportunity to solve real problems and make teammates’ lives easier.
    • Understanding the reality of the system and addressing maintenance work can have a positive impact on the team’s morale and productivity.
    • Managers should create a culture of accountability and ensure that support and maintenance issues are fixed, not just mitigated.

    Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!

  • Convincing People Not To Touch Hot Stoves Is Easy

    Convincing People Not To Touch Hot Stoves Is Easy

    The hard part is convincing people that what they’re reaching for is a hot stove.

    Having developers work for 6 months and then do a single release is appealing because development proceeds in the straightest line over the shortest distance.  When I push for small releases and iteration, the main pushback is that the incremental releases add work.  The tradeoff appears to be one release after 6 months, or 7 monthly releases.

    Seven months is more than six months, so iteration is more work!

    The problem is that the single release after 6 months of work is “hot stove”.  It’s a plan that needs the requirements to be perfect at the start.  Nothing new can be learned during the project.  Customers and the market have to remain exactly the same.  It’s a plan that will leave you burned.

    It isn’t six months versus seven months.  It is 6 months, plus 3-6 months of rework versus seven months.

    Explaining the risks is ineffective.  People will agree that if the plan is wrong, that if we learn new things, or if the market changes then of course a 6 month plan will end up delivering the wrong thing.  Of course you will then need another 3 months to correct.  That’s obvious.

    After agreeing about the risk, they will then declare that those risks don’t apply to THIS project.  This project isn’t at all like those “hot stoves”.  This project plan is correct and everything has been considered.

    Describing the danger doesn’t work if the problem doesn’t resonate with the listener.  Instead of urging people not to touch hot stoves, work on helping them identify the hot stove.

    • Instead of talking about the risks of a single release, talk about the value of iteration.
    • Don’t talk about the likelihood that the new workflows aren’t perfect, talk about the value of early customer feedback.
    • Don’t talk about the risk that the 3rd party system doesn’t really work like their spec, talk about the value of early integration.
    • Don’t talk about holding back valuable features for a gigantic release, talk about delivering value to customers.

    Once people see the risk and the tradeoffs, you won’t have to convince them not to burn themselves.  After all, everyone understands that you shouldn’t touch hot stoves.

  • The Never Rewrite Podcast, Episode Forty-Six, Workshopping Live with Dustin Rea

    The Never Rewrite Podcast, Episode Forty-Six, Workshopping Live with Dustin Rea

    In this conversation, Dustin Rea discusses the challenges faced by one of his clients who has a CRM. The CRM is built on a legacy codebase with issues in quality, scalability, and deliverability. The team has been working on improving the system, but there are still problems with emails, SMS, payments, and automations. The company is mission-driven and relies on the CRM to run their business, so rebuilding customer trust is crucial. They have made some improvements in infrastructure and email deliverability, but there is still work to be done. The conversation covered several topics related to database architecture and system design. The main themes include database optimization, handling heavy reads and writes, improving error handling and incident response, and enhancing observability. The speakers discussed the need for a purpose-driven database, implementing caching with Redis, tracking heavy queries, and addressing inconsistencies in the email service. They also mentioned the importance of logging and monitoring tools like Sentry and Grafana. Overall, the conversation highlighted the challenges and potential solutions for improving the performance and reliability of the system.

    Takeaways:

    • The client’s CRM system is built on a legacy codebase and has issues with quality, scalability, and deliverability.
    • The team has been working on improving the system, but there are still problems with emails, SMS, payments, and automations.
    • Rebuilding customer trust is crucial for the company, as the CRM is core to their business.
    • Improvements have been made in infrastructure and email deliverability, but there is still work to be done.
    • Optimizing database performance is crucial for handling heavy reads and writes.
    • Implementing caching with Redis can help improve performance and reduce load on the database.
    • Tracking and optimizing heavy queries is important for identifying and resolving performance issues.
    • Improving error handling and incident response processes can help address inconsistencies and ensure reliable system operation.
    • Enhancing observability through logging and monitoring tools can provide valuable insights for troubleshooting and improving system performance.

    Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!

  • AI First API Strategy

    AI First API Strategy

    What does BYO-AI mean for your company’s API strategy?

    This article goes into the obvious, and subtle, changes that need to happen to support customers bringing their own AI.

    All Features Need To Be Fully API Accessible

    AI can only interact with publicly facing endpoints.  Any feature that isn’t publicly facing, or is only accessible through the UI, isn’t accessible to your customer’s AI.

    Embracing your customer’s use of AI means making all of your features publicly available.  Now is time to reevaluate past decisions about excluding features for performance, scalability, security, pricing, and other reasons.

    Customers come to you with problems and AI opens up entirely new avenues for solutions; but only if you let AI into your API.

    All Features Need To Be Documented

    AI is going to learn about your API by reading the docs, reading the example code, and then reading StackOverflow.

    If you want AI to do a good job writing code against your API, you need to do a good job telling it how.  Undocumented endpoints, missing parameters, and bad code examples make that harder.

    Outdated docs and libraries need to be scrubbed from the internet.  It is very much like hiring a reputation company to ensure that what AI sees and knows about your API is what you want it to see and know.

    Technical writers and other DevRel positions are notoriously understaffed.  This is a great time to hire talented writers and get them writing for AI instead of human readers.

    Features and Docs are the Easy Part

    Fleshing out your API and producing up to date, readable docs are things you should have been doing all along anyway.  AI doesn’t change that, it merely raises the cost of poor software hygiene.

    Once you get the table stakes out of the way, things get harder.  Or at least take more effort.

    Assume That AI Has No Idea What It Is Doing

    AI is the ultimate expression of the developer attitude, “I don’t want to understand the business, I just want to code.”  AI can’t understand the context and the customer can’t understand code.  This model works out because this is sort of how most software is written anyway.

    These are the conceptual shifts you need to make with your API to support AI development.

    Make Your API Simple and Verifiable

    Users are going to manually confirm API actions through the UI so your API needs to be simple and easy to verify.  Endpoints that take multiple, open ended collections are great for performance and pushing a lot of data.  

    If they can partially succeed, customers will find them a nightmare as they try to understand the results of the AI’s code.

    Lean on strong validation, all or nothing changes, and understandable error responses.

    Things Are Going To Be Rerun.  A Lot.

    Everything will end up being rerun.  Customers won’t understand data models and won’t understand the risks with rerunning things.  

    Embrace it by making everything as idempotent as possible.  

    Make non-idempotent actions all-or-nothing.  AI is going to get close, a lot.  The last thing your customers expect is for all that flailing to wreck their data.  Bias the endpoints to either fully succeed or to take no action.  Don’t do partial updates and then bail on the request.  Change your processing to do full validation, and make changes only when you know you can make the full change.

    Idempotent and all-or-nothing can be shockingly hard changes to implement.  Fortunately, you can iteratively improve your API!

    Error Messages Need To Be Non-Developer Readable

    Put effort into making your errors human readable and reasonable.  AI is better at grokking text than dense error responses.

    Even better, when something fails validation include a link to the documentation where you’ve explained the correct structure.

    Shifting to An AI First API Takes Time

    Shifting to an AI first API isn’t complicated, but it will take time and effort.

    Start with the fundamentals – make all of your features available through your API, and get your documentation up to date.  Then, if need be, work to make you API AI friendly.

    Customers are going to bring their own AI to your API and do amazing new things that you would never have imagined.  Embrace the change.

  • The Never Rewrite Podcast, Episode Forty-Five, Year In Review

    The Never Rewrite Podcast, Episode Forty-Five, Year In Review

    Isaac and Jeffrey celebrate the one-year anniversary of the podcast!

    In this episode, Isaac and Jeffrey look back on past year and discuss lessons learned and highlights from interviews. If you’re new to the show, this is a great episode to find out about the back catalog.

    Takeaways:

    • Consistency and scheduling are key to maintaining a successful podcast.
    • Finding interesting guests and stories is important to keep the podcast engaging.
    • Forcing functions are often necessary for successful rewrites.
    • Rewrites can be risky and discourage learning about the current system. The investigation process is crucial for understanding code and improving its quality.
    • Tidying up code and making small improvements can make it easier for future developers to understand and work with.
    • Iterative workshops can be an effective way to transform and improve code.
    • Developers should strive to understand the business side of software development to make better decisions.
    • Developing human skills is important in the age of AI.

    Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!

  • BYO-AI API Support

    BYO-AI API Support

    Long before AI becomes good enough to take over today’s developer jobs, it is going to unleash an explosion of programming, opening up the unprofitable low end. The transformational use case for AI development will be letting non-developers use APIs.  For SaaS companies, expect to see AI tools become your most common 3rd party integrator.  Not the highest volume and certainly not the best integrator, but when it comes to customer enablement, nothing else will be able to compete.

    If you’re a SaaS, your customers are going to demand BYO-AI support!

    Major Implications for SaaS Companies

    According to best practices, every SaaS company has been API First for a decade now.  That means that the APIs and UIs are separate.  Everything the UI does is through the API and in theory a customer could write their own UI.  SaaS with white labeling and reseller revenue streams may even have bespoke UIs.

    In the real world however, public facing APIs are rarely complete.  Tech debt, security, business models, and other concerns result in some features not being accessible through the API.

    What’s a SaaS to do?  

    Anything not available through the API is not available to customer’s AI developers.  Building AI features into the UI offers a partial solution with three large drawbacks:

    1. The implementation is limited to how the SaaS’s product designers think about using the service, not how customers want to use the service.
    2. The SaaS has to pay for the AI access instead of the customer.
    3. The AI is locked in the UI and can’t interact with the rest of the customer’s workflow.

    The first point is the most important – any SaaS that sets up internal AIs will always be playing catchup with the ideas of their customers and competitors.

    Alternatively, being truly, completely, API First frees a SaaS to celebrate customer creativity.  AI development and crazy implementations can become community building activities.  Watching what customers build is free market research about what features the UI is missing.  Build those features for the less savvy customers.

    First, Go API First

    The first step for every SaaS is to get every feature fully into the API.  Make everything available to your customer’s AI of choice.

    Embrace BYO-AI and let your customer’s visions soar!

  • First, AI will do unprofitable programming work

    First, AI will do unprofitable programming work

    In a recent episode of NeverRewrite, Isaac and I discussed the implications of AI doing rewrites for developers.  

    So what if things change?

    AI will probably replace most of the coding that developers do day-to-day today.  Today’s developers don’t punch punch cards, don’t design circuits, or write assembly code.  The nature of development has changed in the past, and will change in the future.

    The first killer app for the PC was Lotus 1-2-3, which made spreadsheets widely accessible.  People used Lotus to perform a lot of accounting functions, but we still have accounting software.  Spreadsheets didn’t replace special function software, they created a new category of general purpose tools.

    Rather than eating software development jobs, AI is a general purpose tool that will enable an explosion of software at the unprofitable low end.

    What is The Unprofitable Low End of Software Development?

    Simply, software developers are expensive.  There is a lot of potential software that could be written, but wouldn’t be profitable enough to hire someone to write.  There are tons of single use, single purpose tasks that could be replaced with a small bit of code, but only if you happened to have a developer nearby.

    No Code / Low Code solutions help narrow the gap, but there are tons of tasks that are still too small to be generalized.

    For example, I recently took a 3,500 product catalog in CSV form and wrote some python to upload the products to Shopify.  It also performed a second step to pull in the images from the factory’s website and associated them with the new products.  Shopify has great tools, uploading the file would be easy, but adding the images would have been a daunting and dull manual task.

    Easy for a developer though; I ran it once, and will probably never touch the code again.

    Trust, When You Can Verify

    AI can write mostly correct code today.  When you can verify the results, you can accept them.

    In the Shopify example you can verify by checking to see how many products were created.  You can check to see if they have images.  If it showed up on Shopify, then the software is correct.  It is running locally so performance and security pretty much don’t matter.  Optimized string functions and defensive string escaping aren’t concerns when you are only dealing with one file, once.

    Prediction: AI Will Increase API Usage

    As a development tool, AI will turn the average user into a mediocre API developer.  There will be an explosion of non-technical people using company’s APIs to solve bespoke problems in ways that couldn’t be done profitably before AI.  This won’t reduce the need for developers, it will increase the demand for people who can write APIs that are resilient, performant and secure.

    Expect a flood of customers using your API in ways you couldn’t imagine.  The question isn’t how to have AI write code for your company, the question is how to make it easy for AI to use your API.

    I can’t wait to see what people come up with!