When your SaaS Scaleup can't scale, I can help
Isaac and Jeffrey discuss ways to improve yourself outside of traditional means like college, courses, or reading.
Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!
The hard part first
If you’re trying to reduce risk, do the hard part first. That way, if it fails, you’ll have minimized your time and effort.
On the other hand, if you’re looking for buy-in and commitment so you can get through the hard part, do it last. People are terrible at ignoring sunk costs, and the early wins and identity shifts that come from the easy successes at the beginning will give you momentum as you go.
The hardest part of scaling a system is getting pieces of the current system to work with the new scaled out versions. Design and implementation are usually much more work, and take much more time.
Design and implementation don't involve working with the imperfections of past design and implementation. They don't have political agendas, and they aren't too busy to make the changes you need.
From my experience, the hardest part of any project built in isolation, is bringing the system from isolation to production. Whether you are bringing in a scaling framework, doing a rewrite, or building a new product; you and your team can write the software. You can shape it how you want or need because it is in isolation. Getting out of isolation requires changing your software to match production, and changing production is very hard.
Save yourself and your team, get your software into production from the beginning. Otherwise you may find yourself throwing away a year's worth of work because getting into production is the hard part.
My previous post about a creating lightweight garbage collector for settings was a little obtuse. This post is an example implementation for companies using LaunchDarkly and Gitlab.
Remember, this is a technique for getting a handle on your settings AFTER you’ve created or inherited a distributed mess. Not making a mess in the first place is always a better idea.
This technique is to help you get started cleaning up a mess. A mess is an ambiguous problem, it doesn’t have a central mass. There’s no “worst” and no particularly good or bad place to start.
A Garbage Collector will help you get started by showing you the easiest things to clean up!
Create a shell script, extractSettings.sh
#!/bin/sh
#Pass in API key, don’t check it in
apiKey=$1
settingsFile=“settings.txt”
#Get the flags in json, massage it so that you’re left with just flag names, one per line
## Explanation of the bashfoo:
## 1. Download a list of the flags from launch darkly in json format
## 2. Use jq to extract just the flag name key
## 3. Use grep to get rid of the remaining json and just have “key”: “flag name”
## 4. Use awk to get rid of key and be left with just “flag name”
## 5. Use sed to get rid of the “s so we have just flag name
curl -X GET “https://app.launchdarkly.com/api/v2/flags/default” -H “Authorization: ${apiKey}” | jq “.items[] | {key}” | grep key | awk ‘{print $2}’ | sort | sed ‘s/“//g’ > $settingsFileCalling ./extractSettings.sh <your api key> will give you a list of setting names, one per line.
These are the unique strings that you need to search for in gitlab.
Create the second shell script, settingsToProjects.sh
#!/bin/sh
#Pass in API key, don’t check it in
apiKey=$1
settingsFile=“settings.txt”
#Create output dir if it doesn’t already exist
mkdir -p ./settings
#Iterate through $settingsFile - each line is a single launch darkly flag name
while IFS=“” read -r p || [ -n “$p” ]
do
## Explanation of the bashfoo:
## 1. Use curl to download json of all the projects containing the feature flag string
## 2. Use grep to break from json to a single “key”: value per line
## 3. Use sort and uniq to get a unique list regardless of the number of references
## 4. Write the results to ./settings/<settingname>.txt
curl --header “PRIVATE-TOKEN: ${apiKey}” “https://<your gitlab install>/api/v4/search?scope=blobs&search=${p}” | jq ‘.[] | {project_id} | grep ‘project_id’ | sort | uniq > settings/${p}.txt
done < $settingsFileCalling “./settingsToProjects.sh <your api key>” will give you a directory full of files.
Note: You need an admin level API key so that the script can see all projects. Otherwise you might miss some references in projects you can’t see.
Each file will be “settingName.txt” and contain results like this:
“project_id”: 1
“project_id”: 15
“project_id”: 243
This doesn’t even need a script, run:
> wc -l settings/* | sort -rThis will give you a list of every setting and the number of projects it is used in, sorted highest to lowest. Anything with 0 lines is used in your codebase, 1 line means that it is referenced by one project, etc.
Start with the zeros. Any file with 0 lines is either brand new, or no longer used. That’s an easy check in LaunchDarkly.
Next, check for the single project use cases. Some of those will be unused strings in constants files. Clean up the constants files, and clean up the settings that fell to zero.
Because your garbage collector is tracking references, you can stop at any time. You’ve automated the massive data gathering element, which makes cleanup cheap and manageable.
In service oriented architectures adding new application settings like environment variables, constants, and feature flags costs almost nothing. Cleaning them up, however, is expensive.
Ideally developers would keep track of settings and remove them when they are no longer needed. More often feature flags get set to 100%, everyone moves on, and the old code path lingers for months. I have personally spent hours updating code, only to realize that it is effectively unreachable and needs to be deleted instead.
Talk about wasting time and money!
Doing nothing seems like a cheap solution. The code gets a little bloated, the services use a little more memory, and API calls send a few extra kilobytes. No single unused feature flag or environment variable has any impact.
Eventually you end up with hundreds of unused settings, on thousands of servers, distributed to hundreds of thousands of customers.
The cost of those crufty settings add up in terms of performance, development time, and outages.
A key disadvantage of Service Oriented Architecture is that settings get passed from one service to the next. It’s rarely safe to remove unused settings because they may be gathered into a collection and passed along to a service that does use them.
Figuring out what is used where across multiple services is a slow process. When you’ve got hundreds of settings to investigate, it’s daunting and demoralizing.
A Garbage Collector keeps track of references to objects, and when there are no more references, cleans them up.
You can make a very basic collector by scripting out a few API calls to your git repository.
The three basic steps are:
That’s it! You now have a rudimentary system that tracks references to settings across your repos. Any setting file with only one entry isn’t used anywhere else and can be deleted once the parent app is done with it.
A garbage collector is overkill if you’re curious about a handful of settings, just use regular search. But when you’re facing hundreds of settings and you need to figure out which can be targeted for cleanup, a garbage collector might be just what you need.
A non technical story of deception, good deeds with terrible consequences, and weak HR.
Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!
Sometimes an upgrade to a new version of a framework takes weeks or even months. What do you do when you need to make changes to areas that are being affected by the upgrade?
Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!
A common component of code rewrites is The Pause. No new development, just for a few months, so that developers can replace the existing code. After the rewrite, development will be so much faster! We will quickly come out ahead of where they would be without The Pause.
I have seen managers, directors, and even CTOs go to stakeholders and convince them to allow a pause. No matter what everyone agrees to, new development is rarely fully paused, the rewrite is never finished by the original date, and unforeseen events always occur.
Put aside the reality of why The Pause won’t work for a moment and examine the premise itself. The Pause is all about getting tech what it wants on the assumption that this will make things better for everyone in the long run. The Pause excludes all other departments. Basically, The Pause says:
Yes, in the past, we failed to understand and anticipate changes from the rest of the business. We made our life so difficult that we can’t keep making changes for you. But don’t worry your pretty little head, we are so much better at understanding you now!
We are going to go off by ourselves for a few months and create new technology that will handle and anticipate all of your future changes. We understand you so well, and our work is so difficult to explain, that we are going to do this work without you.
The Pause is an insulting fool’s errand. If the technology group were capable of anticipating future changes by themselves, they wouldn't need a rewrite.
Instead of asking for The Pause, try a dialog with the rest of the organization. Bring them in instead of shutting them out.
Special Guest Aadil Farid joins us again to continue his discussion on the harsh realities he confronted when he led a team that computerized ticketing on India’s Railways in the 1980s.
This episode goes deep on empathy for your customers and fighting internal corruption with code. If you love stories about fighting the good fight, you'll love this epsiode!
Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!
In Episode 26 of Never Rewrite, Special Guest Aadil Farid joins Isaac and I to discuss the harsh realities he confronted leading a team that computerized ticketing on India’s Railways in the 1980s.
Learn how Aadil had to get around hardware constraints, corruption, and software issues. In some ways everything was different about development 40 years ago, but in many respects things are exactly the same. Maybe not the part where a computer crash caused a riot that needed the national guard to put down.
If you want to go deep on how software can be a positive force for change, you won't want to miss this one!
Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!
In episode 25 of Never Rewrite, Isaac and I are joined by special guest, Brian Genisio, from Khan Academy. Brian walks us through their successful 3.5 year rewrite that wrapped up in early 2023.
Learn how Brian’s team was able to achieve the nearly impossible act of successfully completing a rewrite using many of the techniques we’ve talked about in the past, plus some brand new ones! If you want to learn about what it takes to do a rewrite, buckle up!
Watch on YouTube or listen to it at NeverRewrite, Spotify, Apple Podcasts, or Amazon Music, and let us know if you have ever been involved in a rewrite. We would love to have you on the show to discuss your experience!