My previous post about a creating lightweight garbage collector for settings was a little obtuse.  This post is an example implementation for companies using LaunchDarkly and Gitlab.

Context, You’ve Got a Mess of Distributed Settings

Remember, this is a technique for getting a handle on your settings AFTER you’ve created or inherited a distributed mess.  Not making a mess in the first place is always a better idea.

This technique is to help you get started cleaning up a mess.  A mess is an ambiguous problem, it doesn’t have a central mass.  There’s no “worst” and no particularly good or bad place to start.

A Garbage Collector will help you get started by showing you the easiest things to clean up!

Step 1 – Extract Your Settings From LaunchDarkly

Create a shell script, extractSettings.sh

#!/bin/sh

#Pass in API key, don’t check it in
apiKey=$1
settingsFile=“settings.txt”

#Get the flags in json, massage it so that you’re left with just flag names, one per line

## Explanation of the bashfoo:
## 1. Download a list of the flags from launch darkly in json format
## 2. Use jq to extract just the flag name key
## 3. Use grep to get rid of the remaining json and just have “key”: “flag name”
## 4. Use awk to get rid of key and be left with just “flag name”
## 5. Use sed to get rid of the “s so we have just flag name

curl -X GET  “https://app.launchdarkly.com/api/v2/flags/default” -H “Authorization: ${apiKey}” |  jq “.items[] | {key}” | grep key | awk ‘{print $2}’ | sort | sed ‘s/“//g’ > $settingsFile

Calling ./extractSettings.sh <your api key> will give you a list of setting names, one per line.

These are the unique strings that you need to search for in gitlab.

Step 2 – Search Gitlab for Setting Strings

Create the second shell script, settingsToProjects.sh

#!/bin/sh

#Pass in API key, don’t check it in
apiKey=$1

settingsFile=“settings.txt”

#Create output dir if it doesn’t already exist
mkdir -p ./settings

#Iterate through $settingsFile - each line is a single launch darkly flag name
while IFS=“” read -r p || [ -n “$p” ]
do

  ## Explanation of the bashfoo:
  ##  1. Use curl to download json of all the projects containing the feature flag string
  ##  2. Use grep to break from json to a single “key”: value per line
  ##  3. Use sort and uniq to get a unique list regardless of the number of references
  ##  4. Write the results to ./settings/<settingname>.txt

  curl --header “PRIVATE-TOKEN: ${apiKey}” “https://<your gitlab install>/api/v4/search?scope=blobs&search=${p}” | jq ‘.[] | {project_id} | grep ‘project_id’ | sort | uniq > settings/${p}.txt

done < $settingsFile

Calling “./settingsToProjects.sh <your api key>” will give you a directory full of files.
Note: You need an admin level API key so that the script can see all projects.  Otherwise you might miss some references in projects you can’t see.

Each file will be “settingName.txt” and contain results like this:

“project_id”: 1
“project_id”: 15
“project_id”: 243

Find the Unused Settings

This doesn’t even need a script, run:

> wc -l settings/* | sort -r

This will give you a list of every setting and the number of projects it is used in, sorted highest to lowest.  Anything with 0 lines is used in your codebase, 1 line means that it is referenced by one project, etc.

Take Out The Garbage

Start with the zeros.  Any file with 0 lines is either brand new, or no longer used.  That’s an easy check in LaunchDarkly.

Next, check for the single project use cases.  Some of those will be unused strings in constants files.  Clean up the constants files, and clean up the settings that fell to zero.

Because your garbage collector is tracking references, you can stop at any time.  You’ve automated the massive data gathering element, which makes cleanup cheap and manageable.