2022 Year In Review

It’s the time of year to look back, reflect, and highlight some interesting pieces you might have missed.

By The Numbers

I set a personal goal of publishing every week, or at least 50 articles for 2022.
All told, I published 32 articles - 64% of my goal.

I may not have hit my goal, but I'm very proud of the pieces I did write!

Top 3 Articles From 2022

These were the most popular articles that I wrote in 2022:

  1. My series of articles on SaaS Tenancy Models
  2. The Opposite Of Iterative Delivery
  3. The Chestburster Anti-Pattern

Top 3 Articles In 2022

These were the three most popular articles in 2022 that I wrote in past years:

  1. Links VS Tags, A Rabbit Hole
  2. The 5 Ws For Developers
  3. Your Database is Not A Queue

Two Published Articles

I became a published author this year, with 2 pieces on leaddev.com

  1. The Dangers Of Pulling Rank as a Staff Engineer
  2. How to Break the "Get Me Everything Cycle"

Goals For 2023

I'm going to spend more time writing in 2023. My goal is to publish 100 pieces, approximately 2 per week.

Happy Holidays and I'll be back in the new year!

The Only Client Experience

The only client experience you can offer clients is the one you have today.  Yesterday’s experience is a memory, and tomorrow is a promise.

Delivering iteratively, baby step improvements to your client’s experience, makes the experience better today, tomorrow’s promise more believable, and memories of things improving.

Promising a moonshot keeps the only experience your client’s have exactly the same; and if it was a good one, you wouldn’t need a moonshot.

When Your Customers Struggle

When your customers struggle with a non-intuitive interface, would they rather:

Wait 6 months while you work on an amazing redesign and brand refresh, or get weekly tweaks that make using your product easier and more intuitive?

Branding and color schemes are about you, improving the UI is about your customers.

Don’t put off customer steps while working on a moonshot like a branding refresh.

Moonshots vs Baby Steps at A Scaleup

You’re A Scaleup!  What’s a Scaleup?

You become a Scaleup when your SaaS’s service offering becomes compelling and you start attracting exponentially more clients.

All at once you have a lot more clients, clients with a lot more data.

Solutions that support 1,000 clients buckle as you pass 5,000.  Suddenly, 25,000 clients is only months away.

Services that support hundreds of thousands of transactions a day fall hopelessly behind as you onboard clients with millions of transactions.

You finally know what customers want.  You quickly find the edges of your system.  Money is rolling in from customers and VCs.  You can throw money at the problems to literally buy time to find a solution.

But you’re faced with a looming question - moonshots or baby steps.

Moonshots Are About You, Baby Steps Are About Your Clients

It’s not about you or your SaaS, it’s about your client’s outcomes.

Moonshots are appealing because they take you directly to where you need to be.  Your system needs to scale 10x today and 100x next year; why not go straight for 100x?

Baby steps feel like aiming low because the impact on you is small.  But it’s not about you!  Think about the impact on your clients.

From a technology perspective, sending emails 1% faster is ::yawn::

But for your clients, faster emails means more engagement, which means more sales.

Would your clients rather have more sales this week, compounding every week for the next year, or flat sales for a year while you build a moonshot?

Clients who churn, or go out of business, won’t get value from the moonshot.  Even if you deliver greater value eventually, your clients are better off getting some value now.

Are you delivering value to your SaaS or your clients?

2 Developers, A Mathematician and a Scrum Master Walk Into a Bar

And come up with “The Worst Coding Problem Ever” dun dun dun!

Imagine getting this whopper in an interview or a take home test:

The United States has been conducting a census once a decade for over 200 years.

Imagine you can iterate the data at a family level, with the family data being whatever format/object is easiest for you. 

Find the family with the longest fibonacci sequence of children.

The most fundamental issue is that it’s not clear what the answer looks like.  In fact, the 4 of us had 3 different interpretations of what the answer would look like.

Is the question looking for children’s ages going forward?

That would be an age sequence of 0, 1, 1, 2, 3, 5, etc

Or a newborn, a pair of 1 year old twins, a 2 year old, 3 year old, 5 year old, etc

Or is it looking for children born in the sequence?  (This is the inverse of the first answer)

A 6 year old, 5 year old twins, a 3 year old and a newborn

Or is it asking about the age gap between children?

In that case you’d be hunting for Twins (gap of 0), a gap of 1 year, a second gap of 1 year, a gap of 2 years, etc.

There are so many ways to be the family fibonacci.

Many Technical Problems are like this

Fairly straightforward computer problems with meaningless mathematics sprinkled on top.  Being asked by people who won’t know the implications of any of the 3 answers. 

But what’s the answer?

If you are presented with this question in an interview, the correct answer is to thank the interviewer for their time, wish them the best of luck in their search, and end the interview.

Scaling is Legacy System Rescue That Pays 4x

In my last article, You Won’t Pay Me to Rescue Your Legacy System, I talked about my original attempt at specializing, and why it didn’t work.  I bumbled along until I lucked into a client that helped me understand when Legacy System Rescue becomes an Expensive Problem.

Rather than Legacy System Rescue, I was hired to do “keep the lights on” work.  The company had a 3 developer team working on a next generation system, all I had to do was to keep things running as smoothly as possible until they delivered.

The legacy system was buckling under the weight of their current customers.  Potential customers were waiting in line to give them money, and had to be turned.  Active customers were churning because the platform was buckling.

That’s when I realized - Legacy System Rescue may grudgingly get a single developer, but Scaling gets three developers to scale and one to keep the lights on.  Scaling is an expensive problem because it involves churning existing customers and turning away new ones.

Over 10 months I iteratively rescued the legacy system by fixing bugs and removing choke points.  After investing over 50 developer months, the next generation system was completely scrapped.

The Lesson - Companies won’t pay to rescue a legacy system, but they'd gladly pay 4x to scaleup and meet demand.

Infrastructure Consolidation Drives Early Tenancy Migrations

For SaaS with a pure Single Tenant model, infrastructure consolidation usually drives the first two, nearly simultaneous, steps towards a Multi-Tenant model.  The two steps convert the front end servers to be Multi-Tenant and switch the client databases from physical to logical isolation.  These two steps are usually done nearly simultaneously as a SaaS grows beyond a handful of clients, infrastructure costs skyrocket and things become unmanageable.

Diagram of a single tenant architecture become multi-tenant

Considering the 5 factors laid out in the introduction and addendum - complexity, security, scalability, consistent performance, and synergy this move greatly increases scalability, at the cost of increased complexity, decreased security, and opening the door to consistent performance problems.  Synergy is not immediately impacted, but these changes make adding Synergy at a later date much easier.

Why is this such an early move when it has 3 negative factors and only 1 positive?  Because pure Single Tenant designs have nearly insurmountable scalability problems, and these two changes are the fastest, most obvious and most cost effective solution.

Complexity 

Shifting from Single Tenant servers and databases to Multi-Tenant slightly increases software complexity in exchange for massively decreasing platform complexity.

The web servers need to be able to understand which client a request is for, usually through sub domains like client.mySaaS.com, and use that knowledge to validate the user and connect to the correct database to retrieve data.

Increased complexity from consolidation

The difficult and risky part here is making sure that valid sessions stay associated with the correct account.  

Database server consolidation tends to be less tricky.  Most database servers support multiple schemas with their own credentials and logical isolation.  Logical separation provides unique connection settings for the web servers.  Individual client logins are restricted to the client’s schema and the SaaS developers do not need to treat logical and physical separation any differently.

Migrations and Versioning Become Expensive

The biggest database problems with a many-to-many design crop up during migrations.  Inevitably, web and database changes will be incomparable between versions.  Some SaaS models require all clients on the same version, which limits comparability issues to the release window (which itself can take days), while other models allow clients to be on different versions for years.

Versioning and Migration Diagram

The general solution to the problem of long lived versions is to stand up a pool of web and database servers on the new version, migrate clients to the new pool, and update request routing.

Security

The biggest risk around these changes is database secret handling; every server can now connect to every database.  Compromising a single server becomes a vector for exposing data from multiple clients.  This risk can be limited by proxy layers that keep database connections away from public facing web servers.  Still a compromised server is now a risk to multiple clients.

Changing from physical to logical database separation is less risky.  Each client will still be logically separated with their own schema, and permissioning should make it impossible to do queries across multiple clients.

Scalability

Scalability is the goal of Multi-Tenant Infrastructure Consolidation.

In addition to helping the SaaS, the consolidation will also help clients.  Shared server pools will increase stability and uptime by providing access to a much larger group of active servers.  The client also benefits from having more servers and more slack, making it much easier for the SaaS to absorb bursts in client activity.

Likewise, running multiple clients on larger database clusters generally increases uptime and provides slack for bursts and spikes.

These changes only impact response times when the single tenant setup would have been overwhelmed.  The minimum response times don’t change, but the maximum response times get lower and occur less frequently.

Consistent Performance

The flip side to the tenancy change is the introduction of the Noisy Neighbor problem.  This mostly impacts the database layer and occurs when large clients overwhelm the database servers and drown out resources for smaller clients.

This can be especially frustrating to clients because it can happen at any time, last for an unknown period, and there’s no warning or notification.  Things “get slow” and there are no guarantees about how often clients are impacted, notice, or complain.

Synergy

There is no direct Synergy impact from changing the web and database servers.

A SaaS starting from a pure Single Tenant model is not pursuing Synergy, otherwise the initial model would have been Multi-Tenant.

Placing distinct client schemas onto a single server does open the door to future Synergy work.  Working with data in SQL across different schemas on the same server is much easier than working across physical servers.  The work would still require changing the security model and writing quite a bit of code.  There is now a doorway if the SaaS has a reason to walk through.

Conclusion

As discussed in the introduction, a SaaS may begin with a purely Single Tenant model for several reasons.  High infrastructure bills and poor resource utilization will quickly drive an Infrastructure Consolidation to Multi-Tenant servers and logically separated databases.

The exceptions to this rule are SaaS that have few very large clients or clients with high security requirements.  These SaaS will have to price and market themselves accordingly.

Infrastructure Consolidation is an early driver away from a pure Single Tenant model to Multi-Tenancy. The change is mostly positive for clients, but does add additional security and client satisfaction risks.

If you are enjoying this series, please subscribe to my mailing list so that you don’t miss an installment!

Build Allies By Asking “How did you solve”?

Intractable technical problems are a fact of life.  Architectures make seemingly easy use cases impossible.  Critical code won’t scale, because of meaningless choices made years ago.  There are endless tech problems that defy easy and obvious solutions.

Earnest, well meaning, developers who come up with solutions that won’t work are also part of life.

After you’ve been thinking about a problem for months or years it can be tempting to tell a developer why their solutions won’t work.  Lead the developer down your chain of reasoning.  Show them how much more you’ve thought about the problem.  How you’ve considered their solution, and a dozen others. Be superior and dismissive.

Prove that the intractable problem can’t be solved until you get the developer to say “Ok!  Ok!  You’re right!  It won’t work!”

Or, you could be open to being wrong and build an ally.

Agree with the basic idea of the solution.  That seemed like a good idea to you too! Then, instead of telling them why they can’t, ask the developer how they got around the intractable problem.

“How did you solve it?” is interested and hopeful.  It tells the other person “I have spent time thinking about this problem too.  Here’s where I got stuck.  I’d love to hear how you think we can solve this problem.”

There’s almost never a solution.  Most of the time the other person had no idea that the architecture wouldn’t support it, that the message size is too big, or any of the subtle technical reasons why the solution won’t work.

Asking about the solution, instead of preaching the problem, puts the two of you on the same side.  It makes you allies against the problem and sidesteps the teacher/student power dynamic of “That won’t work.”

Sometimes, a fresh perspective does produce a solution to the intractable problem.  Since you avoided staking a claim that there was no solution, you won’t feel the sting of being wrong.  You won’t end up defensive, and can embrace the developer’s solution.

“How did you solve it” builds allies. 
“That won’t work” harms you both.

Testing For BitRot – 2022

Welcome back to regular Wednesday posting!

Starting next week I will have new content every week, mostly on Wednesdays.

This blog covers scaling SaaS software: common gotchas, anti-patterns, performance and scaling strategies. I cover the technical and social aspects of these problems because it is often harder to convince people than implement code.

Thank you for reading!

Site Footer