I keep oscillating between these two camps. Simple health checks which return true tell you very little about the actual health of your service and are mainly there to just keep the load balancer happy. More complex checks might result in the situation where a temporary blip in a dependency ends up with all your nodes being taken out of the load balancer.

Or something like this happening.

These are the kind of outages that stay with you. Thankfully, some load balancers implement a special strategy called fail-open to deal with the case when all servers start failing health checks. This from the Amazon Builders Library.

For example, the AWS Network Load Balancer fails open if no servers are reporting as healthy. It also fails out of unhealthy Availability Zones if all servers in an Availability Zone reports unhealthy. Our Application Load Balancer also supports fail open, as does Amazon Route 53.

Anyways, there are two issues with thinking about health checks in this binary way.

• The health of a service lies on a spectrum.

• The perception of health needs to include the client side perspective.

Health of a service is not binary

This insight is picked almost verbatim from Cindy Sridharan's lovely essay. From the essay:

The “health” of a process is a spectrum. What we’re really interested in is the quality-of-service — such as how long it takes for a process to return the result of a given unit of work and the accuracy of the result.

The essay goes on a little beyond the point of health checks and makes some very interesting points. The key takeaways for me:

• For the layer that determines whether to give traffic to a particular node or not, its more interesting to understand whats the ability of the node to handle the work thats being sent its way, rather than just whether it's up or not.

• A large percentage of outages can be avoided by using various graceful degradation techniques. This involves creating some mechanism of applying back pressure to signal that the service is overloaded. One such example is Qalm from Uber.

• Back pressure needs to be propagated all the way up the call chain, if its not there would be some degree of queuing at some component of the ecosystem.

So yes, build some backpressure in your normal workflow is one way to indicate distress while not making a bad situation worse.

Health lies in the eye of the client

From Steve Yegge's famous platform rant

monitoring and QA are the same thing. You'd never think so until you try doing a big SOA. But when your service says "oh yes, I'm fine", it may well be the case that the only thing still functioning in the server is the little component that knows how to say "I'm fine, roger roger, over and out" in a cheery droid voice. In order to tell whether the service is actually responding, you have to make individual calls. The problem continues recursively until your monitoring is doing comprehensive semantics checking of your entire range of services and data, at which point it's indistinguishable from automated QA. So they're a continuum.

If the perception of a service's health is completely internal it will miss a lot of outages. One of the most common things I advise developers to do when setting up SLOs for their service is to set it up from the client perspective. A very similar point is also made in the Metrics that Matter talk from Google.

If you are to take away anything from this, remember to instrument your metrics from the client side, those are the ones that really matter.

Measuring Service Health

So, how do we think about service health and health checks. The Amazon Builder's Library again gives us a great framework around this. The define 4 kinds of health checks:

Liveness Checks: These test for basic connectivity and the presence of a server process.

Local Health Checks: These check resources which are not shared by the peers of the server. (So, no dependency checks). Examples include checking the ability to do disk I/O, checking supporting daemon processes which put functionality at risk in subtle, difficult to detect ways.

Dependency Health Checks: These checks check for the ability of the server to communicate with dependencies, if any metadata that is consumed by the server is stale or other issues. If any automation is built around the dependency health checks, the right amount of thresholding needs to be built in to prevent the automation from taking any drastic action unexpectedly.

Anomaly Detection: These check for if a server is misbehaving compared to its peers. Examples include clock skew, old code running, extra reported latency etc.

The first two are good candidates for a traditional /health endpoints. The latter two are health checks which are performed from a point of view that is external to the service. These are good candidates to monitor and alert some human operator on.

I think that settles the debate, at least in my in my head, about service health checks

Just like Rob’s dilemma in High Fidelity, what came first in distributed systems is an hard question to answer. This is because coming to an agreement on what time it is now in a network of computers is difficult. 

In this post, I want to discuss the two kinds of time that are relevant when we talk about distributed systems, the characteristics of both, the different kinds of problems that they solve and their usage in real world distributed systems.

We will discuss:

• Physical time

• Logical time

• How Dynamo and other dynamo like systems use vector clocks

• How Spanner provides the highest level of consistency using Truetime API

Physical Time or Wall Clock

The first kind of time is time as we know it, the time we get by looking at a clock. In most cases, a process on a computer finds the physical time using the gettimeofday() system call. Physical time on a machine is usually calculated with the help of quartz crystals that vibrate at known frequencies when electricity is applied to them. Like with everything you get what you pay for.

In an ideal world, all the machines would be perfectly synchronized since that would give us a mechanism to order events no matter what node they originated in. For a more detailed treatment of how synchronized clocks make a lot of distributed algorithms easier, look at this paper from Barbara Liskov. Our world is far from perfect though and locks in different machines can gloriously go out of sync. Hence the need for clock synchronization algorithms.

Most clock synchronization algorithms work on probabilistic assumptions about clock rate and message delay. So they synchronize clocks with some skew epsilon i.e. they guarantee that if c1 and c2 are clocks on two nodes of the network the difference between the time at c1 and the time at c2 does not differ by more than epsilon with some very high probability.

The most popular synchronization algorithm is NTP which is used to bring a computer’s clock to within a few milliseconds of UTC. (Sidenote: Whenever needed to make a choice of a timezone, always use UTC.) 

NTP uses a strata of clocks with the lowest stratum time server (stratum = 0) synchronized with a high quality clock such as an atomic clock or GPS. NTP also has to deal with leap seconds. Leap seconds are 1 second adjustments that need to be applied to UTC to make sure it doesn’t fall out of sync with observed Solar Time. The default is to pause the clock for a while but smearing the leap second over 24 hours is becoming more popular and is used by both Google and Amazon.

Even with a functioning NTP running on your servers, the skew between locks range from anywhere between 100 -250 ms. There are certain algorithms which can provide tighter skews at the cost of not being internet scale such as PTP, which is used in the High Frequency Trading world. There is also the Google TrueTime service which we will be discussing a bit later while discussing use of physical clocks in real world systems. 

Getting a tighter skew costs a lot more though and sometimes all we are concerned about is what happens within a distributed system. Enter Logical time.

Logical time and Vector Clocks

Let’s say we want to determine the order of events within a distributed system. We can use physical clocks, but as we just discovered physical clocks are not a panacea. So is it even possible to define what happens before what without using physical clocks. 

This is exactly what Leslie Lamport covered in his seminal paper Time, Clocks, and the Ordering of Events in a Distributed System. Lamport defines his world of distributed systems as n nodes running and exchanging messages with each other. He then defines the happened before relationship ‘→’ as follows.

(1) If a and b are events in the same node, and a comes before b, then a → b. 

(2) If a is the sending of a message by one node and b is the receipt of the same message by another node, then a → b. 

(3) If a → b and b → c then a → c. 

Two distinct events a and b are said to be concurrent if we cant say whether a happened before b or b happened before a.

He then introduced the concept of a clock, where the clock is an arbitrary number assigned to an event. The Lamport clock condition L is as follows.

If a → b then L(A) > L(B).

These clocks, also known as Lamport Clocks,  therefore can be implemented as monotonically increasing integers which are incremented on every operation in a node. 

While Lamport clocks give us a way to capture potential causality they are very limited. If L(A) < L(B) then the only thing we can say for sure is that B did not cause A. They do not help us understand whether A and B were concurrent. 

To solve this, vector clocks were introduced which are a generalisation of Lamport clocks where each node tracks the maximum Lamport clock that it knows of, of every other node. 

Checking the happened before relationship between two events x and y translates to checking if each entry in the vector of x is smaller or equal to the corresponding entry in the vector of y and one is strictly smaller. If not, they are concurrent. 

The authors of this article provide another way of thinking about vector clocks in terms of tracking causal histories. 

Causality can be tracked in a very simple way by using causal histories. The system can locally assign unique names to each event (e.g., node name and local increasing counter) and collect and transmit sets of events to capture the known past.

For a new event, the system creates a new unique name, and the causal history consists of the union of this name and the causal history of the previous event in the node. For example, the second event in node C is assigned the name c2, and its causal history is Hc = {c1, c2}(shown in figure 2). When a node sends a message, the causal history of the send event is sent with the message. When the message is received, the remote causal history is merged (by set union) with the local history.

In our example, here’s how the tracking of causal histories would look like. 

The astute reader might notice that Vector Clocks are essentially a compact encoding of causal histories.

This article provides a thorough treatment of vector clocks if you are interested in looking into them more.

I should mention there is a huge class of algorithms called consensus algorithms such as Paxos, Viewstamped Replication, Zab/Zookeeper, and Raft. They also provide ways of defining an ordering of events across a distributed system even though physical time cannot safely be used for that purpose. I intend to discuss this soon in a futue blog post.

Vector Clocks are great if all we want to do is to track order of events within a system, but the moment we want to introduce an observer outside the confines of the distributed system (for example a client), the limitations of vector clocks becomes quite apparent. Katy Perry captures the limitations of vector clocks very well in the following lyrics.

There are things which only make sense in the realm of physical time. One such example is the concept of failure of a node. Typically we say a node has ‘crashed’ when the client has been waiting too long for a response. Too long is a concept that only exists in physical time. Without physical time it’s impossible to say whether a node is dead or merely pausing before doing its next thing. 

Another such concept where physical time is necessary is the concept of external consistency while talking about transactions in a database.

“A violation “of external consistency occurs when the ordering of operations inside a system does not agree with the order a user expects.”

Or in other words, External Consistency basically says that for two transactions T1 and T2

if T2 starts to commit after T1 finishes committing, then the timestamp for T2 is greater than the timestamp for T1.

External consistency is a tighter guarantee than the C promised in CAP because its defined for transactions whereas the C in CAP is equivalent to linearizability defined only for a simple read write register and not objects. 

So, we understand physical time and the problems associated with it. We understand Vector clocks and their limited applicability to certain problems. Next I want to look at how real world systems use both these concepts and the implications that they bring.

Use of Vector Clocks in Dynamo

This paper was published by Amazon in 2007. It described an unapologetically eventually consistent key value store used internally within Amazon. Eventually Amazon released this as a managed data service called DynamoDB. This paper inspired a lot of dynamo style systems such as Riak and Cassandra and is quite seminal work in the NoSQL space. I strongly recommend you read it. 

For concurrent updates Dynamo as described in the paper offered two reconciliation mechanisms. 

• Client specific Reconciliation where when conflicting versions were detected the responsibility to resolve them was passed to the client application.

• Last write wins reconciliation where the object with the biggest physical timestamp is retained as the latest version.

The client specific reconciliation uses vector clocks to detect when divergent versions of an object exist. 

“Dynamo uses vector clocks in order to capture causality between different versions of the same object. A vector clock is effectively a list of (node, counter) pairs. One vector clock is associated with every version of every object. One can determine whether two versions of an object are on parallel branches or have a causal ordering, by examine their vector clocks. If the counters on the first object’s clock are less-than-or-equal to all of the nodes in the second clock, then the first is an ancestor of the second and can be forgotten. Otherwise, the two changes are considered to be in conflict and require reconciliation. In Dynamo, when a client wishes to update an object, it must specify which version it is updating. This is done by passing the context it obtained from an earlier read operation, which contains the vector clock information. Upon processing a read request, if Dynamo has access to multiple branches that cannot be syntactically reconciled, it will return all the objects at the leaves, with the corresponding version information in the context. An update using this context is considered to have reconciled the divergent versions and the branches are collapsed into a single new version”

The diagram below shows an example of this might work.

This concept was also borrowed in other dynamo style systems, most notably Riak. The problem with using vector clocks is that you might end up with a very large number of concurrent versions or siblings as riak calls them and you will need to trim them and its critics argue that this is unnecessarily complicated.

The publicly available version of DynamoDB only seems to have the last writer wins reconciliation mechanism so it seems like the critics (most notably Cassandra) can feel justified in their opinions. 

Spanner and TrueTime

Spanner is Google’s highly scalable globally replicated database. It’s unique in providing external consistency guarantees at a global scale. One of the key aspects of its design is the Truetime API which explicitly exposes the uncertainty around time. 

The TrueTime API has 3 methods

TT.now: This returns an interval [min, max] in which the absolute time at which this function was invoked truly lies.

TT.after(t): Returns true if the time is definitely after t.

TT.before(t): Returns true if time is definitely before t.

The Truetime API provides much tighter guarantees around the uncertainty of time than NTP does. The uncertainty in Truetime is around 7 ms, the equivalent of NTP has an upper bound of 250ms. Truetime achieves its tight time bounds by using underlying time resources of GPS and atomic clocks which provide highly accurate time.

With the uncertainty of time being so tight, Spanner achieves external consistency with a very simple trick which it calls commit wait. Let’s revisit the definition of external consistency again

if T2 starts to commit after T1 finishes committing, then the timestamp for T2 is greater than the timestamp for T1.

In Spanner, for every transaction, a node assigns a commit time to the transaction which is the upper bound of the TT.now() call. Before a node is allowed to communicate that a transaction has been committed, it must wait out the uncertainty, typically around 7ms, which guarantees that we are now definitely past the commit time assigned to the transaction. 

Recall that by definition the “true” start time of any competing transaction T2 will be after the commit time assigned to the original transaction. The timestamp assigned to T2 will be greater than or equal to this “true” time as it also has to wait out the uncertainty. Voila! we have external consistency. (It took me a while to wrap my head around this). 

There is another system, CockroachDB,  which uses this concept of uncertainty of time but on commodity hardware, to provide consistency guarantees which are second only to Spanner. From the blog

When CockroachDB starts a transaction, it chooses a provisional commit timestamp based on the current node's wall time. It also establishes an upper bound on the selected wall time by adding the maximum clock offset for the cluster \[commit timestamp, commit timestamp + maximum clock offset]. This time interval represents the window of uncertainty.

As the transaction reads data from various nodes, it proceeds without difficulty so long as it doesn't encounter a key written within this interval. If the transaction encounters a value at a timestamp below its provisional commit timestamp, it trivially observes the value during reads and overwrites the value at the higher timestamp during writes. It's only when a value is observed to be within the uncertainty interval that CockroachDB-specific machinery kicks in. The central issue here is that given the clock offsets, we can't say for certain whether the encountered value was committed before our transaction started. In such cases, we simply make it so by performing an uncertainty restart, bumping the provisional commit timestamp just above the timestamp encountered. Crucially, the upper bound of the uncertainty interval does not change on restart, so the window of uncertainty shrinks. Transactions reading constantly updated data from many nodes may be forced to restart multiple times, though never for longer than the uncertainty interval, nor more than once per node.

Since CockroachDB runs on commodity hardware running NTP, this uncertainty interval can go upto 250 ms, which sucks but this should not happen for most transactions.

That concludes our journey of time in distributed systems. As a followup, an interesting area of research in distributed systems are CRDT(Conflict free Replicated Data Types) which attempts to avoid the need to worry about time and order. They provide a kind of safety that is sometimes called strong eventual consistency: all hosts in a system that have received the same set of updates, regardless of order, will have the same state provided that the updates to the system are commutative, associative, and idempotent. The research tries to explore how expressive we can be within these constraints. I want to cover CRDTs in my journey eventually.

There were a huge amount of resources which helped me in understanding all the nuances around time. I am sure, I probably missed a nuance or two. If you want to look at them further here are the ones I recommend:

• Time, Clocks and Ordering of events in a distributed system - The original paper describing Lamport clocks by Leslie Lamport

• Practical uses of synchronized clocks in distributed systems - A paper by Barbara Liskov describing how one can take advantage of synchronized clocks in certain distributed algorithms

• Logical Clocks are easy A handy way to understand and reason about Logical Clocks.

• There is no now Another interesting article exploring the problem with time in distributed systems.

• Time is an Illusion A paper exploring physical time and synchronization problem in more details.

• These flamewars and discussions around the use or lack there of of vector clocks in Riak and Cassandra. Bonus: Cassandra’s comparison with DynamoDB.

• The Dynamo and Spanner papers.

• Falsehoods programmers believe about time.

If you have reached until here, please consider following me on twitter.

Similar scenarios have been discussed by Charity Majors in  Know your One Job and Do it First and Tanya O’Reilly about glue work. I think there is a distinction between the two protagonists as described. 

The protagonist in Charity’s post was involved in a variety of tasks without having clarity on what the organisation cares about, whereas the protagonist in Tanya’s post is doing the invisible work required for the organisation to be successful. The latter is glue work. It’s important to distinguish it from filler work. Filler work is low quality work which fills up the day but its value is questionable. 

When I reflect upon what I was doing, it was a lot of filler work. In the aftermath of me not getting promoted, I tried to develop a framework of what I should spend time on.

Who is the work important for? 

A mentor gave me some advice which is to always 3 set of objectives:

• One set of objectives that are important to the organisation

• Another set of objectives that are important to your manager

• Final set of objectives that are important to you.

I think this is a useful framework to manage expectations mostly with yourself as to how you can expect to be rewarded. The first two you can reasonably be expected to be rewarded for by the organisation, the third you are doing for yourself. Any reward is a nice bonus. The ideal state is when there is significant overlap in the 3 things, but reality is seldom that clean.

I would also like to add that it’s important to distinguish between important and urgent tasks. Spending time on urgent things might feel that you are doing important work, but in reality you are not. If you find yourself spending time on urgent things too much of the time, take a step back and evaluate.

Figuring out what’s important to you sounds probably the easiest of those problems but is harder than you think. The best framework I have found so far is this post from Will Larson on structuring a 40 year career. A lot of things which fall under glue work - stakeholder management, writing great documents can also reasonably fall under here. While you might not be rewarded immediately, this is an immensely useful muscle to develop for leadership roles.

The first step in figuring out what’s important for your manager is to ask them. It’s also important to determine which of these objectives are must haves for them and what are good to haves. If you choose to spend time in the good to have objectives be prepared to not be supported from your manager. It’s good to have a continuous sync with your manager if any of these realities have changed. 

When it comes to what's important to the organisation we have the things that are stated explicitly in terms of business goals. The real signal here is though what the organisation rewards and more importantly what it does not reward.   

For example while a lot of organisations say hiring might be important for them, few are structured in a way to incentivise engineers spending significant time on it. Mentoring people is another that the organisation might not structurally reward, but mentoring has a lot of long term benefits, so my suggestion to most people would be to stick to it.

I sincerely wish managers and organisations would be to have their actions match their words. People ultimately figure this out anyways and a gap between the two causes a lot of erosion of trust. One particular insidious failure mode which I have seen of this is where leaders pile on their personal projects on the teams whilst not having alignment amongst themselves. 

What about changing what's important for the organisation?

Most overachievers have a strong desire to bring about change where they work. If you are successful in bringing about this change it can have significant upside but conversely the way there is very attritional. 

Let’s be clear about something though -  the fact that you want it to be important doesn't necessarily mean it actually is. Managers often make this problem worse by not giving their reports feedback about what's important. Often because it’s unclear to themselves. 

If I distil my advice, it’s basically to do something important for someone important. It is very simple but it’s hard to get right, especially among the noise of busyness in big organisations.

If you are here, maybe you should follow me on twitter!

As we make changes in these systems, we increase their Software Entropy. As stated in the seminal paper, even against the best efforts of the most architecturally conscious organisations, most systems tend to end up as a Big Ball of Mud. A lot of software strategy is essentially about managing this entropy.

I daresay, software strategy comes a little bit later in the organisation’s life cycle, as initially it needs to succeed as a viable business. This makes total sense.

So let’s assume we are a successful business and own a set of systems which power this business. We are ready to form a strategy to deal with the entropy of these systems. Most typically we start with a system that is of most importance to the business. We choose an approach to deal with the entropy of this system and it seems to work. 

Often, having seen some success with the approach, we declare victory and rollout the same approach across all our systems. These attempts often end up in failure and at huge costs. I personally am guilty of going down this route.

Turns out, this is so common that there is a law about this. It’s called Ashby’s Law also known as The Law of Requisite Variety.

Or put in a more approachable way

“In order to deal properly with the diversity of problems the world throws at you, you need to have a repertoire of responses which are (at least) as nuanced as the problems you face.”

Essentially, we need a variety of strategies to manage different systems in our software portfolio. Depending on the various attributes of each system, we can then apply a strategy to manage their respective entropy.

Let’s look at some common strategies that one can apply to each system in your software portfolio.

Do nothing

This is a valid strategy when the domain that you are trying to explore is new to you. This applies to new systems being built in new domains that your business is navigating. It’s suited for exploration. 

This is also a commonly adopted strategy by people when the software has not been tended to for a long time.Then it becomes more of a by-product of inertia. 

Refactoring

Once you have an understanding of the domain, before the system gets out of your hand, you need to make sure that the software now represents the domain and the problem you are solving. This activity needs to be done continuously and with discipline. 

In practice, I have seen refactoring often fail. This happens because of a variety of reasons. If you let your system grow too much, if the person or the people doing the refactoring are not using the right abstractions, if the abstractions are not communicated well in the team, if the tools you have to protect your refactoring from getting smudged all of this plays a part into this.

Rewrites

Rewrites, while being extremely desired by engineers, are an extremely risky choice.  As Joel Spolsky wrote in his essay, engineers love this because it’s harder to read code than it is to write it. 

A not exhaustive list when they make sense are:

• When the abstractions that you originally started designing the system with, no longer hold.  While not strictly a rewrite, the evolution of Stripe’s API is a great example of  this. The Stripe API initially started out  with the idea that payments are finalised instantly as their primary use case was credit cards. Finalisation means that the user has sufficient confidence is guaranteed. However with most payment methods, including crypto, finalisation takes a while. Stripe’s API was then redesigned with this abstraction in mind. 

• The other case in which a rewrite makes sense is if you are going for a paradigm shift. Uber rewrote their app to fundamentally utilise functional/reactive patterns and redesign their UI to allow multiple product teams to work together. Personally, I worked on a project at work, where to keep a handle on latency of our user funnel, we rewrote the backend so as to not make any database queries. All the requisite data was stored locally on each box and any updates propagated through kafka streams. The astute reader will notice this is just CQRS in practice.

However rewrites are extremely risky. At best they slow you down massively during the rewrite, at worst they can be threatening to your very existence. The current system has acquired its calluses the hard way and those calluses serve a purpose. Translating that into a new system always takes longer than you thought. Maintaining backwards compatibility is hard. You might end up replacing your system with an over-engineered monstrosity - which Fred Brooks in the Mythical Man-Month called the Second System Syndrome.

You have been warned. 

Reclaim

For the longest time, I thought that you only had the 3 choices that I mentioned above. However, I recently came upon this article by Will Larson. Will suggests translating your beliefs about your system into the desired properties and behaviours. Then one can try to implement validations and assertions of these desired properties and behaviours and eventually you would be able to reason about your software.

Another approach of reclaiming systems which have regressed into incomprehensible monstrosities is to break them apart into logical services. Then we implement SLOs and SLIs, clean up the APIs and iteratively try to reduce the entropy in each individual system. Sounds a lot like Will’s idea, doesn’t it?

This strategy makes most sense for systems which, while chock full of important business logic, have fallen into a state of disrepair due to lack of attention. This can also be a very valid strategy before embarking on a full rewrite. Once you can reason about the system slightly better, you can choose how best to rewrite it and which parts.

Replace

I think the best software to have in your portfolio is one whose maintenance you are not responsible for. You could probably use X as a service especially when X is not one of your core capabilities i.e payments, communications platform etc. If it doesn't bring you competitive advantage, you probably should not be building it.

If you have already built it, chances are this system is more likely to decay, for the simple reason that it’s not your priority. Replace it with a commoditised service. Increasingly, there are also offerings like Retool which let you quickly build internal tools.  This is a topic I want to explore more in the future.

These are the strategies I have commonly seen applied. While the top 3 are quite common, I think Reclaim and Replace are an incredibly useful lens with which to look at your system.

Humans have a tendency to crave one size fits all strategies. Perhaps we deal with complexity by pretending that the complexity does not exist. We end up fighting against Ashby’s Law and losing.

Let’s remember, the world is complex.

And Variety beats Variety.

Based on how students reason about complex systems, they were placed in 4 categories in order of increasing sophistication: Completely Clockwork, Somewhat Clockwork, Somewhat Complex and Completely Complex.

“Clockwork responses are those that show deterministic, linear, single-cause, non-networked, centralized, or static system interactions or states, whereas complex responses are those that demonstrate nondeterministic, nonlinear, multiple causes, networked, decentralized, or dynamic system interactions or states”.

In my experience this translates extremely well to the progression of a developer and their understanding of the complexity of the ecosystem they operate in. So, I wanted to attempt to derive a learning progression for developers.

At each level of development, I recommend a few books. Each book hopefully is a thread you can pull on, if you want to go deeper into a particular topic.The book recommendations are subjective, but I believe that one can chart their own curriculum following the basic model. The recommendations that I make are just based on my own experience and aim to serve merely as guidance.

When it comes to progression to the next level, my general advice would be to master the level you are operating on and then aim to understand whats needed to operate at the next level.

Completely Clockwork

At this level most of the work is involved in making changes in a single service or component. This is where most of the ‘coding’ happens. The amount of time that will be spent reading your code will be multiples of the amount you would take to write it. You will work more with existing code rather than writing new code. A corollary of this fact is that you need to know how to make your code easy to change.So your focus should be on writing readable code and making it easy to change.

Your scope of work will probably within your team and its very likely that your circle of influence, impact and visibility would mostly be your team as well. The abstraction at which you operate in is your team.

The advice I would give at this level is to optimise for seeing problems solved through different reference points.

My recommendations for this level would be:

• Refactoring

• Clean Architecture

Somewhat Clockwork

As you begin to build familiarity and also gain experience, you might see that increasingly you begin to look outward from the immediate scope of your team. You begin to try to understand how the surroundings affect the goals of your teams. You start going on organisational side quests to achieve your team’s goals. The abstraction at which you operate is still typically your team but the way you look at things becomes more outward looking.

In modern software development, distributed systems are ubiquitous. Even when you are making a change to a single service, that service most likely lives in an ecosystem of other services and a plethora of other distributed tools. As you grow in seniority you need to understand how best to interact with other services and also the tradeoffs involved with each tool. 

Books for this level

• Designing Data Intensive Applications 

• Building Microservices

Somewhat Complex

As you gain more experience, you would have ideally developed some calluses, had a ring side view of some heart stopping close shaves. This will prepare you to operate at next abstraction: teams of teams. You would be advising on how to organise software and teams best to make teams successful in aggregate. You could be either leading migrations to address technical debt or reclaiming unreasonable software. It will also probably be your call as to when to do what.

A practitioner operating mostly at this level needs to be able to use models to explain the overall system or domain they are operating in and its interactions with its neighbouring domains. These models are not just to explain the software system but also the teams that surround that software. You need to understand the best models for organising teams to drive maximum impact. Additionally, as the operator of complex systems they wear two hats : that of the producer and that of the protector against failure. The practitioner needs to be able to handle this duality well. 

Books for this level

• Domain Driven Design

• Team Topologies

• SRE Book 

• Accelerate 

Completely Complex

The abstraction that you operate in now is the entire organisation or in bigger companies, an entire division or department. Your focus is primarily on what to build rather than the how. You will be doing a lot of writing, since you need to communicate at scale.

According to the paper, a completely complex system is a dynamic process, which is constantly in a state of flux. Any patterns here are emergent. Changes have both short term and long term repercussions. You also realise that the system is self organised for the most part. This sounds a lot like the modern organisation. At this scope you are typically operating as the technical leadership. You need to be able to form technical strategies grounded in the needs of the business. 

Books for this level

• Wardley Maps

• Enterprise Architecture as Strategy

So to summarise, this is what we end up with as a progression plan for developers over varying degrees of complexity. 

There is a pithy saying - “All models are wrong, some are useful.” The model I have described here is not perfect. The skills and responsibilities needed at different levels are at a continuum, not discrete. I merely aim to expose you to the different abstractions that engineers operate in based on where they are in their career. I hope you found this useful.

If you are here, follow me on twitter.

Over time, it was established that the best way to organise teams is to set them up to be self sufficient. This meant a people with different aptitudes form a team and ideally own the end to end delivery of a product.

Yin and Yang

Simon Wardley of the Wardley Maps fame also settled into this conventional wisdom but found that there was a lot of conflict between his teams. He hypothesized that the conflict was caused because there were fundamentally two types of work - new development and core operations. A similar theory is behind Gartner’s Bimodal IT assertion which asserts that there are two streams of work — one focused on predictability, the other on exploration.

This setup is extremely popular with the rise of platform teams supporting product teams. It was even formalised in an AWS service (AWS Proton) announced at ReInvent 2020.

What Simon found was by grouping his multi-aptitude teams into two groups, somehow it increased the amount of infighting but more importantly the evolution of services/components was not happening.

Evolutionary flow of a Service

One of the central thesis around Wardley maps is

Standardization of components enables creations of better complexity.

This means services evolve from Genesis -> Custom Built -> Product -> Commodity. The services that you choose to evolve basically defines your strategy.

Anyways, so back to the story. Simon Wardley found that in his bimodal world, the evolution of services wasn’t happening. Instead, the operations people kept complaining how lassez-faire the new development people were and the new development kept complaining how much the operations people had a stick up their a**.

Pioneers, Settlers and Town Planners

The hero of the new development people is the ‘Pioneer’. The Pioneer is great at breaking through the organisational ennui and bringing change about. He does get bored quite easily and cant be trusted to write proper documentation.

The hero of the operations people is the ‘Town Planner’. The town planner obsesses about operational health and minimising risk.

You need both of them in your organisation but if you stick them together, they fight. This is something I have seen from personal experience and it benefits neither of them.

The missing ingredient to make them work are the ‘Settlers’. The settlers get along with both pioneers and town planners. They are the Salieri to the Pioneer’s Mozart. They recognise the brilliance of what the pioneers build and make it functional. They give their creation enough structure that Town Planners can take services from them and commoditise them.

The trio also setup a nice cycle for the evolutionary flow of the services. Settlers take services from pioneers and make success happen. Town Planners steal from Settlers and commoditise services. Upon these commoditised services pioneers build their newest innovation. A healthy food chain for the enterprise.

So, Simon realised that the best way to organise team was to make teams of people which had the same attitude (pioneer/settlers/town planners) and different aptitudes (PO/dev/designer). This both kept the peace and allowed for the evolutionary flow of services.

I will be honest, I haven’t seen this in practice anywhere, but I have seen all of these personas. The most successful team I could think of which was a combination of the three but with immense respect between each of them which probably made it work.

What I have instead seen is this happening more organically, because pioneers get bored and moved on. Pioneers usually have a settler than hangs around them who then takes over. Occasionally some of these services are commoditised, though being a town planner is truly an art form. Many town planners often end up designing ghost towns. On the other hand the pioneer’s lassez faire attitude and the zombie systems they left in their wake rubs enough people the wrong way for them to become a political hot potato. Being a settler is a tough balancing act.

I hope that this writeup makes more managers realise the need for all three attitudes and to manage the evolutionary flow of a service. You can do worse than following Simon Wardley on twitter or read through Wardley Maps.

For what its worth very similar ideas first expressed by Robert X Cringely in Accidental Empires, first published in 1993 about the type of personalities that drive innovation in Silicon Valley. The equivalent personality types were Commandos, Infantry and Police.

Also, you should probably follow me on twitter.