Observability 360 Editor
Any attempt to map the observability space is probably doomed to be, at best, a heroic failure. The space is a ceaselessly moving target, products don't easily fall into neat categories and nobody will agree with your classification system. At the same time, there is probably a basic human instinct at work. To have some kind of organising model to comfort us that there is some order and structure in the particular cosmos we inhabit.
We have made the initial launch a "beta". The reason for this is simple. Observability is the Madagascar of IT. It is pretty hard for one small group of individuals to have a the full take on so many different species in such a rapidly evolving space. There is no canonical classification system and there is no supreme court making final judgements. But dialogue, feedback and the hive mind can help to plot out this unique and highly active corner of the tech universe.
There are a number of other mappings out there and this is not an attempt to be "right" or to argue that other categorisations are wrong. It is a just a different perspective.
Unfortunately, in a two-dimensional image, it is difficult to reflect the fact that many observability applications are multi-dimensional and operate along a number of different axis. That is before we even approach the elephant in the room - what is an observability product in the first place?
Observability 360 has always taken a broad view of this. Observability is a toolkit, not a one-size-fits-all platform. From a specialist tool such as Zymtrace to a full spectrum platform such as Datadog - there is room for all of them in the big tent.
Famously, naming is one of the hardest problems in computing. To be honest though, classification is not far behind. For example - how do you classify a product such as Cilium? The capability of deep inspection into the Linux kernel gives it almost unlimited possibilities and applications. It is not a dedicated Kubernetes observability tool but if you want to understand or configure connectivity on your cluster, it is now probably the default option.
Similarly, Grafana is a shapeshifter which can mean many things to many people. It can mean the visualisation product, the open source stack or the SaaS platform. In this iteration we have settled for placing the Grafana name in a position between Full Stack and Wide Spectrum whilst also allowing seperate listings for Loki and Tempo in the Dedicated products space. .
There were also many other vendors whose products did not easily fall into a particular category. We will just look at two examples - Embrace and Elastic - although we could have selected a number of other edge cases. Embrace started out as a mobile observability specialist but have now extended their offering to include Front End observability. There is obviously no single category that encompasses this profile. Elastic are also very hard to pin down to one category as their identity is rather multi-faceted. They are a very capable application observability platform, but they are also a highly sophisticated platform for AI and search. These are separate axis but it is not really possible to plot this in a two dimensional plane.
It is also interesting that the distribution of stars around this cosmos is not even - like the physical universal itself, it is somewhat lumpy. There are clusterings of stars, but there are also large areas of empty space. We would expect to see the Meta Tooling space to become more populated but other areas are harder to predict. Is the AI SRE sector saturated or is there still room for new entrants. We will know the answers before long.
In schemas such as the Gartner Magic Quadrant, the positioning of a product along a certain axis is often taken to imply a value judgement (even if that is not what Gartner intend). The positioning of a product on this map does not imply any kind of qualitative scoring. If one product appears above another that is just a coordinate, not a score. At the same time, horizontal positioning can have some meaning.
As you will see, across the central belt of the cosmos we have mapped a progression from dedicated systems, to full stack and then on to wide spectrum observability. This is a progression which is designed to imply an increasing range of functionality. It doesn't imply that a product which is further to the right is "better". It is a just a progression from a dedicated tool to a swiss army knife.
This initial iteration covers 125 products across 16 categories. The categories themselves will inevitably evolve as the market does. Click on the links below for a brief summary of each category.
If you want to quickly find the location of a product just scroll down the list on the left hand side of the screen. When you hover over the product name you will see its star pulsing proudly. If the star is burning bright yellow you can click on it and see a product card.
This is not a complete or final picture. During the course of building the map we have repeatedly changed the names of classifications and found ourselves adding new classifications. As vendors have expanded the scope of their products, they have also had their position re-plotted.
Our aim is to publish a new iteration each quarter alongside a summary of new entrants, movers and even leavers. Hopefully there will be very few of the latter.
Watch the skies!
The excellent Alex Ewerlöf blog is now back in full swing and in this latest article he turns his attention to dealing with a real SRE curveball - how to build reliability engineering for an air-gapped system. We are talking hermetically sealed - not even a maintenance window for external connectivity. This task involved a high-security military facility where installing updates meant physically handing over an archive file to a system operator.
The constraints were pretty stringent - no logs, no metrics, no traces, no remote access of any kind.This was an extreme case and, ultimately, the solutions had to be both human-centred as well as relatively low-tech. How would you address the challenge? Hit the link below to read about the solution that Alex put in place.
In the past month or so the GitHub platform suffered a number of well-documented outages which resulted in loss of service for users. In the spirit of transparency, GitHub CTO Vlad Fedorov published this article on the GitHub blog, explaining the causes of the outages and the lessons learned as well as detailing the remediations that GitHub engineers will be putting in place.
The article really brings home the challenges of orchestrating the components of a global technology infrastructure - as well as the compounding effects of working at very large scale. The investigation revealed a perfect storm of edge cases, hidden tipping points and unforeseen knock-on effects. It’s impossible not to feel for the engineers sweating in the war-rooms as the dramas unfolded - after all, watching your failover fail must be pretty gut-wrenching.
If the Observability world had a code of secrecy akin to that of the Magic Circle, then Charity Majors might be in danger of being banished to exile and ignominy. In a single slide deck, she has blown the gaff on a whole trove of insider knowledge. It is the “What they don’t teach you at Harvard Business School” of observability knowledge. Not the abstract theory or technical detail but lessons and insights from the o11y frontline.
The deck in question was used in a talk at the LeadDev event in Berlin earlier this month and its 52 slides are an illuminating distillation of observability wisdom. We actually weren’t present at the talk and only came across the deck thanks to a mention in Michael Hausenblas’s excellent olly news newsletter. However, the slides contain sufficient clues (and images of unicorns) to easily re-construct the narrative and win friends and influence people as an observability savant.
If you are an SRE, when an outage happens you will know about it pretty quick. With security breaches the picture is rather less clear as, by their nature, they are designed to go undetected. Intrusion detection therefore is often based on a mixture of tools designed to spot unusual spikes, suspicious patterns or failed logon attempts.
This article by Fatih Koç argues that one of the major difficulties involved in identifying attacks is that of correlating signals across multiple sources such as Falco, Prometheus, Kubernetes Audit Logs etc. In this article, he outlines a strategy for extracting relevant data from each of these sources and pulling it together into a single observability dashboard.
The first line of cyber defence is normally at the perimeter - preventing attackers from entering your network in the first place. The next line of defence is intrusion detection. This can often take the form of anomaly detection using a variety of heuristics.
There are also some more creative possibilities, such as the canary solution adopted by Grafana. Just as the canary in the coalmine sings to alert underground workers to the presence of toxic gases, Grafana’s canary was designed to alert them to the possible presence of intruders in their domain.
Load testing can be simple in theory but in modern distributed architectures, it involves a lot more than throwing requests at an individual service. This article on the Airbnb engineering blog looks at how the company’s engineers use the Impulse load-testing framework to handle a number of more complex requirements such as dependency mockingand managing messaging and asyncronous calls.
Unfortunately, at the moment Impulse is just an internal Airbnb framework, so you won’t be able to get your hands on it at present. At the same time, the article provides a valuable blueprint for tackling advanced, real world load testing scenarios.
It’s an announcement that might have seemed unthinkable not long ago, but the porting of the revolutionary eBPF technology to Windows is now a reality. The ability to bring safe programmability to the kernel has resulted in enormous gains in fields such as security, networking and observability for Linux hosts, so applying the same principle to the Windows ecosystem is obviously an attractive proposition. It is not, though, without its own difficulties. There were a lot of hurdles to overcome and, inevitably, given the differences in OS architecture, this is not a full-fidelity replica of the Linux implementation.
This possibly foundational article by Pavel Yosifovich guides you through the steps involved in boldly going where few have gone before and creating your first eBPF program for Windows. One paragraph in the article begins with the sentence “this is where things get a bit hairy“ - for some that will likely be a challenge rather than a deterrent. This may not be cooking up nuclear fusion in your bedroom, but it does feel pretty radical.
As well as rolling out their Open AI observability solution, Elastic have also been very active within the OpenTelemetry project. C++ has a reputation for being something of a fearsome foe for observability practitioners. In this article on the Elastic blog, Haidar Braimaanie dons his protective gear and attempts to tame the beast with a soothing dose of OpenTelemetry instrumentation.
Unlike languages built in frameworks such as .NET, C++ does not have a standardized runtime environment that supports dynamic instrumentation across all platforms and compilers. C++ also uses a variety of build systems such as Makefiles and CMake, so that implementing instrumentation can be difficult and error-prone. In the article, Haidar looks at adding OpenTelemetry support to a C++ application running on Ubuntu 22.04. He also includes sample code for instrumenting the project with database spans and then observing the application in APM.
After reading this article you may want to give the C++ developer in your life a hug.
Even if you are not familiar with the name of Brendan Gregg, you are almost certainly familiar with the fruits of his labours. Brendan is the creator of the Flame Graph - one of the most important and iconic visualisations in the observability toolkit.
We featured the Flame Graph in our recent tribute to the work of UX designers in the observability arena - but you should also visit Brendans’ web site.
Brendan’s latest innovation is the AI Flame Chart. This is an evolution of the original flame graph and its ambitious aim is to help reduce the vast financial and environmental costs entailed in the use of LLM’s. This means that whereas the original flame graph was focused on CPU cycles, the latest generation sets its sights on reducing GPU load. The article discusses the considerable complexities involved in mapping GPU programs back to their corresponding CPU stacks.
The names of some of the instruction sets look intimidating to the uninitiated but the basic concept of the graph is quite simple - the wider the bar, the more resource it consumes.
If you have ever had to grapple with a 3,000 line Helm chart to deploy your observability infrastructure, you may be forgiven for thinking that there must be a better way to do this. Whilst YAML has a certain formal elegance, its syntax struggles to express the architectures and relationships embedded in highly complex systems.
Whilst Pulumi have tackled this problem by enabling the use of high level programming languages for IaC, System Initiative are taking a fundamentally more radical approach. Their goal is nothing other than completely reinventing IaC from the ground up. The blog article for the launch of the product is an incredibly ambitious statement of intent. The terms ‘game changer’ and ‘paradigm shift’ tend to be thrown around somewhat casually, this might be a case where their usage is appropriate.
So, what are they proposing? Well, System Initiative is IaC without the code. It is a kind of digital canvas where you manipulate digital twins of your systems. Is the future here or is this the Platform Engineering equivalent of science fiction? Read the article and decide for yourself!