AI is no longer just a feature on the observability roadmap – it is reshaping the roadmap itself. This quarter's crop of products shows the range of that change: agentic root cause analysis, baseline-driven system health, tools that plug straight into coding agents, gateways for governing LLM traffic and a refreshingly visual approach to taming OpenTelemetry Collector fleets.
Over the last few years, a number of full-stack vendors have extended their reach into incident and on-call management. It makes sense to build these workflows into a single, integrated solution – even if it does seem to be eating the lunch of the specialist vendors. However, this is a two-way street. Accurate root cause analysis is pretty much the philosopher's stone of AI observability, and everybody is after a piece of the action.
In their latest release, Incident.io have introduced Investigations – an agentic AI RCA tool that springs into action as soon as an incident is triggered. The agent has a holistic view of your enterprise, reasoning across telemetry, code, deployments and previous incidents. This is a very slick piece of tooling, and it makes for a persuasive proposition: sophisticated incident management combined with AI insights.
Of course, before you can explain what went wrong, you need to know whether anything has gone wrong at all. Otterman takes a new slant on this problem. Rather than reacting to ad hoc outages or individual metrics, it learns the patterns and shape of your system and establishes baselines – for Ottermon, the baseline is the metric. This allows it to intelligently judge whether an error is a critical failure or just background noise. Like a number of modern systems, it also attempts to build up the full context of your business operations, rather than relying solely on the traditional observability signals.
AI-driven development is the new normal, and as velocity increases, feedback needs to arrive faster and earlier in the SDLC. We covered Control Theory back in issue 33 of the newsletter, when they emerged as a player in the burgeoning observability pipeline sector. Like other vendors in that niche, they are harnessing AI to redefine their position, and have now launched Dstl8 – which they describe as a "Runtime Feedback Loop for AI-Speed Engineering". One of the great challenges for this kind of tooling is identifying which signals actually represent a meaningful regression. Dstl8 tackles the problem by evaluating telemetry not just against anomalies and patterns, but also against "sentiment".
While Dstl8 works from the runtime back towards the developer, IronBee goes straight to the source. Much AI observability tooling takes an outside-in view, providing retrospective analytics on agent behaviour. IronBee takes a radically different approach, hooking into the running agentic process via MCP, which gives it the ability to reach into every aspect of the agent's activity. It can then insert itself into the agent's verification loop in real time, generating a highly structured context payload that the LLM can reason about without exceeding its context window or burning excessive tokens. The product integrates with source control and CI/CD systems and supports all of the major coding agents – in a fast-moving field, it is probably ahead of much of the pack.
AI-driven development has fundamentally altered the surface area of governance and observability. Interaction with external LLMs now happens throughout the development lifecycle and across all of your environments. Dedicated LLM gateways are gaining traction with organisations that want greater oversight and control of the traffic flowing to and from LLM providers.
Reiver provides an LLM gateway, but it positions itself as something broader – a full AI observability control plane, with tracing, model routing and agent SDKs. Onboarding does not even require a human in the loop: you just point your agent at the docs. Step away from the keyboard, human, we've got this covered.
At the other end of the pipeline, even modest deployments are likely to be running more than one OTel Collector, and as the number grows, fleet management becomes an increasingly important concern for observability teams. If you are in the market for fleet management tooling, then Telflo should be on your shortlist. It is an open source project designed to help you build, configure, test and deploy OpenTelemetry Collectors.
Telflo is built with usability in mind, so you can create sophisticated configurations through a beautiful and intuitive UI rather than hand-crafting thousands of lines of YAML – although viewing and editing in YAML is fully supported. The editor is a real pleasure to work with: you drag and drop components onto a canvas and then refine them in a properties window, and an AI assistant lets you build configurations with natural language commands. Other powerful features include a visual builder for OTTL functions and a testing framework, so you can verify that your telemetry transformations are working before you build and deploy your fleet.
This is an article from the CNCF blog that, although relatively brief, tells the story of an epic IT project. Atlassian are a huge presence in the DevOps space and they operate a correspondingly large-scale metrics infrastructure – 100k hosts running across 14 regions. The task facing the engineering team was to migrate the entire system from their legacy gostatsd (no - that’s not a spelling mistake :-)) implementation to OpenTelemetry.
With thousands of services running across multiple different platforms, the option of simply ripping out gostatsd and switching to OpenTelemetry in one big bang project was not on the table. As the authors note, “the obvious plan (tear out the old pipeline, get every team to re-instrument on the oTel SDK, flip the switch) is a pipe dream”.
This is a really illuminating article that records the engineering decisions that the team made to ensure the successful delivery of a major project.
A lot of conversations about metrics tend to take a widescreen view – dealing with issues of cardinality explosions and hyperscale ingestion. This article by Jesús Espino at VictoriaMetrics takes a more atomic view, looking at the anatomy and lifecycle of a single metric as it is created in memory, travels over the wire and journeys through an observability backend.
This may sound dry and technical, but it unfolds almost like a nature documentary - all that’s missing is a David Attenborough voiceover as our humble metric metamorphoses and migrates across habitats. This is an article that really sheds light on the intricacies of metrics processing and the engineering that goes into maximising performance and optimising storage efficiency at a low level.
AI SREs seem to be everywhere at the moment – both as standalone products and integrated into observability platforms. There is also something of a mystique around them – their algorithms are the IP of the vendor and their training is a closed book. That makes it hard to judge whether a product fits your environment, or even which questions to ask.
As Severin Neumann argues in this blog article, one of the best ways to learn more is to build your own. Even if it never sees the light of day in your production environment, the lessons learned along the way will be worth the effort. This is not just a theoretical post, there is a hands-on lab that will take you step by step through the whole process.
Copying hyperscalers’ engineering practices wholesale is a familiar anti-pattern. Sometimes, though, their solutions do scale down. Metrics sprawl is a near-universal headache, and Pinterest’s account of building a unified metrics layer has lessons for those of us operating at humbler scales. With many teams minting custom business metrics and running experiments, duplication was rife, and practices for monitoring and managing metadata drifted apart. The solution was a metrics board – a semantic layer ensuring that metrics were defined consistently and exposed through a single interface. Pinterest work at a scale that meant their implementation required some sophisticated plumbing, but the core idea of a centralised discovery layer as a single source of truth for metrics would probably deliver an ROI for most teams.
The excellent Alex Ewerlöf blog is now back in full swing and in this latest article he turns his attention to dealing with a real SRE curveball - how to build reliability engineering for an air-gapped system. We are talking hermetically sealed - not even a maintenance window for external connectivity. This task involved a high-security military facility where installing updates meant physically handing over an archive file to a system operator.
The constraints were pretty stringent - no logs, no metrics, no traces, no remote access of any kind.This was an extreme case and, ultimately, the solutions had to be both human-centred as well as relatively low-tech. How would you address the challenge? Hit the link below to read about the solution that Alex put in place.
In the past month or so the GitHub platform suffered a number of well-documented outages which resulted in loss of service for users. In the spirit of transparency, GitHub CTO Vlad Fedorov published this article on the GitHub blog, explaining the causes of the outages and the lessons learned as well as detailing the remediations that GitHub engineers will be putting in place.
The article really brings home the challenges of orchestrating the components of a global technology infrastructure - as well as the compounding effects of working at very large scale. The investigation revealed a perfect storm of edge cases, hidden tipping points and unforeseen knock-on effects. It’s impossible not to feel for the engineers sweating in the war-rooms as the dramas unfolded - after all, watching your failover fail must be pretty gut-wrenching.
If the Observability world had a code of secrecy akin to that of the Magic Circle, then Charity Majors might be in danger of being banished to exile and ignominy. In a single slide deck, she has blown the gaff on a whole trove of insider knowledge. It is the “What they don’t teach you at Harvard Business School” of observability knowledge. Not the abstract theory or technical detail but lessons and insights from the o11y frontline.
The deck in question was used in a talk at the LeadDev event in Berlin earlier this month and its 52 slides are an illuminating distillation of observability wisdom. We actually weren’t present at the talk and only came across the deck thanks to a mention in Michael Hausenblas’s excellent olly news newsletter. However, the slides contain sufficient clues (and images of unicorns) to easily re-construct the narrative and win friends and influence people as an observability savant.
If you are an SRE, when an outage happens you will know about it pretty quick. With security breaches the picture is rather less clear as, by their nature, they are designed to go undetected. Intrusion detection therefore is often based on a mixture of tools designed to spot unusual spikes, suspicious patterns or failed logon attempts.
This article by Fatih Koç argues that one of the major difficulties involved in identifying attacks is that of correlating signals across multiple sources such as Falco, Prometheus, Kubernetes Audit Logs etc. In this article, he outlines a strategy for extracting relevant data from each of these sources and pulling it together into a single observability dashboard.
The first line of cyber defence is normally at the perimeter - preventing attackers from entering your network in the first place. The next line of defence is intrusion detection. This can often take the form of anomaly detection using a variety of heuristics.
There are also some more creative possibilities, such as the canary solution adopted by Grafana. Just as the canary in the coalmine sings to alert underground workers to the presence of toxic gases, Grafana’s canary was designed to alert them to the possible presence of intruders in their domain.
Load testing can be simple in theory but in modern distributed architectures, it involves a lot more than throwing requests at an individual service. This article on the Airbnb engineering blog looks at how the company’s engineers use the Impulse load-testing framework to handle a number of more complex requirements such as dependency mockingand managing messaging and asyncronous calls.
Unfortunately, at the moment Impulse is just an internal Airbnb framework, so you won’t be able to get your hands on it at present. At the same time, the article provides a valuable blueprint for tackling advanced, real world load testing scenarios.
It's an announcement that might have seemed unthinkable not long ago, but the porting of the revolutionary eBPF technology to Windows is now a reality. The ability to bring safe programmability to the kernel has resulted in enormous gains in fields such as security, networking and observability for Linux hosts, so applying the same principle to the Windows ecosystem is obviously an attractive proposition. It is not, though, without its own difficulties. There were a lot of hurdles to overcome and, inevitably, given the differences in OS architecture, this is not a full-fidelity replica of the Linux implementation.
This possibly foundational article by Pavel Yosifovich guides you through the steps involved in boldly going where few have gone before and creating your first eBPF program for Windows. One paragraph in the article begins with the sentence “this is where things get a bit hairy“ - for some that will likely be a challenge rather than a deterrent. This may not be cooking up nuclear fusion in your bedroom, but it does feel pretty radical.
As well as rolling out their Open AI observability solution, Elastic have also been very active within the OpenTelemetry project. C++ has a reputation for being something of a fearsome foe for observability practitioners. In this article on the Elastic blog, Haidar Braimaanie dons his protective gear and attempts to tame the beast with a soothing dose of OpenTelemetry instrumentation.
Unlike languages built in frameworks such as .NET, C++ does not have a standardized runtime environment that supports dynamic instrumentation across all platforms and compilers. C++ also uses a variety of build systems such as Makefiles and CMake, so that implementing instrumentation can be difficult and error-prone. In the article, Haidar looks at adding OpenTelemetry support to a C++ application running on Ubuntu 22.04. He also includes sample code for instrumenting the project with database spans and then observing the application in APM.
After reading this article you may want to give the C++ developer in your life a hug.
Even if you are not familiar with the name of Brendan Gregg, you are almost certainly familiar with the fruits of his labours. Brendan is the creator of the Flame Graph - one of the most important and iconic visualisations in the observability toolkit.
We featured the Flame Graph in our recent tribute to the work of UX designers in the observability arena - but you should also visit Brendans’ web site.
Brendan's latest innovation is the AI Flame Chart. This is an evolution of the original flame graph and its ambitious aim is to help reduce the vast financial and environmental costs entailed in the use of LLM’s. This means that whereas the original flame graph was focused on CPU cycles, the latest generation sets its sights on reducing GPU load. The article discusses the considerable complexities involved in mapping GPU programs back to their corresponding CPU stacks.
The names of some of the instruction sets look intimidating to the uninitiated but the basic concept of the graph is quite simple - the wider the bar, the more resource it consumes.
If you have ever had to grapple with a 3,000 line Helm chart to deploy your observability infrastructure, you may be forgiven for thinking that there must be a better way to do this. Whilst YAML has a certain formal elegance, its syntax struggles to express the architectures and relationships embedded in highly complex systems.
Whilst Pulumi have tackled this problem by enabling the use of high level programming languages for IaC, System Initiative are taking a fundamentally more radical approach. Their goal is nothing other than completely reinventing IaC from the ground up. The blog article for the launch of the product is an incredibly ambitious statement of intent. The terms ‘game changer’ and ‘paradigm shift’ tend to be thrown around somewhat casually, this might be a case where their usage is appropriate.
So, what are they proposing? Well, System Initiative is IaC without the code. It is a kind of digital canvas where you manipulate digital twins of your systems. Is the future here or is this the Platform Engineering equivalent of science fiction? Read the article and decide for yourself!