Monitoring Akka.net Based Systems

Building maintainable and reliable systems is hard because of their impermanent and fragile nature. In order for systems to solve important problems or provide business value they have to function properly.

During the development process unit tests, use cases and domain users provide important feedback to ensure what is built meets the desired outcome. That is the reason for methodologies like Agile, TDD and Lean but once a system is in production how can you ensure that it is still functioning properly?

The path to success can be found by embracing the fact that software is fragile and just like during development feedback is key. It's not enough to just monitor if a daemon or windows service is running. What is truly important is that it is running smoothly and not stuck in some kind of chaotic state.

There is a great blog post by Ian Malpass titled Measure Anything, Measure Everything which details a project he started called StatsD. I think this quote from the post sums up StatsD.

StatsD is a simple NodeJS daemon (and by “simple” I really mean simple — NodeJS makes event-based systems like this ridiculously easy to write) that listens for messages on a UDP port. (See Flickr’s “Counting & Timing” for a previous description and implementation of this idea, and check out the open-sourced code on github to see our version.) It parses the messages, extracts metrics data, and periodically flushes the data to graphite.

A few months ago I a became acquainted with the Akka.net project for creating Actor based systems listening to .Net Rocks. As a result of my interests in the project I decided to contribute by creating an extension, based on the Java version of Akka, that create Actors using dependency injection. While on that journey I got some great feedback from Bartosz Sypytkowski and Aaron Stannard not to mention Roger Alsing.

When Aaron was giving me feedback on my documentation he provided me with a link to his akka-monitoring project as a reference. The akka-monitoring project provides monitoring for Akka.net based systems using StatsD right out of the box. It was from this project that I came across the for mention blog post and was my inspiration for this post.

Akka-monitoring is really simple to use and provides an array of automated measures but some of the measures require the programmer to implement some simple code as demonstrated here.

https://github.com/Aaronontheweb/akka-monitoring

In Aaron's FAQ I saw the following question that got my interest.

Any plans to automatically collect actor lifecycle and message received data? 

That depends largely on how much traction Akka.Monitoring gets - we're considering subclassing UntypedActor and TypedActor to automatically provide lifecycle and receive instrumentation, but we want to see how other people use it first.

This got me thinking would be possible to solve the for mention question without subclassing but instead with using AOP.  So I decided to create a thing called an Interceptor, which is Castle Windsor's way of doing AOP, and wired that up to Akka.DI.CastleWindsor.

The interceptor creates a proxy around the actual object that it represents. Each method executed on the object, which is an Actor in the case, is intercepted prior to executing allowing you to call the various akka-monitoring methods without actually touching the Actors directly. Here is a very basic example.

https://github.com/jcwrequests/AkkaMonitoringSampe/blob/master/MonitorInterceptor.cs

I have created a working sample based on the akka-monitoring examples using my own simple console monitor to demonstrate the concept presented in this post without the need for a StatsD server.  This sample is for demonstration purposes only. If you are interested in pursuing this for your Akka based system then I would recommend following the best practices in the CastleWindsor documentation and consulting the akka-monitoring project where you will find excellent documentation on setting up a StatsD Server as well as implementing your own monitor if StatsD does not fit your needs.