A year or so ago, my energy meter at home got replaced with a “smart” one. Smart, in this context, means it periodically sends energy usage data over GPRS to a central registration server. This central service is in turn used by various websites where you can register your meter and they’ll provide you with some nice graphs and dashboards. Although this sounds great, I didn’t feel quite happy with this idea. What if someone would intercept the data, or manage to register my meter? It would allow a burglar to see when I’m on vacation by just seeing that I’m consuming little to no energy. And since it was time to start experimenting with Akka and Scala, a new project was born!

smart meter

Figure 1: A smart meter, manufactured by Kaifa

Connecting the smart meter

First of all, I needed a way to physically connect the smart meter to a computer. My meter, a Kaifa MA105, is equipped with an RJ11-connector on the front side. RJ11 might be familiar from landline phones, but it can be used for different purposes as well. First, I bought a small USB-to-RJ11 convert on eBay which was shipped from China. It didn’t have much specs, expect the two sides of the converter were respectively USB and RJ11. It seemed like a great fit.

Custom wiring

Figure 2: Custom wiring

When the connector arrived and I used it to connect the smart meter to my Raspberry, nothing happened. I asked for help on a hardware-related forum and it turned out to be a Bad Idea. I could easily have destroyed either my Raspberry or my meter due to the unknown pin connections! The connection “not working” was actually the least disastrous outcome, so I guess I’m happy with that.

So, I studied the specifications that Netbeheer Nederland has created. It’s called the DSMR, which stands for Dutch Smart Meter Requirements. At the time of writing, v. 4.2.1 is the most recent version. The DSMR describes the exact pin layout for connecting the meter:

 Pin layout for the RJ11-connector

Figure 3: Pin layout for the RJ11-connector on a smart meter

Great! Since I’m not quite good with soldering irons and the like, I decided to not make the same mistake again. I’ve ordered a prefab cable from a Dutch webshop which is designed to connect the smart meter and tested as well.

Prefab cable

Figure 4: Prefab cable

When the prefab cable arrived, I connected the smart meter again. This time, dmesg on my Raspberry indeed showed some output indicating that an additional serial port was found!

Figure 5: dmesg output: we’ve got a connection!

[   1.979124] usb 1-1.2: new full-speed USB device number 4 using dwc_otg
[   2.105808] usb 1-1.2: New USB device found, ...
[   2.107861] usb 1-1.2: New USB device strings: ...
[   2.109856] usb 1-1.2: Product: FT232R USB UART
[   2.111719] usb 1-1.2: Manufacturer: FTDI
[   2.113519] usb 1-1.2: SerialNumber: ASYXDBF7

Reading data

The next challenge is to actually read useful data from the serial connection. According to the DSMR, the speed of the serial connection is fixed at 115200 bps. The protocol is read-only; by providing a signal on the second pin, you will receive data on the fifth pin. This has been implemented by the chip that is embedded in the cable, so we don’t need to worry about that.

Instead, we can use a small Unix utility called cu to read a serial connection. When we start it, we must specify the connection speed and the serial connection: cu -l /dev/ttyUSB0 -s 115200 --parity=none. The last switch indicates that there is no parity bit. At first, cu will simply say “Connected”, but nothing happens. But within a second or ten, something like this pops up on the terminal:

Figure 6: cu echoing data from the smart meter

/KFM5KAIFA-METER

1-3:0.2.8(42)
0-0:1.0.0(1604161128545)
...
...
!4016

The terminal will keep on printing new data, until you disconnect the session by typing ~..

The wait time is at most 10 seconds, because the DSMR specifies that data must be sent every ten seconds. Each batch of data is called a “P1 Telegram” and follows the same structure.

Interpreting data

Although we have more-or-less readable data by now, it’s hard to interpret such a telegram. Apart from the first and last line – “” and “!4016” – all lines do seem to have some kind of structure, but it’s hard to guess what it means. Luckily enough, the DSMR comes to rescue, again! These lines all share the same structure:

Figure 7: Structure of lines inside a telegram

<digit>-<digit>:<digit>.<digit>.<digit>(characters)

Two lines do not follow that pattern: the first line and the last line. The first line, /KFM5KAIFA-METER, is a header: KFM is an identification for the make of the meter, 5 a fixed value and KAIFA-METER is how the meter identifies itself. The last line, !4016, contain a checksum after the exclamation mark. You can use that to verify you’ve received the telegram correctly.

Since these lines are very structured indeed, it should be easy to parse them. Scala has a very powerful mechanism for that: parser combinators. As an example, think of the situation where we want to process a line from a P1 Telegram that looks like this: 1-0:1.8.1(001371.351*kWh). In this case, we’re not interested in the numbers, dashes and dots up to and including the opening bracket. Nor are we interested in the asterisk, kWh (unit of measurement) and the closing bracket. What’s left, 001371.351, is of interest to us, and we want to have that value as a BigDecimal.

In this context, a parser is something that attempts to convert structured text into a different representation; for example, it can convert any text that matches the regular expression d*.?d* into an instance of the BigDecimal class. The code for that is pretty easy:

Listing 1: Trivial parser

def number = """d*.?d*""".r ^^ { BigDecimal(_) }

Listing 1 creates a regular expression by calling .r on a String. Since that String was created with triple quotes there is no need to escape e.g. backslashes, making it a lot easier to read for human beings. The ^^ says to feed the input that matches this regex into the block that follows it, where the _ is a placeholder variable for the input.

A combinator, on the other hand, is a function that accepts two parsers and returns a parser as well. You could think of it as a combination of the two parsers.

Listing 2: Combination of three parsers

def elecCons1 = "1-0:1.8.1(" ~>
   """d*.?d*""".r <~ "*kWh)" ^^ { BigDecimal(_) }

Listing 2 requires some explanation. The ~> combinator creates a parser that processes input that matches the parser that precedes it, followed by the parser that follows it. It will, however, only yield that latter. Similarly, the <~ combinator creates a parser that processes input that matches the parser that precedes it, followed by the parser that follows it. This time, it will only yield the former. So the result is a parser function that processes the whole line we’re interested in, but it will extract only the part that was matched by the middle parser, and map that to a BigDecimal.

This might look like a very complex setup; why not just matching whole lines of text using a simple regular expression? The answer is in reusability. By combining parsers, it is easier to re-use parser definitions for frequently used types. BigDecimal might not be the most challenging type to parse, but the P1 Telegrams also contain a custom date-time format, yyMMddHHmmss; it makes sense to implement that only once.

This way, starting at the lowest level, we can build a combination of parser functions that will eventually parse the whole P1 Telegram. Lines can be combined into parts of the P1 Telegram, like the header, the metadata, the actual data and the checksum. On the highest level, parsing a P1 Telegram is a combination of invoking the parsers for these elements, as can be seen in Listing 3. It uses the ~ combinator, which works similar to ~> and <~ except it keeps both parts. The last bit, starting with the |, is for error handling: if the input doesn’t conform to the format we expected, we invoke a parser that always fails with the specified message.

Listing 3 Parsing a complete Telegram

def telegram = header ~ metadata ~ data ~ checksum
def parser: Parser[P1Telegram] = telegram ^^ {
   case header ~ metadata ~ data ~ checksum =>
      P1Telegram(header, metadata, data, checksum)
   } | failure("Not all required lines are found")

Processing data with Akka

Introducing Akka is far beyond the scope of this article; I assume you have a basic knowledge of the concepts behind it. If not, “A Map of Akka” and “Introduction to Akka Actors” will get you started.

To read the serial line from Akka, a lot of work is needed. It even involves some native code to access the serial line, since the JVM cannot do that itself. Luckily enough, this path has been paved already by Jakob Odersky’s excellent akka-serial project. To use it, your application must start a “manager actor” using IO(Serial) – here Serial is the Akka IO extension from akka-serial. This manager actor will take care of opening the port and making sure a dedicated actor, the “operator actor” will handle further interactions with it. That operator will manage interaction with the native code and send Serial.Received messages to the actor that created the manager actor. See Listing 4 for an illustration of these interactions.

Listing 4: Interacting with the serial line from an Akka actor

private val operator = IO(Serial)(context.system)

  override def preStart = {
    // Specify the serial port as well as baud rate
    // Other settings omitted for brevity
    operator ! Serial.Open("/dev/ttyUSB0", serialSettings(115200))
  }

override def receive = {
    case Serial.Received(bytes) =>
      // Do something with the bytes that came in, e.g. read an UTF-8 string
      val s = bytes.utf8String
      log.info(s"Received bytes $s")
    // Handling of other messages omitted for brevity
  }

  override def postStop(): Unit = {
    operator ! Serial.Close
  }

These Serial.Received messages contain just raw bytes. There is no guarantee as to how many bytes you will receive in one message. That means we need to buffer these bytes until we know we have exactly one P1 Telegram in memory. We can be sure the P1 Telegram is complete after we received the checksum which starts with the exclamation mark. At that time, the aforementioned parser is invoked that will attempt to parse the received text to a nice object structure. Now that we have a P1 Telegram in memory every ten seconds, there’s a lot of things we can do with that.

But before we do so, we need a way to distribute these P1 Telegrams. The solution to that is a pretty simple actor that acts like a distributor. Other actors can register and de-register themselves at the distributor. Any other message the distributor receives is forwarded to all actors that are currently registered at the distributor. This simple trick makes it very easy to create various other actors that implement the interesting features we’re looking for, all taking their data from one central place.

Distributing P1 Telegrams with Akka

Figure 8 Distributing P1 Telegrams with Akka (simplified)

Pitfalls

During the project, I’ve encountered quite some challenges to deal with. Apart from unfamiliarity with certain Scala features, three of them are particularly worth mentioning.

  1. Don’t try to be cheap and use some good-locking converter. The risk of blowing up either your Raspberry or your energy meter (or worse, both!) isn’t quite attractive.
  2. There are many versions of the DSMR specification. Make sure you have the right one, since many parts changed over time. This includes, but is not limited to, the transfer speed (9600 bps vs 115200 bps) and which lines to expect inside the data-block of the telegram.
  3. If you’re new to Akka, always include a “fallback” into your receive() method. If the last line is something like case a: Any => log.debug(s”Ignoring message $a”), it is easier to troubleshoot where certain messages have been sent to and why they are not acted upon.

Conclusion

In this article, we have seen how we can build a flexible and extensible energy dashboard using modern techniques like Scala and Akka. The architecture is extensible in the sense that allows for easily adding new features without having to reshape existing parts of the software. We did not dive into adding these features, or even building a web application that visualises the data. That is left as an exercise to the reader – or maybe the topic for another article. Stay tuned!

If you want to check out the code, you can! It is published on GitHub: mthmulders/hyperion. Feel free to fork it, play around and see how stuff works. If you have a Smart Energy Meter as well, there’s a little disclaimer: I’ve tested the software on my meter and it works. When you connect it to yours, Your Mileage May Vary.

Outsmarting the Smart Meter

| Mind the Geek| 881 views | 0 Comments
Profile photo of mthmulders
About The Author
-

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>