Category Archives: Tools

Interactive Web Display of Real-time Sensor Data

The biggest psychological impediment to my work on wireless sensor network frameworks is data. What do I collect? What do I do with it? In short, why do I even want a WSN? (If you can answer that, but want help making it happen, please get in touch through one of the options on my about page.)

Collecting data that’s never looked at is a waste of time, so good tool support for viewing and analyzing time-series data is important. The pieces underlying this are the databases and the graphs.

The traditional database approach is RRDTool. Been around forever, does the job very well.

I’m not in love with its API, which involves providing text representations of the observations through an argc/argv command-line interface even from within C, which on the surface introduces a lot of overhead. But “I just don’t like it” is an inadequate reason to reject a time-proven solution. In fact, a text-based API has certain benefits: I recently got a patch merged to RRD that significantly simplifies specification of archive retention, but the new feature can’t be used in collectd configurations because the parameters are no longer the integer counts that collectd stores and sorts to reformat into RRD arguments.

Where RRDTool falls down is in the display of the data. Well, really, in constructing the specification that’s used to define the graphs. The rrdgraph tool that comes with it actually makes some very powerful graphs, but it’s in desperate need of a GUI to help select among data sets, combine them, change the time bounds, etc. There’s cacti, but it wants to do data collection too, I don’t like kitchen-sink solutions, and it’s not as good as collectd. The days of the fat client are past; there are some web apps like Collectd Graph Panel that make it a bit easier, but not enough to really be nice to use.

Or so I thought until I read this blog and found out about statsd which led me to graphite.

This is what I’m talking about: dashboards with pre-configured graphs showing the information I want to see, automatically updated as the data comes in. A web application to dynamically construct graphs, adding and removing from all available data sets, applying transformations to each source in turn, etc.

Turns out the graphite project provides three layers:

  • whisper and ceres are back-end databases for time series data, similar to RRDTool
  • carbon is a daemon infrastructure that receives samples over a network connection and dispatches them to whisper or ceres based on a text name, which by convention encodes a hierarchy such as collectd.server1.cpu0.load.
  • graphite-web is what generates the graphs; it can read data from whisper, ceres, and RRD databases.

It’s all written in Python and runs as a virtual host under django, so it’s pretty easy to configure on Ubuntu 12.04 or 14.04. Even getting django to run in Apache isn’t nearly as hard as everybody makes it out to be.

Graphite’s capabilities are awesome.

Carbon has some really neat architectural features including automated creation of databases as metrics arrive (with customized retentions based on the metric name), centralized aggregation of metrics, and daemons to perform meta-aggregation and relaying to other servers.

Whisper is…inadequate.

Now whisper exists only because of a couple of issues that the graphite developers had with RRDTool. Paraphrasing the explanation:

  • can’t back-fill data that wasn’t available in a timely manner. Here I have some sympathy. This limitation on RRDTool may explain why collectd’s architecture can’t handle plugins that record multiple observations within a sampling interval. Fixing this in RRDTool would be tough. Probably the only viable solution is to integrate the capability into rrdcached, and enhance rrdcached so it acts as a fetch server without first flushing everything to disk.
  • not designed for irregular updates. In short, incoming data to RRD gets interpolated to align with the primary data point timestamps. If you don’t get data often enough to do that interpolation, RRDTool can’t do the alignment, and data gets dropped. I’m sympathetic to this issue, though it doesn’t affect my use cases as much as it does, say, StatsD and other system-monitoring applications.

On the other hand, whisper has a few problems of its own:

  • Each file stores only one metric, which wastes storage when a sensor provides a multiple metrics (e.g. temperature, humidity, pressure, wind direction and speed) and multiple consolidations (AVERAGE, MAX, MIN over various retentions)
  • As a consequence, when aggregating for lower-resolution periods only one consolidation function is allowed (normally “average”). You lose the extreme values (such as daily highs and lows) unless you configure to create separate databases for those. (Carbon does support this if carbon-aggregator is used, but that’s another daemon/point-of-failure.)
  • In my case, sensors have a limited ability to store data locally, so if the off-sensor database stops functioning and I’m told about it in time I can restart things and back-fill the missing material. This is exactly what nagios is for, but figuring out when the last update was received by a whisper database requires an O(n) search because the value isn’t stored in the database header, even though a related problem was a motivation for rejecting RRDTool!

The biggest problem with carbon, and the graphite project as a whole, is lack of leadership and active management. Carbon has over two hundred open issues and pull requests, some of which have been apparently been addressed but the issues left open. There was a major rework called “megacarbon” but that’s been dormant for six months. There are incompatible changes being made on the 0.9.x maintenance branch relative to master. whisper is supposed to be superseded by ceres, but requests for information on project status and schedule are left unanswered for months. If you noticed the link I gave above for graphite was to an outdated page: it’s because the new one doesn’t have the FAQ or any of the pieces that tell people why they should even care about the project.

This an unfortunate but common failing with open source projects, where nobody’s compensated for their effort and maintenance naturally drops when the original developer can only respond “it works for me” or “I don’t even use that software anymore”.

Regardless of all that. RRDTool is a robust tool that’s worked for years and continues to be maintained: a dozen enhancement patches I submitted were promptly integrated for the next release. As an open source solution for web access to display real-time data I don’t think graphite-web would survive a serious challenge from an actively-managed alternative, but it works well enough that there’s really no motivation to develop a competitor.

I’ve set up permanently running rrdtool, graphite, and collectd systems on both my stable and development servers. They’re already recording whole-house power consumption at 1Hz from a TED 5000, interior temperature and humidity from a daemon running on a raspberry pi (stored in RRD databases because I care about that data), and collectd statistics from internal hosts (stored in whisper databases because it’s easier to customize the retention periods and I don’t care about that data).

Since I finally have a way to visualize the data, and I have all these wireless microcontrollers and sensors, it’s probably time to start collecting more.

On using current toolchains

An excerpt from a discussion on the TI MSP430 Forums regarding trade-offs between sticking with an old toolchain that you’re used to, and continually updating:

You can stick with an existing, “well understood” system, and assume that you’re safe because it passes what you think is important to test. Or you can keep up to date with what’s provided by a vendor (who sees a lot more use cases and variations than you do). This is a management choice.

All I can say is that, in my own multi-decade experience, the biggest long-term source of destabilization comes not from regular updates to the current toolchain, but from staying with old tools until something happens that forces you to make a multi-version jump to a new compiler. (And I agree that a new version is a new compiler and cannot just be assumed to work; this is why one should develop complete regression suites with test harnesses to check the “can’t happen but actually did once” situations.) I can’t see what happens in proprietary systems, but it’s been many years since an update to GCC has resulted in my discovery of an undesirable behavioral change that wasn’t ultimately a bug in my own code, with the fix improving quality for that code and all code I’ve worked on since.

If you’re operating in a regulated environment where the cost of updating/certifying is prohibitive, so be it. Best approach in that case is to keep with the toolchain used for original release of the product, and release new products with the most recent toolchain so you’re always taking advantage of the best available solution at the time.

I’m not saying there’s a universally ideal policy, e.g. that you should always use the current toolchain. I am saying that a shop that develops and releases new products using old toolchains without a strong reason behind that decision is not using best practices and is likely to produce an inferior product. If management thinks they’re saving money and reducing risk by not updating, there’s a good chance they’re being short-sighted.

Editing XML schemas on Emacs with nXML

While looking into DocBook recently, I discovered that GNU emacs finally has a high-quality XML editing mode that includes validation of the documents. nXML mode is integrated into emacs23, and it comes with RELAX NG grammars to support docbook editing, though only for DocBook 4.2.

For work on PyXB, though, I really need something that handles XML Schema Definition (XSD) documents. emacs nXML doesn’t come with XSD support, but the RELAX NG homepage points to Jeni Tennison’s schema as a candidate.

This is a great start, but when I tried using it with xmllint from libxml2 to validate some schemas supported by PyXB it said they were invalid. There are a variety of subtle issues the original version didn’t quite get right (and a few cases where my example schema were wrong). I’ve updated the schema to fix those issues, and made it available on github.

emacs nXML comes with an XSLT RELAX NG schema, but only for version 1.0. As XSLT 3 is nearly complete at the time I’m writing this, I was hoping to find support to validate against other XSLT versions as well. Turns out Norman Walsh has provided a unified solution for XSLT 1.0, 2.0, and 3.0 on github.

So: To support XSD and XSLT editing with nXML in Emacs 23, I put this in my .emacs file:

I copied the original schema from the rng4xsd and xslt-relax-ng repositories, and used Trang to convert from the standard RELAX NG XML syntax to the compact syntax used by nXML. Then the following goes into ~/.emacs.d/nxml-schemas/schemas.xml:

Now when I write my XSD schemas in emacs, the mode line tells me when they’re invalid, and I can use C-c C-n to jump to the error, with the explanation placed in the message line. Very nice.

C++ Unit Test Frameworks

One of the first decisions in a new project is which unit testing framework to use. Traditionally I’ve used CppUnit, so I pulled down the current release and started working.

This left me unhappy as the first test produced this compile-time error:

For a couple days I worked around this by casting the integer literal to a type that satisfied the calls, but eventually I got fed up.

So I looked for alternatives. I found fault with the first two choices, but joy with the third. Herein are some examples with discussion of what they reveal about the choices. The files are available as a github gist.

The Test Criteria

Three specific assertions were found to cause trouble with various solutions, so the examples used below show all of them:

  • Comparing a std::string size() with an integer literal;
  • Pointer-equality testing for char * values;
  • Comparing a floating point result to a specific absolute accuracy

In addition, these criteria are relevant:

  • Verbosity: how much boilerplate do you have to add that isn’t really part of your test?
  • Installation overhead: is it easy to build the library for specific compiler flags or is the assumption that you build it once and share it? This matters when playing with advanced language feature flags such as -std=c++1y, which can affect linking test cases together.
  • Assertion levels: when a test fails can you control whether the test keeps going or aborts (e.g., when following assertions would be invalid if the first fails).
  • Assertion comparisons: can you express specific relations (not equal, greater than) or is it mostly a true/false capability?

CppUnit

Originally on SourceForge, this project has developed new life at freedesktop.org.

CppUnit comes with a standard configure/make/make install build process which installs the headers and the support library into the proper directories within a toolchain prefix. You need to provide a main routine to invoke the test driver.

CppUnit provides only one level of assertion: the test case aborts when it fails. It also has limited ability to express specific requirements (for example, there is CPPUNIT_ASSERT_EQUAL(x,y) but no CPPUNIT_ASSERT_NOT_EQUAL(x,y).

Here’s what the tests looks like with CppUnit:

There’s a lot of overhead, what with the need to define and register the suites, though it didn’t really bother me until I saw what other frameworks require. And I did have to do that irritating explicit cast to get the size comparison to compile.

The output is terse and all tests pass:

Boost.Test

Boost is a federated collection of highly-coupled but independently maintained C++ libraries covering a wide range of capabilities. It includes Boost.Test, the unit test framework used by boost developers themselves.

Boost.Test can be used as a header-only solution, but I happened to install it in library form. This gave me a default main routine for invocation, though I did have to have a separate object file with preprocessor defines which incorporated it into the executable.

Boost.Test also supports three levels of assertion. WARN is a diagnostic only; CHECK marks the test as failing but continues; and REQUIRE marks the test as failing and stops the test. There are also a wide variety of conditions ( EQUAL, NE, GT, …), each of which is supported for each level.

Here’s what the tests look like with Boost.Test:

This is much more terse than CppUnit, and seems promising. Here’s what happens when it runs:

Um. Whoops?

Boost.Test silently treats the char* pointers as though they were strings, and does a string comparison instead of a pointer comparison. Which is not what I asked for, and not what BOOST_CHECK_NE(x,y) will do with other pointer types.

Boost.Test also does not provide a mechanism for absolute difference in floating point comparison. Instead, it provides two relative solutions: BOOST_CHECK_CLOSE(v1,v2,pct) checks that v1 and v2 are no more than pct percent different (e.g. 10 would be 10% different), while BOOST_CHECK_CLOSE_FRACTION(v1,v2,frac) does the same thing but using fractions of a unit (e.g. 0.1 would be 10% different). Now, you can argue that there’s value in a relative error calculation. But to have two of them, and not have an absolute error check—that doesn’t work for me.

Boost.Test also has a few other issues. The released version has not been updated for four years, but the development version used internal to the Boost project has many changes, which are expected to be released at some point in the future. From comments on the boost developers mailing list the documentation is generally agreed to be difficult to use, and has produced a rewritten version (which, honestly, is what I had to use to try it out).

All in all, I don’t feel comfortable depending on Boost.Test.

Google Test

Google Test is another cross-platform unit test framework, which supports a companion mocking framework to support unit testing of capabilities that are not stand-alone.

The code comes with configure/make/install support, but also provides a single-file interface allowing it to be built easily within the project being tested with the same compiler and options as the code being tested. You do need a separate main routine, but it’s a two-liner to initialize the tests and run them all.

Google Test supports two levels of assertion: failure of an ASSERT aborts the test, while failure of EXPECT fails the test but continues to check additional conditions. It also provides a wide variety of conditions.

Here’s what the tests look like with Google Test:

Even more terse than Boost.Test, because it doesn’t use something like GTEST_TEST or GTEST_ASSERT_EQ. To avoid conflict with user code I normally expect framework tools to provide their interfaces within a namespace (literally for C++, or by using a standard identifier prefix where that wouldn’t work). Both CppUnit and Boost.Test do this for their macros, but for unit test code that doesn’t get incorporated into an application I think it’s ok that this isn’t done.

And here’s what you get when running it:

A little more verbose than I’m accustomed to from CppUnit, but it’s tolerable. The most important bit is the last line tells you the overall success, so you only need to scroll up if something didn’t work.

Conclusions

Summarizing the individual tests for each criterion, with a bold answer being preferable from my subjective perspective:

FeatureCppUnitBoost.TestGoogle Test
Handles size_t/int comparesnoyesyes
Handles char* comparesyesnoyes
Handles absolute float deltayesnoyes
Verbosityhighlowlow
Installationtoolchainheader-only or toolchainproject
Assertion Levelsonethreetwo
Assertion Conditionsfeweverymany

So I’m now happily using Google Test as the unit test framework for new C++ projects.

In fact, I’ve also started to use Google Mock, which turns out to be even more cool and eliminates the biggest limitation on unit testing: what to do if the routine being tested normally needs a heavy-weight and uncontrollable supporting infrastructure to satisfy its API needs. But I can’t really add anything beyond what you’ll can find on their wiki, so will leave it at that.