Introducing panflute-tests

The bad news is that I won’t be releasing the next version of Panflute at the end of the month as I had planned; that’s been pushed back to the end of October. That’s because it took much longer than I was expecting to get a new backend feature working: panflute-tests.

As of ths writing, Panflute supports 15 different music players, with even more planned: Amarok, Audacious, Banshee, Exaile, Guayadeque, Listen, MOC, MPD, Muine, Quod Libet, Rhythmbox, Songbird, VLC, XMMS, and XMMS2. Manually testing Panflute with the latest version of each is impractical; expanding that testing to include the last n versions of each of those is completely unreasonable. As a result, I had largely been slacking off doing proper testing, meaning that bugs, especially when the new version of a player changed its interface somehow, were common.

panflute-tests is a tool designed to automate the process of compatibility testing as much as possible. It allows a developer to define a series of test configurations, which are basically a particular installation of a particular version of a player, such as Banshee 1.7.4 installed under $HOME/players/banshee-1.7.4. For each defined configuration, panflute-tests will initialize the player with a known “music” collection and run a series of tests against it. The tests themselves are simple; for example, one checks that when the Play, Pause, and Stop commands are issued, the player responds by reporting the appropriate state change.

Even though each test is simple, the ability to automate running them against lots of configurations makes overall compatibility testing much more thorough and repeatable. Already, it’s discovered bugs that hadn’t been reported to the Panflute bug tracker, such as not retrieving metadata from recent versions of XMMS2 correctly.

Running panflute-tests yourself

panflute-tests is very useful, but also very dangerous. Running it will destroy any configuration settings you have for the music players installed on your system. This is by design; it’s essential to start each test with the player in a known state, so configuration settings get rewritten to load a specific music connection. For this reason, if you insist on running panflute-tests yourself, I implore you to do so in a virtual machine dedicated to using panflute-tests. I myself have a VM of Ubuntu Lucid where I currently have approximately 60 different players installed. As an added precaution, panflute-tests will refuse to do anything unless you specify the deliberately ominous --destroy-my-data option on the command line.

panflute-tests configuration

Adding a new test configuration is straightforward: click the Add button and fill in the player name, its version, and the prefix under which it was installed. I recommend installing each player to be tested to a unique prefix; I’ve been using $HOME/players/name-version, so that in the screenshot above I have Amarok 2.3.1 installed under /home/paul/players/amarok-2.3.1. At the very top of the window, you’ll also need to specify the installation prefix of Panflute itself.

That’s really it as far as setting up a test configuration goes. panflute-tests takes care of the ugly, nasty, difficult work of editing each player’s configuration settings appropriately before a test, or dealing with any other trickery to make everything work. Select the tests you want to run and click Run, then wait for the results.

panflute-tests result

Each test has three possible outcomes. PASS means it worked, FAIL means it didn’t, and SKIP means the player doesn’t support the feature being tested at all. In the event of a FAIL result, panflute-tests will show the exception backtrace that triggered the failure.

In the above screenshot, we see that the Seek test FAILed for Amarok 2.3.1. It turns out that this is because of quirky behavior of Amarok itself rather than a bug in Panflute — during a seek, Amarok’s reported elapsed time jumps around a bit before it settles down, and this confuses the checks that panflute-tests is making. That’s why panflute-tests lets you add a comment to any test or configuration. This is very useful to document which tests are known to fail due to no fault of Panflute’s. Also in that screenshot, we can see that the combined set of tests for Amarok 2.3.2 PASSed, suggesting that this quirk of Amarok was fixed in the current version.

As an undocumented feature, it’s possible to use panflute-tests to start a player under its test configuration without running any tests. To do this, invoke panflute-tests along these lines:

panflute-tests --subprocess Amarok instdir ~/players/amarok-2.3.1

where instdir is the Panflute installation prefix, ~/players/amarok-2.3.1 is the player installation prefix, and Amarok is the name of the player. This is sort of an abuse of how panflute-tests works internally, but I’ve found it useful when trying to diagnose why a test case is failing. Be warned that it will still destroy your data when invoked in this manner.

As another undocumented feature, panflute-daemon’s stdout and stderr get redirected to /tmp/panflute-daemon.out and /tmp/panflute-daemon.err during a test. This is also helpful when trying to figure out why something didn’t work.