Introducing panflute-tests

The bad news is that I won’t be releasing the next version of Panflute at the end of the month as I had planned; that’s been pushed back to the end of October. That’s because it took much longer than I was expecting to get a new backend feature working: panflute-tests.

As of ths writing, Panflute supports 15 different music players, with even more planned: Amarok, Audacious, Banshee, Exaile, Guayadeque, Listen, MOC, MPD, Muine, Quod Libet, Rhythmbox, Songbird, VLC, XMMS, and XMMS2. Manually testing Panflute with the latest version of each is impractical; expanding that testing to include the last n versions of each of those is completely unreasonable. As a result, I had largely been slacking off doing proper testing, meaning that bugs, especially when the new version of a player changed its interface somehow, were common.

panflute-tests is a tool designed to automate the process of compatibility testing as much as possible. It allows a developer to define a series of test configurations, which are basically a particular installation of a particular version of a player, such as Banshee 1.7.4 installed under $HOME/players/banshee-1.7.4. For each defined configuration, panflute-tests will initialize the player with a known “music” collection and run a series of tests against it. The tests themselves are simple; for example, one checks that when the Play, Pause, and Stop commands are issued, the player responds by reporting the appropriate state change.

Even though each test is simple, the ability to automate running them against lots of configurations makes overall compatibility testing much more thorough and repeatable. Already, it’s discovered bugs that hadn’t been reported to the Panflute bug tracker, such as not retrieving metadata from recent versions of XMMS2 correctly.

Running panflute-tests yourself

panflute-tests is very useful, but also very dangerous. Running it will destroy any configuration settings you have for the music players installed on your system. This is by design; it’s essential to start each test with the player in a known state, so configuration settings get rewritten to load a specific music connection. For this reason, if you insist on running panflute-tests yourself, I implore you to do so in a virtual machine dedicated to using panflute-tests. I myself have a VM of Ubuntu Lucid where I currently have approximately 60 different players installed. As an added precaution, panflute-tests will refuse to do anything unless you specify the deliberately ominous --destroy-my-data option on the command line.

panflute-tests configuration

Adding a new test configuration is straightforward: click the Add button and fill in the player name, its version, and the prefix under which it was installed. I recommend installing each player to be tested to a unique prefix; I’ve been using $HOME/players/name-version, so that in the screenshot above I have Amarok 2.3.1 installed under /home/paul/players/amarok-2.3.1. At the very top of the window, you’ll also need to specify the installation prefix of Panflute itself.

That’s really it as far as setting up a test configuration goes. panflute-tests takes care of the ugly, nasty, difficult work of editing each player’s configuration settings appropriately before a test, or dealing with any other trickery to make everything work. Select the tests you want to run and click Run, then wait for the results.

panflute-tests result

Each test has three possible outcomes. PASS means it worked, FAIL means it didn’t, and SKIP means the player doesn’t support the feature being tested at all. In the event of a FAIL result, panflute-tests will show the exception backtrace that triggered the failure.

In the above screenshot, we see that the Seek test FAILed for Amarok 2.3.1. It turns out that this is because of quirky behavior of Amarok itself rather than a bug in Panflute — during a seek, Amarok’s reported elapsed time jumps around a bit before it settles down, and this confuses the checks that panflute-tests is making. That’s why panflute-tests lets you add a comment to any test or configuration. This is very useful to document which tests are known to fail due to no fault of Panflute’s. Also in that screenshot, we can see that the combined set of tests for Amarok 2.3.2 PASSed, suggesting that this quirk of Amarok was fixed in the current version.

As an undocumented feature, it’s possible to use panflute-tests to start a player under its test configuration without running any tests. To do this, invoke panflute-tests along these lines:

panflute-tests --subprocess Amarok instdir ~/players/amarok-2.3.1

where instdir is the Panflute installation prefix, ~/players/amarok-2.3.1 is the player installation prefix, and Amarok is the name of the player. This is sort of an abuse of how panflute-tests works internally, but I’ve found it useful when trying to diagnose why a test case is failing. Be warned that it will still destroy your data when invoked in this manner.

As another undocumented feature, panflute-daemon’s stdout and stderr get redirected to /tmp/panflute-daemon.out and /tmp/panflute-daemon.err during a test. This is also helpful when trying to figure out why something didn’t work.

Complaining about Metroid: Other M

The core gameplay of Metroid: Other M isn’t bad, at least from what I’ve seen of it so far (just before entering Sector 2), but man, some of the design decisions are questionable.

The rationale given for why Samus doesn’t have all her equipment from the previous game is indefensible. Of course for gameplay reasons Samus can’t start off fully loaded with equipment, since that would gut the defining characteristic of the series: exploring the environment to find upgrades to enable further exploration. Except in Other M, she does have them, but decides not to use them until Adam “allows” her to.

This, by itself, could have been made to work. It’s not unreasonable to say Samus should avoid using devastating weapons like Super Missiles or Power Bombs when she and the GF troops on the so-called Bottle Ship haven’t assessed the degree of structural damage or whether there are any survivors nearby. It would have been far more interesting had these prohibitions been made voluntary for the player, with the consequences of using (or not using) certain weapons having an effect on how things play out. It’s just aggravating to see an upgrade lying there that could be accessed with a Super Missile, and knowing Samus has Super Missiles, but not being allowed to fire one.

But what makes this excuse for de-powering Samus inexcusable is how it applies to all of her equipment. She’s prohibited from using her Ice Beam until she’s already been fighting enemies in the requisite fire-and-lava level of Sector 3. But given how the GF marines have been using their own Ice Beam weapons from the very beginning, why wasn’t Samus allowed to use hers? Even worse, she’s not allowed to use her Varia Suit‘s heat shields until she’s already been running around in Sector 3 taking heat damage for several rooms, and Adam knows this.

WTF? What possible excuse could Adam have for prohibiting Samus from using a purely defensive and life-saving feature of her armor? And more importantly, why would Samus let herself be subjected to this level of dickishness? The first time Adam tried to make her step foot in a lava-filled chamber without the Varia Suit, she should’ve cut off all contact with him, quit cooperating with the GF marines, and struck out on her own. Anyone willing to needlessly put her in harm’s way like that is no one she should be taking orders from. Especially when she’s not under his command to begin with!

I’m not sure I’m looking forward to any forthcoming cutscenes that might try to justify this dysfunctional relationship. The cutscenes so far have ranged from mediocre to painful, especially with the drearily flat narration by Samus’s voice actress. For example, how many times can we make Samus say “baby” during the opening cutscene? Are we supposed to take that bizarre and overly-long thumbs-down cutscene seriously? Unless I see evidence otherwise, my interpretation is that Adam always resented Samus’s asinine youthful-rebellion nonsense while she was under his command, and is using his newfound control over Samus to exact revenge. Meanwhile Samus, who thought her displays of insubordination were somehow being respectful and endearing, is subjecting herself to Adam’s abuse because she’s desperately seeking approval from the person she subconsciously sees as a father figure. It’s a pretty drastic change from how Samus had been portrayed as a self-reliant unflappable warrior in pretty much every other game, and it’s no wonder why some reviews call the plot downright sexist.

Also, why are there vast lava-filled caverns on a space station anyway? Wouldn’t that kind of melt the station’s hull? The holographic emitters hiding the walls in the “outdoors” areas of Sector 1 made sense, but holographic lava wouldn’t be hot, and certainly wouldn’t have giant monsters swimming around in it. No doubt one of the other sectors will turn out to be the ice caverns. What space station doesn’t have those, right?

Adam’s dickishness also extends to locking doors at various times, keeping Samus largely limited to following a linear path through the station, cutting out a lot of the opportunity for free exploration of the environment. The Metroid Prime games didn’t fall into this trap: there’d be a hint of where you were supposed to go next, but the game wouldn’t actually prevent you from going wherever you wanted to explore or look for items. At some point Samus will no doubt be allowed to revisit the areas, if only because there’d have to be some way to collect all the items she’s had to pass up on account of not being allowed to use the ability needed to reach them.

There’s a pretty good game under here, but you have to suffer through the presentation to find it.