Testability, repeatability, fairness: testing preps is all about asking the right questions

We do a lot of product testing here at The Prepared, so we know how difficult it is to design a test that’s repeatable, fair, and actually answers the burning questions we have about a product. In fact, in many cases we spend far more time designing product tests than we do actually running them. So when we see a well-designed durability test of a product, we appreciate the effort because we know how hard this is.

A while back, DayZero contacted us about reviewing their UV-H2O-Box, a device that lets you decontaminate a gallon of water using a UV bulb powered by a hand crank.

We basically told them “we don’t like hand-crank products because the cranks break.” This is a common problem with “survival” and “emergency” devices that get their electricity from an internal dynamo that’s run by an external hand crank. Whether it’s a radio, a flashlight, or some Frankengadget that has a bunch of different functions, if you have to crank it to power it, then odds are that crank is going to break sooner than you’d expect.

More on the topic of water in emergencies: 

But instead of taking “no” for an answer, DayZero designed and built a powered test rig for figuring out if their products could consistently stand up to 10,000 turns of the crank.

We built a test stand that uses an electric motor to turn a wooden disk. The handle of the generator crank fits loosely through a rectangular hole in the disk. We added a timing circuit and two counters. The timing circuit runs the motor for 68 seconds and then turns it off for 22 seconds, and repeats the cycle until power is switched off. The 68-second ON time accounts for the 8-10 turns it takes to get the UV light fully on, followed by one minute to achieve the required 40 mJ/cm2 dose. The short 22-second OFF time permits the test to be completed in about 3 hours. One counter records the number of revolutions of the wooden disk, and the other counter records the number of 68-secs-on-22-secs-off cycles.

This sort of test rig is trickier to build and use than it looks. I spent three days on a similar rotation-based test rig for comparing the abrasion resistance of different brands of paracord, and ended up scrapping the test entirely because I couldn’t get consistent enough torque and abrasion across test runs to make the test fair.

Just four of the many abrasion test setups we tried with different tools and bits.

These kinds of rotational tests have all sorts of pitfalls and land mines — issues from torque, to wobble, to timing, to speed. All of this has to be controlled, and it’s difficult to do without some combination of a machine shop and enough money to throw at the problem.

Analyzing the results

OK, so all the samples DayZero used in the test run stood up to 10,000 rotations of their hand crank with zero failures. This is good as far as it goes, but I have to admit I’m not actually all that satisfied by the result.

The problem is that “number of rotations” feels like a vanity metric. When designing a test for a preparedness product, we really like to zero in on exactly what the unit is supposed to be doing. In this case, the UV-H2O-Box is supposed to purify a gallon of water. So what I care most about is, how many gallons can these units purify without failing?

An even better, but slightly more general metric is: how long will this unit last me in a grid-down scenario?

Luckily, we can back into these numbers from DayZero’s own blog post. The test runs lasted between 120 to 172 purification cycles (each cycle purifies one gallon of water), so that’s an average of about 146 gallons purified without breaking.

At one gallon per person per day, that’s about 146 person-days of water purification. I personally would like to see at least ten times that number before really feeling good about the hand crank — so about 100,000 rotations per test sample.

So while we appreciate the effort, DayZero’s numbers still don’t tell us what we really want to know.

When we design our tests here at The Prepared, we’re always trying to zero in on what’s testable, repeatable, and above all practical. Now that our forum is up, in future reviews we’ll be able to solicit reader help in both formulating questions we want to answer with testing, and designing the tests themselves.

As for the UV-H20-BOX, we are getting one to try out as part of a bigger project on treating water. So stay tuned for that, and keep an eye on our forums if you want the chance to help us design a useful set of tests for this and other products.


  • 1 Comment

    • Sbesch

      Very keen insights into the process of testing.  I’d like to see them test 10 units to failure, then determine a “mean gallons to failure” figure.  If they never test to failure, they don’t know where their weak link might be in the design.  If they find that weakness and improve it, they might multiply the longevity by a factor of 10 with one simple change.  Thanks for advocating for consumers by demanding more informative testing!

      7 |