StoveBench: A Stove Testing Protocol for Comparing the Performance of Backpacking Stoves

[Ryan Jordan]

Companion forum thread to: StoveBench: A Stove Testing Protocol for Comparing the Performance of Backpacking Stoves

StoveBench defines a protocol that is used to measure two important backpacking stove performance features (power and efficiency) in order to determine a single, quantifiable performance factor called the StoveBench Score.

[Jerry Adams]

I like the thermal images of the stove run at different speeds

[Eric Blumensaadt]

I see these tests as being highly objective as laid out by Ryan.

However (you knew there was a “however”) the use of a titanium pot v.s. an aluminum pot introduces the factor of the much higher heat resistance of titanium and the much lower even heating on the pot bottom. (More of a “hot spot” with Ti pots.)

This ti hot spot problem is very evident when comparing a ti skillet to an aluminum skillet when frying, for example.

Then there is the matter of the pot width-to-height ratio. There is a “golden mean” for pot efficiency which is somewhere in the neighborhood of bottom width : side height of 2:1. What ratio of pot height-to-width will be used?

Run a thermal imaging device from an overhead view and this problem of the ti hot spot becomes evident.

 

 

[Ryan Jordan]

Eric, all good points.

The choice of titanium was made for the standard protocol because Ti is considered the gold standard of lightweight cooking pot materials, esp. in our community. The choice of the Ti pot used in this control test was motivated by its geometry – it’s W:H ratio is about 1.3. With 500 ml of water in it (the amount of water used in the control test) the water column W:H ratio is about 2.3. I didn’t want to use the common “tall and narrow” solo cookware because it makes the system even less efficient. As a comparison, a Toaks 850 with 500 ml of water in it has a water column W:H ratio of about 1.3. It’s gonna be fun to compare StoveBench Scores between these two pots :)

What can be done from here, now that we have control test data, is run tests to compare different pot geometries, different materials, etc., and compare those StoveBench Scores to the control score, and get a much better handle on how these types of changes improve or reduce the performance of the cook system.

Thermal measurements on the bottom of the pot in these tests (where the stoves were turned up to full throttle) revealed no major differences in the size of the hot spot on pots of this size (5 in dia), with one exception – the little tiny-burner stoves like the BRS3000t, when they are running at a low throttle.

So we’ll probably see less difference for Ti-vs-Al with smaller pots, higher throttles, and larger burners. It will be cool to actually run those tests and look at the differences quantitatively, though.

[Barry B]

What about alcohol stoves?  Surely ultalighters like these! I have used Trangia and now Vargo for a few years. Recently I tested ethyl versus methyl alcohol and found that ethyl boiled faster and also burned longer, which surprised me.

[Ryan Jordan]

Hey Barry – I have protocols for solid fuel, alcohol, wood, integrated canister stoves, inverted canister stoves, and pump-pressurized liquid fuel stoves, and have performed tests with all of these types of stoves. The protocol is totally adaptable to any stove type (see the “General Test Procedure” section above).

I need to do some more rigorous statistical testing and tweaking to the alcohol and wood stove protocols before publishing them, though. The other protocols are pretty dialed and we’ll be releasing those as each of the respective stove gear guides is getting ready to be published.

[Ben H.]

so this is efficiency divided by time to boil…. and then multiplied by the lower heating value of the fuel and divided by the specific heat and density of water to get a really confusing dimensional parameter.

You could just take time to boil and divide it by efficiency and call it efficiency corrected time to boil which would only have the units of time.  The problem with all of these is you have to exactly boil the water to get a good reading which means elevation will skew your results.  Do you want a better stovebench number? just hike up the mountain!  OK… on a backpacking forum maybe that does make sense.

I think it makes more sense to compare efficiency corrected power (= stove power/efficiency) since you are trying to optimize power and efficiency.  If you use power then you can calculate a reliable number in real time throughout the test and your test protocols are not dependent on bringing the water to a boil.

In the end it will be interesting to see how your proposed linear combination of power and efficiency (as opposed to a power of either efficiency or power) identifies as “optimal” stove systems.

[Scott Chandler]

Did you find variation in weights of full canisters, or were they pretty consistent? If you used different brand canisters, would you share “full” and empty weights of each?

[Ryan Jordan]

@bzhayes: the formula actually doesn’t depend on boil time, elevation, etc. – just a heating differential between two temps over the course of some time interval. It’s normalized on a per-degree T basis.

Ice water and boiling were chosen as the two reference points because they are pretty easy to validate with consumer grade thermometers, or for estimation purposes, even without thermometers at all – just prepare ice water, start with that. Wait for the water to boil, and end with that.

I looked at calcs involving efficiency (based on the technical definition, see below) divided by time to boil but the physical description of that parameter no longer has a strong basis in evaluating the reality of stove performance, which (defined here) equals output divided by input. At some level, of course, it becomes a semantics debate.

Also, fun fact about efficiency: from a technical standpoint, efficiency is defined by the theoretical amount of fuel used in a test (if all of the heat in the fuel was transferred to the water) divided by the actual amount of fuel used in a test. For the 25 canister stoves we are reviewing right now for the gear guide, we’ve measured efficiencies from 20% all the way up to about 50%. There’s a lot of variability here, and this is where stove design and system design really come into play. We’ll present this data in the upcoming gear guide as well.

[Ryan Jordan]

@blueklister: We’re using MSR IsoPro 227g net weight canisters in all tests. Mainly because they’re the only ones I could afford to send to a gas chromatography lab for analyzing impurities (they have less than 4% n-butane which is a really good purity for consumer grade gas).

I’m hoping to do that analysis at some point on all of the gas brands, but we gotta save up a little cash for that one, the tests aren’t cheap.

The gross weight of these canisters (canisters + fuel) doesn’t vary by more than about +/- 2 g. Some of that may be in the retail pricing labels :D I’ll have a better handle on the empty weight variability as soon as I start emptying all these things and weigh them! But the average of six empty MSR 227 net canisters that I have here now is 150.1 g, with a StDev% of 0.6% – and that could be fuel residual. Very difficult to empty these all the way.

[James Marco]

Barry, Alky stoves generally conform to the same limitations as all other stoves. By using Ryans F values within the tolerances he specifies, it should be easy to compute the various values. However, most of us kitchen testers it is more of a matter of testing what we have in the configuration we will be using. (I personally noted differences in the shape of pots, heat exchanger systems, color of pots, metallic make up as Eric alluded to above, with/without a heat shield, a large difference between high and low cannister settings, etc. Mostly these were small 4-5% values in each case except cannister settings, but present. Anyway, getting back to the testing values… You will find that Ryan’s calculations will not vary much among the same type of stoves, but, you cannot use them to compare stoves using different fuels. For example compare his best F values with his worst F values. Assuming the times remain consistent, we see that just the amount of fuel used changes the equation by about half: F-1.56 vs F-0.73 Likely, it will be more of a difference since around 10gm/L is a rough guess for Canister fuel and around 40g/L is needed for alky and it would take a LOT more time…perhaps an F-0.50 would be a really optimistic goal to shoot for…not even in the same class.

As far as heat value of methanol vs ethanol, you will find that methanol has a lower heat density than ethanol. Here is a public reference: https://en.wikipedia.org/wiki/Heat_of_combustion
Methanol has about 9800BTU/lb, Ethanol has about 12,800BTU/lb or roughly 25% more heat per volume. It is easy to see that Ethanol will be more efficient since both burn roughly equivalently. The overall mass density is about .78 or so. “Ounces” is based on water density. 1 fluid ounce of water weighs 1oz. Alcohols/WG/Kero weigh around .78oz per fluid ounce. This means that a 16floz ‘coke’ bottle of ethanol will actually weigh slightly more than 12oz, discounting the weight of the bottle.

Or, in other words, 1floz of methanol is roughly the same as 3/4floz ethanol in a stove. Anyway, drop me a line offline and we can discuss it if this is a poor explanation.

[Ryan Jordan]

you cannot use them to compare stoves using different fuels

Well you can but the solid fuel/alcohol guys may not feel good about it 😂

But we can cater to them by noting the F divided by system weight values. A canister stove plus the empty weight of a canister is much heavier than many alcohol stove, windscreen, pot support, and fuel storage bottle combos.

[James Marco]

Ryan, unfortunately I do not agree with the full throttle protocol. It is probably the easiest and perhaps the only way to objectively test various stoves, though. But, as you know, turning the heat down can save as much as 30% of the fuel you would use on high. Some canister stoves cannot be turned down to a slow burn. Example: the MSR Reactor. The actual heat output also depends on the jet size. Some stoves simply do not produce a lot of heat (example: JetBoil Sol) and would be penalized by the time to boil yet produce hot water quickly with a minimum of fuel in the field as designed.

I really like the idea of starting with icewater. This is fairly consistent. More than this would require distilled water to be used for ice and for the water to boil. The DeltaT  normalizes off standard boils due to altitude and is a good number, also.

I do not worry so much about type of fuel. nButane/Butane/Propane/WG/Kero all have very similar specific heats. I just lump these together as petroleum fuels. Pressure, however, WILL effect the high settings you are using. As you discussed, a water bath to stabilize the cannister fuel temp would help. Typically in a lab setting, this would be a constant flow of some Temp of water, but difficult to set up in the kitchen labs. Anyway, this is an excellent try at standardizing the testing. Thanks!

 

[Vance P]

Ryan –

This is an excellent step in standardization of testing. Thanks for the work! I’m glad to see that you had Roger, Hikin’ Jim, Gary and Jerry help with input. Guru’s are they all!

Just as you note in your comment to Eric above, once there are benchmarks, then expanded comparisons can be made. The F number is essentially focused only one tiny (but significant) portion of the entire set of system components and actions – the flame. I think the next important move is incorporating some kind of total system weight into a further efficiency rating.

Of all the zillion other variables that can cause us to lose sight of standardization for meaningful comparison, the main factor for comparison is adding in a factor regarding weight – not just basic total system weight but also how that weight changes across the duration of an outing – that would be part of practical efficiency measurement we need to help us make good decisions about what cooking system will work for which outings and back country environments. A total system weight needs to include, at least the pot, the fuel as we will start our outing (often full), fuel container, any windscreen and a fire-starter.

[For myself, I’ve gone through a few stoves through the years: A Svea 123 (jet rocket!), a Camping Gaz Trident (the ‘low’ setting is NOT simmer), MSR Windpro, a Primus Gravity, a Optimus Vega, a Kovea Spider (using Jon Fong’s wonderful kit) – but the one stove that always kept me thinking “there must be a better way (weigh?)” was the Coleman PowerMax stove – and fuel. It was so easy (even our scouts) to squeeze efficiency out of those things! But, of course, it was the fuel container that set it apart – and led to it’s ultimate demise. I solved my yearning for a similar approach to the lightweight fuel container when I read on Jan Rezac’s isobutane solution to Roger’s winter stove – https://backpackinglight.com/forums/topic/83643/. That combo kills on weight-related efficiency when compared to anything needing a STEEL can for containment*.]

The total system weight comparisons I’ve done, both at trailhead weight and end-of-outing weight, show that fuel & fuel container make a huge difference in practical efficiency.

>>>>Keep up the great work!!   -V

* – MSR Iso-Pro (20/80) 8-oz steel canister = 72.72 btu’s per gram of canister weight; ; Ronson 5.82-oz aluminum canister (100% isobutane) = 158.12 btu’s per gram of canister weight.

[Ryan Jordan]

@jamesdmarco:

I do not agree with the full throttle protocol. It is probably the easiest and perhaps the only way to objectively test various stoves, though. But, as you know, turning the heat down can save as much as 30% of the fuel you would use on high.

I totally agree. Once we have a full throttle benchmark, we can evaluate from there how to optimize the systems accordingly.

Some canister stoves cannot be turned down to a slow burn. Example: the MSR Reactor.

Right, as is true with most pressure-regulated stoves like the Reactor. More accurately, the range of throttle is limited at the high end.

Some stoves simply do not produce a lot of heat (example: JetBoil Sol) and would be penalized by the time to boil yet produce hot water quickly with a minimum of fuel in the field as designed.

And therein lies the beauty of the StoveBench Score – it rewards efficiency. Although some “stoves simply don’t produce a lot of heat” (which isn’t so great in adverse conditions), “stoves that don’t lose a lot of heat” is even better.

Pressure, however, WILL effect the high settings you are using.

Yes, for sure. That’s why it’s really important to start with a canister temp at a consistent temperature, it levels the playing field a bit and rewards stoves with pressure regulation, which is a very good design feature to improve cold weather performance and extend canister life (# boils) without *as much* need for manual control and monitoring of the valve.

As you discussed, a water bath to stabilize the canister fuel temp would help. Typically in a lab setting, this would be a constant flow of some Temp of water, but difficult to set up in the kitchen labs.

Yeah, I thought about adding this to the protocol, but eventually canned it for exactly that reason. I want people to be able to come up with StoveBench Scores on their own and have it be as reasonably comparable to this protocol as much as possible.

Anyway, this is an excellent try at standardizing the testing. Thanks!

Thank you, great feedback!

[Ryan Jordan]

@vance:

The total system weight comparisons I’ve done, both at trailhead weight and end-of-outing weight, show that fuel & fuel container make a huge difference in practical efficiency.

Yes, for sure – this is the kind of analysis that needs to be done in comparison reviews, and information about “how to choose a stove” etc.

[Jon Fong / Flat Cat Gear]

First off, kudos to the team, this kind of analysis is rarely done.  The first results will be pretty interesting.

I understand the concept of testing at full throttle as “tuning” each stove to a metered output would be extremely difficult.  I do have concerns about using a small 0.8 l pot as a benchmark as the diameter is pretty small for larger burner head (MSR WindPro).  Given the squirrel full throttle test, it may be more relevant to use a larger pot (8″ in diameter) to reduce the noise level in the test.  Yes, it doesn’t emulate a typical backpacker but, very few backpackers would run at full throttle.  The test results could be leveraged over to snow melters who are trying to produce volumes of water quickly.

From my own experience with the Kovea Spider, I can boil 2 cups of 70 F using about 6 grams of fuel on a 1.3 liter titanium pot.  Wide open, I have seen the numbers increase to +12 grams.

 

Best regards, keep up the great work.

[Eric Blumensaadt]

As these tests go forward over the next few years we will see that Ryan’s BASE DATA will be truly a benchmark for comparison tests of:

->various fuels

->various shapes of pots

->different pot metals

->different burner shapes

->flame distances from pot bottoms (esp. with different fuels)

And here I submit for your approval new tests to be included:

->the efficacy of religious incantations on boil speed

->the efficacy of cursing on boil speed

->the bliss of complete ignorance of stove efficiency (In which we test the blissful cook’s blood pressure and compare with the BP of those doing the testing.)

GOOD LUCK Ryan!

[James Marco]

Eric, not so bad as all that.   Come, come, now… we know that prayers, magical incantations or curses have little effect on the stove’s F value, though they may indeed help warm up the person and surrounds a bit depending on how energetically or vehemently they are performed. StoveBench numbers do not represent much beyond the confirmation of the manufacturers specs in most cases. The question is, what does the F value actually represent, and the corollary, why should we be testing for it here at BPL?  Unfortunately,  I was hoping for a more simple and complete testing protocol rather than the bits and pieces being proposed here as simply a part of a larger whole. But, science is always plodding, ponderous testing, testing and more testing…

Yes, I agree with Ryan totally in the sense that these are valuable numbers to have and verify against the different manufacturers. But, his calculations for a single F value at this stage is premature. Yes, he limits variables and potential sources of error. The fuel, water, pot, set-up conditions, etc read like the first paragraph of a lengthy research paper.

But, his implication that we could then just calculate and enter the Fvalue into a database is incorrect as is his assumption that the raw data should be combined to produce an F value that has meaning beyond the collection of data. As one of the first steps in database normalizations, you NEVER enter a “calculated” value into a database. If needed, you can always just produce this number on a spread sheet (or from a group of tables in a report.) It becomes an unimportant number to enter and track, rather it belongs on the LAST page(or as a member of further calculations) on a report if needed at all.

Does this protocol answer what stove is best for the conditions each of us hikes in? No. It is a lab protocol for testing stoves…a really good start on a standardized procedure. Using icewater as a starting point is a great idea. A known starting point will help us all…easy to do, easy to use. I highly recommend this as a starting point for any stove tests and/or assume a 1C/34F using the ice water. I am not sure about the ending point. As Ryan demonstrated, you have to know your elevation. Storm systems can effect air pressure. Boiling points can vary due to superheating, too.  Typically in a lab, a magnetic stirrer would be used but in a home lab, this is a bit much. I would suggest a simple 76C/169F ending point. This avoids all the altitude issues and minimizes vaporization/heat losses due to near boiling temps. (Note the range works out to an even 75C degrees.) Icewater is a good start.

Defining efficiency is a matter of comparing actual work done with potential work done. For that, we are not getting a good comparison between stoves. As Eric mentions, there are a LOT of variables involved there. Even with the pot Ryan suggest we use, it is easily overdriven on high with some wider burners with flames spilling over the sides. (Indeed, I have seen this with some stoves actually increasing boil times/fuel usage.) On the lowest possible setting, you get heat losses from the actual pot sides and differences in BTU(KW/h) outputs from the stove itself needed to maintain the low setting. In between, settings are arbitrary and not well defined, nor, easily reproducible. Both increased time-to-boil and fuel-use on high, as used in the F calculation, can lead to misleading values. Not good…I don’t have any good suggestions there.

Most of us are more interested in a good high power stove system (capable of winter use) that has high efficiency on low settings (capable of three season use) and is extremely light in weight (capable of all season use) and takes up a small volume in a pack (there isn’t that much room in my pack.) What stove should I buy (or make?) This is perhaps a more important question.  Given the number of variables even with this, it is far from a simple answer. Let us not loose site of the goal.

[Ken Thompson]

So with the nearly endless combinations of pots, stoves, temps, etc.. what is the practical application here?

I know that I get routinely 14-15 boils per canister. What am I missing?

Edit: I see my education is lacking.

[Gary Dunckel]

Ryan, the final draft looks great. While some of the readers might have doubts that this type of testing protocol reflects their own personal use of canister stoves, the StoveBench approach allows one to rather accurately compare the performance of every stove system in his/her arsenal. One of the challenges of any scientific inquiry is to assure that the results can be standardized, and that they can be replicated. The StoveBench Protocol essentially minimizes human error and subjectivity when comparing various stoves.

While it is true that almost none of us start with 32* F (0* C) water, and that factors like altitude and wind will alter a stove’s efficiency, at least we can objectively compare the performance of each stove.

Several years ago I devised my own (amateurish) study of my favorite stove/pot systems. I was mainly interested in knowing how they all compared at the same ambient temperature and altitude with no wind, using a standard water temperature (2 cups/boil from my refrigerator). I was most interested in time to boil, grams of fuel consumed, and also the total system weight (stove, pot, and canister). I simply multiplied (time to boil) X (fuel consumed to achieve the boil) X (system weight). The lower the number, the better. Later I decided that the time to boil wasn’t all that important to me – 3 minutes, 4 minutes, who cares? So I just settled upon (grams to boil) X (system weight). Anyway, this gave me a rough idea of how my stoves compared.

But the StoveBench Protocol now offers a definitive method of accurately evaluating the differences in performance of any stove out there. The user can determine which stove will best meet his/her requirements. We can all work with the variables of altitude, wind, ambient temperature, and starting water temperature, etc., but at least we will know with some confidence which stove should perform the best for our intended purposes.

[Ken Thompson]

Thanks Gary.

[Scott Chandler]

The bottom line to these initial tests are “How much fuel does it take this stove to raise the temperature of the water 1 degree?” That’s the constant we are all looking to compare. Next comes the detailed testing of pot size, flame height, throttle opening, fuel types, weight of the stove itself, etc.. Great first step. Look forward to seeing the results within the scope of this test.

[Jon Fong / Flat Cat Gear]

Next question.  What about sample size?  When I test stoves, I typically use a sample size of at least 6 runs.  Without really getting into statistics, that sample size is pretty low from a confidence/reliability standpoint.  It does give a reasonable, first look output if you are looking at mean and standard deviation.  Then there is the question of what value are you reporting?  Mean minus one standard deviation to account for maybe 67% of the population?  It starts getting steep and deep if you need statistical information however, it is pretty solid information.  The bottom line is a sample size of one is not very valuable.  My 2 cents.

[Ryan Jordan]

@jonfong – sample size is *very* important! It’s probably the elephant in the room for most kitchen-counter stove-testing enthusiasts and cottage/garage stove makers.

The short answer is that 2 is an absolute bare minimum – IF you have instrumentation with good measurement resolution. Two samples tell you qualitatively that your method might be OK.

Three samples are required for rudimentary statistical analysis, and 6 are required for rigorous statistics that have a believable standard deviation.

If your measurement accuracy is not great, the sample size may need to go up in order to get a valid average.

I don’t get too worked up with people not having fancy instruments when I’m reviewing others’ stove tests, but I do wish they’d provide repetitive samples and disclose their statistical data so we all have a feel for how much scatter there is in the tests. I’ve purposely performed rudimentary, poorly-measured tests in the field under a range of conditions (ie not a controlled environment), and then use that scattered data to ballpark my probably low and high fuel needs for a trip. That’s a very valuable exercise for any stove enthusiast.

[Author]

Posts