Gear Review Standards??

[Eric Blumensaadt]

In the recent solo tent state-of-the-market review I posted calling for STANDARDIZED reviews for all backpacking products – not author-generated review criterion.

That way we can easily compare, say, an older solo tent design like the TT Contrail to a newer one like the SMD Skyscape.

Letting each reviewer set their own comparison standards is chaos and is confusing when we "grunts" are researching products we want to buy or recommend.

The same should go for member-generated gear reviews. Some standards exist but if we make them the same as for staff reviewers (but perhaps abbreviated in detail) it again, makes comparisons easier.

OK, let the slings and arrows begin. :o)

[R K]

BPL used to do this a long time ago. They had categories weight, usability, features etc. on a scale of 0 to5. That was only for packs though. It could easily be adapted for other products.

As far as the member reviews I don't think that would work for the same reason the wiki never took off … nobody could agree on how to organize it. I prefer the freestyle either way, it shows the users personality a little.

[Eric Blumensaadt]

RK> I think the staff reviewers have plenty of leeway for getting their personality into a review in the text alone.

When comparing gear in the same category, stoves, for ex., I do NOT want a reviewer's personality to enter the ratings (as much as possible). They can always indicate favorites and reasons for them in their text.

What I am asking BPL for is COMPARABILITY from review to review of gear in the same category. That's not asking for the moon from reviewers or BPL.

[R K]

Eric,

I agree with you that the BPL reviews should be more standardized for comparability purposes. Perhaps use what they've used in the past and improve upon it.

I also agree that would be nice for member reviews. I'm just stating that I don't think its likely to happen, thats all.

[Bob Bankhead]

To see a standardized gear review process, check out Backpack Gear Test.

http://www.backpackgeartest.org

Click on "How to Become a Tester" in the blue left-hand column. Read through the several pages of procedures and standards required just to submit an Owner's Review.

After you've had two owner reviews published, you can apply to be a regular gear tester and receive gear directly from the manufacturer for the express purpose of writing a published, standardized review. Yes, reviewers' opinions are a critical part of the review process.

It is left to the reader to determine the degree of relevance of the review to the reader's own location and hiking style. For example, I may be curious about a particular down jacket, but have little or no interest in or practical use for owning any such item.

[Eric Blumensaadt]

RK >

Yep, yer keerect, it ain't likely to happen (review standards, that is). Too bad. I also, upon re-reading your posts, bow to the reality that member reviews need not be too standardized lest it discourage review posts.

Bob >

Yep, "Backpack Gear Test" DOES have a standard for reviewing which they require of their reviewers. I don't pay to read those reviews but they are very well done and always informative.

But I DO pay to read BPL's reviews and thus my expectations are a bit higher for staff reviews. Thus my "request" for review standards for various gear categories and even across all categories.

So BPL staff? The courtesy of a response will be appreciated.

(After all, many vendors post responses to questions about their gear. Maybe you could deign to do the same on this topic.)

[Nick Gatel]

How can one standardize a review? Weight, material, etc. are easy. I want to know things like how it is integrated into a kit. While Skukra might rate usability/convenience of a stove a 1, many would rate it a 5. What I want to know is why Andy rated a 1.

I have owned a couple UL packs that are highly rated by almost everyone in the UL community. But guess what? They just didn't work for me because the placement of pockets and a couple other items reduced my efficiency. Didn't realize this at first, but over time I got to hate those packs. How would one rate that? Also, I actually sold those used packs for exactly what I paid for them because they are in high demand due to all the great reviews. However, I did find a couple folks who had the same complaint (after the fact).

Most regulars here can pick their own gear, there is plenty of information on the web. What they often look for are anecdotal information from users they respect.

[Mary D]

I agree that reader reviews should be unrestricted. However, I've found more and better reviews by searching the forums here, although it takes a lot longer. I suppose it's hopeless to try to steer folks into posting on the Reader Reviews instead of in the forum when reporting on a piece of new gear?

A few of the reader reviews that I checked when trying to make "buy" decisions weren't worth the space given them. Here's one of those (slightly summarized and with many changes to protect the guilty), but it really was no more informative than this:

"I took this [piece of gear] to XYZ State Park last weekend. The weather was beautiful and we enjoyed using the [piece of gear]. This is an awesome [piece of gear]."

Somehow I didn't find that review very helpful! Most, though, contain much more detail and information about performance in inclement conditions, which is the sort of info I believe most of us look for when making decisions.

I would be fine with official BPL reviews (the ones for members only) if they used similar criteria as past ones, such as the recent SOTMs on lightweight and frameless packs (even if the poor ULA Ohm did get left out of both articles due to differing criteria on what constitutes framed vs. frameless), the one on down jackets and the classic SOTM on single-wall tents a few years ago.

All reviews necessarily contain a subjective element, and this is to be expected. There's plenty of subjectivity in the BPGT reviews, too, even with their standardized format. I'd rather read the subjective views and then make up my own mind as to whether those opinions affect me than end up with a simple rehash of the info on the manufacturers' websites. The same is true of regional reviews. Different regions of the US have a lot more in common than some think. An example is the eastern US and the Pacific NW–both get lots of rain, just not at the same time of year.

A lot of the past SOTM reports had numerical ratings, too, such as the one on single-wall tents a few years ago. IMHO, that one was the gold standard and the reason we were so hard on the SOTM that came out this week. That one and the one on down jackets definitely affected my "buy" decisions and I wasn't sorry for either purchase.

Remember the review of the original NeoAir? Remember how we all rejected the criticisms stated in it, especially the one about noise? (Just in case you think the "negativity" about this week's SOTM is something new!) Well, that was one item I was sorry that I bought (fortunately, I had sense enough to buy it at REI so I could return it). It wasn't for the reasons stated in the review, though, but simply that I could never get comfortable on it–I could never find the "sweet spot" between my hip bone hitting the ground and the pad's being too hard–and that every time I rolled over it "bucked me off." I think one of the reviewers did mention rolling off the pad, and I should have paid more attention to that!

[Eric Blumensaadt]

Mary,

You're very astute on saying that staff reviewers inject their own biases into reviews. Given that a standardized review format (chart type) could help alleviate that somewhat and let the reviewer air his/her biases more in the text.

Nick,

As I said above, I feel that if there is enough commonality in gear to group them into a review then they can be sorted and rated by a standard category for that gear type, say packs or tents. And the standards I want to see are in the CHART COMPARISONS. As I said previously, if Motor Trend can do it on far more complex autos BPL can do it here for backpacking gear. They explain their ratings in the text, just as reviewers can do here.

On the Neo Air I had exactly the same problem as you Mary, plus it deflated too much in the cool night air at 9,000 ft. Like you I took it back to REI. I now sleep better in a 1 oz. heavier Thermarest ProLite reg.

[David Chenault]

Your first post contradicts itself, Eric. Data can be generalized quite easily and well across time and between writers (weights, dimensions, etc). Evaluative criteria cannot, and having an excessively routinized template (like BPGT) predisposes reviews towards being both boring and less rigorous (agin, like BPGT too often is).

For example, having me use the same subjective categories as Will to compare different backpacks in articles five years apart only pretends to increase generalizability, when in fact more often than not it does the opposite, especially for the uninitiated reader. Writers should highlight their favored conditions and biases as they relate to the review in question, and from that point reader discretion must take over.

[spelt with a t]

What if definitions were developed for categories of gear, and then spreadsheets were made that list all items of gear and their specs in that category as they come onto the market. Who could do this would have to be worked out–at ratebeer.com, users are allowed to enter new beers and each one is checked by an admin to ensure the accuracy of the database. Admins also keep the database up to date by retiring beers and correcting information errors. I get the impression BPL would not be keen on that level of user control; then again, apparently there was once a wiki, so at one point community-edited content was considered a good idea.

Following the spreadsheets, categories of rating could be defined, and within those a set of features/performance that a review would be expected to address, e.g. Packs>>Frameless packs>>Frameless packs under 40L>>Features>>Exterior Pockets, where there would be pictures and commentary on some standard things like how much a hipbelt pocket will hold and if bottles can be reached in the side pockets without removing the pack. There would of course be room for a reviewer's subjective opinions as well.

Official reviews could be added as they are completed and their progress noted, thus allowing rolling reviews while still giving readers some idea of when the full review will be complete. When complete, reviews could be wrriten up as articles as well. There could also be an area where members could add links to their own reviews (posted on BPL forums or their own blogs) so that people who like certain reviewers can find their work. A check to compare function would make it easy to look at the 3 tents you're interested in without having to scroll through reviews and specs for all the ones you're not.

Non-paying members could see the sheets of specs, while paying members could log in to access the reviews.

Obviously this would be an immense project, but people here LOVE data, the more the better. I imagine if something like this were well-designed, implemented, and kept up to date, it would be extremely popular.

My idle 2c on a Friday afternoon.

[Eric Blumensaadt]

Like "Spelt" sed, "rolling reviews. Same chart standards, different (new) gear in the same category added to the review as it "rolls" along.

IF some info. is not available leave that space blank and, if data can be had later fill in the blanks & indicate with an * that it was added.

Thank you Spelt, great term, "rolling reviews". Does you wife/significant other admit that you're really smart? My wife is always calling me smart – well "smart@ss", anyway.

[James holden]

most of it is subjective anyways …. especially for stuff like packs, shoes, clothing that is highly fit dependent …

most decent modern gear will work decently to one degree or another IMO … especially if you do a quick search of REAL customer reviews … if it doesnt simply make sure you buy it from somewhere with a no questions asked return policy

the real value is in MEASURED standards such as BPL used to measure R values, true volume, etc …. the rest of the blah blah blah … i ignore like charlie brown sleeping in class …

[Tom Lyons]

I think that it might be useful to take a step back and have a real look at this entire review concept.
Do we really need it?
I say, no we don't.

I originally came here a while back to get some good direction on gear purchases, and some helpful tips.
I am not a subscribing member, and I have not paid to read any of the "articles".
I have gotten all the information I need from user comments and reader reviews, and I have absolutely no need to pay for an "expert" opinion which may be not be needed for me to make my decisions.

In all of these forum type websites, it's the readers/contributors which drive the site.
It is a "grassroots" game, and not a "top down" enterprise that makes sites like this work.

What I see happening now is a situation where people paid to become "members" and the site is now seeming(according to some members' comments) to be less of a priority to the site owner, and there are some disgruntled responses to it all, because there are expectations arising out of the payments. The complaints about quality/content of reviews has been ongoing for a while now, and it seems as if each new review just rubs more salt in the wounds of paying members.

IMO, the answer to this is to return to a user/reader commentary and review system on the forums, and let the readers glean their information from the many real users of various experience levels that come to this site.
I found that to be quite satisfactory, and I return the favor by writing my comments about the gear on the forum pages, so that others may benefit from my experiences with the gear. I see no need to pay and burden certain people with the job that is cheerfully done by users on this forum every day, and can be conveyed in simple postings in the forum pages.

I personally think the whole "membership" thing was a mistake, and it might be best to find a way to move away from that problem.

[Author]

Posts