Behavioral-Science Foundations of the Backpacking Light Member Gear Review System as a High-Fidelity Trust Signal

[Ryan Jordan]

Companion forum thread to: Behavioral-Science Foundations of the Backpacking Light Member Gear Review System as a High-Fidelity Trust Signal

Backpacking gear is an experience good: performance and reliability are hard to judge before purchase. This article explains how the Backpacking Light Member Gear Review System creates stronger trust signals by separating recommendation, field performance, and re-use intent, then pairing each review with experience level and days in the field. Product-level aggregates preserve nuance, helping shoppers interpret fit, risk, and credibility.

[Terran]

I’m getting an error message.

There may be a few other variables. For example a person’s size will will effect sleeping pad comfort. When AK speaks of cold, it’s on a whole different level than when I speak of cold. Age can make a difference and so forth. It’s understandable that some folks may be reluctant to provide personal details, but it does help with the reviews.

 

[Jon Fong / Flat Cat Gear]

Sorry that I have to say this but to me it read like the author is trying to defend the BPL review system.  It also comes across as book smart, reality, not so much.

The BPL reviews were flooded by staff members and people associated with BPL.  If you look at the initial reviews, a majority of them were glowing 9’s and 10’s out of 10.  First of all, the reviews were not anonymous.  That being said, people who have a differing view may not want to respond to prevent people from hurting their feelings.  Forum members (in general) tend to be polite.  I know myself; I did not respond to a lot of reviews with my thoughts for that very same reason.

As was pointed out earlier, since the review was based upon that person’s opinion, how does that relate to my own experience?  For example, I have slept on the snow in a NeoAir in a 30-degree quilt and have been perfectly fine.  My wife would be freezing.  How do people evaluate the ratings the same way?  One person‘s good may be a 6 while another is an 8.

Finally, good surveys and reviews are one of the most difficult things to craft.  There are people who do this professionally and there are courses that are taught on how to get the desired information.  In my career, I have worked with people in Human Factors who’s job is to evaluate initial Alpha and Beta products to see if we met the customer demands.  This is a huge effort that takes a lot of skill.  If BPL want to craft a review, I suggest that they seek out professionals to help in this matter.

While a lot of thought may have gone into this article, IMO is is a swing and a miss.  I do not see the credibility of the BPL Gear Reviews.  In fact, I see it causing more harm than good.  My 2 cents.

The Backpacking Light Member Gear Review System

Product Recommendation Score – 2/10

Field Performance Score – 4/10

Re-Use Intent – 2/10

[Ryan Jordan]

Jon –

For example, I have slept on the snow in a NeoAir in a 30-degree quilt and have been perfectly fine. My wife would be freezing. How do people evaluate the ratings the same way? One person‘s good may be a 6 while another is an 8.

Yes! So, you get it, clearly.

The goal here is to move *towards* an evidence-based execution of a review “environment” that increases trust between reviewer and reader. The claims made in the article are based on a pretty large body of research in behavioral and consumer psychology studying consumer behavior. This field has been well-established for many decades, and the internet has introduced many complex anamolies that we’re all trying to sort out.

The effectiveness of any review system will result from aggregation and user contribution.

You seem like the perfect profile for a review contributor, because you are able to break the barrier of not wanting to hurt anyone’s feelings, and are critical-minded. That’s exactly what we need in modern technical gear reviews – not affection for the gear.

Contributions from users like you will strengthen the system – you have expertise, experience, and an honest lens.

[Drew Smith]

A suggestion for calculating aggregate scores — use medians rather than averages (if you are not already doing so).  Medians are not sensitive to outliers, one anomalously high or low score cannot drive the aggregate rating.

[Drew Smith]

Jon – you make some good points. I would add another source of distortion – people tend to become advocates of gear as a way of affirming their own judgment. No one likes to admit “I was a dope for buying this”.

That said, I am not as alarmed by the proliferation of 9’s and 10’s on reviews. I think other graybeards on BPL will support me in saying that backpacking gear is objectively waaay better today than it was decades ago. The design, materials and manufacturing processes employed now range from pretty good to outstanding for almost all products. The market is sufficiently competitive that you can be fairly confident that value corresponds with price. Not always — but that’s one of the things reviews are for.

No review system can be perfect. But ask yourself which you trust more — results of a Google search for “best backpacking gear” or BPL reviews. The question kind of answers itself.

[Terran]

I appreciate the reviews. I find them handy and a starting point for further research. I don’t necessarily take them at face value. I find comparisons much more valuable. It’s important to be a wise consumer and do your diligence. I think most here are here for that reason. Learning how to read reviews is an art that should be learned.

[Jon Fong / Flat Cat Gear]

I am going to apologize in advance to the OP reviewer.  It is not my intent to denigrate your review.  I just want to point out what I consider issues.  Below are 2 reviews that caught my eye.  I had thoughts, but didn’t respond for obvious reasons.

 

First, the reviews were done by staff members, and the ratings were extremely high (IMO).  Needless to say, staff members tend to have more authority than a lot of participants.  This is probably true for new subscribers to the forum.

 

Second, at the top of the article is a section on where you can purchase these items: Garage Grown Gear for one.  This raises concerns about quid pro quo.

 

The GSI Outdoors Compact Scraper

 

A Staff/Associated member rated this a 9/10/10.  My first thought was “is this a joke?”.  This is a site for light weight backpacking where we are aiming to reduce weight by thoughtful selection of gear and value-added pieces.  If I need to, I use sticks, sand and stones to scrub out my pots.  For tricky dishes, I often use parchment paper as an anti-stick coating.

 

SilverAnt 0.8L Titanium Water Bottle Review

 

A Staff/Associated member rated this a 8/9/9.  It $87.99 I guess it solves a problem when money is no concern.  If you are going to go that route, you would probably be better with thin-walled Stainless steel vessel.  I do agree that micro plastics are a big concern.  Leading culprits are paint and car tires.  How much microplastics do we get from water bottles?  Spit balling here, but in the course of a year, it is probably less than driving on the road.  There are some plastic bottles that have lower microplastic contributions.

 

At the end of the day, these products were given a high rating.  Followed up by an article about Behavioral-Science Foundations of the Backpacking Light Member Gear Review System as a High-Fidelity Trust Signal.  To me there is a big mismatch between what is desired and what has been stated.  Will it correct itself over the long run?  I doubt it. My 2 cents.

[David D]

Most common drawbacks with gear reviews online whether backpacking or other:

– reviewing not stating their needs and preferred tradeoffs as part of the review

– reviewers not stating their use cases.  Do they move fast and sweat, go long and slow, sleep warm, sleep cold, etc etc

– reviewers not sampling breadth of readily available equipment then allocating 10s.  Makes the scale almost meaningless.  10/10 compared to what?

Finally, like Jon, “In my career, I have worked with people in Human Factors who’s job is to evaluate initial Alpha and Beta products” (extensively for years).  This is hard work.  The absence leads to the last and most often ignored liability of online reviews, lack of meaningful performance targets.  For example, love the MVTR tests, but what’s the lower limit for a good MVTR and under what use cases?

These are the norm.  Online these then get upvoted when they confirm the bias of the upvoter.

Its a trade off. Of course its polite to not say this stuff.  But is it polite or helpful to others to not share helpful improvements?

 

[Dan]

I agree with much of what Jon posted. I’ve seen some reviews recently that seem totally off-the-wall. But personally, I would be very unlikely to post a review that disagreed with a review posted by the owner of the site or by a staff member – i.e. authority figures. And in general, I would hesitate to post a review that disagreed with the OP of that review, because it would seem like a personal insult, with little upside. I found myself wanting to add a review to the products I liked, and not wanting to contribute to the products that I didn’t appreciate. So in the end, I decided not to contribute. The format of the review threads, with an initial review and responses, is problematic in this regard.

So just based on my own emotional response, I suspect that there are strong biases built into this system. Nobody expects otherwise, to be honest, but I don’t think the system is worthy of a pseudo-scientific meta-analysis.

[Terran]

There will always be flaws. Nothing is perfect. You strive for perfection knowing you’ll never reach it , but if you don’t the results can be catastrophic. As teachers and as students, analyze everything. Realize what works for one person might not work for you and visa versa. If a certain ad sets off a red flag, take that into account. If you see it, realize others do as well. If there’s a better way or if you believe there’s a better product or one that may better serve someone’s needs, bring it up. You can critique or you can improve and make it work. Realize the average consumer searching for reviews is pretty savvy.

[Jerry Adams]

maybe have more numbers – size, weight, when did you get it, link to manufacturers web site

you could have a link to where it can be purchased and say that you get a small fee if the product is purchased there – it costs money to run a website and it’s reasonable to have ads to pay for it

I don’t look too much at the number of stars a products was given by the user, but I do look at the negative reviews to determine of that characteristic would be important to me.  So, maybe have a section “this is why I would buy this” and “this is why I wouldn’t”.  Like, maybe a product wore out early but it was used in an extreme way that wouldn’t apply to me, like mountain climbing or arctic expeditions

[Jerry Adams]

or someone could say “this pack is no good because it doesn’t have any pockets”

I don’t want pockets so I would know to ignore the low number of stars for that

[Jon Fong / Flat Cat Gear]

It starts at the beginning

This article explains how the Backpacking Light Member Gear Review System creates stronger trust signals by separating recommendation, field performance, and re-use intent, then pairing each review with experience level and days in the field. Product-level aggregates preserve nuance, helping shoppers interpret fit, risk, and credibility.

IMO, the review results do not match the tone and objective of the main objective.  There are a lot of references and discussion about Human Behavioral Science as a Foundation for this format.  What many people do is read through the comment sections and see if that aligns with their viewpoint/ perspective.  So as I said earlier (IMO) the current process is a swing and a miss.  There are better results but just looking up gear reviews and sorting through the wheat and chaff yourself.    The above process overstates and underdelivers (show me the data how this method is better than other reviews and give examples).

Believe it or not, a good review system IS ROCKET SCIENCE!  it takes a tremendous amount of work , energy and refinement to get it right and trustworthy.  That, and you need a large sample size to fill out the space.  My 2 cents.

[Jon Fong / Flat Cat Gear]

Here is an example of the SilverAnt 0.8L Titanium Water Bottle Review on Amazon.  I does not have the hthree tiered rating system of BPL, but BPL will not have the statistical numners to provide a robust viewpoint.

4.1/5 Stars 240 reviews

And here is a review that aligns with my thoughts. The person bought it for show, fancy titanium that he paid a ridiculous price for as a status symbol as he notes there are much cheaper options out there.

[baja bob]

Does it really need to be this complicated to review a piece of gear? I think this is the first time in the history of internet forums where there needs to be a pseudo-scientific article about gear reviews and a long drawn out debate to justify the gear review format that includes affiliate links to the product reviewed by the website staff.

BPL should go consult with a marketing expert to come up with a better way to monetize its product. Slipping in affiliate links everywhere tied to a “review” does not lead to credibility. Quite the opposite. The Zen Bivy paid research article that not surprisingly had Zen Bivy as the top performer is a prime example.

The Reddit Ultralight subreddit has a basic review template. It works and no one questions the veracity of the reviews.

 

[Terran]

I don’t think price is a variable when determining quality or use. Nor do I see it as a status symbol. If I pull my high dollar backpack out of the trunk of my 13 year old Toyota with peeling paint, please don’t I’m being a snob.

[David D]

The gear reviewed and recommended is usually top price.  Comparisons with top performing but much less expensive gear would be a big help.  Sometimes that last couple percent costs an arm and a leg.  Context is helpful.  I’m a gear nerd so I almost always find excellent much lower cost alternatives.  But I think high trust high utility reviews wouldn’t leave so much of this as an exercise for the reader.

[Terran]

Etiquette. I feel this is the place to discuss it. I feel a second companion thread is sometimes necessary to prevent thread drift. Does an alternative review of similar products belong in the same thread? Comparisons are valuable. Should they be brought up? If so how?

[Dan]

Etiquette. I feel this is the place to discuss it. I feel a second companion thread is sometimes necessary to prevent thread drift. Does an alternative review of similar products belong in the same thread? Comparisons are valuable. Should they be brought up? If so how?

Absolutely, I think that the threads are set up to accommodate multiple reviews for a given product. But the point I was making in the other thread is that IMO a post that reads like an alternative review should be an actual review from someone who has used the product.

[Terran]

I don’t know the answer. I find some value in the opinions. I think they may lead to thread drift. Maybe I’m doing that here?

[Dan]

Maybe you’re right. People should just post what they want to post, and the readers can sort it out.

[Terran]

I’m not right. If a thread drifts too far away from actual reviews, it will start losing value. If I was researching, I’d maybe read the first post and ignore the rest, then spend my time looking elsewhere.

[Matthew / BPL]

This is a good discussion and potentially we should fork the conversation to a new thread but I’ll keep it here for the moment.

Here’s my thought: if the thread is reviewing ABC water bottle then posts should be reviews of and questions about ABC water bottles not why users think that water bottles in general are useless/useful or that they don’t like water bottles or that they prefer DEF brand or why they think water bottles are _______.

The thread is for people to review that brand of bottle (and some follow up questions). Sound good?

[HkNewman]

9’s and 10’s

Also figure most gear sold is tested in “normal” backpacking conditions.  It’s not like the typical ultralight backpacker is running from polar bears in the Arctic Winter.   I gave pretty much the ma on the Nu-25 headlamp thread, but it pretty much works as advertised.

So think there will to be a base amount of functionality when retailing gear, but always some tradeoffs.   Still that’s assumed, so maybe a lower temperature used for quilts and battery operated electronics, user height for shelters (assuming they’ve used the manufacturers guidance in selecting the right shelter for their height).  Think there’s some quantitative measurements that can be added for categories of gear.

That said, there’s intangibles (such as liking an elasticized opening for a quilt to keep out drafts even though one bonus of a quilt is freedom to move limbs, though other users may “feel” differently.  Some review thoughts will need to be descriptive, anecdotal evidence (liked so much, .. bought another if the original was lost), etc…

[Author]

Posts