Mission Trumps Bias

Been a bit of a dust-up lately in the personal audio enthusiast world regarding reviewer bias. Most notably this thread (now locked) at Head-Fi, and his youtube rant (which appears to be withdrawn). But the topic runs deep in the audio community and much has been said. For example this article by Roger Scoff, and the excellent series of articles by Scot Hull in response here, here, here, and here. And going further back in time, Stereophile has published numerous articles on the subject including this.

I think these are very good topics of conversation—an open dialog along these lines will continue to put pressure on reviewers to up their game. But I find one topic consistently missing from the conversation: The Reviewer's Mission. It's perfectly understandable, people want reviewers to be impartial and as objective as possible. But I think that's a misplaced desire. Impartiality and objectivity run counter, in my way of thinking, to the dimensions of passion and personal interest with which enthusiasts approach and appreciate gear. I think a reviewer, like any professional, should be on a mission to do their job well. All subjective reviews are a refection of a variety of personal bias. A mission is exactly the sort of thing needed properly align personal bias to produce a useful review. And focus on your mission is exactly the thing needed to consciously reject undesirable bias.

Mission, Power, and Focus
Polite company finds the discussion of personal power quite unseemly. To paraphrase a quote from I movie I saw recently, "You can have Truth, or you can have happiness." It may be uncomfortable, but discussing a reviewer's power, and how s/he wields it, is at the center of this topic. As a reviewer grows in reputation they gain in power to mold opinion—it's the nature of the beast. There's no doubt, a very well-respected reviewer has the ability to make or break a product. I won't beat around the bush, that's power.

A competent reviewer will recognize this growing personal power and manage it. Power is not something in which to take pride, it should not puff up our ego. Power is a gift gained over time and given freely by readers, but it can also be taken away in a heartbeat. The power of a reviewer is essentially found in the reviewer/reader relationship: How many readers trust the reviewers opinions? Does the reviewer address issues with a set of motives and values that is strongly held among readers? Are the creative thoughts of the reviewer aligned with the desires of readers? To my mind, the most powerful approach to these issues is for the reviewer to have a very strong sense of mission, and to be brilliantly focussed on that mission. When well chosen, readers will "get" the reviewers mission and will happily follow along delivering pageviews. Pageviews is the metric measuring the success of the reviewer, but an interested reader is the actual product of a good reviewer. And like the Blues Brothers, a man on a mission can be quite interesting, effective, and entertaining.

My Mission
In reading the comments, criticisms, and concerns expressed in the articles above and in the community in general I find a good deal of focus around some common problems, namely: review samples; relationships with manufacturers; lack of negative reviews, and personal subjective bias versus objectivity. While there are common concerns, I'm not sure there are common solutions to the problems. From my point of view, I find I can only speak for myself and how I personally address these issues. Reviewers are individuals living within their own paradigms, and as a result each reviewer will have their own methods for dealing with these potential stumbling blocks. So, for the remainder of this article I will try to address how I, personally, deal with them. I'll start with what I feel is my mission.

"To help people better access the art of music by finding and describing the benefits of quality personal audio gear."

That's my mission; that's what I focus on. I find gear that is likely to allow people to enjoy their music to the fullest. I'll point out that it has a lot to do with audio fidelity, but there are other factors like comfort and styling that can dramatically effect one's ability to enjoy their music. If it's not comfortable, you're not going to wear it long enough to enjoy a full-length symphony. If it's ugly as sin you'll feel too much embarrassment while you're out and about to fully enjoy your music. I review products that are likely to satisfy a goodly portion of my audience; I try not to negatively review unsatisfactory product as it doesn't further my mission. It's a waste of time.

I also write opinion pieces when I feel they will influence the industry at large in producing better headphones in the future. For example, my articles on the Harman Target Response curve are an effort to applaud the serious work done by Harman to improve headphone listening, hopefully encouraging other manufacturers to do the same. I hope to bring attention to the subject, which will then provide feedback from readers allowing their voice to be heard at Harman as they continue their work. In these cases I hope to both write interesting material for readers, but to also act as a powerful proxy voice for readers back to industry. So, influencing manufacturers from an enthusiasts viewpoint is part of the long-term activities I take up in my mission.

My Methods
Trust in Myself Through Experience - First and foremost, I know I'm just one person with an opinion. I can be truthful to one thing only: myself. I can try to be authoritative, but in the end the only thing I can author is my well-considered personal opinion. Fortunately, I've been doing this a long time and have a well developed sense of the relationship between my personal experience and pleasure to that of others. I've come to trust that. It's not so much that I think others will have the same experiences that I have, but rather, that I have a fairly stable value system and have a lot of experience with other people's value systems. I know, for example, that I like things just a little more relaxed and polite than the average person might, and I try to frame my experience in such a way as to allow others to see where I'm coming from and adjust their impressions accordingly. For example, I positively reviewed both the Shure SRH1540 and B&W P7 even though they were somewhat too "U" shaped in frequency response for my liking; but both cans did that particular signature quite well and would be enjoyable to those with tastes along those lines. Anyway, over a long period of time I've come to trust my particular way of evaluating the sound of headphones and where that is relative to the norm, and (this is the important bit) when something gets in the way of my particular mode of evaluation, I don't like it.

Let me give you an example of when my paradigm was shaken up by a product and gave me some trouble: When I received the Oppo PM-1 and heard it I felt deeply uncomfortable. I liked it a lot, but I also knew its sound was quite unusual and would produce a wide variety of opinion. None the less, I went ahead and wrote a positive review carefully remarking that the sound of these cans would no be for everyone. Then the time came to figure out whether or not to put these cans up on the "Wall of Fame". It's my policy to return as much product as possible to manufacturers after evaluation or review (many times, for low and medium cost headphones, PR departments don't want them back for hygiene and cost reasons), but when a product reaches reference quality for that price and type, I ask for an extended loan so I can put it on the "Wall of Fame" for future reference purposes. The PM-1 was one of those cases where I wasn't sure it warranted a slot on the WoF, but I sure as heck felt a great desire for that headphone, and I trust that desire. And if it didn't go up on the WoF, I'd have to return it or buy it. Well, I don't buy headphones (we'll get to that in a minute) so they would be returned if they didn't go up on the WoF, and that bugged the hell out of me. Why? Not because I'd personally miss having it (I've got dozens of good headphones at my disposal) but because it provided an unusual but very pleasant listening experience, and maybe more importantly, provided ergonomic and efficiency advantages not available in its category. The personal bias it exerted on my was uncomfortable because it stretched what I professionally consider the norm. But my inner headphone geek—that I trust—said it was cool. It caused me to rethink my professional value system and come to understand there's room for comfort, pride of ownership, luxury, and a sonic character slightly away from neutral in the polite direction as a perfectly acceptable benchmark. After all, I'm willing to say there are good "U" shaped signatures for some folks.

The take-away of the last two paragraphs is that I don't really lust after headphones much anymore, and when I do, I don't particularly enjoy it as it makes me have to work hard to properly understand why I really like something and how that fits in with the tastes of others. I do trust that inner headphone geek and I do feel the bias pulling on me, but because I'm on a mission the bias causes me to spend time figuring out why the product is so satisfying, rather than just wallowing in lust.

Wall of Fame - Over time I've become more and more convinced that the WoF is the focal point of my work and a terrific regulator of my opinions. I use these headphones for comparison all the time when I get a new headphone in for review consideration. I don't consider the headphones on the WoF as mine, but rather as tools of my profession. I simply couldn't do my work properly without them. One viewpoint that lay-enthusiasts don't quite understand is that, at least for me, I no longer have the kind of lust for headphones that many enthusiasts do. There's an old saying, "When you turn your hobby into a profession, you need to go find another hobby." My hobby is a garage full of old motorcycles. My profession is headphone evaluation, and an understanding of the headphone market and my roll in it.

I certainly can see how some of the budding reviewers out there might easily be swayed in their opinions with the thought of acquiring headphones at low or no cost, but I doubt seasoned reviewers are often similarly tempted. As a reviewer, you're mostly looking for the next cool product, and less concerned about bulking up your collection. In fact, my biggest annoyance is having to deal with piles of headphones manufacturers don't want back. Sometimes they go to neighborhood kids, I've occasionally given them away to friends, I've even given a pair to the cable guy when he came to fix my internet connection—I suppose that might be considered personal gain—but for the most part they just sit in boxes and eventually make their way into the trash.

So, I do have a desire to acquire headphones for the WoF, and that desire is completely aligned with my mission to find headphones that serve others in their pursuit of experiencing the art of music.

Applauding the Good, Ignoring the Bad - One of the most common criticisms leveled against reviewers is the focus on writing positive, but not negative, reviews. My mission is to give people recommendations for products that deliver an good listening experience for the money. Warning them away from junk isn't anywhere near as effective as highlighting good product. It's really as simple as that. I do however make efforts to tell people about poor product performance, but I don't do it with reviews. I measure as many headphones as I can get my hands on and I briefly comment on headphones that aren't up to snuff in my "Update" posts. I figure that's a pretty fair solution.

I will occasionally write negative reviews, but only when it aligns with my mission. For example, when I heard and measured how poor the original Beats Solo was knowing that hundreds of thousands were being sold, I felt warning people off and suggesting alternatives might be a worthy endeavor. When enthusiasts were drooling over the luscious lambskin of the Ultrasone Edition 10 and hobbyists invested in the large outlay were expressing their joy and tempting others, I felt I needed to have a listen. When I did and found the cans deeply flawed, I felt I needed to warn folks away so they could spend their hard earned bucks elsewhere. Both the above cases were egregious, and I felt I could do something to stop the bleeding, so I did.

Here's a case where I'm a bit conflicted about being biased. I did write a negative review about the AKG K812. I wasn't going to, but after posting the measurements and a few comments in a Head-Fi thread I was overwhelmed with requests to do the review anyway—even someone from AKG who was active in the thread asked me to. I knew the K812 was a hot topic in Headphonedom at the time, so I knew the pageviews would be good. Given so much encouragement and the lure of good traffic I decided to go ahead and do the review. I was fair, I think, and the review wasn't a bashing, but it was negative on the whole. I still feel a little conflicted about that one.

There's no doubt in my mind that, at least temporarily, I could publish more negative reviews and might be able to pump up my pageviews with sensationalistic negativity. But sorry, I feel focussing on my mission will be a far more successful route in the long run...and I'm in it for the long run. I'm not here for short-term gains in pageviews or personal gear lust, I'm here on a mission, a mission I believe in, a mission I'm willing to carefully invest my personal power into. I want to make a difference in peoples ability to experience the art of music. That mission, that intention and motive, helps keep my biases aligned.

Relationships with Manufacturers - Sometimes people complain the a reviewers relationship with manufacturers will soften their criticisms in reviews. Well...the alternative is to think of manufacturers as faceless institutions, but they're not. They are collections of real people with passions and emotions. I think it's better to realize that, and deeply feel the difficulties of being a good manufacturer. There's not a lot of good reasons to hurt a manufacturer with a negative review, these are the people that, in the future, might deliver much better product.

Let me give you an example: A few years ago Sennheiser came out with the HD 700, a headphone I found deeply flawed. I measured it, and made a short comment in an update, but that's where it ended for me. There's no reason to try and drag Sennheiser through the mud, they've produced many fine headphones, and will produce more in future I'm sure. It's far better for me to publicly applaud the many headphones of high quality they do produce, and to encourage folks to buy those models so that Sennheiser can continue to develop headphones.

So yes, my relationship with manufacturers and knowledge of them as well meaning people with feelings does bias me away from writing bad things about them, but that's fine as it aligns with my mission and how I see it being effective.

Summary...for now
While I've got no problem discussing reviewer bias as time goes on, I'd prefer to do it in concert with a discussion of reviewers motives, intensions, and sense of mission as I believe this acts to clarify the job of a reviewer. Dialog about reviewer bias, it seems to me, is somewhat meaningless without an understanding of a reviewers stated mission.

I realize this post does not address this issue comprehensively, I see it as more a starting point for me to develop a fuller understanding of the topic in preparation for an official "InnerFidelity Reviews Policy" and guidance for myself and other InnerFidelity contributors. I'm very interested in continuing this conversation in the comments below, I'd love to hear your opinions.

COMMENTS
lachlanlikesathing's picture

Hi Tyll! I have to say thank you for the excellent thoughts on this topic. I wish I had the time to more fully respond to some of your thoughts but I'll just write a bit for now.

Firstly, because some people interpreted my comments as an attack on your reviews, I want to repeat what I said in the original article.

I have the utmost respect for you as a fair and objective reviewer, and I think you have done amazing things for the industry. I am glad that you acknowledge that relationships with manufacturers may possibly unconsciously or even consciously bias you towards moderating your comments in reviews. This is something I acknowledge myself but I have seen other reviewers repeatedly deny again and again any such bias occurs in response to my comments.

Again I repeat I do not think there is anything sinister in this, it is a natural outcome of having relationships with manufacturers which, as you point out, are good folks too.

I want to also make sure that it is completely disclosed that I started a campaign to crowdsource funding for review units so as to remove myself from dependencies on manufacturers. So I have a self interest in this topic, and this is partly the reason why my thread and my comments were moderated on Head Fi - though I don't believe at any point I was directly trying to solicit the community.

My major concern that I expressed in my article, and perhaps I didn't express it clearly enough, is that if there is an overall tilt towards positive reviews, this means that manufacturers do not receive enough negative feedback and they are not exposed to enough scrutiny. I acknowledge that you prefer to applaud the good and ignore the bad, but I am concerned that this overtime produces 'halo effects' for different brands and propels the industry towards cycles of hype and positivity.

We can disagree about this, and that's fine too, but I think reviewers and enthusiasts have more information and are more discerning than the average consumer, and I am concerned about the harm the lack of negative reviews does when the average consumer consults the available pool of knowledge. Of course I don't think sensationalist negative reviews are a good thing either, but I wish there was a better balance.

I did indeed delist my video rant about Head Fi, because while I stand by my comments, I felt that the negativity expressed in the video (which I made when I was indeed very emotional) did not really get across what I am trying to say. Specifically in regards to Head Fi I am concerned that the lack of transparency and the sponsorship in the forum creates bad incentives.

I think a lot of this is about having explicitly stated ethics standards - and sticking to them. This is something that sites like theverge.com do well. I would applaud any effort on your part to make your policies clear and create a best standard practice for other people to follow.

Before it was locked, towards the end of my thread on Head Fi I pointed out that of 13 reviews featured on the Head Fi front page, 9 of them didn't follow Head-Fi's own guidelines about disclosing the source of units. While in most cases this was because the unit was personally purchased, 2 of those cases were reviews made the administrators of the website who I would argue have more reason than anyone else to follow their own guidelines and set an example for their own community.

There must be a community recognition that review ethics is important and cannot simply be dismissed with the idea that people can make up their own minds whether or not to trust reviewers. These things are not black and white. All I am asking for is more transparency so that people have better information.

I do not mean to start a witch hunt. You are absolutely correct that with open dialogue and transparency we can improve the industry.

Tyll Hertsens's picture

First and foremost, Lachlan, I want to thank you for your bravery in thinking out loud. I like that.

With regard to Head-Fi: It's a jungle out there. We all must find our own way. Head-fi is finding it's . (Is the possessive of "it's" there proper?) I've seen the evolution of the headphone community and it's schisms, and I've been a moderator on a motorcycling site about the same size as Head-Fi (advrider.com), and I can tell you that it's not as simple as it looks.

You propose a certain approach, but you simultaneously criticize the nature of other's approach. My article above is saying I understand that that's easy to do when you look at the symptoms, but when you look at it from the side of mission, you will gain a larger layer of meaning.

I like that you had an idea for croudsourcing headphone for review purposes. I think those types of activities need to stand on their own two feet, however, promoting it from within Head-Fi tends to strain the diffuse, and somewhat noisy and unguided, nature of this common ground.

It's hard for Head-Fi to have clear mission. It's kind of easy for me at InnerFidelity...but it's mostly just me and a few other guys. There's a bazillion people at Head-Fi. As a bazillion people your mission has to be pretty diffuse and nonjudgmental to remain useful. To expect Head-Fi and it's participants to conform to any one thing strongly---however reasonable---just won't happen. Head-Fi, to some extent, by it's very nature, has to be a common denominator to exist. In important ways, Head-Fi is the town square and your opinion is a vegetable stand. Head-Fi doesn't have an opinion you or I much apart from occasionally shining a light on your stall on it's front page if it decides it's good for business. But you also can't be throwing rotten tomatoes at other peoples stalls---it's bad for business. You can't strain things in the town square.

Now, I'm not saying you were throwing rotten tomatoes at all, I'm just saying you were bad for business downtown. If you do really have something to say, if you do have a mission, then it cannot be on Head-Fi because that's bigger than just a stall. You just need to name your own street.

Anywhooo, that's all just a long intro to asking:

What's your personal mission?

That's the telling thing.

"Of course I don't think sensationalist negative reviews are a good thing either, but I wish there was a better balance."

Good on ya, mate, you add your shovel to the load. I've watched a bit of your stuff and like it. You should do it your way and see if people come read it.

Gotta respect the town square though.

lachlanlikesathing's picture

Thanks Tyll, I really appreciate your comments. I think there are many things I could have probably done differently and with a different approach the dialogue on Head Fi could have been more positive. In the end though, you're right - at some point you gotta get out of town.

While I have had some thoughts about my mission ever since I started and more thoughts as I learnt more, your thoughts do press me towards articulating a clear statement of purpose.

Thanks for your guidance and your kind words about my work - please keep doing what you do. I'm a huge fan!

Tyll Hertsens's picture
....of people energetically contributing to this activity and I see you doing just that. Keep after it mate, find a mission, make a mark.
utopianemo's picture

Tyll, thank you for clarifying why "it's" is used incorrectly EVERYWHERE. I never could understand why people misused it so much until now.

Rule of thumb you can utilize for the rest of your life: "It's" ALWAYS means "it is". ALWAYS(unless you're talking about something that belongs to Cousin It).

When you're referring to an entity possessing something, such as "Head-Fi is finding ___", you use "its". No apostrophe. As in, "It's amazing how the internet can't get its collective apostrophe placement correct".

I never could understand why that one is so often incorrectly used, but I guess it has a certain logic to it: any other possessive uses the apostrophe, so it would stand to reason that "it" would.

But it doesn't! :)

doublea71's picture

I appreciate that you think about this stuff and are serious about trying to do your job in what you feel is the right way. Negative reviews are a touchy subject because, like you said, companies are made up of people and the impact of an online evisceration of a set of headphones goes well beyond the satisfied ego of the writer. I think the best approach is to offer what amounts to constructive criticism that addresses the flaws and pointers on how to make improvements for the next version while also pointing out the strong points. That way, the reader is informed about the product and the manufacturer has something to work with when they head back to the drawing board.

As a teacher, I undergo formal observations once or twice a year and this is the exact approach my managers take when it comes to the feedback stage. If they only focused on weaknesses, my confidence would be shot and I'd be walking on eggshells going forward. Instead, I'm given some praise for what I did well, and informed of areas that need to be addressed. Not only that, I'm given specific advice about how to fix and improve those areas. I walk out of the meeting with my confidence intact and with some useful strategies that will make me better. A similar approach in reviews would benefit all involved imo.

Tyll Hertsens's picture

Wow. Great comment.

Maybe the real trick is in very discerning praise?

doublea71's picture

I think it's a matter of considering the bigger picture and not throwing something completely under the bus just because it doesn't suit one's taste or has a few warts here and there. I do think we all have our limits, however. If somebody charges $1000 for a something better suited to a landfill, then all bets are off and I'd be happy to helm the wheel of said bus. Maybe these are my own personal rules, come to think of it...I think the best any reviewer can hope to achieve is consistency, transparency, and to generally have a system of principled bias, for lack of a better term.

HeadphoneAddictCOM's picture

As a teacher do you also only give out good grades or don't grade poor work because your students are made up of people, and poor grades might affect them bad later in life?

doublea71's picture

No, we fail students if they don't meet the criteria, but when possible, we try to avoid humiliation by instilling in them the idea that there will be bumps in the road to learning a language and that persistence pays off. Preserving learner motivation, especially when facing failure, is paramount.

HeadphoneAddictCOM's picture

I fully agree about preserving learner motivation, but at the end of the day you fail students who doesn't do well enough (I hope this still applies when a student does the best s/he can, but it still falls short). The grade you give them might have huge negative implications for their future, but you still give it to them, which is correct.

Headphones (at least most of them) are the product of highly intelligent and skilled professionals' efforts. Their mission is to make something good which is well received and will make the company that pays their salary money. It's not like they have to sit in a vacuum and invent the wheel all over again, they have a vast pool of data on the internet (and hopefully internally) to make the best they can.

If they still come up short, do you really think reviewers should just let it fly? Shouldn't a reviewer keep manufacturers on their toes when they overprice something or flat out make something that performs poorly?

doublea71's picture

My previous posts answer both questions.

HeadphoneAddictCOM's picture

Please explain how it does answer these questions, I don't get it.

HeadphoneAddictCOM's picture

Didn't notice you wrote this comment: http://www.innerfidelity.com/comment/494056#comment-494056

So then we basically agree. :)

Jazz Casual's picture

I think product reviewers should declare how they came to receive the product for review and what will be happening to it once the review has been completed/published. They should also reveal any other incentives offered by the company in exchange for reviewing a product. These are minimum ethical requirements that enable readers to be make better informed judgements about the merit of reviews and the integrity of reviewers.

Tyll Hertsens's picture
When I first started InnerFidelity I got a lot of headphones from HeadRoom for measurements, got a few from Headphones.com as well. They were all returned. Now I mostly get cans from PR people. They all have different terms. Some make you fill out a loaner agreement (Shure, Denon, Bose) , but most just send a sample when I ask. I usually try to send gear back, you wouldn't believe the number of time I simply get no answer. But as I mention the simple economics of headphone under, say, $300 is that it's not worth the time and energy to reprocess the incoming loaner, for most cans. I'll add that when I took the job I had to sign a corporate code of ethics spelling out the limits of and gifts or expenses receiver from manufacturers. It boils down to nothing over $100 in gifts and travel, lodging, and meal expenses can be paid for factory visits. For example, Philips paid for the trip and expenses to their research labs.
tony's picture

I have been a Retailer , Distributer and am now a Manufacturer . I've read reviews for the last 5 Decades . Now , today , I feel confident in saying the Quality of Headphone Reviewers in the persons of Tyll , Steve G. and that joker fella that does IEMs to be the finest reviewing in the history of reviews . The only possible addition I could make would be Alex Dykes of TTAC who is easily the finest Auto Reviewer in the Car business .
I am in the 3 Trillion Dollar Transportation Industry , not the Audio Business , Headphones is my delightful Hobby .

Great work lads , I love reading you-all , I'm a headphone hobby guy because of you lads ( first saw you on a RMAF panel a couple of years back . I remember thinking back then : "wow , these guys have integrity" , I still think that after reading most of what you two comment on , great reading !! Thank You !
Sure , reviewer bias exists , my industry exploits it to our fullest abilities , we nurture our advocates with plenty of perks , the stuff you headphone lads will never have any shot at because of the size of your tiny industry ( a couple of Billion ? , certainly nothing we would be interested in chasing , a couple of Billion wouldn't move the needle for us ) . Free headphone ? , how about a free "All expenses paid loaner SUV for a year and a trip to Monaco for a Product Release , First Class" plus tons more goodies , just for saying a few nice things . I think you get the point . Jeese , you guys have to bring you're own beer . I smile when I see your Show Videos , you holding a tiny Camera , a one man show , a jiggle cam , basic sound , 720P or lesser , super low budget stuff . Nobody is buying-out you guys , it's obvious , so obvious that it's refreshing to watch , I pointed you out to my wife the other day , looky-here Judy , this guy is telling the truth , on camera no less , we watched you interviewing some guy at a Show booth table , flower shirt , fun smile , it was great ! Judy said "well , you won't see that sort of thing everyday" .
As I read your stuff I feel like I'm peering into this product thru your eyes , mind and experience , learning what you have discovered from the context of decades of this sort of work , wonderful stuff , I love it . Thank you .

Tyll Hertsens's picture
Thanks kindly for the complements, and your dandy light-hearted post. I'm glad you're enjoying the coverage and the activity as a whole. Now, about that trip to Monaco....
Claritas's picture

I appreciate your insight and candor. The only part I'm having a little bit of hard time with is the section on the PM1. I understand you to be saying that you regard products on the WOF as tools of your trade. If so, why not simply buy them like other tradesmen? I sense that there's a distinction there I'm not quite getting.

Tyll Hertsens's picture

Well...I am paying on an $80,000 lease to purchase my headphone measurement gear, and I do that out of my own pocket. So there's the real tools of the trade purchase.

Maybe my voice of words "tools of the trade" are poorly chosen. I've put myself over years into the position where I can erect a "Wall of Fame" and maintain it as a feature in the public's consciousness. And I have enough power to make it have a goodly degree of credibility. In essence, that's my job. Manufacturers want their headphones on the Wof, it's their pleasure to provide a headphone for the wall should I dean it worthy. It's my mission to make that wall meaningful, and so I'm biased to protect the integrity of the wall.

The PM1 was troublesome because my mind said it didn't really have a place on the wall due to my pre-existing internal definition of how to rank headphones. But my heart said it was a very worthy headphone in its own way. So, I had to have an internal struggle about valuation. In that process a read a lot of peoples impressions of the PM-1 and how much they looked a premium can that had a good degree of portability, and I read a lot of comments about how people at first thought it was too polite and later came to find themselves strongly attracted to the sound. Even though it's a bit rolled-off it also retains enough perceived detail to be very satisfactory. These were all things I found in my own experience. At that point I felt there was enough desirability that it warranted a place on the wall as a stander of sorts for this type of headphone...there isn't much out there like it.

Claritas's picture

I can see a case for that. The analogy that comes to mind is that your measurement rig is your tool, and the reference headphones are considerably less consequential parts.

It's pretty clear, though, that you're an/the exception amongst reviewers, so you can see why I would hope that ordinary reviewers decide after the review either to buy or return the sample (or at least make a sincere effort to do so).

tony's picture

No Industry would sell anything to a Reviewer , it is never done , period . It's a Long term loan basis , it won't end up on ebay , it belongs to the Manufacturer . We can't have reviewers selling our stuff , competing with our distribution system , very bad for business , all the way around . Reviewers can gift our discontinued products to thin out a bit , but please no selling .

mikemercer's picture

Tyll,

I will write a more in-depth response soon (family - the 4th and all) but I wanted to say that, as a person who literally grew up in the audio and music industries (starting at 17 at TAS and now, at 39, doing my own thing w/ Audio360 and for various publications) that I've also decided not to waste my readers time bashing a product. I made that choice years ago simply because that product I might bash, well, somebody else will probably like it, and that product was also somebody's art - their blood and sweat went into its development. So I don't want to pick apart something that others might enjoy - and I know my opinion is NOT absolute! Also: I'm a confessed music addict - so my mission is to find gear that connects me to the music, and allows me to forget about the componentry while basking in that connection. I feel I can do a better job of conveying that journey when a product excites me.

I'm also so glad to read your words about your mission!

and I agree that Lachlan is free to voice his feelings on any topic,
but he should've left it on his own website - not in the forum.

More to come...

Tyll Hertsens's picture
I'd love to hear your mission statement, and what methods you employ to effect that mission.
Magick Man's picture

"and I agree that Lachlan is free to voice his feelings on any topic,
but he should've left it on his own website - not in the forum."

Amen. His approach only stirred up drama and wasn't appropriate. As the old saying goes, "if you want to change something, you need to start with yourself".

johnjen's picture

A friend once asked me what ethics meant.
I answered, 'doing the right thing for the right reasons, for all involved'.

To me it's a sign that as, an individual, a business, an industry, an institution, focuses upon ethics as an important aspect of, what they do, and why they do what they do, that a sense of responsibility is or has become a key ingredient in their forward progress.

And responsibility is a common theme that all of these topics are involved with. It's a concept that is multifaceted, which is another way of saying it can get really complex, all to quickly.

These are NOT trivial nor 'easy' issues or aspects of what and how the efforts and productivity of individuals who create, should be codified, let alone be able to understand the full impact of just what all of this means not just now, but also thru time.

But the interesting thing is that those who adhear to a balanced sense of ethics are usually easy to spot, and it usually doesn't take all that much time to do so.

I'll cut this short by closing with this thought.

When there is recognition, then understanding, then responsibility, then action, the results be they short or long term, usually work out best for all involved.

Ethical considerations and actions are like that.

JJ

Currawong's picture

Great write-up and great comments in reply, but I have to correct one thing Lachlan said about the contents of the front page. Most people who write reviews on Head-Fi are members who bought a product. They don't have anything to declare. I wrote the guideline, so I should know!

That is the point about Head-Fi though: It is a community of members who enjoy sharing their enjoyment of listening to music and their means of doing so, alongside people who design, build and sell that gear, who are most often enthusiasts themselves.

As for negative reviews: There are forums focussed on trashing what the members see as bad products, but that just turns them into sites for negative and hate. That isn't fun. We're here for enjoyment and sharing that enjoyment. When you think about that, one's choices are pretty easy. Nobody is trying to sell Amway to their friends here. ;)

Tyll Hertsens's picture

I don't think I'd characterize Changstar as "forums focussed on trashing what the members see as bad products, but that just turns them into sites for negative and hate. " I will agree that there is a lot of pointed criticism there, but I also see pointed praise as well. I see a lot of disagreement amongst members there, but I don't see a lot of that disagreement resulting in flame wars---they've got a pretty healthy cultural ethic of being able to agree to disagree.

It's just different, Amos. Saddly, we all too easily look at others from our own narrow perspective and ascribe poor motives on others. They say you're milk toast, you say they're bashers. From 30,000 feet, ya'll are just different organs in the body of the hobby and serve different functions.

I'm often called the granddad of the hobby. I can tell you one of the things that sticks in my craw is all the kids in-fighting.

Currawong's picture

...but I wasn't thinking of Changstar.

Tyll Hertsens's picture
Oh well...it doesn't much matter. The answer is the same, different people, different needs, different places. In every case there a flaws and benefits. One thing I say quite often, "I've never met anyone who wasn't incredibly beautiful and fatally flawed." It holds true for all the things we humans do. Oh well....
Claritas's picture

My understanding is that it's not much of an exaggeration to say that Head-Fi is a business, sponsors are its clients, and the community are relative spectators. Or something like that; obviously, it's not so simple. It didn't start out that way, but if that's more or less true now, then as a company it should make that clear to the public.

Currawong's picture

Head-Fi doesn't have clients. Huddler has sponsors as clients, if you want to put it that way and the members, including staff, have nothing to do with that. Head-Fi is a community of members, the same as it was from the start.

TMRaven's picture

You'll never see a negative review or a review that offers constructive criticism on the front page of head-fi, especially not if it deals with new and/or praised high dollar headphones. Head-fi is allergic to such kind of things, they're too busy trying to please their sponsors.

Tyll Hertsens's picture

But I'd say it's because they're trying to create a place where a bazillion people can just get along. It's too diffuse to be pointedly critical...and, as I said in my article, being strongly critical might be a loosing proposition anyhow.

I mentioned I used to moderate on a huge motorcycle forum (advrider.com). When I first started there it was a rawkus place. Most of the board is dedicated to various types of bikes and the sub-forums are a wealth of tech info on bikes. But in the basement (members lounge) things could get really heated, and funny as hell. Problem was, some people took the ribbing too seriously and did stupid things like show up at the board owners business and start complaining about stuff that was happening on the board. And worse, a number of folks decided legal actions were required agains other members. It was a mess. The only thing to do was clamp down on a lot of the shitstirring and grab assign going on. It dramatically changed the flavor of the place, and I personally didn't like it---it wasn't as much fun any more---so I quit.

A splinter board was built and many of the "cool kids" had an exodus to the new (private, invitation only) board. It's where I now hang out with my motorcycle buddies. It's excellent. But is it better than advrider.com? No, it's just different. Advrider is still the way better place to get tech advice on your bike. there's simply way more people and you can get better advice. Both sites have their place.

Head-Fi is like that...the parallels are pretty striking to me. I've come to believe Head-Fi is very good at what it does. And yes, it's not good at some things. I definitely think that Changstar.com is much better at nitpicking and laying bare the technical characteristics of a headphone. They have an important roll as well. When I'm researching what others have said about a headphone I find myself going to both places. Head-Fi is going to have many more posts about any particular can and you'll get enthusiast impressions everything from "Hey! I just spend $200 on this and it's totally awesome! Yippee!" to "Man I'm sick of these cans, they don't fit my ears." So you read with a bit of a filter, but you do get a pretty good survey (because of the statistically significant number of posts) and generally you can get a very good idea of how the headphone impresses users.

For a headphone, Head-Fi is like a nice day out for a walk in the park; changstar is an obstacle course/WWF match.

Both have their place.

Magick Man's picture

What is crying and bashing a company going to accomplish? When the HD700s tanked on release the silence over them was deafening, only a few diehard Sennheiser fans were talking about them while everyone else was going, "oh hey, did you see the Mets game last night?" (insert other subject-changer, if you so desire). The same happened with Denon with the AH-D7100s, although they were even more polarizing. At one time I was active in a criticism thread on H-F for the latter, that was mostly constructive, and it went on for over a year. Did it need to be publicized on the front page? Heck no. However, they didn't try to hide it either.

That's much, MUCH worse for a company that relies on word-of-mouth and reputation to make sales. If they can hear a pin drop, they know they've flat-out screwed the pooch and that it's back to the drawing board.

Tyll Hertsens's picture
The headphone enthusiast community at large does a pretty good job of being negative when needed and the word gets out. Absolutely one of the reasons I think negative professional reviews aren't vital.
DaveK1977's picture

Since discovering this site through the video reviews on YouTube, I have to say that I feel InnerFidelity's reviews are excellent and useful for a lot of the reasons you've mentioned above. Your expert opinion based on time spent listening and in the lab is an admirably complete process that is unlikely to steer somebody to something they'll hate if they'll compare a few of your reviews. I think that the Wall of Fame and Stuff We Like are both ways to maintain a changing list of worthy products across the categories that headphone buyers need. I don't want to call it a wall of shame, but I think that there's something to be said for keeping some duds around for comparison, especially if they're huge sellers in the category. The best example of this would be the Monster Beats Solo. Now twice superseded, Beats sold tons of these, and products in a similar price and size range should be compared thoroughly to benchmarks of both quality and market success. Your horror at their sound was obvious (backed up by the careful test graphs that are a great part of tests here), but they were equally obviously the one to beat for a couple of years in the big selling $150~ closed on-ear category. There should be a way for you to justify keeping one that's compelling to an audience in some special way without putting it on the Wall of Fame.

Tyll Hertsens's picture
I've got the new Beats Solo2 in house. It's pretty darned good actually. I reckon there's a pretty good story going on at Beats at the moment. You'll hear more from me about it soon.
DaveK1977's picture

I've seen a lot of things that suggest the Solo 2 is like something from a different company, but at its price so far puts it at $200 and against stuff like the Q701 and Beyerdynamic's DT pro models instead of at $150 and against the much less appealing but more numerous competition that it was ruling consistently. I mentioned that it my original comment because I have a friend who thinks his Monster Solos are the most amazing sound ever, and clearly Beats was doing a dominant job at $159.

sszorin's picture

Being at the moment in an anti-social mood excuse me saying plainly that the new Beats Solo2 is the same old whore in a new dress and with a fresh make up. Some enterprising people around the OEM in China that has been making the Beats line of headphones set up for themselves a separate counterfeit Beats manufacture [...probably on the same assembly line]. The interesting thing is that they were not happy with the sound of Beats and they tweaked the product to make it sound better. The fake Beats became successful in China and they are now available even on Ebay. There are comments on Youtube from people who bought the fakes that they sound better than the originals. 'Dr. Dre' asked his manufacturer / supplier to improve the sound of 'genuine' Beats; He must have found it grating that fake Beats sound better than the real ones. Their market ground zero is in the shopping malls in Shenzhen. In effect Dr. Dre asked the pirate manufacturers to make their product an official one. The new Solo2s have an 'audio quality' circuit in them to improve the sound signature, particularly to give an impression of better clarity by boosting treble.

veggieboy2001's picture

I think any review should be taken with the proverbial grain of salt. I think that's what makes what you describe as your your mission statement so important.

When consumers start to look at reviews for the 1st time, it is overwhelming. Partially because there's no way to filter out the B.S.. The more you read, the more you get a "sense" of the reviewer (sound preferences etc) and you get a clearer idea of where the review is coming from. A mission statement only solidifies that.

To think any review could be 100% scientific seems unreasonable if not impossible. We all have biases (as well as physical differences which effect performance). As long as those biases are up front (giving context... as with a mission statement) I have little problem following a review.

I think negative reviews can have their place (Lachlan definitely has a point about manufacturer feedback) but, as you point out (and doublea71 echos in a way) does this further your mission? How does it help the audio community? Blind bashing doesn't, & I won't read those kinks of reviews.

Any way, I do enjoy your perspective (and reviews) and your mission statement only clarifies your reviews. Thank you!

Gelocks's picture

Will Lachlan then only review "bad" performing headphones to prove that he can and will indeed say things like they should be said? (you know since all other reviewers have "created an industry wide, systemic bias towards positive reviews...")?

Will Lachlan only review what people want him to review and NOT what HE wants to review... (i.e. crowd-funding purchases for his pleasure and not for his "viewers" pleasure ... that's what they are paying for right?)

Also, how will he fight his own biases and preferences and still provide an objective point of view? He clearly wants to avoid bias by not accepting unit reviews because, well "biases hurt a community that needs open and transparent information to make decisions..." But not everyone will see his POV as correct or may just THINK that he is praising way too much a product if he's too positive anyway (you know, the thing he's trying to stop and/or go against!)

I don't know... what I'm trying to say is crowd-funding/sourcing for someone to review gear sounds pretentious, egotistic/centric and just plain wrong to me. You want to review gear because you LOVE the hobby, not necessarily because viewers are clamoring for you to review all products that come out (I think...) and if so, why not buy them or ask for the review units and STILL be objective that way? You can indeed provide constructive criticism (as doublea71 smartly remarked) without being 'hurtful' if that's something you don't want to do...

I definitely understand some of your points and heck even agree on some of them, but I also believe that the current system is Ok. It's up to EACH INDIVIDUAL to decide if x or y reviewer is a shill or not AND if their preferences match!! . Is the system flawed? Of course it is, but I tend to believe that audiophiles are intelligent individuals who can discern and decide... so they will be able to see if Head-fi is trying to push products just because they want to clearly do more $$$ or because the products are actually really good or good enoough! And if head-fi is doing that, so what??? Again, it is up to each and every individual to decide what to read, what to think, what to ask on the boards and what to buy! I'm pretty sure that most visitors that sign in for an account to ask about a product they saw on the front page probably DON'T EVEN BUY THE FRONT PAGE PRODUCT because others on the board pointed to OTHER alternatives!!! That's something you are NOT taking into consideration. HECK, I don't even go to head-fi's front page usually! I didn't even know a review of mine was on it until I received a PM! LOL

Personally, I have bought and tried a LOT of headphones. I have bought some that Tyll has liked (and I haven't... like the Sennheiser Momentums) and have had some that I liked and Tyll hasn't (like the Shure SRH940s). With that info I know more or less his preferences and I can align and take his reviews for what they are! Is he shilling a product? No. Is he giving them too much praise? Maybe sometimes, usually he's on point! But still I appreciate his thoughts and love that he provides impressions subjectively and objectively.

Will you provide both as well? Are we supposed just to believe your subjective, "unbiased" impressions just because? And the answer is no! So what's the difference then between a review being on the front page of head-fi than on your page or youtube channel? That they are PROBABLY making money and you are not? Again, maybe it's a cultural thing on my part, maybe it's the language barrier, but I just don't get what a new "system" would prove, people will still try to align their taste to yours and decide that you are probably "wrong", shilling and not being un-biased anyway...

So that's my whole .02 cents.
Note Lachlan, I like you, I've enjoyed your posts on MF's Diary, and also your long-arse reviews on head-fi (lol) but I just cannot support your POV because I think that it is as "flawed" as the current system and the flaws are the same, people are the ones that decide...

Thanks..
Hope I made some sense... glad I didn't had to use a translator nor dictionaries! :)

lachlanlikesathing's picture

Gelocks, thanks for your comment. I want to address your points in a general way.

The bias I am talking about is not about being a shill for manufacturers. I am talking about subtle and subconscious biases that might moderate how we approach reviews, so that we might soften otherwise negative statements. I acknowledge these biases, so does Tyll. It is not a black and white proposition.

Tyll does not think that these biases are severe or get in the way of what he is trying to do, and that is fine. My point of view is that I am uncomfortable with these biases. I do not believe I can actively control them anymore than I can actively control the placebo effect in a blind listening test. I feel I am better off eliminating this particular source of bias completely.

I originally did start all my reviews and my Youtube channel by talking about things that I had bought and the things that I owned personally, and the majority of my reviews are still personal purchases. Over time I was asked by my viewers time and time again to review certain products, so I began requesting review units. But overtime I recognised what I feel are flaws with the current system and decided to do something about it.

For every discerning consumer I am sure that there are 10 other lurkers who don't do as much research, and I am sure that they help sustain the industry in unhealthy ways. If the pool of information is overly positive, my argument is that this leads to badly informed or irrational purchasing decisions. In recent years I feel like a bubble is building in the personal audio community (where we are starting to see kilobuck products with alarming frequency) and my feeling is that review units and unclear ethical standards propel that irrational growth.

Crowdfunding has its flaws. I am going to discover those flaws over time and I'll report on them the same way I talked about problems with review units. But on a basic level, here is my rationale.

Manufacturers have little incentive to have negative reviews posted of their products. Consumers have strong incentives to see negative reviews, since it informs them. I want to help consumers and I do not feel like I can be fully on the consumer's side if I have a conflicting interest in making and maintaining relationships with manufacturers. I don't think that it's right that I am in a system where I can freely praise a product to high heaven but can't be as negative when it is called for if I want to keep getting review units.

So I want to collapse those incentives, and as part of that I'm asking the people who like what I do to help me keep doing what I do.

Maybe it does seem egocentric. But I do put a lot of effort into my videos and I respond to my viewers every day. It takes a lot of work and I don't feel as though it is a huge imposition to ask the people who enjoy what I do and who can afford it to help me out. I don't feel like I can in good conscience ask for people's money without also disentangling myself from the system so that I am responsive to what my audience wants. Of course I will maintain my independence - that is also why my audience exists in the first place. In time we will see how this all turns out.

Please keep in mind that a lot of the stuff I want to do on my channel going forward is not even specific reviews of products. I have plans for all types of content, again because it's fun and I want to make interesting stuff.

I'm not asking anyone to trust me more or other reviewers less because of my comments. As you say, in the end it is always going to be up to the viewer. But I want to act in ways that I feel most comfortable with. I do not see why I have to put myself into a position which I regard as ethically uncomfortable if an alternate solution exists.

Jazz Casual's picture

I've not seen your youtube reviews but I have read a couple of your reviews over at Head-Fi. They are thorough and thoughtfully written, if overly long for my taste. I've also read the now locked thread there. It was interesting to read Jude's and Currawong's responses. I found myself agreeing with them and also with Brooko's posts. Your crowd~funding proposal doesn't sit comfortably with me for the reasons they have given. There is also a whiff of self-indulgence about it all, but I confess that may be a generational thing on my part. Good luck with it anyway.

hutnicks's picture

Simply put. No one ever walked into an audio store and said "Tell me what I shouldn't buy". Negative reviews fall into the same scope. Not many really want to have their search for new equipment start on a downer. We want this hobby to be a positive experience so working from reviews which highlight a products capabilities is of far more use than something that harps on a products inability to do the job.

If you are looking at a piece of equipment that's been around a while and cannot find a good review, that in itself should speak volumes to you.

mikemercer's picture

And you shall have it sir! Thanks for asking - I don't know if you ever read our collective mission statement at Audio360 (me and Warren wrote it when it was just me, Warren, Stan and Michael) but I'd love for you to read that in the meantime. I'm going to respond ASAP at The High Fidelity Report! With family for the weekend for the holiday - and really looking forward to catching up with you soon!

Here's our mission statement at Audio360.org (if the link doesn't work because of settings in here - you can just goto Audio360.org and read our About page - its unlike any other standard-issue About page you'll find, and I hope you like it):

http://audio360.org/about.php

tony's picture

Probably everyone in the Auto Industry has the Toyota Camry and Corolla on their Wall of Fame yet neither product is a stellar performer , these are Vanilla Cars but they represent the best Vanilla a person can buy , the pair of them dominate the categories they represent .

Sennheiser have the same position with their 580/600/650 headphones . I own the Sennheisers and now own the OPPOs , the OPPO is better in every way , maybe only by a little bit but my guests will point to the PM-1 if they are offered both , both sound great and both work well with the Asgard 2 .

My summary is that anything that can sound as good as the Sennheisers belongs on the Wall , especially if it doesn't require a special Amp / Cable system .

Maybe the W of F should feature two categories :

1). the regular street stuff
and
2). the Full Race 24 Bit stuff like the Mad Dogs and Audeze with their attendant High Octane Electronics .

The OPPO product belongs and has earned it's position as a Top Street Performer.
Probably the Audeze 2s should remain as the better entry level Race Product .
There is room for both offerings up there , both earned their places , the Audeze 2 can stay as the starting point to Balanced Cable Big Dog stuff .

This Headphone World is a place where us hobby people keep winning , soooo many great affordable things to be passionate about . What Fun

HeadphoneAddictCOM's picture

Firstly, I've been thinking about this a lot lately.

Secondly: Tyll, you write that mission trumps bias. By doing this, you're framing the discussion around the mission with no regard to the ethical guidelines needed to avoid the systemic bias and the incentive structure which is skewed in favor of the manufacturers under the current model.

It is my belief that ethics trumps mission, hence implicitly bias trumps mission.

Personally, I would like to see someone like you or a group of people like you come together and create a set of standardized ethical guidelines as to what a professional review should include. These guidelines should promote transparency so people can judge for themselves if they trust any given review.

Such guidelines could include (1) if it is a review unit, (2) if a reviewer has a relationship with anyone from the company, professional or personal (3) if the reviewer bought the unit, at what price.

A statement could be something in the lines of:

"This review unit was lent to me for the purpose of this review by long time professional acquaintance PR rep. John Doe at Headphones INC. I got to keep the unit free of charge."

That little paragraph makes ALL the difference in the world imo, and I will certainly begin following it myself.

RPGWiZaRD's picture

Every reviewer should just learn to write reviews in an objective manner.

Example (subjective): I like that the highs are slightly bit veiled so it prevents me from hearing sibilance that I'm sensitive to.

Example (objectively written): The headphone has a slight bit veiled highs which helps to avoid sibilance issues.

A professional review shouldn't include bunch of subjective opinions, you're supposed to DESCRIBE the sound which require extensive experience/knowledge among various headphones so you can classify roughly where the different headphones stand in various aspects. Objectively written and comparison against other headphones (not necessarily have to point out that but the review should be written in a way that you internally reflect whereabouts the various sound aspects stand like compared to all those other headphones you've tested). You shouldn't take much stand to whether the different aspects are good or bad, I mean some people may prefer a more up-front soundstaging and others may prefer a more laid-back extensive soundstage, you are not the person to tell which is "better" in that regard, that's up to the reader to decide, you're just supposed to describe how it is! :)

RPGWiZaRD's picture

Can't we edit our posts anymore? (I'm the person who needs typically like 10 edits before I'm satisfied getting my message through).

Anyway to further strengthen my point here is that I don't want to read about a person raving how Musician X's voice is giving you more eargasm than anything else before. Instead I want to read things like, the soundstaging is quite large, it's maybe slightly bit more oval shaped, so slightly bit wider sounding than deep. The imaging is fairly good, it handles most music well except very busy passages it can get a bit crowdy. The dampening seems particularly good, there's no audible resonance peaks or ringing through the mids and highs, possibly slight bit excessive energy in the bass range. The bass is fairly punchy in the character and has decent speed. The highs are slightly bit soft but resolution remains fairly good.

^ that is just some example. That should obviously internally reflect your experiences with all the vast headphones you've reviewed and this works for persons like Tyll which indeed have tested many many different headphones so their ability to judge various aspects in sound and classify them roughly how good they are compared to the other headphones in those aspects.

castleofargh's picture

sometimes the line between opinion and fact is thin. in those moments the reviewer should best know what to do.
when the difference is clear, I also firmly believe that opinions(clearly stated as such) are entitled to the reviewer, that's his own freedom to think how he does and talk about what he wants. if that means focusing on the positives of products, so be it. working with people has its rules and you can't just go and spit all day long into the hands that fed you the demos. ethic isn't something that starts only with the readers.
and relationships between reviewer and public relation guys cannot be objective and sterile. I wish it could, for the objectivity of reviews. but that just can't happen. talking about those relations openly in the review is problably the closest thing to unbiased review we can expect.

for your own reasons and choices, you show what you think isn't up for game by actually not showing it. I wish the signal were clearer at times, because of timely reasons mostly.
when something is new and getting on the FOTM bandwagon, at some point we want one, because we're that kind of creature. it's hard to know if you didn't review it because you didn't get one, because you got one and thought it was crap, or because you have too much on your hands at the time and will publish a passionate review a month later. we're hanging there not knowing when the other reviews available at the time are obviously the bad ones made too fast by enthusiasts tripping with endorphin from having the new famed toy.
sometimes you mention you're working on something while posting on something else and that gives me enough to postpone my purchase frenzy until the review comes out. but when you don't mention a thing, I tend to presuppose that you just didn't like that one headphone.
I realize that's my fault for giving meanings to your silences, but you can see how you're choice to focus on good stuff, makes a good foundation for my wild interpretations of silence. talk about bias ^_^.

well that's for opinions and personal choices.

on the other hand facts shouldn't be filtered by positive thinking or anything else. or they stop being facts. as simple as that. so good or bad if we're talking about a headphone, facts should never be silenced because they're negative ones.
and when a headphone comfort is below average it's not just a matter of opinion or bias, you had enough stuff on your hears to make an actual unbiased statement for comfort.I for one didn't buy a lot of great sounding cans because they were heavy or not comfy, so you taking down the LCD2 for a lighter headphone does talk to me. I simply love the sound, I don't buy one because I would never stand the weight.
so as long as you're biased that way, keep it going ;).

avens's picture

It's widely known that lots of the most popular headphones are not of "hi-fi" sound quality and are also overpriced, because they are fashionable accessories.

With that said, doesn't getting more traffic, while being objective, necessarily mean bashing headphones?

On a second note, I'd like to see more comparisons, face to face, and recommendations in general.
1. For one I'm searching for good to very good in-ears with microphones. Before hand that seems an easy search, as they are some of the most popular...right? After searching it turned out to be the opposite, with very few products available that actually sound good and that won't cost $1,000.
2. For some reason lots of people that are into headphones-only think everything is so subjective that for an 'outsider' it's practically impossible to find a good sounding product, but things aren't like that. In the end what I want to say is that if someone is looking for something better than beats, that info should be available here and in an easy to access to way.

alex77456's picture

Even a biased positive review is way much lesser evil than classical advertisement, as long as the bias is not due to personal gain. I somewhat like the idea of "steering" consumer interest for the sake of long-time industry improvement.

I love the wall of fame. Hope the retirees never find their way to the bin!

Also Tyll, your statement that AKG employee (as far as i understood) asked you to make a K812 review - even if negative - made me think whether was it because negative attention is better than no attention, or out of expectation that your review will be gentle?

I understand that there are quite a variety of headphones and only limited time, so it's a very understandable choice to focus on the best ones.

thelostMIDrange's picture

but the gangsters that have taken over every sovereign state are no joke. While rome burns around the world, the headfi community wrangles with pseudo politics and unreasonable fear of honesty coupled with a seeming inability for the latest generations to process anything 'negative' or contrarian. It's states of affairs like these that signal a needed implosion and rebuilding from more solid beginnings. DIY. Corporations do not care about you. It's inherent to its structure. I still say crowdsourcing would pay off your 80k testing gear although I admit it would take a while given the lack of disposable income the average joe has in his piggy bank these days, which brings us right back to the start of this post, with much more real and pressing concerns you'all might want to look at if you can stomach reality instead of burying a head in a headfi sandbox

thelostMIDrange's picture

is short sighted and dangerous, as did the writer of this article. When truth and reality cannot be processed you know it's time to make that reconciliation your top priority and not how to get a free headphone. It probably has poor fit and substandard or odd sound in any case. Life is short people. Defining a headphone mission is fine but not if it's at the expense and neglect of how to live a meaningful authentic life, as hard as that is today I understand.

Long time listener's picture

Tyll, one thing I always appreciate about your reviews--one thing that makes them useful--as that when you do mention a drawback or a shortcoming of a product, you do so directly, but in a context that makes it balanced. After reading Stereophile for many years, I finally got fed up because so many reviews, especially of speakers, finessed any negative aspects to the point that reviews were often useless, or even misleading, since you would never know what the drawbacks were until you yourself bought the product. And Wes Philips always had to cap off his reviews with a glowingly positive comment--even if utter nonsense--such as "And it sounds more than twice as good as most speakers costing less than half as much, and if that's not a value, I don't know what is!"

On another topic you've mentioned the Harmon response curve. Am I correct that this represents a kind of average of listener preferences arrived at through listening tests? I hope it won't become a standard that all headphone makers have to hew to in order to be respected--or to have their headphones reviewed. I've found that my own preferred curve is often quite different than this average, and very different from what reviewers such as ljokerl prefer (though I find his reviews helpful). I hope that manufacturers can continue to produce phones with different, distinct sound signatures.

Tyll Hertsens's picture

Developing a standard for audio reproduction standard is a complex issue, and really warrants it's own post. The problem with not having a standard is that if recording studios aren't mixing on neutral systems we end up with a "circle of confusion" where no one knows where neutral is. I'll point you to Floyd Toole's book "Sound Reproduction: The Acoustics and Psychoacoustics of Loudspeakers and Rooms". Where he goes into great detail on where neutral can be found.

TVs have standard for color balance, contrast, brightness and the like so that they can be calibrated to be faithful to the original image. So should speakers and headphones, IMO. I don't think speaker and headphone makers necessarily HAVE to follow these curves, but they should know what the target is.

Yes, the Harman target response curve was developed with blind subjective preference testing. One of the things they found is that the desired response is held much more commonly than many would expect. There really isn't a desire for more bass, as an example, by young folks that some claim there is. Just like people really don't want too much red in their TV's response.

Long time listener's picture

"Yes, the Harman target response curve was developed with blind subjective preference testing. One of the things they found is that the desired response is held much more commonly than many would expect. There really isn't a desire for more bass, as an example, by young folks that some claim there is. Just like people really don't want too much red in their TV's response."

Tyll, I understand your arguments, and I accept the outcomes of the Harmon testing. But that leaves me in a quandary. Because I listened to one of the Focal headphones you reviewed recently that is supposed to match the Harmon curve fairly closely, with my own music and my own equipment. I was eagerly expecting that finally I'd find a near-ideal headphone, balanced and voiced to meet a verifiable standard. But before long, I realized I didn't care for it--and I'm sure I'm not the only one. I'm also sure I'm not that much of an outlier.

TVs may have standard settings, but they also have adjustable controls for contrast, brightness, sharpness, etc.--which people use because not everyone sees things, or wants to see them, the same way (despite what the Harmon people think). Even though I listen mostly to classical, I've oddly enough over the years come to like a deep, solid bass very much. And I don't like IEMs as bright as the ones ljokerl tends to like. I don't want just one flavor of headphone that everyone is supposed to like. I want all 31 flavors, so I can choose.

Impulse's picture

Technically speaking, TVs have calibration controls because they're often horribly calibrated at the factory (if at all); not so much because people see things differently per se. No one REALLY wants to see a football game where the grass is neon green or a TV show where everyone looks sunburnt.

Proper calibration costs more at the assembly line tho and takes time. Same reason that high end PC monitors (with better out of the box calibration) tend to cost 2-3x as much... There are tools for calibrating displays to a standard, test patterns and special DVDs etc.

Some people might like oversaturated colors or a really bright display initially (specially under store lights) but most quickly realize it isn't accurate and just looks off later on. Video is just treated differently IMO tho...

If you start looking at something like photography, the concept of art and personal preference suddenly starts overriding the drive for utmost accuracy. Everyone will tell you that you need a calibrated display for proper workflow yet at the same time they'll admit something as basic as color saturation varies from one manufacturer to another (and can easily be tweaked endlessly if shooting RAW).

There's some interesting trends across the different consumption mediums involving how much each one is influenced by art vs precise reproduction... Headphones and personal audio gear seem to straddle that line more than other consumer markets IMO.

Impulse's picture

Having said all that, the fact that you pointed out TVs have calibration controls bears pointing out as a unique analogy... TVs have calibration controls indeed, because most decent sets are very adaptable. If you have a ton of windows in your living room you can crank the brightness, if you have a source with a color issue you can calibrate and adjust for that particular input.

If I buy a Panasonic camera but like the colors Olympus or Canon produce better I can shoot RAW and adjust them to my liking without losing anything at all. If my camera didn't meter properly under certain lighting I can adjust the white balance quite easily later on as well. Even something like video games offers endless options to fine tune the experience (at least on PC).

If I get a headphone whose sound I don't like I could EQ a bit or mess with different amps but chances are it'll never be close to a different headphone I would've liked more off the bat. There's a few tunable headphones and IEM but they aren't very common... Maybe that does mean there's room in the market for all sorts of different headphones, or maybe it means headphones aren't nearly as flexible (overall) as other more developed goods and could still benefit from a ton of research.

RPGWiZaRD's picture

I like Harman target response in terms of timbre, I'm looking for a natural response, not too sparkly or smooth, realistic and I do find Harman seem to represent quite well that but I still need my bass boost with headphone listening a great deal above the harman but the 300Hz+ range I don't mind following that target response. :) The thing with bass that appeals to me personally though is adequate punchiness, bass on its own isn't that exciting, I'm more into the initial landing impact/hit of a drum or EDM bass beat.

Bennyboy's picture

Hey Tyll, what happens to all the gear you review? Do you just keep it? If so, isn't that a bit selfish, when I am quite poor and also bored? Can you therefore send some to me please?

Tyll Hertsens's picture
....if I had someone salivating for my leftovers it might bias my system. :)
arnaud's picture

I get tired of seeing new social medias sprouting here and there with "reviewers" that have tastes so far off the map and/or conflicts of interest due to closeness to manufacturers so strong it's not funny. So what do I end up doing? Just what you said Jazz Casual: put a large pinch of salt on anything I read from this part of the "ecosystem", or more simply, I just go back to inner fidelity as I am yet to find a review from Tyll that very much deviated from my own experience (can't say the same of all reviewers out there lol).

We all hear differently, it's all about subjectivity, yeah ok. What I really hope is that time will clean this up in a natural process so that whoever doesn't deserve to be listened to will eventually loose all credibility, else become irrelevant. That'd what happens to Charlatan's Audio(TM) brands in general, so it should be no different for reviewers.

Hence all is good really, and long live Tyll's blog :).

Lawk's picture

At university you are made familiar with academic research principles, and those really make you question things, I suppose this might have triggered Lachlans post on head-fi. Or just healthy critical thinking.

Having worked at a online marketing / reputation management company in the past

(while I have no first hand experience in the audio/headphone industry)

I can assure you that in other industries there are MASSIVE efforts to manipulate all imaginable aspects that contribute to a product or service success in order to boost sales and profit.

from dating sites to car manufacturers, sign the non-disclosure agreement, get to work - after a year you will see the entire Internet with a new set of eyes.

And then bringing up youtube reviewer bias of headphones for consumers or audiophiles will be like complaining that the 10x serial killer in the interrogation room might not have been truthful about the brand of coffee you offered him he really prefers.

I mean all concerns are legit, and people are not naive here or at head-fi, but compared to the deeper layers of the iceberg the hairpin tip is rather benign.

There is no simple solution to this anyway. Those who are dependent are vulnerable, those who are independent are vulnerable (higher ups, corporation etc..)

Anyway companies will continue to profit from the amateur youtube authentic feeling reviewers, it's still not a huge thing depending on views, but I bet these companies are busy trying to determine how much they are going to bet.

For established niche reviewers of headphones like Tyll the dynamics are a little different in my opinion. I do believe a negative K812 review really does hurt the product somewhat.

I think a negative review is justified and should also be made sometimes, the Q460 for example is horrible IMO and Tyll said so, same goes for old Beats solo, while the Pro was ok, and Tyll said so.

But in my books Tyll also raves about certain Sennheiser products which to me were a disappointment. But it might not be bias, just different ears, taste.

Personally I found out, that it is essential to try a headphone yourself regardless of review.

dripf's picture

There's no need for this tributary hand wringing.

All headphone forums are sewers.

'Reviews' without valid measurements are graffiti on toilet walls.

Lawk's picture

Measurements only tell half the story.

dripf's picture

I go straight to page 3.

dripf's picture

of Innerfidelity*

Jazz Casual's picture

Where's an eye rolling emoticon when you need one?

Jim Tavegia's picture

I would think that one must keep in mind the pricepoint of the "cans" in question and temper that with the company's marketing hype in their ads and website.

I don't think that a headphone that sells for $50 should be held to the same standards at one at $250 or $500. At $200 or more the standards should be higher, especially so customers are not fooled by marketing hype and style.

If reviews talk about value most customers can decide for themselves at that point. I would hope so.

dpippel's picture

Interesting, thoughtful discussion. I like it. One thing I haven't seen mentioned yet is the VALUE of negative reviews. While rants are in most cases childish and counterproductive, manufacturers NEED to hear negative feedback. Products aren't created in a vacuum. Focus groups only go so far. Once the product has been released into the wild and a larger pool of users (and reviewers) begin to chime in on its pros and cons, the value of both positive AND negative opinions starts to become apparent.

Companies do indeed need to be challenged when products fall short, don't represent a good value, miss the mark, or have inherent design flaws. This is the way to identify deficiencies and produce better products by improving the processes used to design and manufacture them. When all we talk about is the good stuff, the bad stuff gets ignored and progress is impaired. I’m reminded of a scene from “The Incredibles” where the little boy Dash, who has a superpower that allows him to run blazingly fast, is complaining to his mother about not being able to use that power and try out for the school track team. He tells his Mom that his father is always saying that his powers make him special, to which his Mom replies in a tired voice, “Everyone’s special Dash.” He responds, “Which is another way of saying NO one is…”

Headphone manufacturers shouldn't be afraid of good, constructive criticism. They should be encouraging it.

DaveinSM's picture

Tyll, thanks for the sensible explanation of your review approach. I recently acquired a set of Harman's CL Classic on-ear headphones after getting them for a much reduced price. Seems they want to blow these out and my guess is that Harman is going to start discontinuing these like Philips is doing with the Fidelios. I am finding the CL to be bass heavy and a bIt chestier than I was expecting, with somewhat reticent treble, and I would be surprised if they followed the Harman curve themselves. I know you had a listen recently-- what do you think? And are you planning to review the CL or measure it anytime soon? It would go a long way for me to understand how our taste I sound signature may vary... Or not...

Magick Man's picture

I've posted negative reviews, but I won't wax poetic over the company's failure. All you'll end up doing is making them react in a defensive manner. Their people are people too, you know? It's better to focus on what they can do to change the bad in a product, not beat them over the head to fluff your own ego.

Tyll Hertsens's picture
I reckon any ego fluffing gets in the way of conveying useful information. Positive or negative. A good reviewer takes his job seriously...himself not so much...hopefully.
iTroll's picture

I see the relative new flood-of-amateur reviews/reviewers a bit differently ...
Q: What was SOTA (state of the art) in headphones and headphone equipment /before/ head-fi.org, IF... even before the Internet and Tyll's HeadRoom?
A: Some goodies like Grado HP-1, and Ety ER-4S, the now-legendary Koss ES, beyerdymamic 901, etc.

Transducer evolution is not easy -- e.g., basic loudspeaker design has not changed much since WWII.

With headphones, what has happened since the above-mentioned now-classic models first appeared is better economy (more for your $$: Focal, NAD, V-moda, certain Chinese IEMs, etc.)

OTOH, much-improved portable source devices (incl. revolutionary stuff like DAPs, iPods, etc.) -- due largely to Chinese economies of scale, are important contributions to headphones-based audio [but much of this is HUGELY connected to the computer industry]. Who knows where all these forum discussion and "independent" reviews have helped /these/ industries IMPORTANTLY evolve? Too much information is noise -- effective crowd-sourcing and data-mining are not easy tasks.

Seems that head-fi crowd is pretty young and passionate. That comes with young age. Passion is important ... but what's ALSO needed in real-world headphone science/engineering is empirical data, rationality and wisdom. Most of these require $$ (for tools), lots of time (for empirical data: collection and analysis), wisdom comes with age/experience.
When one is young, the passion/drive is there to write reviews ... and that often goes away age/wisdom ... one may write later on in life if the rewards are there ... e.g., you're a professional reviewer or work in related industry [Stereophile, IF, head-fi; work for Sennheiser, etc.]

Bmullock's picture

Isn't it about time that "reviewers" had to demonstrate their professionalism and abilities like all people do who have to give advice, eg. Doctors, lawyers, etc.
Peer evaluation is used in all scientific work and our hobby is scientific. If people claim to have golden ears, let them demonstrate it.
Time to introduce a professional qualification me thinks!

Bmullock's picture

Isn't it about time that "reviewers" had to demonstrate their professionalism and abilities like all people do who have to give advice, eg. Doctors, lawyers, etc.
Peer evaluation is used in all scientific work and our hobby is scientific. If people claim to have golden ears, let them demonstrate it.
Time to introduce a professional qualification me thinks!

Genkishi569's picture

First and foremost, this is only my personal opinion and I expect that people will either agree or disagree with it, and that the argument I make has certain presuppositions in place such as the lack of access to the actual gear itself for auditioning (which is always the best way to do it). Secondly, the existence of the super reviewer who has the best qualities and characteristics that any reviewer should platonically/ideally have as well as the reader's awareness of his/her tastes. Lastly, this argument focuses less on the positive effects of the negative review, and more on the erroneous conceptions of positivity and negativity that have resulted in a severe lack of personal opinion formation "when reading reviews" which gives life to the sheepish, almost mindless, practice of adopting the reviewer/s' opinions as one's own especially when referencing the highly contentious personal aspect that reigns supreme in the realm of audio.

To be brutally honest, I don't see a solution in the proliferation of negative reviews because the difficulty with this proposition is that it simply tries to address a passing need (or more aptly want) from consumers. Give it a good thought and you'll realize that once negative reviews are out and about in droves, people will, in conditioned fashion, start complaining of the "lack of positive reviews" or "the difficulty in finding a great product" or "start referencing to the golden days when tonnes of products were great..." In my experience of reading reviews, the positivity or negativity of a reviewer's opinion is less of a concern (as it is more the exercise of their right to opine and to flavour the work) to me. In fact, my observation for the cause of this clamour for negative reviews might be rooted in the average reader’s conception and exercise of reading reviews.
That I might even be so bold as to say that the average, experienced, and even avid readers of reviews do not know how to read reviews. Why would I say such a thing? (it seems to come off as elitist, but this is less of an ad hominem attack as it is a criticism of a flawed practice) It's because we do not read a review in reference to our own tastes (I'm pretty sure few can articulate their general taste in sound signature), and, hence, commit the mistake of COMPLETELY adopting a reviewer/s' stance/s as if it were our own. Okay, I know that some of you will probably say,"I don't know what the product is...what it sounds like...how it's going to feel..., etc." But the issue here is that you are no longer unaware of the product because you have read/are reading a review. I then propose that in lieu with the reviewer's opinion (which gives the review flavour and character) you must also form your own opinion based on the evaluation of the reviewer's wording (esp. in the case of SQ section), and take it in; process it; and decide whether or not the imagined sound may or may not suit you.

As you can see, not many perform a practice such as this that requires a skeptical mind as well as a real and grounded focus on the personal aspect of audio when it comes to reading reviews. It has come to the point where people simply come to a halt and question the integrity of many, otherwise, excellent reviewers, and, with this, the quality of their reviews for his/her lack of investment to the analysis of said reviews to allow a deeper understanding of the sound signature in his/her viewpoint. By saying this, I mean to convey that it is not always the reviewer who is at fault, and that sometimes it is the consumer's lack of understanding that a positive review can, subjectively, become a positive or negative review given enough scrutiny, and that there is actually no shortage of negative reviews when juxtaposed with one's taste. In fact, the clamour for negative reviews does not, and cannot rule out the fact that untrustworthy reviewers will not simply take advantage of the opportunity and bash an otherwise good product into the fires of hell (notwithstanding the possibility of damage control issues that "might" arise given such action).

The fact of the matter is that the current discourse of "proper" audio review from "trustworthy" reviewers (I won't give a criteria for this since it is beyond my point) is actually quite impartial in the sense that you WILL BE ABLE TO COMPARE IT TO YOUR TASTES because they tell you what is excessive or defective with the sound signature for them, and if you read said reviewers' enough times you will be savvy enough to know their tastes, and, in turn, be able to use their taste as a reference point for your taste so that you might create an opinion which is proper only for you, and finally decide, as a fully informed and self-informed consumer whether a product is worth your money or not.

So to sum it all up, more negative reviews will not solve an ailing system of ethical indiscretions because we will always have that ugly underbelly rife with the undesirable practice fueled by the support of industries and reviewers who both seek personal gain and not community gain through the activity of the review. Therefore, it is imperative that the consumer arm himself with the necessary tools in order to properly scrutinize a review as trustworthy and reasonable so that he/she may be assured that his/her evaluation of the review will lead to a “properly” informed response.

*I know I wasn’t able to cover many things, but to prevent it from becoming even lengthier. I just cut it here. Yes, I’m a meticulous person. Yes, I’m talkative.

ClieOS's picture

I want to thank Tyll for an article that's close to my heart. That's the eternal struggling that should be in every reviewer's heart. By the time one stops questioning whether he or she is biased or not, it is time to quit.

X