In response to the cacophony: Tyll' Stance on Reviewing

Quite the tumult over my Sony MDR-Z1R review. The forums churned. I reckon a little re-cap is in order:

  1. On June 5th, I Published a review of the Z1R and found it to have incredible build quality, styling, and comfort. I also found it intolerably uneven and stated I'd prefer to listen to the ATH-M50X. This is a statement of preference not absolute performance and price is factored into this comparison.
  2. I was heartily rebuffed and, in some cases, reviled and insulted as a result starting about here in the Head-Fi thread.
  3. On June 10th, Jude published measurements contradicting the InnerFidelity measurements showing a significant peak at around 10kHz. Evidencing listening experiences, and frequency response and distortion measurements, Jude concluded:
    Based on what we've heard here (and also on hearing other MDR-Z1R's)--and based on many reviews/impressions that predated the publishing of Tyll's measurements--we'll also assume for the moment that our impressions and measurements are more representative of the MDR-Z1R's that are out in the wild.
  4. This poured fuel on the firestorm of posts in the Head-Fi thread, and spurred a critical SuperBestAudioFriends.org thread. I found the counter-viewpoints of the two threads interesting, entertaining, and enlightening.

  5. On June 16th, I published my response to his findings, in which I show numerous other enthusiast measurements showing a similar 10kHz elevated feature. More gasoline.

While the ensuing mayhem was fun, I walked away concerned: There is a pervasive idea among, what I'll call, the hard-core subjectivists that any peculiar "sound signature" is not flawed if you like it. I have no problem with people liking what they like, but I do have a problem with abandoning a reasonable middle ground from which to evaluate headphones. As I said at SBAF:

Other Enthusiast: The thing is, if we are evaluating the merits of audio gear, as compared to each other, different signatures are fundamental.

Tyll:Every time you say "different sound signatures" I hear "the astonishingly wide array of ways things can go wrong."

Other Enthusiast: Headphones that exist outside of standard models are not necessarily wrong, but admitedly are also probably not going to be well received by the majority of people.

Tyll: "Not well received by the majority of people" sounds like a pretty good definition of "wrong" considering the subjective nature of audio evaluation. It's my job to get it right for "the majority of people." I can't possibly be right for everyone, so I try to give good advice for most.

*snip*

I can tell you in the world of speakers that Sean Olive often sums up the tens of millions of dollars he's spent on researching subjective opinion by saying, "With measurements we can tell with 87% accuracy how people will feel about the sound of a pair of speakers."

These aren't just your average Joe listener either, it's a well rounded sampling of all demographic groups. He's even said there's not much difference when you compare things group to group. With trained listeners, he says his prediction accuracy is much higher.

Now I think Sean tends significantly more to the objective side than I, and I sometimes cringe when he says some things, but he's done the no shit-research to prove the things he says. And what he says is that listeners will basically prefer speakers (in a properly treated room) that: measure flat in an anechoic chamber; are even in their response (no big bumps and wiggles in frequency response); and have well controlled dispersion.

For example, I don't like his headphone target curve in some ways (over emphasizes 600-2.5kHz and 4-8kHz by a few dB) but I do understand why it's drawn the way it is. I'll also note that that target curve is just a preliminary curve they developed to put in the mix of curves to test. That I'm aware of, he hasn't done testing to finesse it.

At any rate, I've got no problem with folks wearing rose colored granny glasses, or high tech, multi-layered, micro-coated, prescription motorcycling glasses (I have some and love them), strokes for folks. The problem arises when the hippy chick in the sun dress looks at you dead serious over her little spectacles and says, "You really need to do some ear candleing now while the moon is in Virgo."

There is a Neutral
Sean's experiments correlating objective measurements and subjective listening satisfaction work for a simple reason: well designed speakers that measure well and do a good job of reproducing audio are what people generally like. As speakers or headphones become neutral and transparent you can better hear the music in its most natural way. The original artist/producer/engineer are driving after something...it often sounds "right." Anytime you add, subtract, or otherwise deviate too far from neutral you run the risk of producing coloration that taints the original musical expression.

I like the word neutral because it doesn't say it's good. But it does say something. Neutral means: Most like real listening. Now we could write a book defining that, but let's just agree it means the reproduced version sounds quite like the cello sounds in front of you. High-fidelity = highly truthful to the original thing. I would generally claim that this is the basis of high-end, reference quality audio equipment, and that some/many aspects of this is measurable.

Let me stipulate, at this point, that there is a good few dB margin for deviation from "perfectly neutral" within which a manufacturer can create an aurally pleasurable experience. Things like second harmonic distortion in singe ended tube amps and simple device circuits a-la Nelson Pass. These things are basically unmeasurable in their gestalt producing effect, as I know it. I'll add that I'm not sure we're able to, or know what to, measure relative to that far more complex interior experience of music listening. So, I reckon there's a good margin for slop around technical measures and still remain pleasurable.

However!

There are measurable performance errors that clearly indicate deviations from the norm that are known to disturb the listening experience for most. These would include: rough, uneven frequency response; significant increases in bass distortion as level increases; significantly noisy impulse response blurring leading edge transients; and inordinate level deviations from known standards (free-field; diffuse field; Harman taget curves).

Now, I have no problem with people liking what they like. I like anchovies on pizza...but I rarely order it if I'm sharing with others. I get it, salty fish on pizza is weird. Similarly, I don't have a problem with people enjoying a "V" shaped response and will even recommend then if tastefully done. (Shure SRH1540) What I do have a problem with is a headphone response that is so far away from neutral that I have no way of enjoying them or figure out how to recommend them to someone with out-of-the-ordinary tastes. If I try to please everyone, I'll please no one as a result.

As a reviewer I feel I have to speak to the norm. It's important that people know where I'm coming from so that I can be used as a benchmark of sorts. People know I tend to like things a tad on the warm side and I try to remain consistant so folks can interpret what I'm saying and put it into their own context. (I'll note here as we evolve toward something like a Harman Target Curve with a bit extra bass and a slight warm tilt I find myself fitting in quite nicely to that norm.) Also, I try not to over-evaluate a headphone to noodle out small, subtle details because I know that people may hear those small things differently than I given variations in taste and physiology. I try to keep it simple, relevant, and, most importantly, straight down the middle.

When I find myself listening to something like the Sony MDR-Z1R with significant deviations from neutral in excessive upper-bass/low-mid emphasis and lack of tight time domain control with large resonances at 3kHz and 10kHz, I'm going to call it flawed regardless of whether or not it was Sony's intention to tune the headphone in that way. It's simply not going to please most listeners in the long run.

So, I promise:

  • I'll keep reviewing headphones the same way I always have. The bar may move sometimes as headphones get better, but my tastes and evaluative criteria will remain consistant.
  • I'll keep measuring headphones the same way so that InnerFidelity's database of measurements will remain self-referentially consistant.
  • The more expensive a headphone is, the more critical I will be. The same modest departure from neutral may be acceptable on a $200 headphones but unacceptable on a $2000 headphone.

Thanks for following along. I'll keep working at it.

COMMENTS
GimmeCans's picture

It isn't 'good' just because it costs a lot or because some advertiser's ox is being gored if you say it isn't (not the first time such a thing has happened on head-fi). For what these cans cost, it shouldn't have flaws that other headphones at a much lower cost avoid. This is Sony's attempt to tap into the 'money-to-burn' market, and if the result is a headphone that doesn't 'walk the walk', too bad for them. Right or wrong, you can at least count on integrity from inner-fi. Stick to your guns.

Visigoth's picture

Well f*cking said!

skris88's picture

Yes. Well said Tyll. Stick to neutral. Natural.

Let those who like sharp, bassy or forward sounds go elsewhere for their learning.

pbarach's picture

I just want to know what you were mumbling to yourself while your head was in your hands...

inventionlws's picture

and keep it up!

GNagus's picture

You would have really made us laugh if they were Grado or Ultrasone headphones

jim in cheyenne's picture

For me it is incredibly important to have a reviewer that evaluates relative to 'neutral'. Otherwise one has no firm base. Along with that you provide measurements. Yes those vary a bit depending on equipment, but as you note, consistency is the issue here. With consistency in the measurements, one gets a good idea what to expect. One element you can't control is variations in the phones from the manufacturer, but you do a good job of reviewing other reviews to see if that is the source.

Keep up the good work!

100VoltTube's picture

They have an audeze carbon fiber headband, and the cups appear to be black, so I guess they're the new LCD-pro. Unless he has a pair of LCD-X's with a carbon fiber headband, or a pair of very dark LCD-4's.

windcar's picture

Good job, keep it up. Ignore the head-fi shrills. Over the years, I found that site loves to hype up plenty of stuff depending on which is FOTM. There are so many noises over there it's hard to get accurate information.

sdecker's picture

Seems most of this Sony tiff could be mitigated if we knew about sample-to-sample variation. Otherwise every reviewer could be hearing varying sonics from their pair and their reporting on what they're hearing is largely accurate. This is especially crucial when a high-price low-volume headphone first comes on the market and the QC consistency just isn't there (yet). They've made a zillion HD600s over 20 years and they all sound the same. Much harder to manufacture planars with much smaller fabrication budgets and much higher price tags deserve special scrutiny in this regard, especially on early production units, even as there are fewer on the market to evaluate.

kais's picture

Headphones don't work like loudspeakers in a room.
Every individual has a different HRTF and it is, opposed to loudspeakers, largely bypassed by using headphones. So what does not fit to one might perfectly fit to someone else.
This is especially true for frequencies from 1 kHz and above, and the differences increases with increasing frequency.
Just look at the primary ear canal resonance around 3.5 kHz:
the frequency and boost largely depends on the pinna and canals physical size so it's different for everybody.
Still, a very strong resonance in the 1 to 5 kHz area is perceived by most and may be considered a defect.
So the individual HRTFs might be one main source for the different opinions on the same headphone.

BlackWolf's picture

At this price point, I would personally view it as Quality Control Problem vs sample-to-sample variation. Variation sounds too passive for this type of money.

My problem is that only a few professionals have a meaningful way measure a frequency curb. What concerns me is that there must be many out there who shell out more than 2k for faulty units and they do not know at all.

At this price point, Sony should have put every units in a through testing before selling to a customer. How much would it have costed Sony to test every MDR Z1r? $20-30 at max? Sony decided to put that extra few bucks in their pocket rather being faithful to their customers who shell out more than 2k... Unacceptable in my opinion.

sdecker's picture

You're right, I was too generous in suggesting sample-to-sample variation when at these prices (and a lot less expensive models too) they better have a solid final product test suite established and reviewed before each phone is packaged. Set a target max deviation for a number of parameters and reject the outliers. Micro mechanical machining is imprecise with a small run of expensive 'audiophile' headphones barely out of development with incomplete manufacturing methods still being optimized, if they ever are or effectively can be (looking at you Audeze).

Ironic that I'm a career electro-mechanical test/quality engineer and I didn't call out a FAIL for clearly half-assed final test results!

wiinippongamer's picture

Audio reproduction is a science. No more no less.

romaz's picture

There will always only be one Tyll. Don't change a thing. Much respect.

headwhacker's picture

I appreciate what innerfidelity is doing with the combination of objective measurements and subjective personal preference evaluating headphones. I may not find myself agreeing with Tyll's subjective preferences all the time but it helps me to find the headphones I like. Like you said just be consistent so that you become a reference point for everyone evaluating headphones according to their personal preferences.

Thanks to your collection of measurements, I was able to find my gems in GR07, Blue Lola and even the tascam TH-02 (TEAC CTH-02) without having to audition them prior to a purchase. Price indeed has no correlation with the headphones SQ.

sunnydaler's picture

a forum full of pro-market idiots.

RPGWiZaRD's picture

The issue with this community boils down to the many various factors:

- Different reviewers have different methods; some take a more objective standpoint, others more subjective approach. We as humans typically are bad at accepting different ways of approaches and opinions as we like to believe "I am the right one", it's one of those bad natures of humanity that lead to wars, especially with people with strong subjective opinions.

- The consistency of manufacturing of headphones is often not given enough analysis, I'm glad Tyll lately have brought this issue forward quite clearly. We may simply be arguing over two headphones that happens to measure very differently so the persons hears a quite different sound and then throwing sticks at each other.

- The human ear and the ear canal is invidual for every person, in speakers, it's more about the room threatment as well as the listeners hearing capability and interpretability but with headphones, the percieved sound from a headphone will always be slightly different so even if it measures perfect in a set of standard "doll head", it's not exactly the same for every person.

- Money & sponsorship. Different sources may be more or less exposed to that. It's often hard for the reader to be able to judge where the reviewer stands as there's often lack of transparency on this part which quickly leads to assumptions. With better transparency this could be improved.

- Psychoacoustics, ie. the science studying the psychological and physiological responses associated with sound. Especially the more subjective oriented the person is, the more important this topic becomes since subjectively oriented reviewers tend to more often go by "what I enjoy, must be good"-sentiments. But when you get into psychoacoustics, you'll quickly discover based on several researches that our brain simply release the "feel good" dopamine based on sounds we enjoy and what triggers this "feel good" response is highly invidual and therefore I personally think it's very important to at least to some extent, be able to separate from personal tastes as well as objective analysis.

- The complex task of translating a personal experience and findings into a well interpretable context for the masses which comes from vary different backgrounds. Being able to write in a way that gives an accurate presentation of one's findings isn't always the easiest task. We often get caught up in our personal experience too deeply and might forget to present several factors which may or may not affect the end result.

What I like about Tyll's approach is his consistency and objective standpoint but still sees value in the subjective analysis. For me personally I find his approach works best for using as guidance to find the headphone which matches your personal tastes. Everyone has to find their preferred sound themselves as I wrote earlier we all have different tastes (due psychology, how our brain associates with certain sounds) as well as different structured ears. It's impossible for anyone else to tell what headphone is right for you, you'll have to do your own research! But once you got a good grasp what sound you are looking for, that's when Tyll's efforts and approach to his way of reviewing becomes extremely useful.

castleofargh's picture

is known to be very effective in relaxing your eyes.

there seem to be a fairly universal idea of neutral speaker nowadays a great deal thanks to the Harman crew. listening levels would probably shake that a little but in a very predictable way following the equal loudness contour of the listener. so it's all good.
meaning that despite physiological variations, our brain finds a way(no idea how but it's amazing)to calibrate the response well enough for most humans to apparently end up hearing about the same thing.

but that logic is exactly why I don't agree with a universal neutral response on headphones. if the brain compensates for the body, then when the sounds come from a different angle and don't get affected by the body and head, what happens? IMO we still have the brain compensation messing up the response and based on our very own body.
and based on HRTF measurements, some of those variations can go well beyond 2 or 3dB. to me they're significant enough to say that 1 neutral doesn't work.

maybe after a long time when we spend a lot of hours with a headphone, the clever brain starts working its magic on that headphone. after all I can walk, I can use roller blades, a bicycle, a car, windsurf... and end up with references and god reflexes for all those different systems. the brains knows to switch when moving from one to the other. so why not the same brain plasticity for headphones after some time? but then IMO it could be an even bigger problem as right now, we all get used to BS responses from headphone responses all over the place.
what if the calibration on those responses is the reason why the next headphone sounds good if rather close in some aspects, or like crap if not? it would mean that people using a similar headphone could tend to like the same new ones more than people who have been used to a different signature.
if that was real, because for all I know it's just my wild hypothesis, then it's a chicken and the egg problem. to help get a universal neutral we might need to first set a universal neutral.

of course getting responses based on our own HRTF data would bypass all that crap and make so much more sense. the answer is something like the Smyth realiser IMO.

as for keeping your calibrations and way of doing things, I'm 133.7% with you. the value of your work comes from a common reference. bith in measurements and in subjective opinions. if tomorrow you started saying that your preference goes with stuff Joker likes most, I would be very confused. he likes colder stuff than I do. you like warmer stuff than I do. I know it and can work things out on my own like a big boy.

castleofargh's picture

wall of words!!!!
even the white walkers can't possibly get past this.

timmyw's picture

Keep it up mate. Your work has helped me a lot over the years.

thefitz's picture

10kHz spike or not - that 300Hz step response was a joke. If a headphone measures with that 300Hz step response, it's going to make your ears bleed. End of story.

JimL's picture

I'd give you three thumbs up but then I'd be a monster.

Three Toes of Fury's picture

THIS is why Innerfidelity is, bar none, my go-to favorite site for headphone and headphone related reviews, discussions, and information.

Thanks Tyll for the brain dump on the subject and your promises to keep doing what you do so well...much appreciated.

Thanks Talkbackers for the great discussion and laughs. There's lots of both today. By and large, y'all have good, open, sometimes critical/conflicting, but most often professional and productive discussions about a subject near and dear to ya.

Peace .n. "Without music, life would be a mistake" -nietzsche

3ToF

skyblazer's picture

First, this is my first comment on this website which I've been reading for a few years now, great website.

Now for the heart of the matter, I feel like I've seen this being talked about a lot of times. Almost ever since I've been reading about headphones (~2011 I'd say, where I've bought my first higher-end headphones), I always see this debate between objectivity (which in itself is already a massive debate as to what is actually neutral for a pair of headphones) and subjectivity ("what's the matter as long as it sounds good to me ?").

And honestly, I think I'm more confused now than ever, since I've started to read reviews of headphones I own. I have read pretty much anything and its opposite depending on when I read it and where. I fear that there is a lot of "bandwagonning" when it comes to appreciating how something feels, rather than actual correct subjective impression. It's probably especially bad now that headphones which used to be top-end (Sennheiser HD650/AKG K701/Beyerdynamic DT880) are now considered middle-of-the-road when it comes to headphones, with so much more higher-end options, and I fear that people really want to justify the added cost of their higher-end headphones without using the most probable law of diminishing returns which probably strikes especially hard past that price point of 200/250$.

stalepie's picture

"I can't possibly be right for everyone, so I try to give good advice for most."

The problem there is that the Z1R is not going to be sold to "most." It's not even available in many stores. It is a specialty item appealing to someone who wants a fancy sound, who is well off and probably already bored with some other established choices (Senn, Audeze, Grado, etc). It's comparing a Jaguar to a Civic or Camry. If someone wants Neutral from Sony without paying a grand or more they buy their pro line (7506, 7520)? They know they're not getting neutral. They're deliberately spending more to avoid accuracy, to get colorization. A flavor, a style, some distinction. To hear their old recordings again in a slightly different way. That last issue is key because people (now with today's perfect modern equipment) are hearing their audio identically over and over again, so it helps to buy something different to hear it again new, a little fresh, with some surprises and so forth.

wiinippongamer's picture

If you wanted "surprises" in your old recordings, in the form of coloration, wouldn't it be easier and cheaper to just use an equalizer? That's a laughable excuse to justify a $2000 purchase.

Besides, if you need coloration to enjoy recordings you've already listened to, you're simply not listening to the right music.

Martin.'s picture

You beat me to this. If you have all that money, just fix it with an equalizer or get something cheap. Buy 10 different headphones for 200 each and then you'll get all the coloration you need. No, not everyone's opinion means as much.

Tyll, the reason I come to innerfidelity is because of this. Good to know you'll keep going as you have.

skris88's picture

Agreed! Why switch 'phones when all you need is a tweak of the EQ??

On my iOS devices I use EQu app, great equaliser. On Android "Music Volume EQ" is a simple, non-CPU hog.

The key is, I use my HD-600s to "calibrate" my ears. Yes, my ears. At Flat, zero EQ. Then, when I switch to any one of a half dozen or so more portable/convenient 'phones I try and EQ it to match the HD-600s. That way I can travel with a cheap, throw away pair, yet not have to put up with shrill, thin, boomy or dull sound as the case my be.

afoam's picture
Quote:

If someone wants Neutral from Sony without paying a grand or more they buy their pro line (7506, 7520)?

Are you implying that the 7520 is neutral and deserves the "pro" label? Have you seen the measurements of the 7520, and have you heard it? Bloated but badly extended bass, oddly shaped presence region, spike at 10kHz. Add to that the scary(!) transient/impulse response (more than twice the mess compared to the Z1R), plus the distortion problems (which could be even worse, I admit). For pro use, such a headphone is simply out of the question. At least I don't want to listen to the mixes made with these headphones.

Pages

X