Researchers conclude, "No correlation between headphone frequency response and retail price."

Just a few days ago in my Sony MDR-Z1R review I said:

When I'd rather listen to the Audio Technica ATH-M50x we've got problems. I don't want to misslead here, it's an okay sounding headphone, but 'okay' just doesn't cut it at $2299.

(Please note this is a statement of preference not performance.)

Boy, did that cause a stir. You can check it out yourself in the thread starting about here. I'll save you the time with a few quotes:

"Tyll does not like the Z1R, prefers lowly M50. OUCH!"

"Yeah sure it has a "tuning" but come on M50's over these, I wouldn't pick the M50's over the Z7 let alone the Z1R"

"Man tyll just pooped all over these headphones lol, i do agree with him compairing to the mr speakers but that part in the end with the athm50x compairson killed me."

"I haven't heard the MDR-Z1R. BUT! Tyll's comment was that he would more preferably listen the ATH-M50Xs is hard to swallow."

It seems to me that part of the astonishment folks express is that it just doesn't make sense for a cheap headphone to sound as good as an expensive one. Okay, that's a rational way to think. How could a $100 Huffy be anywhere near as good as a $2000 Cannondale? It absolutely won't be.

Sadly, in my experience, the headphone world just isn't like that. For a start there's no accepted standardized measurement target for manufacturers to shoot for—the Harman curve is gaining traction, but it's far from a standard.

Subjective evaluation of sound quality is hard enough given biases and psychological accommodation. Headphones are a completely artificial way to hear things, and therefore, I think subjective evaluation is even harder than when evaluating speakers, which uses your OEM listening equipment in a normal way.

Further, people buy and are satisfied more with comfort and styling than they are with sound quality. It's no wonder manufacturers are all over the map with sound quality.

Well, now there's researched evidence of that. In a recent paper published by The Journal of the Acoustical Society of America titled, "No correlation between headphone frequency response and retail price," researchers found no evidence that price has any effect on frequency response performance. They used the Harman response as their baseline, but it really doesn't matter. What the paper shows is that there is no evidence showing expensive headphones narrow in on any target or common response.

Here's a couple of quotes from the paper:

This study quantifies variability of measured headphone response patterns and aims to uncover any correlations between headphone type, retail price, and frequency response.

The market for headphones and earphones has been growing steadily over the last decade, both in value and volume. Market reports indicate a world-wide aftermarket reaching over 11 × 109 units sold per year (Iyer and Jelisejeva, 2016), with growth rates of up to 55% in the 25–49 USD segment. Research suggests that factors influencing consumers' choice as to which model to purchase are mostly based on wireless functionality (Iyer and Jelisejeva, 2016) and attributes such as shape, design, and comfort (Jensen et al., 2016). Interestingly, sound quality does not seem to be a major attribute for purchase decisions.

Based on the evaluation of the mean, variance, PCA, and mean square error with respect to a target function, no correlation could be observed between the measured magnitude response and retail price of headphones. However, the variance in low-frequency response seems to decrease with increasing price, indicating an improved bass response measurement consistency across headphones in the higher price range. [Ed note: in other words, expensive cans have better quality and seal on their ear pads.]

Nevertheless, assuming that the perceived audio quality is largely determined by the spectral magnitude response of headphones, there are plenty of relatively cheap models that match the assumed target function, as well as very expensive ones that deviate significantly from an assumed ideal response.

*sigh*

I do think headphones have generally gotten better over the last five years, but I absolutely agree with the paper's findings that price has little to do with sonic performance.

I gotta get back on the hunt for good, cheap headphones I reckon.

COMMENTS
thefitz's picture

You're paying out the wazoo for leather, carbon fibre, and low batch numbers anyway. THD levels are a different story.

I mean, there's ---> ZERO! <--- quantitative evidence that cables do a damn thing... but here we are.

MRC01's picture

How about a new Wall - the Wall of Accuracy? It represents the headphones having the best measurements.
Score a few key measurements (FR, distortion, Sq wave response, impulse response) from 1 to 100. Score represents how close they are to "perfect".
Combine the scores with weights representing typical audibility, to generate the rank. FR has highest weight because if it's not right, nothing else matters.
Publish the scores & weights so readers can re-compute ranks using their own weights. Or make that an interactive feature of the page.
Price is not part of the score.

Bansaku's picture

Tell that to the electronic engineer I worked with, working out of a multi-million dollar test lab, who show'd me first hand using gear I guarantee no tech review site has access to. I made a joking comment " What difference does 26ga wire make over 28ga? " after I accidentally brought him the wrong spool of the wrong gauge and wire type, and man oh man was I schooled! If my ears weren't enough validation, seeing real numbers on a Xeon workstation's screen in real time convinced me.

thefitz's picture

I'm sure there are several scenarios where using bonkers-thin wire like 28ga over long lengths make a difference.

Headphones is not one of these scenarios. Even speakers have some people rolling their eyes, insofar as the wire impedance should be no more than 5% of the minimum speaker impedance. And we're talking dozens of feet of wire with 4-8 ohm drivers. 10 feet with 20 ohm drivers? 300 ohm drivers? 600 ohms!? Come on!

(And as an aside... how come upgraded cables always seem to "open up the mids" and cure a headphone of whatever flaw it's known for? Cables cure an HD800's 6kHz peak... make Audeze treble clearer... extends HD650 bass... all of this with more open mids of course!)

Bansaku's picture

Because of NDA I cannot disclose too much (I kid you not, even though I am no longer employed at the company). The length of wire I am talking about measured no more than 1m, usually a lot shorter, and the products in question are similar in design to a dynamic driver (minus the membrane) and their power requirements are very similar to headphones. The readings I saw in regards to both length and material (copper, silver, hybrids, and platinum) measured in the 0.XX range. How does this relate to headphones, heck if I know, but my point is material, thickness, and length (as well as insolation and temperature) do play a huge role in how well that signal is passed from the transmitter to receiver and are measurable.

I will agree, 100%, that people's reactions and expectations regarding how well an "upgrade" cable will perform is laughable. In my experience, using a better quality copper cable v.s. stock does alter the sound for the better, but the effects are

    extremely subtle
      in regards to changes in the frequency response and/or soundstage; Noticeable to my ears, but not night and day. While a good cable like 99.99% OFC or OCC will increase transparency (/clarity/smoothness), a bad cable's potential to introduce grain is very real. I tend to think this is why people are hearing what they hear. It's not that x cable increases say bass, or soundstage (because a cable is passive and simply cannot do that), no it's that the cheap cable is masking the signal and therefore cleaning up the signal path makes it appear as an enhancement/increase.

      One last blurb that's going to trigger people. Silver for use with headphone/interconnects in no way enhances anything. Copper is king, period! In real world applications the only reason silver (and platinum) is used is because of factors such as thickness vs signal transmission, and the effects of environmental conditions such as moisture and temperature. In terms of performance, copper slaughters everything. It's only weakness is the fact that it corrodes a lot easier than silver, gold, and platinum.

thefitz's picture

Tyll actually tested the FR and impulse response of several cables on an HD650 and found no appreciable change. One reading of one stock cable was out of whack... but another stock cable was almost identical to all the other cables.

There's a theory postulated that better cables yield better THD. Fascinating concept, truly. However, will one be able to detect a drop of THD when it's less than 1% with the stock cable?

Cats say that THD is stupid, and you need to look at harmonic distortion. Apparently stock cables just so happen to lower the unpleasant harmonic distortion. Fascinating theory again.

But all these are as measurable as the FR and impulse responses were when objectively proving cables do not impact those things.

MRC01's picture

What made me think of this is perusing the headphone measurements here realizing there's no easy to see which headphones have closest to ideal FR.
If such a thing existed it would be a good way to build a short list of headphones to audition, having whatever attributes one values most.

steaxauce's picture

This is pretty annoying and definitely matches my experience. Frequency response isn't all that matters, though.

When using Sonarworks calibration, the ATH-M50x, HD650 and HD800 all have very similar frequency responses as far as I can tell. In terms of overall sound quality, though, post-calibration the HD800 is much better than the HD650, which in turn is much better than the ATH-M50x. At each step you get a better sense of space and more clarity and impact.

More expensive headphones usually come with better drivers, housing and pads. If the designer doesn't completely screw the pooch, those things ought to lead to performance improvements like lower distortion, flatter wavefront, etc. My theory is that frequency response is different, though, because headphones have to shoot for a target curve that isn't flat, and better parts don't reduce the need for tuning to hit that target. That takes significant R&D, which often doesn't make sense for expensive headphones because of poor economy of scale.

Calibrating high-end headphones can lead to really good performance, though.

The Z1R is an exception in that it performs poorly all around. Sony really made me feel like an ass for preordering those.

Bern L.'s picture

I am interested to know if you sent your Z1R's in for calibration to Sonarworks.

steaxauce's picture

I haven't tried that yet, but was thinking about it. Maybe it's too soon to say whether they'll sound good post-calibration.

Bern L.'s picture

Thanks for the reply. Agree with your assessment of Sonarowrks...it saved the HD800 for me. I EQ the mid/upper bass down a bit on the ZR1....I have found them to be the most "speaker like" experience yet in a headphone.

ELPCU's picture

I believe frequency response takes very big role in sound quality, but I do not thing it is everything.

Though I wanna say that I won't buy any headphone which costs 1000 bucks or more without good FR curve. Simply not worth it.

BrooklynNick's picture

I can't seem to find any list of what headphones were tested in that study. Do you know? The STAX SR-009 looks quite a bit better than most in your measurements. I see that the top price was 5120, which is probably the Abyss.

Type35's picture

Ultrasone anyone? Anyone?
And let's not even talk price vs manufacturing cost.
The margins on headphones would make an investment banker blush with envy.

Jazz Casual's picture

The research findings come as no surprise to me whatsoever and nor did the responses from Head-Fi'ers to your MDR-Z1R review.

--------------'s picture

Harman made a similar study with identical results. They only tested IEMs though.

http://seanolive.blogspot.com/2017/02/twirt-337-predicting-headphone-sou...

inks00's picture

The MDR-Z1R rings like a bell at 3KHz and 10KHz. Just look at the impulse response and square wave measurements.

The magnitude of the problem is can be seen on CSD plots... see SBAF. http://superbestaudiofriends.org/index.php?threads/sony-mdr-z1r-measurem...

Expensive headphones that misbehave with significant ringing and resonances should be named and shamed. If you like the literally extra-musical 3KHz sass and 10KHz tizz shooting arrows directly into your skull, then the MDR-Z1R is for you.

thefitz's picture

It's nice to actually see a link to SBAF on a thread without moderators crying ;-)

Johan B's picture

Lets get back to analysis of the drivers. Fast responding, fast dampening and neutral drivers are key to good performance. Then after that the earpads design and "can" design are second order effects. If I may draw the analogy to speaker drivers where frequency response, distortion, resonance frequency, impuls response and in particular the waterfall plots are excellent means to see quality. Have you ever heard high quality speakers with a bad waterfall plot? What measurements are we missing to distil good from bad drivers?

thefitz's picture

Doesn't it cost $250 to replace the drivers on Audeze's ~$2k headphones? People, you're paying for the bling.

And THD readings.

MRC01's picture

Actually Audeze quoted me $400 a couple of months ago. Yet even though this is still relatively cheap compared to the LCD-4, it's the same price for all models and almost half the price of the LCD-2. Even if it were only $100, this is not necessarily a good analogy. For repairs or upgrades to existing customers they are probably billing marginal cost of the next unit produced which doesn't cover R&D and other costs. Most companies recover their full cost and become profitable from new sales, not upgrades & repairs.

The high prices of rare models (whether headphones or anything else) are due in large part to low production numbers. If the price difference between LCD-2 and LCD-4 (or Focal Elear vs. Utopia) were purely the mechanical/electrical differences, they would be much closer in price. But they sell a lot fewer LCD-4s and Utopias. So the R&D and other design work unique to those models has to be amortized over far fewer units.

That's how cheaper "flagship" models give you something like 95% of the objectively measurable accuracy for less than half the price. Plus over time they trickle some of the R&D of the TOTL down to these cheaper models. You mention THD... incidentally, THD of the LCD-4 is admirably low, but not any lower than the LCD-2.

thefitz's picture

I should have definitely elaborated more - you do hit a "saturation point" with THD, but that's a major distinction when claiming price has no impact on FR.

My point is - if you produced 25,000,000 units of an LCD-4 drivers with pleather and aluminum instead of carbon fiber on an assembly line, you'd end up with something much, much cheaper what would sound identical to the $4k LCD-4... assuming you can make two $4k LCD-4s sound identical :D

cspirou's picture

I learned this lesson recently with IEMs. I've been using the re400s for awhile and I enjoy them quite a bit. I was looking for an upgrade and I found the nuforce primo 8 for $250. I thought it was a good buy because it's reviewed on a lot of 'important' sites and it had an original MSRP of $500. Immeadiately I noticed that it had good resolution but the highs were a bit harsh. This went away at lower volumes, but paying 3x the price for this sort of limitation is inexcusable. Comparing directly with the re400 had me enjoying the re400 far more.

While there are certainly $250 IEMs that are worth it, not all are necessarily better.

So far the Innerfidelity Wall of Fame is spot on. While the list isn't definitive, I do think it's a safe list. The re-400 and the HD600s I bought are great and fairly priced.

Three Toes of Fury's picture

This subject is near and dear to my heart...Cost VS Sound Quality.
Its one that i wrestle with all the time...my primary questions are: 1) Couldnt a manufacturer create low/mid cost (lets say $100-$200) headphones that exhibit sound quality of the uber expensive models?? 2) I believe they easily could but dont due to product differentiation. 3) There are, indeed, "diamonds in the rough" out there which are lower cost headphones that give above value sound...thats my favorite thing to find on this site.

Im not negating costs associated with specific materials or process', however when it comes to headphones, lets be real, there are VERY few components. Sure there's bound to be production challenges in making the fancier pairs, but imagine if a manufacturer came out with a pair universally embraced by the community as the high-price-killer? They'd get gobbled up! I suppose the issue here is missing piece from all reviews...individual subjectivity and preference...there is no universal standard for acceptance. Waverforms are one piece of the puzzle.

Anyway...enough of my babbling...Tyll, one thing i think about and appreciate often is the photo of you and your wall of fame. I know, sitting behind you, are countless pairs of >$1k headphones. And yet you regularly provide good, critical, opinions and reviews of great sounding, lower cost cans. I certainly appreciate reading about your experiences with higher cost items but love that you still look for those diamonds in the rough....sooo..in your quest for "good, cheap headphones"...happy hunting!!!

Peace .n. Living in Stereo

Three Toes of Fury

GNagus's picture

Researchers conclude, "No correlation between headphone frequency response and retail price."

Now you should review the new Grado PS2000e and prove these researchers wrong!

Argyris's picture

I would argue that, in the absence of an agreed upon industry standard target (which will likely never happen), it's nonetheless possible to come up with some basic criteria for whether or not a headphone has good sound quality. If any of these is true...

1) It has too much energy around 5-6 kHz, which is where most people seem to be sensitive--sharp peaks are especially bad
2) It has a wonky, lumpy response contour, the importance of which goes without saying
3) It has excessive ringing anywhere in its response, which manifests itself as fatigue and/or that "crinkling cellophane" sound that cannot be EQ'ed away
4) It has multiple large (>10 dB) deviations in amplitude, particularly if they result in suck outs or narrow peaks

...then, regardless of how much it costs, it does not have good sound quality. This allows for personal preference, since well-executed examples of the various popular sound contours (e.g. neutral, v-shaped, mid-centric) still shouldn't suffer from these defects. It also entirely dispenses with the commonly used excuse that expensive headphones make up for often glaring response shortcomings by being somehow inherently more resolving. If the response is uneven, edgy, peaky and/or ringy, and if the listening experience is therefore unpleasant or downright painful, I don't care about your supposed resolution gains (a claim I'm skeptical of in general). Such a headphone is useless to me at any price, and especially at the ever increasing prices we're seeing for flagships these days.

When the HD 6x0 can somehow manage to avoid all four major pitfalls listed above and can be had new for ~$250-$300 if you're patient and know where to look, there's absolutely no excuse for headphones that cost ten or even twenty times as much to exhibit any of these flaws. If nothing else, this study shows we need to start being brutally honest with manufacturers, and, more importantly, ourselves, if we want things to change.

pete111's picture

What this says to me more than anything is that measurements, particularly frequency response measurements, don't tell the whole story. I personally don't believe in paying that kind of money for headphones and there are really really great sounding headphones for 500$ or less, which to me is a sweet spot. But what the study DOES say is that you can't expect a good bass reproduction in very cheap headphones and that does match my experience. Frequency response is just that. It does not say about detail retrieval, dynamics, Imaging... For me if there is one parameter that is a matter of taste is the frequency response and I admit that I do enjoy solid basses and no, I have not heard a 200$ pair that cut it. My heaphones are HE-400i and TH-X00, which I both love. I have to say that to me those Z1R listened at trade shows really got me immediately as immensely enjoyable and involving. They actually did it for me more than the Utopias in fact, I kept going back to them...So...taste... I would rather listen to M-50Xs than any Grados but that doesn't mean anything, I find the Grado house sound tiring that's all I can still see why GS-1000s are great and would never doubt what's the best headphone. Just not for me that's all. I would like to find objective researches that could quantify what makes the great headphones so special. It seams that even today measuring equipment has it's limit. I'm saying this literally, I'm not the snake oil kind of customer, but what if our ears and brains where more accurate than measurment mics? What makes the He-1 so special for example? To me it goes beyond what can be seen on a graph.

brause's picture

I have made a similar experience with earphones, personally and based on observations.

Example 1: The Soundmagic E10Cs ($60 CDN) have been given consistently 5 stars by Whathifi for a few years now. The KZ HDS3 ($6 CDN) sound better at a tenth of the price - which I can confirm.
http://headflux.de/knowledge-zenith-kz-hds3/
These KZs, when tested elsewhere, fare consistently as mediocre compared to other Chinese cheapos in the $10 to $20 range. What does this tell us other than indicating a severe inconsistency? In my own experience, the Soundmagics are vastly overpriced for what they offer. They are mediocre value at best.

Example 2: The Sennheiser Momentum ($129 CDN) in ears have been reviewed as being as good as earphones twice as expensive. I am battling whether some sub-$20 Chinese earphones sound better, such as the VJJB KS4, the DZAT DF-10, and the Einsear T2. The VJJBs are much better built than the Senns. And if these Chinese earphones don't sound better, they come awfully close and may be even more fun. The enormous price difference is definitely not justified.

This raises the question how intuitive the pricing of the "regular priced" earphones and headphones generally is.

Sil's picture

It's very unfortunate, since many of us regularly fall for it, but price is rarely a good measure of worth or quality.

I remember when I lived in Switzerland, there was a company specialized in blind tests.

Whatever the kind, products with low prices were consistently judged better than the higher priced one in blind tests with real people (no professional testers).

Quite amazing for olive oil, where prices can show a differential of 10x, and beauty products where price differential is more like 100x or 1000x.

Unfortunately, no headphones review when I was there ;-)

skris88's picture

Had to laugh reading this. I've just these past few months been through this exact same scenario!

After getting increasingly frustrated with the loose bass of my Sennheiser HD-600 and HD-650s, I purchased in desperation a $50 pair of Superlux HD668Bs that Tyll had commented on to have "scissor-like" treble. Perhaps it's my aged ears but that treble sounded fine to me, what needed was a cut to its bass. A quick -2dB at 65Hz and a -4dB at 32Hz later, this now is my Go To headphones.

The bass punch is tight, midrange neither too forward or laid back, slightly recessed treble (yet not as 'dark' as my HD-6X0s), and that extra sheen that seemed list at the very top end all these years.

The best headphones ever? I'm sure they're not.

My best headphones ever in a 50 years journey of great audio? Yes!

Pages