Mastering Your Domains: The Difference Between Objective Measurements and Subjective Listening

Ed Note: This post is inspired by Michael Lavorgna's recent AudioStream article "The Trouble with Listening", and comments he received.

The creepy thing about the photo above is that you have a human that is part robot, and a robot that has human-like intelligence. We're pretty far down the road with computers and have some pretty good stuff these days, but from what I understand we're still pretty far away from putting anything remotely like human consciousness into a machine.

Why? Because the human consciousness is not in the domain of the objective stuff like computing.

Human consciousness experiences sensory input, it doesn't measure it. A major chord is not understood by our consciousness as a series of three particular harmonically related notes, rather it's heard as a whole and as having a cheerful character relative to a minor chord. People can fully enjoy the experience of music without having any objective idea whatsoever about music theory and harmonic relationships.

Now I'll certainly admit that computers can do a decent job of composing music, and many would be hard pressed to blind test the difference between computer and human generated music. And computers can listen to music and tell you all sorts of things about the structure and qualities of the signal, but a computer is not going to adopt a preference for Gershwin over Shostakovich, or tap it's foot when it feels good about what it's hearing—though you might be able to program it to look that way. A computer isn't subject to experience.

The Objective and The Subjective are simply two completely different domains, and while there are relationships between the two, one is wholly unlike the other. While measurements of audio gear are terrifically meaningful, the actual relationship between measured performance and listening experience is quite various in terms of the measurements predicting the nature of how we will experience a particular piece of gear. With things like headphones and speakers, where frequency response deviations from linear are strong, the measurements will relate more strongly to listening experience. If the measurements say there will be strong, tight bass in a particular pair of headphones, most folks will pretty reliably hear it that way. With things like headphone amps and DACs however, that are more linear and have performance measures many orders of magnitude better than speakers and headphones, measurements don't seem anywhere near as reliable an predictor of subjective experience.

Distortion, or Character?
Let me give you an example: NwAvGuy's O2 amplifier measures extraordinarily well. Do some basic listening to this amplifier and most folks will come away impressed with it's very competent performance. But when I spent a lot of time with it, I found myself feeling it was a bit of a dry and uninvolving. Switching over to, say, the Apex Butte amp designed by Pete Millett, I find the sound more musical and bouncy—it made my feet tap more reliably. The Butte, while not having as good a set of objective measurements, was clearly more pleasing to my ears. What measures as greater levels of distortion can be experienced as having a more pleasing character of sound.

Herein lies the problem: when we compare measurements, we might report amplifier A has less distortion than amplifier B, and make the assumption that amplifier A must sound better than amplifier B, but it's simply not the case. The added distortion may cause the amplifier to sound worse, but it also may cause the amplifier to sound better, and you can't necessarily tell by the measurements. I'll hasten to add that because we're in the subjective domain here, "better" and "worse", in terms of sonic character, is only meaningful to a single listener. While statistical norms may be found and listener preferences will likely cluster, the fact remains that determination of amplifier character and listener preference is done person-by-person and each person's experience is absolutely valid as their experience, regardless of measured performance.


So Why Measure?
While the relationship between objective measurements and subjective experience might be tenuous, it does exist. Subjective experiences are notoriously fickle, and I personally find that having measurements available as I review a product acts as a stable reference point as I listen to and evaluate a product. If I see big bass in the measurements and don't hear it, I might find myself taking some special care to ensure the pads are sealing correctly on my head, and will take some extra time ensuring that my listening impression isn't somehow influenced by the music selected, or a stuffy nose, or a bad night's sleep.

Measurements are also helpful as it's very hard for most people to get their ears on all the headphones available, so having a database of measurements allows potential buyers to do some preliminary investigation by looking for headphones that measure similarly to those they know are to their taste. This, of course, takes some extra effort on the part of the buyer in order to understand the measurements and their relationship to the listening experience. In the end, however, it's the actual listening experience for that particular person that counts.

For electronics like headphone amps and DACs the equation is a bit different. Because the relationship between the measurements and listening experience is somewhat more tenuous with these more linear devices, measurements have a somewhat different value. The greatest value in amplifier measurements I've performed is their ability to have a small window into the technical competency of the product's design. If measurements show significantly excessive noise, hum, distortion, or crosstalk for example, it causes me to think that particular design is likely flawed, and the designer is likely not very proficient. If I see an amp with extraordinarily good performance in those areas, I begin to feel confidence that the product and designer might both be of high quality. But most products lie somewhere between those two extremes, and relatively little is learned by objective testing.

There are pragmatic problems with implementing a measurement program as well. The equipment needed to make reliable and accurate measurements are painfully expensive and technically challenging to set-up and execute. I'm a pretty decent technician, but it's taking me far longer than I had hoped to get my headphone amplifier measurement program up and running. Objective testing is a distinctly different expertise than evaluative subjective listening—which is, to some extent, the whole point of this post: Listening and measuring are two entirely different things.


The Map is Not The Territory
I was inspired to make this post due to Michael Lavorgna's recent post on AudioStream, "The Trouble with Listening", and critical comments thereafter intimating that subjective reviews without objective measurements are somehow incomplete or flawed. I would argue that measurements are a valid mapping of performance, but that map is not the same type of thing as the territory (or domain) of subjective experience.

When I plan a motorcycle ride through the forest, I will consult numerous maps: topographical maps will tell me about the steepness of trails; road use maps will tell me where I can and can't ride; land management maps will indicate where I may be legally allowed to camp. But the best source of information is dialog with another rider who may have been in the territory and can deliver a scouting report. Stuff like where mosquitos might be, and where there's a scenic camping spot may be brought to light. The maps just aren't going to tell you about the experiences available in the territory.

Subjective reviews are like scouting reports. They tell you about the territory of the actual listening experience with a particular product. Objective measurements, while informative and related, are relatively mute when it comes to describing that. When Michael Lavorgna says in his piece...

I would start by asking a question—Is listening sufficient to determine the worth of an audio component? Or to put it another way—Is listening sufficient in determining what we enjoy listening to? Well if we put it that way, of course it is.

...I think he's right.

Sure, I'd prefer it if Michael had a killer measurement program up and running—when I go on a moto trip I take as many maps as I can—but, for a variety of reasons, that's just not the way it is at the moment. Does the fact that he doesn't have measurement program somehow lessen the value of his subjective review offerings? Hell no! The subjective review is at least in the same domain as the listening experience I'd have with the same gear.

Dump a fat stack of measurements on the table and I'd surely give them considerable attention, but when I was done I'd still have a burning desire for a scouting report. That's why for Michael, and me, and John Atkinson, and all the writers in the Home Tech group, we feel the single most important thing we can do is bring you our subjective listening impressions. Measurements are groovy, but it's the stuff that happens in the domain of the subjective listening experience that really counts.

thelostMIDrange's picture

Korzybski's map not representing the territory was addressing how a human being accesses Reality, or knowledge of it. If you read his works you will find that he ends up saying the same thing heidegger does, and unfortunately nietsche as well; your ability to access Reality with a capital R is a direct result of how human you are. and how human you are depends on how much Truth you let into your life. Your map will only be as accurate as your ability to let reality in. And so that pic at the top of the page of a machine with a pseudo human consciousness and a human with a pseudo machine consciousness will both have limited knowledge of reality. Human, all too human as Nietsche wrote, it is rather these folks have their finger on the pulse. their map most correctly corresponds to the territory. And in that sense, the author of this article shows a deep and healthy understanding of what it means to be a human listener and I applaud this understanding. He is riding that knife edge of objective/subjective and realizes that they are different species, not differing points on the same continuum. Our world is riding that same knife edge, and tetering towards falling off the objective side. Humans who have absolutely and objectively failed at becoming human have assumed power in all the highest positions and so we have leaking nuclear reactors all over, a chemtrail filled sky, soil that is unsuitable for growing, total lack of empathy, morals, compassion, understanding, peace. All the things that a human has that a machine does not and cannot be programmed to have much in the same way it cannot be programmed to know when and how to tap it's foot or bop it's head in a human way. And yes, distortion is part of music and therefore part of a great sound system. The author again understands this. it's a matter of where it's placed, how much and what kind of distortion that helps music be musical, not a total lack of it. anyone who plays an instrument that is not digital knows that distortion is color. The kind of bow on a violin, the kind of wood for a guitar or drum kit, the distortion from an amplifier. A little color goes a long way but it's a spice that makes music more pleasing to a human nervous system. If we as a culture continue trying to make ourselves into computers, then yes, in the future we will not feel distortion as color to be pleasing. You would need a really advanced computer to know enough to build into itself the desire for distortion and color and it's precisely this lack of enjoyment for messy things like color that it would not want to program into itself. Whomever programmed nature on the other hand, was wise enough to figure it into the equation. it's only folks who feel there is no supreme energy that feel it's a bane and feel they can assume the role of creator. Delusional thinking I say. But No worries these technocrats and google heads say, we'll just merge with the computer and leave this messy flesh behind and monsanto and it's franken seeds with the pesticides baked into it will be a stopgap until they are able to become one objective creature living forever in the digital domain. Yeah right. Kurzweil is a delusional nutjob that would be committed if we we'ren't iving in an upside society. The average joe has had a serious mind meld done onto them through false history and schooling that he is so far from being a healthy Truth knowing being that he is easily led down any path set before him. I say this to suggest, that headphones are no different. The objective measurements represent the machines' doing what they are good at and for that they are useful if in the hands of a human with a knowledge of reality and how he differs from the machine. In the hands of one of the two 'things' in the top pic however, they are pretty meaningless and misleading sets of data. As the author states, it's only direct experience or at least second hand experience told from another human that is of value and that objective data needs to be understood in light of that experience. Riding that knife edge is precisely this ability to use the data and not let it use us. As a culture we are letting it use us. The machines are taking over. They are not serving us by giving us more leisure time. And in the end it is liesure time that is the most precious of commodities. Time to enjoy music with headphones or staring out into the sky from under your favorite oak tree or from behind the visor of your moto helmet. Headphone measurements that can help one pick the right headphone are therefore supremely useful if they save someone and his/her time from having to listen to dozens upon dozens of headpphones. If we don't quickly come to this same understanding in the larger culture, the biosphere will no longer be habitable for subjective fleshy beings to enjoy music through headphones, or anything else. Each one of us needs to get his map closer to representing the territory and that territory is nature, which includes us. Say no to the technocrats and withdraw your participation from it's machinations wherever you feel they inpinge upon your god given liberties.

Tyll Hertsens's picture

"Each one of us needs to get his map closer to representing the territory and that territory is nature, which includes us."


Heck of a rant there...thanks for posting. 

Seth195208's picture

...ability to observe and reflect, which, in a lot of ways, excludes us from natural world. this is the gist of the problem. 

Hal Espen's picture

I embrace the God-given liberty to use paragraph breaks, non-bold fonts, and the standard capitalization of proper nouns. 

But I'm grateful that you resisted utilizing the Cap Lock key for that extra soupçon of emphatic assertion.

paul's picture

Serif Fonts ... 

Azteca X's picture

Excellent post, Tyll.  Thanks for sharing.  I was in the comments over there suggesting some measurements and think Michael is a great reviewer and I've read just about every post he's made in the last 18 months.  I value his voice as a writer and his approach to music, life, art and the enjoyment of all three.  The measurements would be a great resource, though; he gets a pretty staggering array of great TOTL DACs in there along with all the tantalizing budget options.  The Auralic Vega seems to be a true contender, for example - I'd love to see some measurements to let me judge if it's competently designed in addition to pushing all of Michael's listening buttons.

And while there are those who can splurge on all the latest TOTL stuff, there are even fewer who can measure and measure well, particularly with a Clio or AP device.

Dan S's picture

Tyll, one thing I really appreciate about your reviews is that you cover both the subjective and the objective. For headphones, I think you've struck the perfect balance. I've really learned a lot from reading your articles and looking at your measurements.

This reminds me of the nature vs. nurture debate. about both? Why does one have to win out over the other? They are both relevant.

A huge reason to provide the objective along with the subjective is that it gives you legitimacy and a grounding in something reliable and reproducible. I feel like you are providing a complete picture, and the objective part of your review can be examined and challenged, if necessary. I trust you a whole lot more than if I was just getting subjective impressions.

I also like that you are open with your biases. For example, you prefer headphones that are a little warm sounding. So do I, but if I didn't, I could take that into account when reading your subjective impressions and adjust for that bias.

robm321's picture

Really enjoyed it. I'm glad to know that audiophiles won't be replaced by computers. 

SonicSavourIF's picture

First of all, I think you will hardly find any objectivists that claim, that it is impossible to prefere the sound of gear that measures poor in favour of better measuring gear. Even guys like NwAvGuy and Ethan Winer agree, that you can like the sound of vinyl, tube amps, analog tape, and that is perfectly fine. As long as you stick to subjective claims, when reviewing subjectively everything is fine, because you are talking about subjective opinion. However if one claims that a poorly measuring amp (or a super expensive cable or ...) is higher fidelity/higher quality, then you are in the realm of measurability and make falsifiable claims. Likewise if one claims that gear L gives 'better sound' but is not able to determin the difference in a proper blind test, than the claim is highly questionable and very likely du to placebo and expectation bias effects.

It is funny that today I went through the measurements of the LCD-2/LCD-3, HD800, Amperior and DT1350 headphones and thought that I actually cannot derive from these which can sounds best. (e.g. looking at the measurements alone, the bass of the LCD-2 seems better than that of the LCD-3, and the rest of the frequency response is quite similar. Yet, people prefere the sound of the LCD-3. Likewise, the frequency response graph of the HD800 looks quite bumpy by comparison). So at least for headphones, especially because it is a huge matter of taste, I can understand, why you have to listen to them before you can know if you like a pair or not.

With electronics however, I think it is exactly the other way round (Hate to disagree with you Tyll :-/). The smaller the differences between gear are, the more likely we get prone to expectation bias.  It is true that in the end listening matters and above all personal enjoyment. But if that is the case, to really determin by listening which gear one likes best, you would have to do a proper blind test. The moment you read a review about an UberAmp XY, know that master Yoda designed it and the thing has a 5k price tag (or likewise measures exelently =)) your brain interprets what you hear in one way or the other. You get influenced by others, by your mood, by marketing and so on. I have experienced utterly different listening experiences with my same gear ranging from something like experiencing dull and boring sound, making me want to immideatelly buy new, 'better' stuff, to tears of joy. That leaves me personally with the question how reliable my hearing is in the first place when evaluating gear.

On the other hand, after all I have read on our hearing/listening, flaws of sighted testing and objective measurements, I find that actually  this knowledge substantially improved my listening experience, as I tend to much less worry about the sound I get and in the same time beeing releaved that I don't have to spend astronomical amounts of money to get good sound. So for me as far as electronic gear is concerned, measurements help a lot and are more helpfull to base a buying decission on than subjective reviews.  

As it is so difficult to properly measure gear, I hope to see more amp measurement on this site as guide we can rely on when searching for new stuff.

Sorry for the long post, cheers

pmillett's picture

Tyll, that's a pretty good discussion of measuring vs. listening.

People really need to understand a couple of fundamental facts:

1. If you like it, it's good.  Period.

2. Measurements probably won't tell you if you will like it.  You need to listen to it.

As you know, I'm an engineer.  And I make lots of measurements.  I believe I can almost predict how people will describe the sound of an amplifier by interpreting the measurements I make.  But I absolutely cannot say "this will sound good" based on measurements.  I can usually make some predictions, like "this will sound pretty bright" or maybe even "I bet this will sound awesome on female vocals with HD-800's".  But "good" is in the ear of the listener, not me.

Sometimes, I can say "this will sound like cr@p", though.

Single-parameter measurements - especially THD - really mean nothing.  I think we've all heard stuff that sounds really good to us that has pretty high THD (like 1%)... and stuff that sounded terrible that had 0.05% THD.  The perceived sound is so much more complex than a single measurement.

That said, if somebody tried to sell me a power amp that had 10% THD at 1 watt, I wouldn't even consider it, because I know it will sound bad.  There are limits, of course!

More involved measurements like FFTs can shed a bit more light on the sound.  But no measurements can really model how you percieve sound.

Unfortunately, our business is full of snake-oil salesmen who fully believe their own lies, so you have to view all measurements - and claims that "this sounds better than that" -  with skepticism.

Except for what I say, of course...  ;)

Tyll Hertsens's picture

Glad you liked it, and thanks for the response, Pete. 

You're 100% correct...of course.

Limp's picture

 Does the fact that he doesn't have measurement program somehow lessen the value of his subjective review offerings? Hell no!

Hell yes! 
Subjective reviews are like scouting reports, and sometimes scouting reports bring back tales of mermaids, sea serpents, kraken, komodo dragons and the Blemmyes.

Without measurements we have no possible way to know wether the tales Michael Lavorgna spins are like these, pure myth, or not.

pmillett's picture

"Subjective reviews are like scouting reports, and sometimes scouting reports bring back tales of mermaids, sea serpents, kraken, komodo dragons and the Blemmyes."

Um, Komodo Dragons are real...

Limp's picture

But are they dragons? Do they spew fire? Do they have a mid-range so liquid it's worth dying for?

pmillett's picture

I think we can devise measurements to factually answer the first two questions.  As for the third, it's in the brain of the listener...

So yes, we do need measurements!  But they still won't tell you if the dragon is pretty, or ugly.

Limp's picture

I concur fully.

The best result is attained with the two disciplines working in tandem, which is why I took some exception to Tylls somewhat surprising statement.

thelostMIDrange's picture

or an unfortunately necessary device in between the music and your ear?

To think of it as a musical instrument is an admission of defeat for hi-fi'ers but potentially more fruitful when it comes to designing such gear. It's a game changer to think of it as an instrument with all the attributes we normally associate with instruments. No two are ever the same, they all have color etc. Then it becomes about fine tuning the musicality of it around that neutral point we call hifi instead of searching for some ultimate end point of fidelity which apparently doesn't exist except in our ideal imagination and as elvis costello sang, imagination is a powerful deceiver.



the etymology of instrument is basically, a tool. Heidegger suggests that we only really notice tools when they break down. when they are working they are invisible. Hifi aims at invisibility but it seems impossible as is evident by measurments and whenever they get close to perfect in measurement they seem to be less and less enjoyable, which can be seen as a sign that there was a missed assumption somewhere along the line and the end goal we set up for ourselves an illusion. But there does seem to be a point at which things get pretty darn close to Hifi and then we whiz past it and go too far. A human feature to go too far and take things past their balancing point. When the rubber meets the road, reality is 'what works' and what sounds good. In that way, the original article suggests it's practical to find what sounds best then look at the measurements and find out why instead of the other way 'round.


but one thing is for sure, whenever I hear the tool of a headphone or amp impart itself all over my record collection then I know it is deficient somewhere. When it disappears, then I know it's 'working' and good enough for me. As long as I can listen without fatigue I know i'm near that illusive balancing point of Fidelity. Whether some attribute could be a bit tighter or clearer is the lure that pushes designers to go too far in trying to reach for that little extra and compromise basic fidelity in the process after which the gear again becomes broken and 'hearable' over the top of the music.

DrForBin's picture


a bit about my situation: 1. i am over 60, 2. i rode a motorbike daily without using any real hearing protection for quite a few years and 3. i recently had my hearing tested by an audiologist who noted that i had "injury loss" which accounts for why i don't always understand my wife and daughter, its not that i'm not listening, i really can't hear them. i am down about 50 dB at the frequency of most female speaking voices.

the objective/subjective divide has always troubled me as both sides seem to take the position that their's is the absolute truth. both you and Mr. Lavorgna's articles are presenting a way out of this seemingly irresolvable debate and i would hope that both sides of the fence will take it to heart and stop the ill-will that comes from taking an absolutist position on anything that should be giving us pleasure.

having said that i will enjoy listening to my AKG K550 while i fall asleep, comfortable in the fact that i do not disturb my bedmate, that all the music i like sounds pretty darn good, and that if the tune is right and my mood is right, the experience can bring tears to my eyes. (e.g. SRV "Lenny.")

Jazz Casual's picture

I was blissfully ignorant of headphone measurements when I became involved in this hobby. I soon found that they were often cited at headphone forums in support of the excellent performance of some headphones and to "expose" the poor performance of others.

I set about trying to understand them and became quite immersed in the process. I would compare the measurements of different headphones, including the ones I owned, and compare the objective data with my own subjective listening experience.

It was always an interesting exercise but ultimately quite disconnected from my response to the headphones that I listened to, and my enjoyment of them or lack thereof.

"From what I've read" measuring and listening are integral to designing and building audio components. Whether it's more art than science depends on which designer or design group you talk to I suppose, but the relationship between the two is intertwined rather than mutually exclusive. However, if you're intending to build an audio component, you'll need more than just a love of music and a decent pair of ears. An understanding of circuit diagrams and oscilloscopes would seem necessary at the very least.

However, being a music listener rather than an audio designer, I don't need to understand a design schematic in order to appreciate the sound that the component is producing; just as a car driver doesn't need to know its performance measurements (and no, I'm not referring to the speedometer) or how it works on a technical level, in order to enjoy driving it.

The same can be said for headphone measurements. Of course it can be interesting to learn more about them, and for some audiophiles, this technical knowledge can be a source of fascination and obsession. However, it is not essential for the appreciation of music reproduction from a headphone or a hi fi system - it is an adjunct to it.

Developing an understanding of frequency response measurements has enabled me to better understand what I'm hearing, but it has not enhanced my enjoyment of it - nor should it. It has not given me an uncanny ability to predict how I will respond to a headphone that I am yet to hear or whether I am going to like it or not. Only the experience of actually listening to the headphone has provided me with that certainty. I'm often surprised how much I like a headphone despite how it measures, and that's because the measurements only tell part of the story. The graphs do not show timbre, imaging, soundstage and various other intangibles that we hear and respond to. That's been my experience anyway, and it's why I seldom think to look at headphone measurements these days.

Audio designers/manufacturers must objectively measure the performance of their products, but it's entirely optional for the rest of us, the consumers and hobbyists - unless we harbour secret ambitions to become audio designers/manufacturers too.  



Willakan's picture

I’ve seen this angle, implied with various degrees of obviousness, several times before, and I still find it, if I’m honest, not very compelling. At least, I think it’s the same angle, as much of the logic of your article functions by implication: your link to Mr. Lavorgna’s piece gives me some idea, however, of where to start.

Rather than immediately leap into rehearsing the relevant aspects of the usual “belief systems”, I’ll start with your opening paragraph under domains (“Human consciousness experiences sensory input, it doesn't measure it.”). The distinction here is presumably between the personal/quasi-transcendent experience of listening to music (which is somewhat irreducible) and the waveform/notes or whatever you want to call the physical “thing” we assume to exist; this thing we can test and fiddle with and measure until the cows come home.

Would it be fair to sum up the logical core of your argument, working from here and from your quote from Mr. Lavorgna on the importance of enjoyment, that you view the enjoyment derived from a piece of equipment, in the environment it is used in (IE: listening to music for pleasure rather than engaged in an ABX test or similar), as something qualitatively different from that which is characterised by test protocols, and that this is the ultimate arbiter of the value of a piece of equipment?

Tyll Hertsens's picture

The thing your being is being exposed to can be characterized in part by measurement equipment. The experience is something you have internally that is catalyzed by exposure to the signal. 

Willakan's picture

OK, so assuming I've got this system right, I have some problems with it :D

The first, and immediately striking issue is that, followed properly, it reduces all the most important claims in audio (in terms of epistemological priority) to a state where they are inherently unfalsifiable. Unfalfisfiable claims are inherently inimical to logic; ergo, the domain of audio in its purest sense transcends logic (putting it politely!), and therefore all opinions on the matter are inherently irrational.

Now, I don't think I'm being overly presumptuous in anticipating a number of objections to this, the most obvious being to draw attention to the fact that you do hold opinions on audio that you didn't make up because you felt like it. At the basis of those opinions lies some kind of correlation: your claims about product X are not unfalsifiable, one might venture, because other people can go and listen for themselves. Similarly, when you personally develop the belief that aspect Z of a given product will tend to have effect Y on the sound, you can go and listen to other products with that aspect and see if they sound different. Please say if I'm horribly misrepresenting you here.

The catch, however, is that strictly speaking, none of these things can render your audio beliefs falsifiable. If someone else seems to hear something, how can that possibly relate to whether you hear it? They are not you, isolated in exactly the same spatio-temporal position - their claim has no logical bearing upon yours. Similarly, if we listen to products with similar attributes and think they sound similar, how can we ever isolate that in the incredible flux of personal experience: you are not hearing the product with Aspect Z being the only variable modified. You are not the same person, strictly speaking, as the person who heard the other product earlier.

Again, I think most would be quick to propose a way (and the only way) out of this: if enough people hear X or Y, or if I personally hear X or Y sufficiently consistently, we can start to talk about the 'thing' that is heard...oh dear! I do believe I just proposed that we need a 'thing' for our discussions about audio to have any real meaning whatsoever. And this troublesome thing falls directly into that troublesome category of 'objective' objects in any meaningful sense of the word (no such thing as an 'objective' claim, but there are certainly things that we must take as being virtually 'objective' in order to, well, think!). And that means the ball lands firmly in the court of the 'objectivist', who can then start discussing evidence, and Occam's Razor, and how immensely plausible it is that A is the case instead of B...

So, to summarise, to say anything about audio at all one cannot retreat into what I've previously characterised as "absolute subjectivism". The moment you poke your head out of that position, even by an inch, you move into the realm where where suddenly ABX testing and the like becomes manifestly relevant. I think I've laid this out pretty clearly, but just to tidy up I'd like you to consider some of the implications of "absolute subjectivism", if we imagine that someone somehow manages to think solely within that sphere.

For a start, from the perspective of the "absolute subjectivist" (and you can't claim to logically occupy a middle ground: as laid out above, you're either wholly 'subjectivist' or an 'objectivist' that just distorts the balance of evidence), there is no inherent difference between "real" and "placebo-y" differences. Therefore, I hypothetically consider setting up "Mr. Willakan's Amp Modification Service". I will take your amplifier and your money, hold it for a while whilst promising vague improvements in sound via technically factual statements ("Some customers have reported vast improvements in treble clarity..."), and then send it back to you with absolutely no modification made to the electrics whatsoever. Everything I know about psychoacoustics suggests that people would tend to report improvements post-'modification'. You see, strictly speaking I'm selling, like all hi-fi manufacturers must be from an "Ab.Sub" perspective, an object from which you derive emotional satisfaction, rather than something based on things like electrons and semiconductors. Thanks to human psychology, creating a process whereby the listener anticipates a difference is likely to produce such a difference: so I don't think I'm doing anything wrong. I'm selling the effect of believing your hi-fi is better, and as every "Ab.Sub" knows, hearing is believing/believing is hearing! If I offer a 100% money back promise if you don't hear a difference, morally I'm surely whiter than white? And don't start complaining about the price: as I so often read in the comments of audio cable reviews and the like, components must be evaluated based on what the listeners hears, not silly things like component costs. Right?

Now, obviously, this is a contrived example, but it does serve nicely to illustrate the problems of the belief system in a solid and faintly amusing fashion. Hell, from my perspective, something very much like what I describe above is already happening (although if it weren't the example would still stand). You don't even need to look at the more controversial products: for example, there is a certain DAC which Stereophile, TAS and lots of other audio mags gave rave reviews to. The designer has a HydrogenAudio account, in which he reveals that as far as he's concerned, his DAC is totally transparent under any semi-sane use case, and such transparency has been possible in both analog and digital domains, as far as circuits are concerned, for years.

Yet, the company has recently released a sequel to their DAC, presumably designed by the same guy. Reading the publicity material, one gets the impression that this new DAC is meant to sound better than the old one, closer to the recording, more seems highly improbable the designer believes this to be the case, but then he doesn't write the PR spiel. One could argue that he's selling to the "Ab.Sub" the anticipation of an improvement...

Of course, from where I'm standing, one can make reference to the far less compelling nature of "anticipated" improvements, as shown by the appropriate literature. One can observe that hi-fi has largely died by its own hand, to paraphrase a certain J. Gordon Holt, and as testimony to the nature of these 'improvements' and focuses we have the drastic fall from grace of audiophilia to the point where the most many potentially interested, technically-minded people hear of it is an article in Gizmodo laughing at $10,000 cables. The average consumer may not know much about audio, let alone really decent audio, but they can smell that something's off a mile away, and will usually respond by largely ignoring the area.

Headphones will perhaps yet serve to reinvigorate hi-fi...hence the sadface in my earlier post when I realised what you were saying :(

Tyll Hertsens's picture

I fundamentally agree with everything you've said...and thank you for your wall of text, it was nicely filled with meaning.

In an ideal world, every product review would have measurements, ABX testing scores, and an array of subjective impressions from those tests, but as a practical matter that would be virtually impossible to achieve.

As a practical matter, the least expensive review process is a single, well qualified listener reporting on their listening experience. The vast majority of product reviews are done this way. The next least expensive method of product evaluation is measurements...but it is quite a big step in cost from subjective reviewing only, and it does take quite a bit of effort on the part of the reader to get to the point of being able to interpret the measurements. So the onus of measurement interpretation is on the reviewer as well. The most expensive desirable method of testing would be an ABX comparison study done by numerous subjective listeners. This method, while extremely potent, is basically out of reach for product review publications due to the cost. However, some publications do "round-table" subjective reviews that have some relevance.

So, while I think you're post is correct, one must take into account the practicalities of reviewing in order to come up with compromise solutions. If, as a practical matter, Lavorgna doesn't have the ability to do DAC measurements, does that invalidate his subjective experience? No. Is his subjective experience a complete evaluation of the product? No. Are we ever going to see product review publications do complete product evaluations (measurements, ABX tests, statistically meaningful subjective review studies)?'s just too damned expensive.

I've seen the work done by NRC, Harman, and Philips where the do subjective response studies of listening preferences, and, when done properly the relationship between personal listening preference and objective measurement is tighter than many subjectivist audiophiles might expect...but there is variations in listening preference. Which means there is room for subjective reporting to bring meaning to measured performance.  And again, the measurements do not describe the  the listening experience fully, at some point subjective information must be added to be meaningful to prospective listeners.

Lastly, in order to clarify my position for readers, I try very hard to put one foot into each camp. I think both objective measurements and subjective impressions are valid. This does create what I would consider a paradox in my evaluative process, and this paradox cannot be resolved. It exists because measurements and experiences are in two distinctly different domains, and there's nothing, practically speaking, I can do about it. 

Willakan's picture

I'm genuinely happy to learn that you're not in danger in tipping over into the belief system Mr Lavorgna seems to endorse, for it would make me somewhat sad to see Innerfidelity review, say, magic metal resonance cones you litter around the room and place on USB cables and suchlike to...erm...(they sound better OK?)although it goes without saying that what appears on site is not up to me in the slightest!

I still, however, have some concerns, which I hope you'll humour!

Firstly, just to establish, I largely agree with you on the matter of speakers and headphones. Whilst research like that you mention at Harman does go some way to point out that more emphasis on measurements would be good (I mean, their predictive model for speaker preference is pretty impressive, especially considering that simply FR and directivity dominate the 'inputs': 89% accuracy is very impressive, and would shame a lot of reviewers!), not everyone has the facilities to measure speakers, and for explaining the last 10% or so advanced mechanised ABX rooms are rather hard to come by.

There's then the obvious fact that differences between speakers and headphones tend to be rather large. With appropriate listeners and enough time, one can get useful information: the capacity for various involuntary kerfuffles to re-arrange that information is somewhat offset by the provision of measurements. The system you've got going here for that with headphones is, IMHO, pretty damn good, and is sure to get better as more research on headphone FR perception filters out of Harman and compensation can be tweaked accordingly. Would I like more measurementy-blindy-bits? Quite possibly, but I have to be realistic here and remember that the audience does not consist of me :D

Where we invariably part company is over the matter of DACs and amplifiers. If you'll briefly indulge me, I shall briefly look back to the 1970s, where, as Doug Self describes, it was perfectly normal to read amplifier reviews where measurements were analysed and little mention made of listening tests. Indeed, I found a scan of an early 1970s tonearm review and was surprised at the time to see that it consisted of almost nothing but technical analysis - not to say that they didn't listen to the amplifiers, but just that they related what information they thought was important.

Now, did they occasionally go over the edge (say, OK-ing marginal distortion performance by looking at the numbers with little attention to the spectra)? Invariably, but the important aspect was that nobody puported to be hearing anything staggeringly different from the numbers. It would seem there was an aura of reasonable, if not 100% exhaustive, understanding in the hi-fi mainstream.

We then get, as far as I can tell, 'subjectivism' emerging in the mid-1970s. An article Self references shows just how far it stood from the mainstream: "Subjective Sounds" would be limited in one magazine to a single, eponymous column; the writer felt sufficiently insecure to need to justify his approach in his first version. Before you know it, suddenly amplifiers have intricate sound signatures, speaker cables have too, and the world of audio is a largely blank (erased) map waiting to be discovered.

Now, lest you think I'm engaged in a particularly egregrious piece of historical revisionism, I direct you to the AES papers "10 Years of A/B/X Testing" and "The Great Debate - Ten Years On", the latter of which is written by a certain Stanley Lipshitz, who needs no introduction, and the former of which is written by David L. Clark, an AES fellow.

Both papers are retrospectives on how the attempt to resolve the "Great Debate" introduced by the sudden subjectivist explosion went, the attempt being comprised of the introduction of the A/B/X box. As the hi-fi press had furnished little evidence for their new angle (and it is made clear that this is the opinion of the authors), they reasoned that making blind comparisons reasonably straightforward by the provision of a reasonably-priced tool for the purpose would quickly establish what these individuals are actually hearing.

As we now know, the hi-fi press largely responded by throwing blind testing under a bus, which both authors look upon with surprise and cynicism respectively. The narrative has now been rewritten for many audiophiles: a bunch of people wanting to spoil their fun turned up, and on the basis of a few blind comparisons sought to 'prove' that everyone was full of crap, and were righteously dismissed.

The above viewpoint fails to observe that what really happened is that, as far as the serious (and concerned with scientific integrity) practitioners were concerned, the audio press/audiophiles responded to an opportunity to prove their claims, so far ventured on a startling lack of any compelling evidence whatsoever, that the current scientific conception of audio was hideously attacking those presenting them with the tools to prove their case.

From then on it's just got worse. Many people would be astounded to learn just how unfounded some of the most fundamental audiophile beliefs of the present really are. The fixation on jitter, for instance. As far as I can tell, the jitter bandwagon was started in the audio mainstream by Robert Harley, who appears to have suddenly decided that jitter was really important/special, and proceeded to largely fabricate a lot of reasons why (although he doubtless believed his own spiel). "The Audio Critic", admittedly not the most polite of publications towards such individuals, bought in an engineer and AES fellow from Analog Devices to provide a more level-headed discussion of the matter: said engineer duly pointed out, with the aid of a fair amount of graphs and mathematics, that jitter was a complex issue, but that it really wasn't that all-important  and that Mr Harley was measuring it wrong.

It is Mr Harley's entirely erroneous conception of jitter which has persisted!

For another example, let's try negative feedback. We are told that negative feedback is bad and that we should, for reasons entirely unknown, avoid it. This 'fact' is illustrated with lurid tales of those amplifiers from Asia in the early 1970s that, despite only exhibiting 0.0000001% harmonic distortion, sounded horrible because of too much NFB. As individuals such as Bruno Putzeys have pointed out, such amplifiers as described here appear to be largely mythological - any number of cheap, bad-sounding SS amplifiers may have been knocking around, but most of them never made it to a test bench and largely fabricated their specs. Furthermore, nobody has ever been able to point out exactly what's wrong with feedback: indeed, Mr Putzeys notes that, as far as the science is concerned, the more the merrier!

Guess which opinion (where the individual has an opinion on NFB) is more common - hell, you don't need to guess, you know!

Anyway, the point of this not insignificant diversion is to demonstrate that, as far as proof and evidence and the like are concerned (and those who do not embrace absolute subjectivism must be concerned with such things), much of what is taken for gospel in audio (intricate amplifier/DAC sound signatures) is propped up on, well...nothing! I am not exaggerating when I say that there is more evidence in the scientific literature to support claims of ESP than there is to support those of audiophiles: true, those papers were found to be hideously flawed and are not representative of the vast bulk of scientific enquiry, but they were, on their reception, found difficult to refute, originating from credible individuals and alleging carefully controlled experiements (the individuals involved were eventually found to have twisted the truth and their test protocols). The audiophile masses lack even that!

At the other end of the spectrum, one might ask: "Where are the papers saying that if THD is less than 0.005%, it will be found to be inaudible?" There are two objections and one observation to be made on this point. The observation is that, at the objectivist end, we are occasionally guilty of oversimplification (rules that will hold almost always, but not quite), in an effort to get points that are 'almost entirely right' across. If NwAvGuy had wanted to argue an absolutely, 100% watertight case, he would have made reference to a vast array of pyschoacoustic research and written long articles on distortion spectra - and nobody would have read any of them (I apologise for presuming to explain to you the art of writing for an audience, but there you go!)

The first objection is to note that the question smells of an inversion of the burden of proof commensurate with the revision of the ABXing narrative. The second, to support the first objection (which requires prior consensus for us to be able to talk of the burden of proof), is to point to the vast array of more generalised electrical and psychacoustic information we've accumulated and verified.

Grab a copy of the foundational textbook "Psychoacoustics: Facts and Models", which seeks to summarise the thresholds discovered across the entire field (and it has undergone reasonably little revision to keep it accurate) and you're presented with masking thresholds, FR variation thresholds, all of which apply in a reasonably straightforward fashion to audio, and all gathered over many tests that anticipate most  objections (eg: idea of golden ears countered with enormous sample sizes and retesting of the best subjects). And because modern equipment can be so good, we can make very bold claims about, say, the O2 and its fidelity to the amplified source. And because the evidence is so overwhelmingly one-sided, when I read that the O2 in fact sounds noticably inferior to another, reasonably well-measuring amp, I'm inclined to put the differences down to volume matching and the nature of sighted comparison (Note that I've personally experienced the sensation that you can hear a difference between something, only to find that the difference sods off once I don't know what I'm listening to: at one point I was convinced that I could distinguish between the choice of ASRC filters on my DAC. Just to reiterate that inherently inconsistent hearing is not something I only ascribe to other people!).

Similarly, on the electrical side, we have signal theory, which gives us things like the null test. If the standard tests run on amplifiers are can see where this is going.

So, I suppose the question from my perspective is that, once one has acknowledged that audio must be a realm in which science has authority, why one would conclude that amplifiers, once you've disposed of things like distortion, FR, output impedance ect would sound different? I'm not 100% sure where you stand; if we're contesting that the differences between amplifiers are indeed shown in the measurements, but that measured neutrality does not predict preference and that very small measured differences can be surprisingly audible...well, it's the same/similar question from my perspective: why would you conclude that with an awareness of other, clearly evidenced potential causes?

To briefly swing round again to the applicability of ABX protocols and the like, is volume-matching and a simple, post-general listening (so they can become aware of any differences) ABX test (accomplished with a few dollars worth of RCA cables, a $20 switching box and a few minutes of someone else's time) really not something that could be pursued? I appreciate there is again an element of Audience =/ Willakan, but even so... you're one of the few guys in the audio press I would trust to pull off some properly-blind amplifier comparison with sensible conclusions (the stuff on headphone burn-in was a model of a reasonable test, done right, with reasonable conclusions drawn, when even one of those things is a fairly rare commodity).

Tyll Hertsens's picture

First, thanks for your comments appreciating they way I work. I'll continue to do my best, and it will remain along the lines of what I do now...only better (I hope) as time goes on.

Second, your post is very informative, but at this point I think I've said what I can say, so I'm going to let it stand as is without comment...other than to say I enjoyed it.

Willakan's picture

Glad to be of interest :D

Errata (as the edit button seems to have left the building): section ending "that the current scientific conception of much of audio was hideously flawed."

Should read "that the current scientific conception of audio was hideously attacking those presenting them with the tools to prove their case."

(From Tyll: This edit has been made. Sorry your buttons have gone missing. :( )

There are various other more minor errors, but I'll stop talking now!

Limp's picture

A lot of words, well stringed together.
Too bad it left Tyll somewhat abashed.

Tyll Hertsens's picture

Not feeling "abashed" in the least.

Limp's picture

Oh, I'm just wildly interpreting your lack of a cogent response, you may feel whatever you want to feel. You have my full support.

Tyll Hertsens's picture

....bring some content. Snarky attitude and comments don't add to the dialog. You're one post away from disappearing.

Limp's picture

Snark and brusque is all I've got left.
I'm off to cook some lamb broth, maybe that will calm me down (rosemary, bay leaf, pepper, onion, carrots, maybe a drop of sherry? Yum)

On the whole I gratly appriciate your work, and I have no desire to see you hurt or angry, but still I insist on being a thorn in your side. Why is that?
I'll do some cooking and pondering for a mo, maybe I'll find a way to word my concerns in a more thoughtful manner. Or maybe not. This seems to be a toughie.

Jazz Casual's picture

Tyll doesn't need me to defend him and I doubt that he would want me to, so I'll just say that you may not share his perspective, which is fine, but it is always cogently expressed.

Tyll Hertsens's picture's a sure sign to push the keyboard away.

Plenty of time to consider this "toughie", no need to rush to post.

mikemercer's picture

I admittedly come from the Harry Pearson school of audio (at 39 I may seem a lil young for that, what can I say I met him when I was young, and I agree): Which is to say I don't care what a bench says.  I wanna hear what music the gear reproduces.  Does it reach me, affect me emotionally?  Can I connect with music through whatever particular electronic vehicle I'm using at the moment.

A KILLER piece, as I never argue with people about this, because I think whatever makes you happy is what counts...

and listening to the ALO Studio Six single-ended triode w/ the new Audeze LCD-XC is sending shivers down my spine right now! I enjoy it even more w/ the LCD-X!  IMO: It's their masterpiece thus far - and I LOVE my LCD-3's!! 

pic HERE

Seth195208's picture

... every conceivable listening/ listener situation. The deeper you go, the more you realize how staggering and unknowable it all is.  

JL77's picture

Designing audio circuits for decades, I've proven to myself over and over that better objective measurements do not always translate into predictable sonic fidelity or accuracy. Some of the finest testing small-signal amplifiers can sound thin and imbalanced.

The question is ... "what is the right way to do subjective listening evaluation?"

I've confidently answered that question for myself over those same decades, but I think there's quite a bit of room for individual experience and opinion here.

Tyll Hertsens's picture

How do you do subjective evaluation?

JL77's picture

"How do you do subjective evaluation?"

We use fast A-B comparison testing, using both high-quality headphones and speakers, sometimes in mono, in a 1st class listening environment. We employ a broad sonic spectrum of intimately known program sources, often self-recorded. After 20+ years like this, we've developed a high confidence in our comparison of candidate circuits.

cel4145's picture

I really appreciate this article, Tyll. I think you and I might differ a little in degree in applying the principles here, but we agree in kind. What I've argued before is that audio equipment evaluation is ultimately part of the aesthetic experience of listening to music. Great audio equipment enhances that aeshethic experience, and trying to quantify everything with measurements is like trying to explain through science how the appreciation of a piece of art is changed due to different lighting. It is not capable of completely explaining that. And yet, the science only arguments would have us ignore that aesthetic experience by lumping it under the term "subjectivity."

Long time listener's picture

"...we feel the single most important thing we can do is bring you our subjective listening impressions..."

Tyll, please don't say that, you're scaring me. The best thing you guys can do is give us BOTH listening impressions AND measurements. We can get listening impressions from any Joe on the street, and even someone with your experience may not tell us all we want to know with your subjective impressions. If I went only by your description of the AKG 518LE, "a great choice for those wanting a strong bass for hip-hop and other urban music listening," I would never have known how great they sound with string quartets. But the measurements indicate a headphone that is both full-bodied in the mid and upper bass (for full-bodied cello) with clean, perhaps even polite highs (sweet violin sound). They really do sound great for string quartets.

Most subjective listening tests are too narrow to cover everyone's tastes. There is a Stereophile review of the T+A Power Plant amplifier in which the reviewer listened to some electric guitar CDs and then pronounced it Class A, noting its "smooth" highs. But unfortunately, I've lived with the amp, and for acoustic music you get ear-piercing trumpets, dry, shrill strings, and horrible metallically ringing pianos. Maybe measurements would have indicated the distorted highs that the reviewer clearly failed to note.

Please don't head over to the completely subjectivist side of things.

Limp's picture

He does seem to be deprecating his own brand there a little, doesn't he?
Though I'm sure it's just politeness of some sort (commercial, professional or toward his readers).

Tyll Hertsens's picture

I believe what I said (though one might have to read very carefully as the balance is subtle), the subjective experience trumps measurements.

Limp's picture

Your form of balance looks a lot like self contradiction.

At one point you make it clear that we can't fully trust our senses, because it's never a fully conscious experience, and ones experience is always different from the next persons.
Next you say that Michaels complete reliance on his senses doesn't detract from his trustworthiness as a reviewer, because… he's 'different'?

Tyll Hertsens's picture

I don't believe I ever used the word "trustworthy." 

A subjective review should ALWAYS be suspect. Suspect in two senses:

1) Subjective impressions are always vulnerable to influences beyond conscious control. Bias is virtually inevitable.

2) Your tastes and experiences may be different than the reviewers.

However, the alternative of taking measurements does not actually address the quality of the listening experience. So regardless of whether or not a reviewer has measurements, the listening experience still needs to be reported for a complete review. And a subjective only review is valid for what it is. 

Again, the subjective and objective are two different domains, and while there are relationships between the two, they are forever two wholly different things and must be addressed differently.

That we cannot rely on subjective information like we can objective information is just the way it is.

Tyll Hertsens's picture

I said the "single most important" not "the best thing we can do". Again, I'd love to see AudioStream have a DAC measurement component, but there are VERY good practical reasons that make the unachievable at the moment. (Unless you want to drop $100,000 and a live-in Audio Precision engineer for Michael for a couple of months.) 

Measurements might add another dimension to the reviews, but lack of objective measurements on AudioStream's site does not degrade the reported listening experiences there. It's as simple as that.

anonymous cat's picture

i know that this is not the place but pardon me i need your help awfully

i bought a x1 fidelio headphone based on your recommendation for 300 $ but plz read the following and comment :

bought this headphone last week and i didn't notice anything playing everything at max volume till one day i came across this website for testing headphones


i tried a test called bass shaker on pc with my headphones connected to denon avr-1611 .. while i was increasing the volume i heard a rattle in the right speaker ( at very high volumes ) i increased the volume by 5-7 db and i started to hear it also in the left speaker .. the rattle is just heard in the 1st 5 second of the test


i tried the test on my ipad and mobile but i didn't hear the rattle at all .. instead may be a very low volume ( not noticeable ) buzz in the right speaker


so i went to a friend of mine to test his x1 too .. and this time the rattle started in his left speaker 1st and on increasing the volume it appeared also in the right speaker


soooo what do you think ?!!!! 

i think i pushed the headphones to the limit by this synthetic benchmark .. also i would like to mention that a grado s80i and a sennheiser hd202 did not produce such rattle and their sound was consistent inspite of the high volume


so i would be happy if you share opinions and ideas with me .. specially if you have x1 fidelio headphones and you can simulate the conditions .. you don't have to own a denon avr and you can use any amp unit on max or very high volume


you will help me very much by your trials as i still can send the headphones for a replacement but i live in india so i will pay shipping and taxes to UK forward and back


note : heavy bass mp3 songs and games and movies play flawlessly at very high volumes except in one scene in the conjuring movie where was a deep low rising bass the right speaker rattled for a fraction of second


so finally what do you see is this a faulty headphone or pushing to the limits .. also if you saw it like a faulty x1 what do you advice me to get instead for 300$ homeuse for movies , games and music .. also i prefer closed but not on the expense of the sound .. by the way thanx for your great efforts on this site .. amazing

xnor's picture

With things like headphone amps and DACs however, that are more linear and have performance measures many orders of magnitude better than speakers and headphones, measurements don't seem anywhere near as reliable an predictor of subjective experience.

I think you got that backwards.

People hear what they want to hear due to bias unless they do proper listening tests that eliminate bias. Therefore, the smaller the actual difference the larger the degree of freedom to interpret the sound however you like.


Is listening sufficient in determining what we enjoy listening to? Well if we put it that way, of course it is.

For someone personally it very well may be, but a reviewer shouldn't just post what he/she enjoys and what he/she doesn't.


Does the fact that he doesn't have measurement program somehow lessen the value of his subjective review offerings? Hell no!

Hell yes! x2

How does Michael test the components? Plug them in, sit down, take a listen, and write down whatever comes to his mind?

It's hardly surprising that reviewers doing it that way can come up with nonsense. Measurement data at least helps revealing that.

zobel's picture

"Not everything that can be counted counts, and not everything that counts can be counted." This will always be true, but measuring and counting will continue as tools to attempt to quantify, understand, improve, and learn more about the nature of reality, which can only be experianced subjectively. Albert Einstein; "Reality is merely an illusion, albiet a very persistant one." And something else, "If the facts don't fit the theory, change the facts." I love that.

Einstein also said, "If you cant explain it simply, you don't understand it well enough."

He also said, "The most beautiful thing we can experiance is the mysterious. It is the source of all true art and science."

Another quote, "All religions, arts and sciences are branches of the same tree."

I could go on and on here, but the message is, we are human, and enjoy the mystery while seeking truth. Ok one more, "The true sign of intelligence is not knowlege, but imagination."



Willakan's picture

Thank you kindly for the 'argumentam ad pretentium'. You missed out "There are more things on heaven and earth, Horatio, than are dreamt of in your philosophy"...

It's all very well to throw your hands up in the air and shout "ISN'T EVERYTHING MYSTERIOUS AND SUBLIME?", but the trouble is that once you look into matters, it really doesn't look that mysterious at all.

Jazz Casual's picture

You might be the smartest guy in this room Willakan :) but there's no need to be so dismissive. Of course science demystifies but I think you'll find that there are scientists who continue to be astonished by the unexpected twists and turns that they encounter in their fields of research. Paul Davies, the physicist, cosmologist and astrobiologist, ponders the relationship between science, religion and the concept of God and is prepared to countenance the inexplicable. Oncologists are unable to explain why some of their terminally ill patients enter remission. There are medical drugs that are clinically proven to be beneficial, however the reasons for this are not entirely understood. So there is room for the mysterious or the unknown if you prefer, even in the minds of scientists.

AstralStorm's picture

See, here's the thing: while highly trained and experienced listeners can be as good or better than the measurement hardware (esp. at harmonics and intermodulation), you just can't trust them completely.

  1. They can be bought.
    Everyone loves free stuff and the best way to get it is to write positive or at least non-negative reviews. A true reviewer will destroy a bad product mercillesly. That's even without going tnto outright bribery. Then there's the "shiny new thing" effect.
  2. They can make mistakes.
    We're all human, some days you just write something lhat later you want to take back.
  3. Their opinions can change.
    Especially with experience. Granted, good reviewers rarely change their opinions at the whim, but see below.
  4. They have tastes and bias.
    Some reviewers have very strong tastes and ideas of what a good sound should be. Unless you identify that and take corrective measures, the reviews are strongly biased.
  5. Their ears might get damaged.
    Fact of life - hearing damage can happen and does happen with age and illnesses. True high fidelity reviewer should get his hearing checked regularly and post that data.
  6. Their anatomy and fit might not match yours.
    Depending on the headphone or in-ear in question, they may get a different fit and thus vastly different sound with certain devices.
  7. The language is confusing.
    Until you fully decode the reviewer's idiolect, you can only guess at the meaning the review conveys. To do so, you need multiple reviews.

And a general comment about both measurements and reviews: manufacturing variances are common, few reviewers have more than one unit.

What matters most is a right set of easy to interpret measurements.

You see less THD+N, it's almost always better - these can be weighted where the distortion is more audible and less. You see sharper high harmonics dropoff, it's always good. Frequency response is hit or miss due to various ways to compensate the measurement (but there are ways to derive a decent compensation), but lack of resonances (use CSD) is always good. Then there's the low frequency square wave response - like CSD, but accurate in the low end instead of higher end.
And of course the measurement gear has to be checked and cross-examinated for accuracy. How to weigh these in the grand scheme of things... who knows, not enough data.

The main idea of high fidelity is for the reproduction and recording gear to disappear and the live recording to sound as close to the live performance as possible. This is and has always been the sound reproduction goal. Anyone who actually doesn't believe this doesn't believe in high fidelity. Might still call himself the dirty word audiophile, but he's not a fan of accurate sound.

cdxskier's picture

This really articulates the heart of the issue. Thanks for sharing!

zobel's picture

The more you know, the more you know you don't know.

The Federalist's picture

I appreciate the work that Tyll does and believe that the effort that you've put into headphone and amp measurements is a great contribution. I use the frequency response curves all the time when I go shopping for a new headphone. It isn't full proof but in narrows the field... in an ocean of current choices. Likewise your reviews are usually an early acid test for me.... I ultimately need to hear say the Fidelio X1 but if it's made its way through Inner Fidelity without any serious criticisms.... that's a good sign. 

I think the tenuousness of this issue lies in a cynicism that readers have towards reviewers and the precarious relationship between say Innder Fidelity, it site sponsors (all headphone related audio companies) and its ability to stay objective towards the companies that pay to keep the lights on.

It's a fine line and I think you guys at Inner Fidelity walk it very well.... I think all the reviewers on your team offer a very balanced opinion with a fair amount of honesty and voicing skepticisms of their own. I think objective measurements provide a level of support behind your conclusions.... It lends credence to your voice.... And I think there are challenges to reviewing things like amps and dacs because the differences are so small.

I think some writers tend to paint with too broad a brush and too much color when the differences are negligible. When writers do this.... with no objective test support... readers can perceive it as questionable or even disingenuous.

I personally take DAC reviews with a healthy dose of skepticism because my own experience is that the difference between a $200 DAC and a $1000 DAC aren't as significant as say Michael Lavorgna's or John Darko's have made them out to be.... but I understand that they have to write copy that people want to read... For me the problem is where is that line between drafting an accurate and reasonable review and drafting compelling copy?

Objective measurements may not tell the whole story, but would be welcome in certain circumstances.... Case in point. John Darko has been preaching SPDIF Converters in lieu of USB to your DAC to reduce jitter.... it'd be nice if he could furnish a test result showing the Audiophilleo or Wyred4Sound uLink or Resonnence Lab Concero actually did reduce jitter. But neither he nor Lavorgna who takes a similar position does... but they keep imploring you to buy an additional $400 to $1000 box to add to the signal chain or at least give you the impression that you are missing out if you don't. This leads to a bit of skepticism... A run on the test bench could alleviate that. 

And reasons for this cynicism are everywhere.... Robert Harley from Absolute Sound wrote a supposed "technical article" recently that was supposedly to inform his readership of the reasons they should consider high end cables for their systems. Problem is that the article was literally a reworked version of Audioquests own sales copy... right down to the images he used.... So was Robert really trying to give his readership good information on cables..... or was he trying to move product for Audioquest, who is also a top sponsor at Absolute Sound. It could've been either, I can't say.

But it is these articles that further stoke cynicism from readers.... and in many cases reviewers will act incredulous when challenged to offer objective support. 

I guess what I am saying is that I see why objective types operate with a level of mistrust..... but also acknowledge that they have the choice to not read the damn article too.... so there is that....

But in the end I wanted to commend Inner Fidelity.... I do think that contrasted against the High Fi online, magazine landscape you guys are really a cut above. You offer reasonable reviews, don't seem to have any fanbois tendencies towards any companies and you support your reviews with measurements.... Especially in terms of headphone review sites.... You guys are a cut above.... In my view the premier online site....

So thank you for that.

Tyll Hertsens's picture

I also see the problems with a "Subjective only" approach, but see no way around the need for it. 

Anyway, thanks for the kind words...I'll keep working on it.

eggil's picture

Even the most honest investigator has bias.

A reviewer has biases too, most of them unconscious, but some may not be.

A pretty and shiny amplifier may "sound better" than a DIY product that may not look as nice.

Financial interests and the desire to justify a purchase are hard to get rid of. 

At the end, a lot of what is left is the reviewers reputation, and you are only part of a handful I would trust. 


castleofargh's picture

I'm always happy when I can get more intel, be it measurement or subjective feedback and find it really annoying(to be polite)that we must always end up antagonizing those two.

if we're looking for mistakes and erroneous conclusions, we can find as many from misinterpreted/badly done measurements, as from badly done auditions. diversity is the best way to foolproof a judgement.

if you believe that you know something about a product, let us know. most of us will be grateful and that's the end of it. (but sometimes I'll think you're full of sh***, because I'm a human ^_^ )

Alex Halberstadt's picture

A good scientist's essential attitude is curiosity toward the world. When phenomena are obeserved that cannot be explained and quantified, a scientist doesn't plug up his or her ears and demand that the phenomena cannot possibly exist. And yet that's what plenty of people in this hobby do when confronted with experiences—the varying sound of cables, for example—that cannot be readily measured or explained. They demand that these differences cannot possibly exist, because they currently cannot quantify them. Believing that the entire experiential world (or domain, to use Tyll's term) can—or should—be described by current empirical science is illogical and demonstrably untrue. Philosopher Thomas Nagle called this fallacy "scientism." To quote H.H. Scott engineer Daniel R. von Recklinghausen, "if it sounds good and measures bad, you've measured the wrong thing."

thelostMIDrange's picture


here's an old timer who tunes a classical guitar top by ear and his decades of experience. talks about human compared to machine listening and measuring towards the end of the vid.

Jazz1's picture

Give me a headphones that give me the soundstage of my long gone ProAc speakers and I will bow to the direction of the manufactuer. Fact is that I now listen to headphones 99% of the time because I don't have the room for speakers for what I consider the ultimate listening experiance. Granted for the  $$$ headphones rule when compared to full blown, stand alone speakers. But it does disappointment me that there is still nothing in headphones that can match any of the full blown speaker systems I've owned. I'm not trolling here as I'm dedicated to headpones and the systems that drive them.

That said, mobile systems for personal listening have come a long, long way since I have had a Pioneer integrated amplifier system driving Koss Pro4A's via TEAC reel-to-reel and AR turntable with Shure IV.