Mastering Your Domains: The Difference Between Objective Measurements and Subjective Listening

Ed Note: This post is inspired by Michael Lavorgna's recent AudioStream article "The Trouble with Listening", and comments he received.

The creepy thing about the photo above is that you have a human that is part robot, and a robot that has human-like intelligence. We're pretty far down the road with computers and have some pretty good stuff these days, but from what I understand we're still pretty far away from putting anything remotely like human consciousness into a machine.

Why? Because the human consciousness is not in the domain of the objective stuff like computing.

Domains
Human consciousness experiences sensory input, it doesn't measure it. A major chord is not understood by our consciousness as a series of three particular harmonically related notes, rather it's heard as a whole and as having a cheerful character relative to a minor chord. People can fully enjoy the experience of music without having any objective idea whatsoever about music theory and harmonic relationships.

Now I'll certainly admit that computers can do a decent job of composing music, and many would be hard pressed to blind test the difference between computer and human generated music. And computers can listen to music and tell you all sorts of things about the structure and qualities of the signal, but a computer is not going to adopt a preference for Gershwin over Shostakovich, or tap it's foot when it feels good about what it's hearing—though you might be able to program it to look that way. A computer isn't subject to experience.

The Objective and The Subjective are simply two completely different domains, and while there are relationships between the two, one is wholly unlike the other. While measurements of audio gear are terrifically meaningful, the actual relationship between measured performance and listening experience is quite various in terms of the measurements predicting the nature of how we will experience a particular piece of gear. With things like headphones and speakers, where frequency response deviations from linear are strong, the measurements will relate more strongly to listening experience. If the measurements say there will be strong, tight bass in a particular pair of headphones, most folks will pretty reliably hear it that way. With things like headphone amps and DACs however, that are more linear and have performance measures many orders of magnitude better than speakers and headphones, measurements don't seem anywhere near as reliable an predictor of subjective experience.

Distortion, or Character?
Let me give you an example: NwAvGuy's O2 amplifier measures extraordinarily well. Do some basic listening to this amplifier and most folks will come away impressed with it's very competent performance. But when I spent a lot of time with it, I found myself feeling it was a bit of a dry and uninvolving. Switching over to, say, the Apex Butte amp designed by Pete Millett, I find the sound more musical and bouncy—it made my feet tap more reliably. The Butte, while not having as good a set of objective measurements, was clearly more pleasing to my ears. What measures as greater levels of distortion can be experienced as having a more pleasing character of sound.

Herein lies the problem: when we compare measurements, we might report amplifier A has less distortion than amplifier B, and make the assumption that amplifier A must sound better than amplifier B, but it's simply not the case. The added distortion may cause the amplifier to sound worse, but it also may cause the amplifier to sound better, and you can't necessarily tell by the measurements. I'll hasten to add that because we're in the subjective domain here, "better" and "worse", in terms of sonic character, is only meaningful to a single listener. While statistical norms may be found and listener preferences will likely cluster, the fact remains that determination of amplifier character and listener preference is done person-by-person and each person's experience is absolutely valid as their experience, regardless of measured performance.

131104_Blog_DomainsObjectiveSubjective_Photo_Scientist

So Why Measure?
While the relationship between objective measurements and subjective experience might be tenuous, it does exist. Subjective experiences are notoriously fickle, and I personally find that having measurements available as I review a product acts as a stable reference point as I listen to and evaluate a product. If I see big bass in the measurements and don't hear it, I might find myself taking some special care to ensure the pads are sealing correctly on my head, and will take some extra time ensuring that my listening impression isn't somehow influenced by the music selected, or a stuffy nose, or a bad night's sleep.

Measurements are also helpful as it's very hard for most people to get their ears on all the headphones available, so having a database of measurements allows potential buyers to do some preliminary investigation by looking for headphones that measure similarly to those they know are to their taste. This, of course, takes some extra effort on the part of the buyer in order to understand the measurements and their relationship to the listening experience. In the end, however, it's the actual listening experience for that particular person that counts.

For electronics like headphone amps and DACs the equation is a bit different. Because the relationship between the measurements and listening experience is somewhat more tenuous with these more linear devices, measurements have a somewhat different value. The greatest value in amplifier measurements I've performed is their ability to have a small window into the technical competency of the product's design. If measurements show significantly excessive noise, hum, distortion, or crosstalk for example, it causes me to think that particular design is likely flawed, and the designer is likely not very proficient. If I see an amp with extraordinarily good performance in those areas, I begin to feel confidence that the product and designer might both be of high quality. But most products lie somewhere between those two extremes, and relatively little is learned by objective testing.

There are pragmatic problems with implementing a measurement program as well. The equipment needed to make reliable and accurate measurements are painfully expensive and technically challenging to set-up and execute. I'm a pretty decent technician, but it's taking me far longer than I had hoped to get my headphone amplifier measurement program up and running. Objective testing is a distinctly different expertise than evaluative subjective listening—which is, to some extent, the whole point of this post: Listening and measuring are two entirely different things.

131104_Blog_DomainsObjectiveSubjective_Photo_Scout

The Map is Not The Territory
I was inspired to make this post due to Michael Lavorgna's recent post on AudioStream, "The Trouble with Listening", and critical comments thereafter intimating that subjective reviews without objective measurements are somehow incomplete or flawed. I would argue that measurements are a valid mapping of performance, but that map is not the same type of thing as the territory (or domain) of subjective experience.

When I plan a motorcycle ride through the forest, I will consult numerous maps: topographical maps will tell me about the steepness of trails; road use maps will tell me where I can and can't ride; land management maps will indicate where I may be legally allowed to camp. But the best source of information is dialog with another rider who may have been in the territory and can deliver a scouting report. Stuff like where mosquitos might be, and where there's a scenic camping spot may be brought to light. The maps just aren't going to tell you about the experiences available in the territory.

Subjective reviews are like scouting reports. They tell you about the territory of the actual listening experience with a particular product. Objective measurements, while informative and related, are relatively mute when it comes to describing that. When Michael Lavorgna says in his piece...

I would start by asking a question—Is listening sufficient to determine the worth of an audio component? Or to put it another way—Is listening sufficient in determining what we enjoy listening to? Well if we put it that way, of course it is.

...I think he's right.

Sure, I'd prefer it if Michael had a killer measurement program up and running—when I go on a moto trip I take as many maps as I can—but, for a variety of reasons, that's just not the way it is at the moment. Does the fact that he doesn't have measurement program somehow lessen the value of his subjective review offerings? Hell no! The subjective review is at least in the same domain as the listening experience I'd have with the same gear.

Dump a fat stack of measurements on the table and I'd surely give them considerable attention, but when I was done I'd still have a burning desire for a scouting report. That's why for Michael, and me, and John Atkinson, and all the writers in the Home Tech group, we feel the single most important thing we can do is bring you our subjective listening impressions. Measurements are groovy, but it's the stuff that happens in the domain of the subjective listening experience that really counts.

Share | |
COMMENTS
thelostMIDrange's picture

Korzybski's map not representing the territory was addressing how a human being accesses Reality, or knowledge of it. If you read his works you will find that he ends up saying the same thing heidegger does, and unfortunately nietsche as well; your ability to access Reality with a capital R is a direct result of how human you are. and how human you are depends on how much Truth you let into your life. Your map will only be as accurate as your ability to let reality in. And so that pic at the top of the page of a machine with a pseudo human consciousness and a human with a pseudo machine consciousness will both have limited knowledge of reality. Human, all too human as Nietsche wrote, it is rather these folks have their finger on the pulse. their map most correctly corresponds to the territory. And in that sense, the author of this article shows a deep and healthy understanding of what it means to be a human listener and I applaud this understanding. He is riding that knife edge of objective/subjective and realizes that they are different species, not differing points on the same continuum. Our world is riding that same knife edge, and tetering towards falling off the objective side. Humans who have absolutely and objectively failed at becoming human have assumed power in all the highest positions and so we have leaking nuclear reactors all over, a chemtrail filled sky, soil that is unsuitable for growing, total lack of empathy, morals, compassion, understanding, peace. All the things that a human has that a machine does not and cannot be programmed to have much in the same way it cannot be programmed to know when and how to tap it's foot or bop it's head in a human way. And yes, distortion is part of music and therefore part of a great sound system. The author again understands this. it's a matter of where it's placed, how much and what kind of distortion that helps music be musical, not a total lack of it. anyone who plays an instrument that is not digital knows that distortion is color. The kind of bow on a violin, the kind of wood for a guitar or drum kit, the distortion from an amplifier. A little color goes a long way but it's a spice that makes music more pleasing to a human nervous system. If we as a culture continue trying to make ourselves into computers, then yes, in the future we will not feel distortion as color to be pleasing. You would need a really advanced computer to know enough to build into itself the desire for distortion and color and it's precisely this lack of enjoyment for messy things like color that it would not want to program into itself. Whomever programmed nature on the other hand, was wise enough to figure it into the equation. it's only folks who feel there is no supreme energy that feel it's a bane and feel they can assume the role of creator. Delusional thinking I say. But No worries these technocrats and google heads say, we'll just merge with the computer and leave this messy flesh behind and monsanto and it's franken seeds with the pesticides baked into it will be a stopgap until they are able to become one objective creature living forever in the digital domain. Yeah right. Kurzweil is a delusional nutjob that would be committed if we we'ren't iving in an upside society. The average joe has had a serious mind meld done onto them through false history and schooling that he is so far from being a healthy Truth knowing being that he is easily led down any path set before him. I say this to suggest, that headphones are no different. The objective measurements represent the machines' doing what they are good at and for that they are useful if in the hands of a human with a knowledge of reality and how he differs from the machine. In the hands of one of the two 'things' in the top pic however, they are pretty meaningless and misleading sets of data. As the author states, it's only direct experience or at least second hand experience told from another human that is of value and that objective data needs to be understood in light of that experience. Riding that knife edge is precisely this ability to use the data and not let it use us. As a culture we are letting it use us. The machines are taking over. They are not serving us by giving us more leisure time. And in the end it is liesure time that is the most precious of commodities. Time to enjoy music with headphones or staring out into the sky from under your favorite oak tree or from behind the visor of your moto helmet. Headphone measurements that can help one pick the right headphone are therefore supremely useful if they save someone and his/her time from having to listen to dozens upon dozens of headpphones. If we don't quickly come to this same understanding in the larger culture, the biosphere will no longer be habitable for subjective fleshy beings to enjoy music through headphones, or anything else. Each one of us needs to get his map closer to representing the territory and that territory is nature, which includes us. Say no to the technocrats and withdraw your participation from it's machinations wherever you feel they inpinge upon your god given liberties.

Tyll Hertsens's picture

"Each one of us needs to get his map closer to representing the territory and that territory is nature, which includes us."

Boom!?

Heck of a rant there...thanks for posting. 

Seth195208's picture

...ability to observe and reflect, which, in a lot of ways, excludes us from natural world. this is the gist of the problem. 

Hal Espen's picture

I embrace the God-given liberty to use paragraph breaks, non-bold fonts, and the standard capitalization of proper nouns. 

But I'm grateful that you resisted utilizing the Cap Lock key for that extra soupçon of emphatic assertion.

paul's picture

Serif Fonts ... 

Azteca X's picture

Excellent post, Tyll.  Thanks for sharing.  I was in the comments over there suggesting some measurements and think Michael is a great reviewer and I've read just about every post he's made in the last 18 months.  I value his voice as a writer and his approach to music, life, art and the enjoyment of all three.  The measurements would be a great resource, though; he gets a pretty staggering array of great TOTL DACs in there along with all the tantalizing budget options.  The Auralic Vega seems to be a true contender, for example - I'd love to see some measurements to let me judge if it's competently designed in addition to pushing all of Michael's listening buttons.

And while there are those who can splurge on all the latest TOTL stuff, there are even fewer who can measure and measure well, particularly with a Clio or AP device.

Dan S's picture

Tyll, one thing I really appreciate about your reviews is that you cover both the subjective and the objective. For headphones, I think you've struck the perfect balance. I've really learned a lot from reading your articles and looking at your measurements.

This reminds me of the nature vs. nurture debate. Um...how about both? Why does one have to win out over the other? They are both relevant.

A huge reason to provide the objective along with the subjective is that it gives you legitimacy and a grounding in something reliable and reproducible. I feel like you are providing a complete picture, and the objective part of your review can be examined and challenged, if necessary. I trust you a whole lot more than if I was just getting subjective impressions.

I also like that you are open with your biases. For example, you prefer headphones that are a little warm sounding. So do I, but if I didn't, I could take that into account when reading your subjective impressions and adjust for that bias.

robm321's picture

Really enjoyed it. I'm glad to know that audiophiles won't be replaced by computers. 

SonicSavourIF's picture

First of all, I think you will hardly find any objectivists that claim, that it is impossible to prefere the sound of gear that measures poor in favour of better measuring gear. Even guys like NwAvGuy and Ethan Winer agree, that you can like the sound of vinyl, tube amps, analog tape, and that is perfectly fine. As long as you stick to subjective claims, when reviewing subjectively everything is fine, because you are talking about subjective opinion. However if one claims that a poorly measuring amp (or a super expensive cable or ...) is higher fidelity/higher quality, then you are in the realm of measurability and make falsifiable claims. Likewise if one claims that gear L gives 'better sound' but is not able to determin the difference in a proper blind test, than the claim is highly questionable and very likely du to placebo and expectation bias effects.

It is funny that today I went through the measurements of the LCD-2/LCD-3, HD800, Amperior and DT1350 headphones and thought that I actually cannot derive from these which can sounds best. (e.g. looking at the measurements alone, the bass of the LCD-2 seems better than that of the LCD-3, and the rest of the frequency response is quite similar. Yet, people prefere the sound of the LCD-3. Likewise, the frequency response graph of the HD800 looks quite bumpy by comparison). So at least for headphones, especially because it is a huge matter of taste, I can understand, why you have to listen to them before you can know if you like a pair or not.

With electronics however, I think it is exactly the other way round (Hate to disagree with you Tyll :-/). The smaller the differences between gear are, the more likely we get prone to expectation bias.  It is true that in the end listening matters and above all personal enjoyment. But if that is the case, to really determin by listening which gear one likes best, you would have to do a proper blind test. The moment you read a review about an UberAmp XY, know that master Yoda designed it and the thing has a 5k price tag (or likewise measures exelently =)) your brain interprets what you hear in one way or the other. You get influenced by others, by your mood, by marketing and so on. I have experienced utterly different listening experiences with my same gear ranging from something like experiencing dull and boring sound, making me want to immideatelly buy new, 'better' stuff, to tears of joy. That leaves me personally with the question how reliable my hearing is in the first place when evaluating gear.

On the other hand, after all I have read on our hearing/listening, flaws of sighted testing and objective measurements, I find that actually  this knowledge substantially improved my listening experience, as I tend to much less worry about the sound I get and in the same time beeing releaved that I don't have to spend astronomical amounts of money to get good sound. So for me as far as electronic gear is concerned, measurements help a lot and are more helpfull to base a buying decission on than subjective reviews.  

As it is so difficult to properly measure gear, I hope to see more amp measurement on this site as guide we can rely on when searching for new stuff.

Sorry for the long post, cheers

pmillett's picture

Tyll, that's a pretty good discussion of measuring vs. listening.

People really need to understand a couple of fundamental facts:

1. If you like it, it's good.  Period.

2. Measurements probably won't tell you if you will like it.  You need to listen to it.

As you know, I'm an engineer.  And I make lots of measurements.  I believe I can almost predict how people will describe the sound of an amplifier by interpreting the measurements I make.  But I absolutely cannot say "this will sound good" based on measurements.  I can usually make some predictions, like "this will sound pretty bright" or maybe even "I bet this will sound awesome on female vocals with HD-800's".  But "good" is in the ear of the listener, not me.

Sometimes, I can say "this will sound like cr@p", though.

Single-parameter measurements - especially THD - really mean nothing.  I think we've all heard stuff that sounds really good to us that has pretty high THD (like 1%)... and stuff that sounded terrible that had 0.05% THD.  The perceived sound is so much more complex than a single measurement.

That said, if somebody tried to sell me a power amp that had 10% THD at 1 watt, I wouldn't even consider it, because I know it will sound bad.  There are limits, of course!

More involved measurements like FFTs can shed a bit more light on the sound.  But no measurements can really model how you percieve sound.

Unfortunately, our business is full of snake-oil salesmen who fully believe their own lies, so you have to view all measurements - and claims that "this sounds better than that" -  with skepticism.

Except for what I say, of course...  ;)

Tyll Hertsens's picture

Glad you liked it, and thanks for the response, Pete. 

You're 100% correct...of course.

Limp's picture

 Does the fact that he doesn't have measurement program somehow lessen the value of his subjective review offerings? Hell no!

Hell yes! 
Subjective reviews are like scouting reports, and sometimes scouting reports bring back tales of mermaids, sea serpents, kraken, komodo dragons and the Blemmyes.

Without measurements we have no possible way to know wether the tales Michael Lavorgna spins are like these, pure myth, or not.

pmillett's picture

"Subjective reviews are like scouting reports, and sometimes scouting reports bring back tales of mermaids, sea serpents, kraken, komodo dragons and the Blemmyes."

Um, Komodo Dragons are real...

http://en.wikipedia.org/wiki/Komodo_dragon

Limp's picture

But are they dragons? Do they spew fire? Do they have a mid-range so liquid it's worth dying for?

pmillett's picture

I think we can devise measurements to factually answer the first two questions.  As for the third, it's in the brain of the listener...

So yes, we do need measurements!  But they still won't tell you if the dragon is pretty, or ugly.

Limp's picture

I concur fully.

The best result is attained with the two disciplines working in tandem, which is why I took some exception to Tylls somewhat surprising statement.

thelostMIDrange's picture

or an unfortunately necessary device in between the music and your ear?

To think of it as a musical instrument is an admission of defeat for hi-fi'ers but potentially more fruitful when it comes to designing such gear. It's a game changer to think of it as an instrument with all the attributes we normally associate with instruments. No two are ever the same, they all have color etc. Then it becomes about fine tuning the musicality of it around that neutral point we call hifi instead of searching for some ultimate end point of fidelity which apparently doesn't exist except in our ideal imagination and as elvis costello sang, imagination is a powerful deceiver.

 

 

the etymology of instrument is basically, a tool. Heidegger suggests that we only really notice tools when they break down. when they are working they are invisible. Hifi aims at invisibility but it seems impossible as is evident by measurments and whenever they get close to perfect in measurement they seem to be less and less enjoyable, which can be seen as a sign that there was a missed assumption somewhere along the line and the end goal we set up for ourselves an illusion. But there does seem to be a point at which things get pretty darn close to Hifi and then we whiz past it and go too far. A human feature to go too far and take things past their balancing point. When the rubber meets the road, reality is 'what works' and what sounds good. In that way, the original article suggests it's practical to find what sounds best then look at the measurements and find out why instead of the other way 'round.

 

but one thing is for sure, whenever I hear the tool of a headphone or amp impart itself all over my record collection then I know it is deficient somewhere. When it disappears, then I know it's 'working' and good enough for me. As long as I can listen without fatigue I know i'm near that illusive balancing point of Fidelity. Whether some attribute could be a bit tighter or clearer is the lure that pushes designers to go too far in trying to reach for that little extra and compromise basic fidelity in the process after which the gear again becomes broken and 'hearable' over the top of the music.

DrForBin's picture

hello,

a bit about my situation: 1. i am over 60, 2. i rode a motorbike daily without using any real hearing protection for quite a few years and 3. i recently had my hearing tested by an audiologist who noted that i had "injury loss" which accounts for why i don't always understand my wife and daughter, its not that i'm not listening, i really can't hear them. i am down about 50 dB at the frequency of most female speaking voices.

the objective/subjective divide has always troubled me as both sides seem to take the position that their's is the absolute truth. both you and Mr. Lavorgna's articles are presenting a way out of this seemingly irresolvable debate and i would hope that both sides of the fence will take it to heart and stop the ill-will that comes from taking an absolutist position on anything that should be giving us pleasure.

having said that i will enjoy listening to my AKG K550 while i fall asleep, comfortable in the fact that i do not disturb my bedmate, that all the music i like sounds pretty darn good, and that if the tune is right and my mood is right, the experience can bring tears to my eyes. (e.g. SRV "Lenny.")

wiinippongamer's picture

People could always add that distortion to the O2/any other very well measuring amp at source level, either hardware or software, or add an output resistor to modify FR, etc. And get the exact same result sonically while spending alot less, or MAYBE, just MAYBE................. EQ the damn headphones and avoid needing to compensate for whatever it is they don't like about them. The whole subjectivist deal is just backwards thinking; worse yet is the willful deceiving of customers by many "subjectivists" out there.

But hey, it's their time and money.

Jazz Casual's picture

I was blissfully ignorant of headphone measurements when I became involved in this hobby. I soon found that they were often cited at headphone forums in support of the excellent performance of some headphones and to "expose" the poor performance of others.

I set about trying to understand them and became quite immersed in the process. I would compare the measurements of different headphones, including the ones I owned, and compare the objective data with my own subjective listening experience.

It was always an interesting exercise but ultimately quite disconnected from my response to the headphones that I listened to, and my enjoyment of them or lack thereof.

"From what I've read" measuring and listening are integral to designing and building audio components. Whether it's more art than science depends on which designer or design group you talk to I suppose, but the relationship between the two is intertwined rather than mutually exclusive. However, if you're intending to build an audio component, you'll need more than just a love of music and a decent pair of ears. An understanding of circuit diagrams and oscilloscopes would seem necessary at the very least.

However, being a music listener rather than an audio designer, I don't need to understand a design schematic in order to appreciate the sound that the component is producing; just as a car driver doesn't need to know its performance measurements (and no, I'm not referring to the speedometer) or how it works on a technical level, in order to enjoy driving it.

The same can be said for headphone measurements. Of course it can be interesting to learn more about them, and for some audiophiles, this technical knowledge can be a source of fascination and obsession. However, it is not essential for the appreciation of music reproduction from a headphone or a hi fi system - it is an adjunct to it.

Developing an understanding of frequency response measurements has enabled me to better understand what I'm hearing, but it has not enhanced my enjoyment of it - nor should it. It has not given me an uncanny ability to predict how I will respond to a headphone that I am yet to hear or whether I am going to like it or not. Only the experience of actually listening to the headphone has provided me with that certainty. I'm often surprised how much I like a headphone despite how it measures, and that's because the measurements only tell part of the story. The graphs do not show timbre, imaging, soundstage and various other intangibles that we hear and respond to. That's been my experience anyway, and it's why I seldom think to look at headphone measurements these days.

Audio designers/manufacturers must objectively measure the performance of their products, but it's entirely optional for the rest of us, the consumers and hobbyists - unless we harbour secret ambitions to become audio designers/manufacturers too.  

 

  

Willakan's picture

I’ve seen this angle, implied with various degrees of obviousness, several times before, and I still find it, if I’m honest, not very compelling. At least, I think it’s the same angle, as much of the logic of your article functions by implication: your link to Mr. Lavorgna’s piece gives me some idea, however, of where to start.

Rather than immediately leap into rehearsing the relevant aspects of the usual “belief systems”, I’ll start with your opening paragraph under domains (“Human consciousness experiences sensory input, it doesn't measure it.”). The distinction here is presumably between the personal/quasi-transcendent experience of listening to music (which is somewhat irreducible) and the waveform/notes or whatever you want to call the physical “thing” we assume to exist; this thing we can test and fiddle with and measure until the cows come home.

Would it be fair to sum up the logical core of your argument, working from here and from your quote from Mr. Lavorgna on the importance of enjoyment, that you view the enjoyment derived from a piece of equipment, in the environment it is used in (IE: listening to music for pleasure rather than engaged in an ABX test or similar), as something qualitatively different from that which is characterised by test protocols, and that this is the ultimate arbiter of the value of a piece of equipment?

Tyll Hertsens's picture

The thing your being is being exposed to can be characterized in part by measurement equipment. The experience is something you have internally that is catalyzed by exposure to the signal. 

Willakan's picture

OK, so assuming I've got this system right, I have some problems with it :D

The first, and immediately striking issue is that, followed properly, it reduces all the most important claims in audio (in terms of epistemological priority) to a state where they are inherently unfalsifiable. Unfalfisfiable claims are inherently inimical to logic; ergo, the domain of audio in its purest sense transcends logic (putting it politely!), and therefore all opinions on the matter are inherently irrational.

Now, I don't think I'm being overly presumptuous in anticipating a number of objections to this, the most obvious being to draw attention to the fact that you do hold opinions on audio that you didn't make up because you felt like it. At the basis of those opinions lies some kind of correlation: your claims about product X are not unfalsifiable, one might venture, because other people can go and listen for themselves. Similarly, when you personally develop the belief that aspect Z of a given product will tend to have effect Y on the sound, you can go and listen to other products with that aspect and see if they sound different. Please say if I'm horribly misrepresenting you here.

The catch, however, is that strictly speaking, none of these things can render your audio beliefs falsifiable. If someone else seems to hear something, how can that possibly relate to whether you hear it? They are not you, isolated in exactly the same spatio-temporal position - their claim has no logical bearing upon yours. Similarly, if we listen to products with similar attributes and think they sound similar, how can we ever isolate that in the incredible flux of personal experience: you are not hearing the product with Aspect Z being the only variable modified. You are not the same person, strictly speaking, as the person who heard the other product earlier.

Again, I think most would be quick to propose a way (and the only way) out of this: if enough people hear X or Y, or if I personally hear X or Y sufficiently consistently, we can start to talk about the 'thing' that is heard...oh dear! I do believe I just proposed that we need a 'thing' for our discussions about audio to have any real meaning whatsoever. And this troublesome thing falls directly into that troublesome category of 'objective' objects in any meaningful sense of the word (no such thing as an 'objective' claim, but there are certainly things that we must take as being virtually 'objective' in order to, well, think!). And that means the ball lands firmly in the court of the 'objectivist', who can then start discussing evidence, and Occam's Razor, and how immensely plausible it is that A is the case instead of B...

So, to summarise, to say anything about audio at all one cannot retreat into what I've previously characterised as "absolute subjectivism". The moment you poke your head out of that position, even by an inch, you move into the realm where where suddenly ABX testing and the like becomes manifestly relevant. I think I've laid this out pretty clearly, but just to tidy up I'd like you to consider some of the implications of "absolute subjectivism", if we imagine that someone somehow manages to think solely within that sphere.

For a start, from the perspective of the "absolute subjectivist" (and you can't claim to logically occupy a middle ground: as laid out above, you're either wholly 'subjectivist' or an 'objectivist' that just distorts the balance of evidence), there is no inherent difference between "real" and "placebo-y" differences. Therefore, I hypothetically consider setting up "Mr. Willakan's Amp Modification Service". I will take your amplifier and your money, hold it for a while whilst promising vague improvements in sound via technically factual statements ("Some customers have reported vast improvements in treble clarity..."), and then send it back to you with absolutely no modification made to the electrics whatsoever. Everything I know about psychoacoustics suggests that people would tend to report improvements post-'modification'. You see, strictly speaking I'm selling, like all hi-fi manufacturers must be from an "Ab.Sub" perspective, an object from which you derive emotional satisfaction, rather than something based on things like electrons and semiconductors. Thanks to human psychology, creating a process whereby the listener anticipates a difference is likely to produce such a difference: so I don't think I'm doing anything wrong. I'm selling the effect of believing your hi-fi is better, and as every "Ab.Sub" knows, hearing is believing/believing is hearing! If I offer a 100% money back promise if you don't hear a difference, morally I'm surely whiter than white? And don't start complaining about the price: as I so often read in the comments of audio cable reviews and the like, components must be evaluated based on what the listeners hears, not silly things like component costs. Right?

Now, obviously, this is a contrived example, but it does serve nicely to illustrate the problems of the belief system in a solid and faintly amusing fashion. Hell, from my perspective, something very much like what I describe above is already happening (although if it weren't the example would still stand). You don't even need to look at the more controversial products: for example, there is a certain DAC which Stereophile, TAS and lots of other audio mags gave rave reviews to. The designer has a HydrogenAudio account, in which he reveals that as far as he's concerned, his DAC is totally transparent under any semi-sane use case, and such transparency has been possible in both analog and digital domains, as far as circuits are concerned, for years.

Yet, the company has recently released a sequel to their DAC, presumably designed by the same guy. Reading the publicity material, one gets the impression that this new DAC is meant to sound better than the old one, closer to the recording, more accurate...it seems highly improbable the designer believes this to be the case, but then he doesn't write the PR spiel. One could argue that he's selling to the "Ab.Sub" the anticipation of an improvement...

Of course, from where I'm standing, one can make reference to the far less compelling nature of "anticipated" improvements, as shown by the appropriate literature. One can observe that hi-fi has largely died by its own hand, to paraphrase a certain J. Gordon Holt, and as testimony to the nature of these 'improvements' and focuses we have the drastic fall from grace of audiophilia to the point where the most many potentially interested, technically-minded people hear of it is an article in Gizmodo laughing at $10,000 cables. The average consumer may not know much about audio, let alone really decent audio, but they can smell that something's off a mile away, and will usually respond by largely ignoring the area.

Headphones will perhaps yet serve to reinvigorate hi-fi...hence the sadface in my earlier post when I realised what you were saying :(

Tyll Hertsens's picture

I fundamentally agree with everything you've said...and thank you for your wall of text, it was nicely filled with meaning.

In an ideal world, every product review would have measurements, ABX testing scores, and an array of subjective impressions from those tests, but as a practical matter that would be virtually impossible to achieve.

As a practical matter, the least expensive review process is a single, well qualified listener reporting on their listening experience. The vast majority of product reviews are done this way. The next least expensive method of product evaluation is measurements...but it is quite a big step in cost from subjective reviewing only, and it does take quite a bit of effort on the part of the reader to get to the point of being able to interpret the measurements. So the onus of measurement interpretation is on the reviewer as well. The most expensive desirable method of testing would be an ABX comparison study done by numerous subjective listeners. This method, while extremely potent, is basically out of reach for product review publications due to the cost. However, some publications do "round-table" subjective reviews that have some relevance.

So, while I think you're post is correct, one must take into account the practicalities of reviewing in order to come up with compromise solutions. If, as a practical matter, Lavorgna doesn't have the ability to do DAC measurements, does that invalidate his subjective experience? No. Is his subjective experience a complete evaluation of the product? No. Are we ever going to see product review publications do complete product evaluations (measurements, ABX tests, statistically meaningful subjective review studies)? No...it's just too damned expensive.

I've seen the work done by NRC, Harman, and Philips where the do subjective response studies of listening preferences, and, when done properly the relationship between personal listening preference and objective measurement is tighter than many subjectivist audiophiles might expect...but there is variations in listening preference. Which means there is room for subjective reporting to bring meaning to measured performance.  And again, the measurements do not describe the  the listening experience fully, at some point subjective information must be added to be meaningful to prospective listeners.

Lastly, in order to clarify my position for readers, I try very hard to put one foot into each camp. I think both objective measurements and subjective impressions are valid. This does create what I would consider a paradox in my evaluative process, and this paradox cannot be resolved. It exists because measurements and experiences are in two distinctly different domains, and there's nothing, practically speaking, I can do about it. 

Willakan's picture

I'm genuinely happy to learn that you're not in danger in tipping over into the belief system Mr Lavorgna seems to endorse, for it would make me somewhat sad to see Innerfidelity review, say, magic metal resonance cones you litter around the room and place on USB cables and suchlike to...erm...(they sound better OK?)although it goes without saying that what appears on site is not up to me in the slightest!

I still, however, have some concerns, which I hope you'll humour!

Firstly, just to establish, I largely agree with you on the matter of speakers and headphones. Whilst research like that you mention at Harman does go some way to point out that more emphasis on measurements would be good (I mean, their predictive model for speaker preference is pretty impressive, especially considering that simply FR and directivity dominate the 'inputs': 89% accuracy is very impressive, and would shame a lot of reviewers!), not everyone has the facilities to measure speakers, and for explaining the last 10% or so advanced mechanised ABX rooms are rather hard to come by.

There's then the obvious fact that differences between speakers and headphones tend to be rather large. With appropriate listeners and enough time, one can get useful information: the capacity for various involuntary kerfuffles to re-arrange that information is somewhat offset by the provision of measurements. The system you've got going here for that with headphones is, IMHO, pretty damn good, and is sure to get better as more research on headphone FR perception filters out of Harman and compensation can be tweaked accordingly. Would I like more measurementy-blindy-bits? Quite possibly, but I have to be realistic here and remember that the audience does not consist of me :D

Where we invariably part company is over the matter of DACs and amplifiers. If you'll briefly indulge me, I shall briefly look back to the 1970s, where, as Doug Self describes, it was perfectly normal to read amplifier reviews where measurements were analysed and little mention made of listening tests. Indeed, I found a scan of an early 1970s tonearm review and was surprised at the time to see that it consisted of almost nothing but technical analysis - not to say that they didn't listen to the amplifiers, but just that they related what information they thought was important.

Now, did they occasionally go over the edge (say, OK-ing marginal distortion performance by looking at the numbers with little attention to the spectra)? Invariably, but the important aspect was that nobody puported to be hearing anything staggeringly different from the numbers. It would seem there was an aura of reasonable, if not 100% exhaustive, understanding in the hi-fi mainstream.

We then get, as far as I can tell, 'subjectivism' emerging in the mid-1970s. An article Self references shows just how far it stood from the mainstream: "Subjective Sounds" would be limited in one magazine to a single, eponymous column; the writer felt sufficiently insecure to need to justify his approach in his first version. Before you know it, suddenly amplifiers have intricate sound signatures, speaker cables have too, and the world of audio is a largely blank (erased) map waiting to be discovered.

Now, lest you think I'm engaged in a particularly egregrious piece of historical revisionism, I direct you to the AES papers "10 Years of A/B/X Testing" and "The Great Debate - Ten Years On", the latter of which is written by a certain Stanley Lipshitz, who needs no introduction, and the former of which is written by David L. Clark, an AES fellow.

Both papers are retrospectives on how the attempt to resolve the "Great Debate" introduced by the sudden subjectivist explosion went, the attempt being comprised of the introduction of the A/B/X box. As the hi-fi press had furnished little evidence for their new angle (and it is made clear that this is the opinion of the authors), they reasoned that making blind comparisons reasonably straightforward by the provision of a reasonably-priced tool for the purpose would quickly establish what these individuals are actually hearing.

As we now know, the hi-fi press largely responded by throwing blind testing under a bus, which both authors look upon with surprise and cynicism respectively. The narrative has now been rewritten for many audiophiles: a bunch of people wanting to spoil their fun turned up, and on the basis of a few blind comparisons sought to 'prove' that everyone was full of crap, and were righteously dismissed.

The above viewpoint fails to observe that what really happened is that, as far as the serious (and concerned with scientific integrity) practitioners were concerned, the audio press/audiophiles responded to an opportunity to prove their claims, so far ventured on a startling lack of any compelling evidence whatsoever, that the current scientific conception of audio was hideously flawed...by attacking those presenting them with the tools to prove their case.

From then on it's just got worse. Many people would be astounded to learn just how unfounded some of the most fundamental audiophile beliefs of the present really are. The fixation on jitter, for instance. As far as I can tell, the jitter bandwagon was started in the audio mainstream by Robert Harley, who appears to have suddenly decided that jitter was really important/special, and proceeded to largely fabricate a lot of reasons why (although he doubtless believed his own spiel). "The Audio Critic", admittedly not the most polite of publications towards such individuals, bought in an engineer and AES fellow from Analog Devices to provide a more level-headed discussion of the matter: said engineer duly pointed out, with the aid of a fair amount of graphs and mathematics, that jitter was a complex issue, but that it really wasn't that all-important  and that Mr Harley was measuring it wrong.

It is Mr Harley's entirely erroneous conception of jitter which has persisted!

For another example, let's try negative feedback. We are told that negative feedback is bad and that we should, for reasons entirely unknown, avoid it. This 'fact' is illustrated with lurid tales of those amplifiers from Asia in the early 1970s that, despite only exhibiting 0.0000001% harmonic distortion, sounded horrible because of too much NFB. As individuals such as Bruno Putzeys have pointed out, such amplifiers as described here appear to be largely mythological - any number of cheap, bad-sounding SS amplifiers may have been knocking around, but most of them never made it to a test bench and largely fabricated their specs. Furthermore, nobody has ever been able to point out exactly what's wrong with feedback: indeed, Mr Putzeys notes that, as far as the science is concerned, the more the merrier!

Guess which opinion (where the individual has an opinion on NFB) is more common - hell, you don't need to guess, you know!

Anyway, the point of this not insignificant diversion is to demonstrate that, as far as proof and evidence and the like are concerned (and those who do not embrace absolute subjectivism must be concerned with such things), much of what is taken for gospel in audio (intricate amplifier/DAC sound signatures) is propped up on, well...nothing! I am not exaggerating when I say that there is more evidence in the scientific literature to support claims of ESP than there is to support those of audiophiles: true, those papers were found to be hideously flawed and are not representative of the vast bulk of scientific enquiry, but they were, on their reception, found difficult to refute, originating from credible individuals and alleging carefully controlled experiements (the individuals involved were eventually found to have twisted the truth and their test protocols). The audiophile masses lack even that!

At the other end of the spectrum, one might ask: "Where are the papers saying that if THD is less than 0.005%, it will be found to be inaudible?" There are two objections and one observation to be made on this point. The observation is that, at the objectivist end, we are occasionally guilty of oversimplification (rules that will hold almost always, but not quite), in an effort to get points that are 'almost entirely right' across. If NwAvGuy had wanted to argue an absolutely, 100% watertight case, he would have made reference to a vast array of pyschoacoustic research and written long articles on distortion spectra - and nobody would have read any of them (I apologise for presuming to explain to you the art of writing for an audience, but there you go!)

The first objection is to note that the question smells of an inversion of the burden of proof commensurate with the revision of the ABXing narrative. The second, to support the first objection (which requires prior consensus for us to be able to talk of the burden of proof), is to point to the vast array of more generalised electrical and psychacoustic information we've accumulated and verified.

Grab a copy of the foundational textbook "Psychoacoustics: Facts and Models", which seeks to summarise the thresholds discovered across the entire field (and it has undergone reasonably little revision to keep it accurate) and you're presented with masking thresholds, FR variation thresholds, all of which apply in a reasonably straightforward fashion to audio, and all gathered over many tests that anticipate most  objections (eg: idea of golden ears countered with enormous sample sizes and retesting of the best subjects). And because modern equipment can be so good, we can make very bold claims about, say, the O2 and its fidelity to the amplified source. And because the evidence is so overwhelmingly one-sided, when I read that the O2 in fact sounds noticably inferior to another, reasonably well-measuring amp, I'm inclined to put the differences down to volume matching and the nature of sighted comparison (Note that I've personally experienced the sensation that you can hear a difference between something, only to find that the difference sods off once I don't know what I'm listening to: at one point I was convinced that I could distinguish between the choice of ASRC filters on my DAC. Just to reiterate that inherently inconsistent hearing is not something I only ascribe to other people!).

Similarly, on the electrical side, we have signal theory, which gives us things like the null test. If the standard tests run on amplifiers are insufficient...you can see where this is going.

So, I suppose the question from my perspective is that, once one has acknowledged that audio must be a realm in which science has authority, why one would conclude that amplifiers, once you've disposed of things like distortion, FR, output impedance ect would sound different? I'm not 100% sure where you stand; if we're contesting that the differences between amplifiers are indeed shown in the measurements, but that measured neutrality does not predict preference and that very small measured differences can be surprisingly audible...well, it's the same/similar question from my perspective: why would you conclude that with an awareness of other, clearly evidenced potential causes?

To briefly swing round again to the applicability of ABX protocols and the like, is volume-matching and a simple, post-general listening (so they can become aware of any differences) ABX test (accomplished with a few dollars worth of RCA cables, a $20 switching box and a few minutes of someone else's time) really not something that could be pursued? I appreciate there is again an element of Audience =/ Willakan, but even so... you're one of the few guys in the audio press I would trust to pull off some properly-blind amplifier comparison with sensible conclusions (the stuff on headphone burn-in was a model of a reasonable test, done right, with reasonable conclusions drawn, when even one of those things is a fairly rare commodity).

Tyll Hertsens's picture

First, thanks for your comments appreciating they way I work. I'll continue to do my best, and it will remain along the lines of what I do now...only better (I hope) as time goes on.

Second, your post is very informative, but at this point I think I've said what I can say, so I'm going to let it stand as is without comment...other than to say I enjoyed it.

Willakan's picture

Glad to be of interest :D

Errata (as the edit button seems to have left the building): section ending "that the current scientific conception of much of audio was hideously flawed."

Should read "that the current scientific conception of audio was hideously flawed...by attacking those presenting them with the tools to prove their case."

(From Tyll: This edit has been made. Sorry your buttons have gone missing. :( )

There are various other more minor errors, but I'll stop talking now!

Limp's picture

A lot of words, well stringed together.
Too bad it left Tyll somewhat abashed.

Tyll Hertsens's picture

Not feeling "abashed" in the least.

Limp's picture

Oh, I'm just wildly interpreting your lack of a cogent response, you may feel whatever you want to feel. You have my full support.

Pages

X
Enter your InnerFidelity username.
Enter the password that accompanies your username.
Loading