Mastering Your Domains: The Difference Between Objective Measurements and Subjective Listening
Ed Note: This post is inspired by Michael Lavorgna's recent AudioStream article "The Trouble with Listening", and comments he received.
The creepy thing about the photo above is that you have a human that is part robot, and a robot that has human-like intelligence. We're pretty far down the road with computers and have some pretty good stuff these days, but from what I understand we're still pretty far away from putting anything remotely like human consciousness into a machine.
Why? Because the human consciousness is not in the domain of the objective stuff like computing.
Human consciousness experiences sensory input, it doesn't measure it. A major chord is not understood by our consciousness as a series of three particular harmonically related notes, rather it's heard as a whole and as having a cheerful character relative to a minor chord. People can fully enjoy the experience of music without having any objective idea whatsoever about music theory and harmonic relationships.
Now I'll certainly admit that computers can do a decent job of composing music, and many would be hard pressed to blind test the difference between computer and human generated music. And computers can listen to music and tell you all sorts of things about the structure and qualities of the signal, but a computer is not going to adopt a preference for Gershwin over Shostakovich, or tap it's foot when it feels good about what it's hearingthough you might be able to program it to look that way. A computer isn't subject to experience.
The Objective and The Subjective are simply two completely different domains, and while there are relationships between the two, one is wholly unlike the other. While measurements of audio gear are terrifically meaningful, the actual relationship between measured performance and listening experience is quite various in terms of the measurements predicting the nature of how we will experience a particular piece of gear. With things like headphones and speakers, where frequency response deviations from linear are strong, the measurements will relate more strongly to listening experience. If the measurements say there will be strong, tight bass in a particular pair of headphones, most folks will pretty reliably hear it that way. With things like headphone amps and DACs however, that are more linear and have performance measures many orders of magnitude better than speakers and headphones, measurements don't seem anywhere near as reliable an predictor of subjective experience.
Distortion, or Character?
Let me give you an example: NwAvGuy's O2 amplifier measures extraordinarily well. Do some basic listening to this amplifier and most folks will come away impressed with it's very competent performance. But when I spent a lot of time with it, I found myself feeling it was a bit of a dry and uninvolving. Switching over to, say, the Apex Butte amp designed by Pete Millett, I find the sound more musical and bouncyit made my feet tap more reliably. The Butte, while not having as good a set of objective measurements, was clearly more pleasing to my ears. What measures as greater levels of distortion can be experienced as having a more pleasing character of sound.
Herein lies the problem: when we compare measurements, we might report amplifier A has less distortion than amplifier B, and make the assumption that amplifier A must sound better than amplifier B, but it's simply not the case. The added distortion may cause the amplifier to sound worse, but it also may cause the amplifier to sound better, and you can't necessarily tell by the measurements. I'll hasten to add that because we're in the subjective domain here, "better" and "worse", in terms of sonic character, is only meaningful to a single listener. While statistical norms may be found and listener preferences will likely cluster, the fact remains that determination of amplifier character and listener preference is done person-by-person and each person's experience is absolutely valid as their experience, regardless of measured performance.
So Why Measure?
While the relationship between objective measurements and subjective experience might be tenuous, it does exist. Subjective experiences are notoriously fickle, and I personally find that having measurements available as I review a product acts as a stable reference point as I listen to and evaluate a product. If I see big bass in the measurements and don't hear it, I might find myself taking some special care to ensure the pads are sealing correctly on my head, and will take some extra time ensuring that my listening impression isn't somehow influenced by the music selected, or a stuffy nose, or a bad night's sleep.
Measurements are also helpful as it's very hard for most people to get their ears on all the headphones available, so having a database of measurements allows potential buyers to do some preliminary investigation by looking for headphones that measure similarly to those they know are to their taste. This, of course, takes some extra effort on the part of the buyer in order to understand the measurements and their relationship to the listening experience. In the end, however, it's the actual listening experience for that particular person that counts.
For electronics like headphone amps and DACs the equation is a bit different. Because the relationship between the measurements and listening experience is somewhat more tenuous with these more linear devices, measurements have a somewhat different value. The greatest value in amplifier measurements I've performed is their ability to have a small window into the technical competency of the product's design. If measurements show significantly excessive noise, hum, distortion, or crosstalk for example, it causes me to think that particular design is likely flawed, and the designer is likely not very proficient. If I see an amp with extraordinarily good performance in those areas, I begin to feel confidence that the product and designer might both be of high quality. But most products lie somewhere between those two extremes, and relatively little is learned by objective testing.
There are pragmatic problems with implementing a measurement program as well. The equipment needed to make reliable and accurate measurements are painfully expensive and technically challenging to set-up and execute. I'm a pretty decent technician, but it's taking me far longer than I had hoped to get my headphone amplifier measurement program up and running. Objective testing is a distinctly different expertise than evaluative subjective listeningwhich is, to some extent, the whole point of this post: Listening and measuring are two entirely different things.
The Map is Not The Territory
I was inspired to make this post due to Michael Lavorgna's recent post on AudioStream, "The Trouble with Listening", and critical comments thereafter intimating that subjective reviews without objective measurements are somehow incomplete or flawed. I would argue that measurements are a valid mapping of performance, but that map is not the same type of thing as the territory (or domain) of subjective experience.
When I plan a motorcycle ride through the forest, I will consult numerous maps: topographical maps will tell me about the steepness of trails; road use maps will tell me where I can and can't ride; land management maps will indicate where I may be legally allowed to camp. But the best source of information is dialog with another rider who may have been in the territory and can deliver a scouting report. Stuff like where mosquitos might be, and where there's a scenic camping spot may be brought to light. The maps just aren't going to tell you about the experiences available in the territory.
Subjective reviews are like scouting reports. They tell you about the territory of the actual listening experience with a particular product. Objective measurements, while informative and related, are relatively mute when it comes to describing that. When Michael Lavorgna says in his piece...
I would start by asking a question—Is listening sufficient to determine the worth of an audio component? Or to put it another way—Is listening sufficient in determining what we enjoy listening to? Well if we put it that way, of course it is.
...I think he's right.
Sure, I'd prefer it if Michael had a killer measurement program up and runningwhen I go on a moto trip I take as many maps as I canbut, for a variety of reasons, that's just not the way it is at the moment. Does the fact that he doesn't have measurement program somehow lessen the value of his subjective review offerings? Hell no! The subjective review is at least in the same domain as the listening experience I'd have with the same gear.
Dump a fat stack of measurements on the table and I'd surely give them considerable attention, but when I was done I'd still have a burning desire for a scouting report. That's why for Michael, and me, and John Atkinson, and all the writers in the Home Tech group, we feel the single most important thing we can do is bring you our subjective listening impressions. Measurements are groovy, but it's the stuff that happens in the domain of the subjective listening experience that really counts.