Big Sound 2015 Wrap: What I Learned

When I came to the conclusion that the considerable amount of gear assembled to test the latest high-end headphones must be shared and began inviting people to join me, I really had no idea of how to conduct the experience of participants. As I thought about it, I came to the conclusion that providing some sort of mental reset button to tamp down expectation bias and allow participants to evaluate the gear with an open mind would be cool. I wondered how to do that.

As I said on the home page teaser for this post, "WARNING: This is not headphone science. This is just a very, very complicated headphone anecdote." I've got nothing to prove here, not because I wouldn't like to prove a few things, but because "proof" is a heavy burden—one that can only be addressed by real science. I'm simply not qualified or equipped to offer proof positive. What I can do is hold a somewhat structured listening event and report on my observations. These are my subjective observations and amount only to my opinion and what I personally learned and took away from the experience. I hope the objectivists among you will read this article in that light.

Blind Tests
My ulertior motive for including some blind testing of participants was to shake people up a bit and show that the objective magnitude of difference between a lot of this gear was very small. There are a couple of reasons for doing this: One was to give people confidence as they evaluated the wide array of headphones that the amps and DACs up-stream (within reason) would have litte effect on the natural sound quality of the various headphones. The transducers themselves are, by far, the biggest variable, and the fact that they were driven by different DACs and amps would not substantially alter the inherent character of the headphones. The other reason was to disabuse folks of the idea that amps and DACS made a huge difference in sound quality—and much more so, the differences due to cable and power conditioning/regeneration. (After participating in the initial blind test I would often offer folks the opportunity to blind test the differences between cables. They would often laugh and say, "No way! If I can't tell the differences in amps with the HE-1000, I'm certainly not going to be able to test the differences between cables.") It is very important for entusiasts not to over-enthusiastically claim huge differences in these areas—it makes objectivists crazy...and rightly so.

For example, one excited participant came in obviously having read a lot about the gear and had a number of pre-conceived notions about what he was likely to hear and the supposedly large differences between various products. After an hour of blind testing he openly commented that his preconceived ideas needed to be thrown out—the differences in the gear was much smaller than he had previously believed. Eggselent!!!

The other thing I learned about blind testing was that it essentially made you blind to the small differences between gear. Objectivists will claim blind testing removes biases, but I think you just trade one set of biases for another. For example, most folks experience significant anxiety when faced with the small differences during blind tests—a lot of self-doubt can apear in the listener's mind, which leads to confusion and indecision. Also, when a listener begins to believe they've identified one of the selections, they then begin to superimpose a bias on that selection. The only way to combat these issues, it seems to me, is long experience with the technique of blind testing—and even then it's very hard. If you ever read some pop-press article about how they blind tested a bunch of people and found there was no difference between an iPod and a $1000 digital audio player, don't believe it. Joe Blow off the street is going to be completely lost with this kind of evaluation. So, the Big Sound blind tests demonstrated that the differences between gear is small.

Sighted Listening
Despite the difficult blind listening experience, the Big Sound participants went on to sighted listening tests, and over the course of the remainder of the day would express fairly clearly their preferences and reasons for preferring one amp over the other. An objectivist might say that a biased opinion is easy to come by, but I have to note that even though the sample size is too small to be conclusive, there was a reasonably strong similarity of impressions among the participants. Sure, the bias against the Burson might be chalked up to the loose volume knob, but there was fairly broad agreement that the TTVJ Teton was a great amp—despite its homely looks next to the Woo mono-blocks. Many participants observed the Eddie Current Black Widow sounded somewhat "tube-like" and was quite pleasant to hear, even though it was placed off to one side and wasn't particularly cool or impressive looking.

There were some notable disagreements, but often they could be chalked up to personal preference. For example, Bob Katz described the SimAudio Moon 430HA as "overly soft and fake-tubey", but I hear it as neutral and even-handed; he found the HeadAmp GS-X Mk2 "excellent transient clarity without any high end exaggeration", where I found it a bit tipped-up and a tad brighter than I prefer. Though we disagree from our respective perspectives, we clearly agreed on which amp was faster sounding.

My point here is that blind testing fundamentally showed that people found it extremely difficult to tell the differences between gear; while sighted evaluation allowed people to develop preferences—albeit in accordance to their personal taste. Why is that?

I think humans want to develop relationships with things. "Hey, that's a chair. I can sit in it. It's light enough for me to move. It doesn't look comfortable." and on, and on. Humans make a coat hook in their head for an object, and then start hanging opinions about it on the hook. When we do blind tests we are without the hook. We have a hard time relating with—evaluating—the object. With sighted listening we get the hook back and can begin to develop a relationship with the gear making the evaluation more natural.

Small vs. Subtle
Still, I remain convinced that the differences between amps, and more-so DACs, is objectively small. If so, how are people able to relatively quickly and easily develop preferences? Here's my thoughts:

I remember as a teen walking through the Museum of Modern Art with my father and rounding a corner to see a large Picasso. It took my breath away on first sighting...stunningly beautiful. I'll observe that not everyone would have this response—some will have taste that runs counter to that particular esthetic, and some simply won't have developed sensitivities along those lines in their visual cognition. But for me it was striking.

The thing about great art is that the artist has extremely well developed technical skills and is able to create nuanced and subtle detail. But the thousand subtle details that go into the painting come together as the coherent whole that speaks to our being. The impression we receive occurs without having to observe each detail, but rather comes to us in an instantaneous grok. We relate directly with the character of the complete painting. Yes, we can dig deeper into the painting and come to appreciate the details, but the more important relationship is to the artwork as a whole.

Similarly, I think when we listen for pleasure to a piece of gear we are neither attracted or off-put by the individual details, but rather, experience the sound as a whole. But the sound, as a whole, is made up of all it's constituent small, but subtle, details. My point here is that while individual subtle details may be difficult to consciously discern, the mind will subconsciously sum the myriad subtle but important details and sense the overall character of the product. It is that experience which allows us to come to personally valid impressions.

I need to add that there is also a time dimension to the relationship with a piece of gear. Over time our mind accommodates to the sound of something. The brain modifies its perception to normalize the sound it hears. Because no headphone is perfect—heck, it's actually a completely artificial way to listen—our mind must always make an accommodation. Sometimes the accommodation is easy, and we find ourselves with a piece of gear we like perpetually; and sometimes the accommodation is not so easy, and we find ourselves with an acoustic burr under the saddle...and gear is soon to be sold.

My point in all this is: Small details add to the whole in ways that make them more important than their objective magnitude.

(As a side note, I highly recommend reading this post by Mike Moffat (Baldr), of Schiit Audio, regarding his experience with blind testing.)

A Special Note About Cables
While I think that cables can make a small difference in sound, and while I think small, subtle differences can provide an improved magnitude of enjoyment larger than the objective differences might indicate, I do think that on the whole far too much emphasis is placed on cables by audio enthusiasts. The reason for this is understandable.

Enthusiasts want to tweak their gear; they want to participate in the creation of the equipment that delivers them so much satisfaction. Nothing wrong with that. But it amounts to an insult to the engineers who design audio electronics when enthusiasts place so much attention on cables without regard for the complexities inside the box. Most enthusiasts are not aren't qualified to open the lid and start modifying the gear, so the only thing left for them to do is swap cables and such.

The big problem for me and my objectivist friends is when this cable swapping activity (and all manner of other tweaks) becomes excessively emphasized as an important method for improving the sound of ones gear. Worse yet, this mindset is often encouraged by cable manufacturers claiming dramatic improvement. I very much wish cable makers and enthusiasts alike would dial back the rhetoric on the benefits of expensive cables. Yes, they can make a small but worthy difference, but choosing electronics and transducers that satisfy is the more important driver of sound quality by far. Once the gear is sorted, time spent with folks like The Cable Company—who have a program to ship you loaner cables for audition—is a reasonable last step in your journey toward building a satisfying listening system.

Personally, I would look at the low cost lines from AudioQuest, Blue Jeans Cable, or Kimber Kable for your initial cabling needs, and then wait until your system is complete to audition more expensive conductors.

Conclusions
Big Sound 2015 taught me that blind tests are really only useful for professionals evaluating small details, and really shouldn't be used by casual listeners or enthusiast n00bs. It's better for most folks to keep an open mind about gear; reduce your desire to produce pre-conceived notions from your forum reading; and resist the temptation to jump to any thoughts that there are huge differences between amps, DACs, cables, break-in, and the like. Please be careful about exaggerating your experience in listening comparisons; they're usually not that big; they'll often be specific to your tastes and sensibilities; and they may skew the experience of others and lead them astray.

I also learned that while we may be hearing the same piece of gear, each individual's impression of it may be different based on both physiological variance and in matters of taste. Just because someone's carefully formed opinion is different than yours doesn't make one of you right and the other wrong—at least, insofar as relatively good kit goes. However, there are junk products out there that should be known as such to all.

Next, and last, Big Sound 2015 post: My take on the gear itself.

COMMENTS
ashutoshp's picture

And very gutsy. That post by Mike moffat is absolutely outstanding too.
I think one of the unexplored aspects in this hobby is the actual biology of the human ear. There are just way too many assumptions floating around.
I think manufacturers need to invest money in understanding the ear. There are so many smart scientists out there desperate for funding. It's a win win for both parties.

OBJECTIVIST's picture

The results of the blind tests aren't worthless just because the participants were worried that they wouldn't be able to back up any claims they made have made about the equipment with hard evidence.

NA BLur's picture

Thanks for your summary. Are you going to post any of the aggregated data in graph form? I am primarily interested if there is any overlap in preference as well as any conclusions we may be able to draw, from the data, regarding blind testing.

Tyll Hertsens's picture
I'll see if I can do that for my next post.
ab_ba's picture

This post really cuts through a lot of nonsense on both sides. It's gonna make some people grouchy - but it shouldn't. I take these comments as a rallying cry to headphone enthusiasts to cool down some of the rhetoric - on both sides. I think the stakes are high - if the market for headphones is going to grow (think how many people will spend on big TVs or the newest cars, all the research they do before buying) we’ve got to stop saying things that just strain credulity. When the market for good headphones grows, the tech will improve, and we all win.

Mrip541's picture

Great series! Sorry for the whinge.

castleofargh's picture

"Big Sound 2015 taught me that blind tests are really only useful for professionals evaluating small details, and really shouldn't be used by casual listeners or enthusiast n00bs."

I understand the warning, hate the advice. I agree that people with no experience shouldn't be so quick to draw conclusions from their blind tests. same with measurements, noobs will misinterpret measurements, it's a given. and same with learning how to swim. it doesn't go perfectly on the first attempts.
how can they hope to stop being ignorant noobs if they don't learn and experiment?

the warning not to be overly enthusiast from sighted evaluation is very good, but it doesn't help much in practice. it's been demonstrated that knowing about bias didn't stop us from being influenced by them(tell me about it :'(). so being careful doesn't help. removing as many biases as possible does. at least if the aim is to get one little step closer to truth.

Moffat's post:
"So much for the blind A/B instantaneous naysayers. All that matters is frequency response, they say. People can't hear anything much above 20KHz in their prime, less later. The ear has a short memory, it is all bias, blah, blah. They should take up a different hobby, say stamp collecting."
//////////////////////////////
how can this be seen as anything but pure demagogy BS post? irrelevant amalgams, straw man argument, unrelated conclusion based on one poor anecdote, and all for the benefits of audio... just kidding, all just to badmouth objectivists. oh yeah what a great post deserving of a link!
let me try: the dog is barking, the moon is high, therefore objectivists are idiots. there you go. I really hope I'll get a link too, next time there is a talk on the matter.

short instant switch is not so much for hearing, where obviously listening longer means more DATA to process and more time to think about it. it's because of how our memory starts to make stuff up after 4 to 10seconds. the accuracy of what we recall drops like a stone after that. so I can only think of a very limited number of things to test, where sacrificing memory accuracy for something else will be preferable. one, would be to find out if the sound is tiring. it's easy to understand why. but dismissing a proper method that works for many applications because of one counter example, that's just irrational.

short passages and almost instantaneous switch(which from what I understood you didn't always provide for big sound 2015), are the starter kit of trying to detect audible differences. it should be at least part of any serious gear comparisons.
nobody said it had to be the only test or the only source of intel. but it works pretty well(when used to it), and it has the huge benefit of providing evidence when a positive result is obtained. uncontrolled listening provides zero evidence whatever the result and can only bring an opinion or a feeling. never a proof or certainty of anything.

if subjectivists had an actual listening method that was effective, objectivists would obviously use it too. the point was never to be an objectivist or to glorify blind tests in all their forms. I have enough stupid groups to join on facebook and feel special already. the point is to try and get closer to the truth by whatever means available. if possible convenient, that don't cost a kidney. in exchange for those compromises, we do lose a little in reliability. it's in the game. but it's still a good upgrade compared to nothing at all done with no control.

in conclusion, when I need a screwdriver and use a shovel, it's not OK for me to just go and blame the shovel for not being a good screwdriver.

“Science is what we have learned about how to keep from fooling ourselves.”
Richard Feynman

Tyll Hertsens's picture
"it's because of how our memory starts to make stuff up after 4 to 10seconds"

And that exactly why, among other things, blind testing is such a limiting form of listening. But I'll agree with you that enthusiasts should become familiar with the protocol to improve their listening skills.

xnor's picture

I know you've heard this countless times, but the only limitation is that which you impose on the listeners. Blind testing itself is not limited to 10s excerpts or some track you hate or using blindfolds or trials limited to a few minutes. ;)

I find the "I need a few weeks or months to distinguish components" quite an interesting excuse.
How on earth would you know if after weeks of listening you really hear a difference? What is the control?

What is the difference between that claim and some person claiming that they have healing crystals, but the effect can only be felt after weeks or months?
Chances are high that you'll just convince yourself that your investment was worth it - of course, you will focus more on how you feel and attribute feeling better to your investment.

Now if you really hear a small difference, I found a score of 75% to be the borderline of where you start asking yourself whether you are hearing any differences. Of course such a score is meaningless without statistical significance. Arguably the shortest test with lowest permissible score would be 10 out of 13 trials (= 77%), and only with 16 trials we get to exactly 75% as a minimum.

Of course you can pool results from multiple listeners for that (without cherry picking!), but given enough listeners you can make any score >50% statistically significant.
So you should look both if there was a difference (significance) and if yes then how big it was.

Tyll Hertsens's picture
You do recall, I hope, this paragraph above:

As I said on the home page teaser for this post, "WARNING: This is not headphone science. This is just a very, very complicated headphone anecdote." I've got nothing to prove here, not because I wouldn't like to prove a few things, but because "proof" is a heavy burden—one that can only be addressed by real science. I'm simply not qualified or equipped to offer proof positive. What I can do is hold a somewhat structured listening event and report on my observations. These are my subjective observations and amount only to my opinion and what I personally learned and took away from the experience. I hope the objectivists among you will read this article in that light.

xnor's picture

Yes, of course, and I'm grateful that you post things like that.
I am not saying that if you don't do "real science" it's wrong and useless.
I am honestly curious about the answers to my questions. The rest was a comment on statistics.

There's one more thing I'd like to comment on: I find the subjectivist/objectivist categorization ridiculous.

Whatever your subjective preferences are ... sure, go for it. Someone's hearing may be an outlier and I wouldn't want to force that person to listen to something that he/she don't like. But whenever a "subjectivist" states that component X has sound signature Y, then he/she is making an objective claim.
"Real subjectivists" should ignore any reviews, after all, it's just their own personal subjective impression that counts. Reading reviews would create some kind of expectation bias and probably skew their pristine experiences.
They also should have no problem preferring a much cheaper component to some expensive flagships, if they like it more, and they also should not get upset if you show them abysmal measurements of components they like ... and yet they do.

That's just fake.

I'd even be fine if someones was a "real subjectivist", but without any controls you will go down a spiral of delusion.

What I'm not fine with is that people don't know this, don't think about it, and then there are suggestions like not to do blind tests...
:(

Tyll Hertsens's picture
"I am honestly curious about the answers to my questions. "

Okay.

"I find the "I need a few weeks or months to distinguish components" quite an interesting excuse. How on earth would you know if after weeks of listening you really hear a difference? What is the control?"

There is no control when the point is personal enjoyment. The metric is internal and on axis where a "control" is somewhat meaningless.

"Chances are high that you'll just convince yourself that your investment was worth it - of course, you will focus more on how you feel and attribute feeling better to your investment."

I agree that this problem skews forum impressions significantly. The real answer, I suppose, is people are somewhat random and irrational. Even good science can't cure that; how many readers here have paid for access into the AES journal library?

It's a jungle out there. We just have to deal with it.

"There's one more thing I'd like to comment on: I find the subjectivist/objectivist categorization ridiculous."

Ha! I agree. I was having dinner with John Atkinson some months ago...just he and I. I've known him for 25 years, but never spent more than ten minutes one-on-one with him. It was great. At one point I asked, somewhat tongue-in-cheek, "Do you consider yourself a subjectivist or an objectivist?" We both had a hearty little laugh and went right on with the next subject. It is a stupid, dividing construct to comply with.

""Real subjectivists" should ignore any reviews, after all, it's just their own personal subjective impression that counts. Reading reviews would create some kind of expectation bias and probably skew their pristine experiences."

The problem is that it's hard to hear a lot of the gear out there...even I have trouble doing that. Having voices that you can trust and characterize does allow you to do a little sorting.

"They also should have no problem preferring a much cheaper component to some expensive flagships, if they like it more, and they also should not get upset if you show them abysmal measurements of components they like ... and yet they do."

That's really two points: There are plenty of cases where cheap gear is better then expensive gear, so price doesn't really enter the equation. As to liking gear that measures poorly, I think you have to be very careful to sort out the difference between "distortions" that are euphonic and desirable for some, and distortions that indicate crap sonic quality. That line can get pretty blurry at times. I will agree with your overall point, however, that people can be fooled. (See: It's a jungle out there, above.)

"What I'm not fine with is that people don't know this, don't think about it, and then there are suggestions like not to do blind tests...
:("

Sometimes you just have to push yourself away from the keyboard. "No Fucks Were Given" is an awesomely therapeutic stance.

romaz's picture

I so agree with this one, Tyll.

Jazz Casual's picture

Common sense is in short supply in this hobby and the wider audiophile world. Your contribution is gratefully accepted thanks Tyll.

Gegliosch's picture

The part about blind and sighted testing caught my interest. So apparently it is easier to develop preferences when you know what you're listening to. I would like to share my thoughts as to why that would be:

Sound is transient and it is very hard to memorize a certain sound signature for future comparisons. I would imagine this takes huge amounts of an untrained listener's mental resources. This should be true for both blind AND sighted listening.

However, in blind listening you need even more resources, because you can't just attribute your experience to one piece of gear and move on. You have to memorize a whole series of experiences and compare the details before you can guess what sound belongs to certain equipment.

I think people don't merely like a mental coat hook to build a relationship. I think they NEED it to free some of their capacities. It makes it easier for them to notice and memorize details or experience the sound as a whole.

I hope I'm making sense - this is kind of hard to describe in a foreign language ;)

detlev24's picture

https://youtu.be/01V_7WamALM

It is about loudspeakers, but I like their blind test/listening room.

That is an excellent approach, although it is harder to realize with headphones.

krikor's picture

See the link below. They are trying to do the same with headphones, and like speakers, their findings thus far are that what sounds good does not vary by listener’s experience, age, gender and culture.

http://seanolive.blogspot.com/2014/10/at-recent-137th-convention-of-audi...

johnjen's picture

Eggsellent summation!

And yeah some folks's opinions are likely to get ruffled, such is life, especially for us audiophools.

For me it boils down to, is our subjective experience enjoyable?
And this is independent of the specifics of the gear chosen, indeed in some cases it's in spite of the gear chosen.

Music is to be enjoyed, so when the fun goes missing I usually just turn up the volume and get carried away by the music.

And so it goes.

JJ
ps I laughed out loud when I read MM's take on bind testing when he capped it off with "stamp collecting" hahahahahahahahahha…

ab_ba's picture

Indeed!

The most rigorous standard for evaluating gear is if a n00b can distinguish it from a less-expensive component on a short listen.

Next most rigorous is if a reviewer or an experienced audiophile giving equipment an hourlong listen in a blind test can distinguish one from another.

Less rigorous - but still real - are sighted comparisons. There are specific and repeatable differences that the listener experiences.

The least rigorous way to evaluate gear is the most important: how much enjoyment of your music is it providing you after you’ve used it for hours on end for a few years? This is the domain where psychological accommodation, burn-in, and cables exert their influence. These judgements aren't formed in comparision to other equipment - at least I really hope they aren't. You'll never be able to just let go and enjoy your music.

A pure objectivist stops at the first standard. Everybody else is comfortable with some degree of subjectivity in their pursuit of musical enjoyment.

Johan B's picture

"Big Sound 2015 taught me that blind tests are really only useful for professionals evaluating small details, and really shouldn't be used by casual listeners or enthusiast n00bs"

So now I am quite happy with my cheap TEAC UD-H01 DAC and my Philips X1. I am not going the hear the difference with a more expensive DAC. I have saved a lot of money by reading this article.

It is only professionals who should buy the gear that is more than $350 because only these chosen ones can hear the difference. This article has killed off an entire industry and also my interest to hear rambling on about jitter and distortion. The only thing of interest to me now is the target response curve. Surely this must be audible to n00b?

detlev24's picture

Of course it is audible, once you know what it sounds like and that is the difficulty. IMO, there simply is no headphone available that emits a flat frequency response.

If you have the possibility, listen to good studio monitors (JBL Professional, Genelec etc.) somewhere, where they are correctly set up (that includes acoustic room treatment). Listen to good recordings (you are familiar with) and you will get to hear what the mastering engineer wanted you to hear. In the best case that is what the microphones captured during an excellent recording session. Now, you may like it or not. Certainly you will have to adapt to it and regarding your X1, they are more on the bass pronounced, thus fun side. That means bass might sound - depending on the recording - too soft to your ears in the beginning.

Johan B's picture

Tyll is right about the Headphone being the point of variation. The issue remains on how an amplifier handles the cans phase turn and impedance. Yes any n00b can hear the difference of drivers and poor drivers.

The X1 may sound fuller bass but I would argue that many cans are not bassy enough. if I listen to a couple of KEF reference speakers or studio monitors then I must say that it is not exxagerated at all.

The issue with the X1 is that you can actually hear the low end disortion at higher levels. At lower levels this may actually color the sound as a bit more woolly. Now you read reviews on the X2 that philips has been brave by going somewhat bass light compared to the X1 and competition.... I think they just engineered out the disortion. The frequency response curve comparison and disortion curves seem to confirm that. I find all this again a compelling argument to buy plannar cans as these have limited disortion.

Goran's picture

Hi Tyll,
I've been reading this site for about 3 years, and appreciate it as one of the very few objective and truthful places about audio in general.
Now I must admit I was not overtly enthusiastic about headphone listening (I am 46 and have grown up with speaker sound), but based on your honest reporting and great enthusiasm I bought my first headphones (NAD VISO HP50) last December, and the experience is different, but still great!!!
Now, keeping in mind that I listen to music (flac files) from my smartphone (HTC One M7), I need your opinion about whether I will experience a small/subtle/big jump if I buy a headphone amp/dac combo from the Big Sound selection.
I understand that the differences between amps and dac's are small, but I never read about subjective or objective difference between good smartphone and desktop amp/dac.
Keep in mind that the Viso are sensitive enough for my phone (I never have to turn up the volume more than 10-12 out of 15), and that where I live, I can not experiment with the gear, so if I buy, I will buy based on reviews and recommendations, as I did with Viso.

Best rgds and keep the excellent job!

bogdanb's picture

I also bought the Nad Viso HP50 based on the article without being able to hear them, but I listened to a lot of cans. I'm not disappointed and I am also recommending them. (some issues with build quality, I had to service them, the cable connecting L to R broke and some metal part, but solved relatively cheaply)
They don't scale up if I remember correctly from the article and with my little knowledge I would say the same, you can't really hear any differences.
I play my headphones from my iPhone 5, or from my receiver (yamaha). I won't bother investing in an external amp.

Now I only wish for a good in ear thing, so I could listen to music when I go to sleep or when I lay down and resting my had on a pillow

ab_ba's picture

I also purchased the Nad Viso because I wanted on-the-go headphones. I would say the difference between the Nad Viso and anything in the room at Big Sound is not small. It's big. I think when Tyll was talking about small differences he meant among the equipment at Big Sound, not just beteween any two pieces of equipment. He's not saying there aren't plenty of big differences out there. Now, the Nad Viso out of a smartphone is a perfectly wonderful experience. The LCD-3's out of a dedicated amp will be a big improvement in sound - but for a huge multiplier in price...

bogdanb's picture

If price wasn't an issue... I might bought something else
well maybe. At the time there weren't many good cans with microphone iPhone compatible, and I would have really hated to have an awkward experience when receiving a phone call while listening to music.

Question.
Does any of the portable amps support the call function / mic on the headphone cabe?

Jim Tavegia's picture

I still teach Math in middle school and still learn new ways to reach my kids every day. I teach a very modest digital audio class to get them to think about how important mathematics is to them every day as they listen to their "Digital Music" as the ones and zeros are pretty magical. At first they think I'm kidding, that is how little they really know about the importance of math. The all generally wear the right "size" clothing to school, so on some level they or their parents get part of the numbers game.

I do consider myself an audiophile, but don't spend crazy money on my gear, but have enjoyed the "recording" part of my being an audiophile much more over the last 15+ years so having Bob Katz a part of this has been great reading for me as I really respect his work.

So my modest recording gear consists of a pair of Tascam DR-2ds in 2496 always and my older DR-07 (redbook) SDHC card recorders. I have a couple of small Mackie mixers that have decent VLZ3 mic preamaps and make very nice recordings with them and my assorted Rode, AKG, and Sennheiser mics.

I have added this year a new Yamaha MG-16XU mixer that also had 24/192 usb recording ability. It sounds nice as a usb playback device, as good as my Steinberg UR-22 (up to 24/192) USB computer interface I use everyday for my music files. With it I have paired my new Tascam DR-680 mk2 6 track recorder that can do 6 tracks of 2496 or 2 tracks at 24/192. This thing sounds so good for only $599. I have had great fun now just tracking recordings, keeping my levels below "0" and then running those tracks into my Yamaha MG-16 and mixing and mastering those tracks for proper stereo placement and adding effects as needed.

My headphones this year added a pair of Focal Spirit Pro's and a pair of AKG k271 mk2's to my 2 pair of Sony 7506's, my AKG K-701s, and set of Shure SE-215's for music on the go. All of this has upped my game and I now have a much greater sense of how hard it is to make great recordings and how much you must agonize over all the details to even attempt to do it right. Your exercise had made this more illuminating with all your expensive gear and even the smallest change has made improvements you have noticed. It all matters.

I had never been into power conditioning, but this week I bought an affordable Furman PL-8 through Ben Porter at SweetWater Sound off his recommendation as he has owned both Monster and Furman. PS Audio is out of my league. It took me all of about 10 seconds to hear a clarity I had never heard before from my Tascam DR-680 mk2/Yamaha MG16 rig. It was not just subtle. How could this be for only $192.00 ? Is my ( and our) AC so dirty that something so affordable can make such a difference? Evidently so. It may be the best $200 I'll spend this year.

I also had owned an ART Headphone 4 and a PreSonus HP4 headphone amp stations with 4 headphone outs with independent vol controls for each. The Art is good for $65 and the PreSonus a step up in clarity for $129. I added this week a Art Headphone 6 Pro with 6 channels of headphone amps with controls for each and added bass and treble controls for each channel...again a huge step up from the PreSonus, but now each performer can adjust the EQ to their liking. Each of these headphone amps benefitted from being driven through the Furman PL-8C in a major way. I would have never guessed it.

So it is clear that with great cans and hearing deeper into the music that everything matters including how clean your AC is as no one gets closer to the music than headphone listeners.

All of this Tyll has done for us is more than just a science experiment for him and I am so glad he has shared all of this with us. It has not only moved me onto consider all that I do in my listening, but also has benefitted my recordings greatly. Thanks for all your hard work.

Jim Tavegia

detlev24's picture
Seth195208's picture

This is the best article I have ever read on this subject.

Alex Halberstadt's picture

Tyll, the one notion overlooked in this otherwise superb article is *why* we listen. After several decades of audiophilia, what matters to me—these days it's about the only thing that matters to me—is how well a piece of equipment can get me submerged in my music. To connect to the art of the great musical artists... or even small-time musical artists. And the ability of a piece of electronics to accomplish this, in my experience, doesn't have all that much to do with the way it *sounds*. This ability is certainly not something discernible in blind tests, where the anxious mind begins searching for small differences in sound and becomes almost indifferent to what's being played. This is a distinction you described very well in your Pono review. Why does that modest plastic device make listening so immersive and fun? The answer is probably not something a blind test, or even a frequency response graph, will reveal.

Jazz Casual's picture

But wasn't Neil Young's objective for the Pono to raise the bar for sound quality in a portable music player and make it accessible to a wider audience?

Tonmeister's picture

Tyll

I think this series has provided a great public service helping to educate people about the diminishing audible differences as you go from transducer (headphone) to amplifier to DAC. Congratulations!

When the audible differences become difficult to discern that is where I think controlled listening tests have a the biggest role because other factors (level differences, auditory memory, expectation biases, etc) can swamp out the small effects you are trying to measure.

The notion that blind tests make us less sensitive to measurable differences has been addressed in an experiment I conducted back in 1994, which was reported in an AES preprint "Blind vs Sighted Listening Tests" We did a blind test on four loudspeakers and repeated the same test sighted a few days later using Harman employees. In the blind tests listeners responded to loudspeaker effects, as well as program and loudspeaker position effects. In the sighted tests, listeners were influenced by knowledge of the speakers they rated (the scores when up for the big expensive JBL products) but were less responsible to program and positional effects. When they saw the products you could modify the sounds (via program and loudspeaker position) but listeners didn't change their ratings.

The visual clues of the loudspeakers clearly affected their expectations and sound quality judgements.. They were less sensitive to changes real changes in sound quality when the products were in plain sight, I suspect in the sighted tests, people knew they were listening to the same loudspeakers and decided to give the same ratings regardless of where the speakers were positioned or what programs were played,

see http://seanolive.blogspot.com/2009/04/dishonesty-of-sighted-audio-produc...

SonicSavourIF's picture

I also don't see that phsycological stress is removed when you do sighted testing. When testing sighted, you might feel the same stress and fear of not hearing all the differences in the gear everybody talks about in articles and reviews, that is, fearing in essence that you might not have those exceptional ears you thought you had. So in sighted listening you have the same perceptional mind problems getting in your way that you have, when you blind test. However, in sighted listening, you get all the well known bias effects on top of these problems.

Because it hasn't been mentioned in the comments yet. Regarding the „the details add up to an overall quality which is indistinguishable in a test that focuses on details“. This claim can be assesed by doing a null test with the signals, thereby elliminating the listener as a measurment factor: If the difference of two signals is noise weaker in level than, say -100dB, there is no reason to believe that one signal has a hidden quality that the other signal is lacking and that will be audibly distinguishable.

Finally, I don't even understand why people WANT to believe that they miss a lot in their music if they don't spend thousands of euros/dollars on gear.

It is a RELIEF that you don't have to!

To me the outcome of Big Sound 2015 (thanks for all the effort and work!), as well as all the scientific oriented Books and papers I have read (S. Olive, F. Tool, E. Winer, C. „Monty“ Montgomery, NwAvGuy and others), amount to this: Apart from headphones, you actually don't need to worry to much about which amp or which DAC you use. Especially, there is no need to purchase insanely expensive gear. And that is phantastic news, because we have awesome sound out of reasonably priced gear. Life is good. Hurray.

dweeb4's picture

A couple of questions, Sean. Apologies if you've dealt with these in the AES paper - I don't have membership of AES

- did the participants previously undergo Harmon listener training?
- were they evaluated for normal hearing acuity & discernment?
- were any controls/calibration used?
- I'm thinking of something like adjusting the amplitude until a difference was perceived sighted Vs blind? This might give an idea of the sensitivity of each type of listening.
- the point being made by Tyll is that long term listening is more discerning than short term - your tests were all short term, right?
- any tests done on long Vs short term blind listening?

Tonmeister's picture

Answers
1. Most of our tests are done using trained listeners with normal audiometric hearing. We often also tested untrained listeners to make sure their preferences are similar to those of trained. So far, we've found they like the same loudspeakers, headphones and codecs as trained listeners. They simply give less discriminating and consistent responses. Untrained listeners also give higher ratings in general,
2. Answered above
3. Most aspects of the test are controlled. We sometimes include a reference or anchor it that is what you mean.
4. Our How to Listen software measures listeners' ability to discern and identify distortions added to music. So there is a way to quantify how good you are as a listener.
5. I think long-term listening may reveal certain problems that aren't apparent in short-term tests. Like a buzz or rattle that wasn't apparent because of different source material used

However, there is also contrary evidence that people adapt to sonic impairments of speakers and headphones over time. We don't hear the impairments until we compare the speaker to a better model. Also, being able to switch between different headphones or loudspeakers allows listeners to focus and hear differences related to the headphones and not the program material. Why? Because human perception tends to ignore stimuli that has constant features and focus on things that are changing. The program and to some extent the room acoustics fall into the background of perception Long-term listening usually implies single stimulus listening and I believe we are less discriminating under these conditions.

6. Have't done much long-term listening because of the reasons stated above

dweeb4's picture

Thanks for the answers.

I don't agree that long-term listening usually implies single stimulus listening - I'm pretty sure it is done to get a handle on the characteristics of the "new" sound & then to remove it i.e revert back to the previous sound (which had previously been subjected to long term listening) to judge any difference/preference.

I would agree with your statement "I think long-term listening may reveal certain problems that aren't apparent in short-term tests" but would not agree or limit it to the issues in your follow-up statement "Like a buzz or rattle that wasn't apparent because of different source material used"

In relation to your last point about long-term listening "issues", I note that Toole has observed that we all seem to gravitate towards the same speaker sound (the smooth freq response on & off axis). He surmised that this was due to some reference of sound that we have internalised. Is this internal reference sound not built up over time & therefore could not be subject to the issues you state are inherent to long-term listening (otherwise, it wouldn't be a common reference that we all share)?

Jazz Casual's picture

where the differences in sound are in all probability even less significant. Is your objective testing confined to loudspeakers only?

Tonmeister's picture

If your comment is directed at me, I've done blind tests on loudspeakers, headphones, automotive audio systems, amplifiers, up-mixers, MP3 enhancement algorithms, stereo 3D enhancers, and even speaker cables.

Jazz Casual's picture

Thanks for replying. Are your results for the speaker cables blind testing published? I'd be most interested in seeing them.

Tonmeister's picture

I've not published any speaker cable tests. I'm not very interested in researching things that produce little or no effect. If there were huge effects why don't the speaker cable manufacturers publish these test results in academic journals...?? I think you know the answer to that question.

Jazz Casual's picture

Yes, I think I do. Thanks again for replying.

Jim Tavegia's picture

People will spend $600 on a phone, but very few would buy Pono as they just don't get it. To them Beats and a Phone are the unbeatable combination. It would be nice if it was different.

Jazz Casual's picture

but I believe that was Neil Young's lofty goal regardless of how naive it might be in reality.

Jim Tavegia's picture

What I noticed is that the participants were clear in stating that it was often hard to head a difference and often could not. Some really were working hard, but could at best do only 50/50. What I found interesting is the unit to unit differences heard in the same model of cans showing that QC for some brands needs to improve. How would any of us know where the pair we bought falls in terms of production quality?

xnor's picture

By doing measurements.

xnor's picture

Why?!?

Comparing equipment under blind conditions is such an eye-opening experience that anyone who works with audio should have experienced it. Especially professionals should make use of this tool.

I know that an analogy with medicine is not the best, but would you want medicine that is sold based on the random diary entries from patients? Welcome to homeopathic "proving" (yes, it's named that way).
... and in the end you get really expensive watered down water or sugar pills.

krikor's picture

I would take exception with your equating of the Picasso painting to audio equipment. The "art" is the music you listen to through the equipment. It is the experience of that music, and all its "nuanced and subtle detail" that takes your breath away when you can "grok" it as a whole. The audio equipment is a means to that end, just as the gallery/museum is a means to viewing the art work (placement, lighting, hell even temperature). I'm not saying there isn't art in the equipment itself, but it is only in service to the real art that is the music.

Simon T.'s picture

Thank you Tyll, wished this world has more reviewers like yourself.

Unfortunately, people like myself who works in an undeveloped country, does not have the privilege to audition all the gears I purchase. And, more often than not, have to rely on product reviews.

Having people like you, who share their honest view/experience is really priceless.

Cheers

The Federalist's picture

I appreciate alot of what you said about amps, dacs and cables impact on sound quality... It's a very rare thing to read these days.

I've heard a lot of different dacs both north of $2k and south of $500 and find the differences between the least and greatest to be pretty nominal in the big scheme of things. Likewise with cables... I own some expensive ones and they haven't improved anything but the visual appeal of my system.

Problem is that the blame for perpetuating these myths or exaggerations can't be placed strictly at the feet of manufacturers... A lot of it falls at the feet of the audiophile press. And I think it ends up damaging the credulity of the websites and business and hobby as a whole. And it obviously creates division in the community. Are these people trying to provide useful information to us or are they just trying to move product for their banner flying sponsors? Some editors and writers scoff at notions of conspiracy theory, but if the public feels they are being lied to?

And the writers just get more defensive or patronizing or start name calling or treating people like their views are childish (which I've seen a lot of lately). And I get that readers would do well to remember its a human being on the other side of that comment screen.

So I appreciate your words... There is plenty here I could argue about (I'm sure others will) but in the midst of all this argument fodder are some pretty powerful statements... So I'll just tip my hat and say thank you for once again being a cut above the rest.

Tyll Hertsens's picture
I think I did make an error by not identifying the audio press as part of the problem. And I guess I'll agree they may be a major source of exaggerated claims...on reflection, they may indeed be a major problem.

In todays world, there is little barrier to entry for aspiring reporters---buy a domain; set up a web site; ask for gear to review; and post glowing reports to get in good with suppliers. Yes, that's a big source of seemingly useful information...but it's often not. Sad.

Bobs Your Uncle's picture

To echo expressions of many earlier commenters: Yours really is a terrific, very insightful post, Tyll. And to quote The Federalist from comments just above "... thank you for once again being a cut above the rest."

I'm exceedingly confident in my personal objectivity when I state that a truly objective analysis will reveal my fullest life experience to be an experience that is unavoidably & undeniably subjective. And how could it be otherwise when, after all has been said & done, I am & will remain, my own frame of reference!

The best that any of us can hope & strive for, I believe, is to expand our individual frames of reference over time so that we encounter, perhaps even encompass, the life experiences of as many different & diverse people as possible. An embrace of expanding ones frame of reference (& an embrace of the knowledge that accompanies such a broadened scope) is an unending pursuit.

As "Perfect Knowledge" does not exist, it is impossible to obtain. But knowledge is gained & enhanced through pursuit of understanding. And any such sincere, honest pursuit is founded in acceptance & acknowledgement that we as beings are, by nature, incomplete & imperfect.

Of course you grok all of that stuff. And Kudos for your perfect application of the term "grok". That usage by itself says quite a bit!

bang.hs's picture

Really appreciate your massive and ever-increasing contribution to the community, Tyll. You provide bright minds the right kind of fuel for clarity. Without you in the community there would only be more poets, people grasping at straws, people groping in the dark, people on the fence, and people mad at what they've got for the buck. People I would have been a part of.

FLTWS's picture

As I recently returned to the world of (quality)headphone listening for music and film viewing, your article has given me a wealth of information to ponder as I search for a personal best solution for me in this area. I've been involved with audio (and now video)reproduction in my home for close to 50 years and appreciate your efforts at this difficult task that you've tackled so well. Thank You.

X