Important words from Floyd Toole

As I'm in the midst of producing what is essentially a large subjective review (an opinion piece on my experience with equipment), I wanted to offer up this video by Floyd Toole. He is a giant in the world of audio and I have the deepest respect for his work. In this video he explains the difference between opinion and science.

I'm going to jump past an enormous amount of dialog and cut to the chase:

There is no audio review publication that does scientific product listening evaluations. It's simply far too expensive an undertaking to finance with something as fickle and scarce as advertising dollars. Science can be done by big corporate concerns because they do have a revenue stream from commercial sales that can support the effort...but even then, only for the largest of companies.

I have, numerous times, done work that has a psudo-scientific approach (Pono player and AKG K701 burn-in blind tests), but the truth is that there are all sort of holes that could be poked in those test by people who do understand the nuance and difficulty of real science.

My point here is that even though I think it's good to make an effort at times, trying to pass off any thing I do as real science is disingenuous at best, misleading at worst. Despite Floyd's cautionary words about sighted subjective reviews, it is, unfortunately but realistically, the best a journalist like myself can do. I'll add that when you take a piece of gear home, you do not listen to the gear in a blind environment...you listen sighted just like I do.

My experience leads me to believe that a competent, experienced subjective reviewer with known preferences can provide a good ballpark evaluation of a piece of gear. A good reviewer's—or, I should say, three good reviewer's—opinion will probably get you maybe 70% of the way to a real scientific result. At about 2% of the cost of a real scientific study, I need to add. Bottom line: No consumer publication will ever be able to afford the time and effort needed to do real scientific comparative evaluation of all the gear out there. Not even the big corporations could—and if they did they would certainly not openly publish their results as it would be expensive and proprietary information.

In the end, we have to settle for the subjective opinion of (hopefully) trusted reviewers. This is exactly why I am determined to get more than just my ears on the Big Sound 2015 project. There won't be time for anything but developing sighted subjective opinions, so more opinions, with a wider diversity of bias, is better than mine alone.

Here's Floyd: He's right, but remember the methods he holds up as science really aren't attainable by journalists in review publications.

COMMENTS
Dreyka's picture

1. You don't need to do it for everything but you need to do it for SOMETHING. If you are unable to tell apart a $100 amplifier from a $1000 amplifier then that is REALLY IMPORTANT. If you cannot tell apart an Objective2 from a Schiit Ragnarok under blind conditions then that is of huge importance. Part of your job is illuminating the way so others can spend their money more wisely.

1. As a reviewer you are fundamentally about informing your readers of a product that THEY will spend money on. "It's too difficult" is not all well and good when their money is being spent on your spurious claims of how X cable sounds 'smoother' than Y cable.

1. The industry is full of spineless individuals who won't say anything negative and others who have terrible critical listening skills and terrible methodology. Most of these people aren't even really aware of it either. There needs to be some that are different but all I hear from PROFESSIONAL reviewers is excuses for them to carry on ONLY doing sighted.

1. Rickety science is better than none. Walking with a dim candle is better than nothing. A piece of equipment will have hundreds of reviews that are contradictory and other reviews where they've just reworded what someone else has written. You can't do everything but you can do something. Pick what is important.

1. Someone needs to push the status quo because it's all hype-train BS where people treat the Sony Z7 as amazing until it actually turns out that it has significant flaws.

1. Saying negative things about equipment is important. If all you hear is silence and praise then people are going to end up buying things they'll be disappointed in.

Beagle's picture

seriously?

Theriverlethe's picture

Very seriously.

bang.hs's picture

Tell the sane ones about it, dude.. I will never understand how people are able to say so many things in favor of amplifier A($1500) vs amplifier B(300$) even if the latter is also very well designed and the only real difference between the two being, lets be generous, a solid ten dBs of THD+N, say -95 vs -105, but both at ear-splitting power?
Just because you can splurge on A doesn't entitle you to claim that you hear a difference, because that's a huge stretch even in theory(of how your ear works).
Another cold hard fact is that moving your headphone to a different position on your head as you listen(even if you're just adjusting it) shall have a significant and very measurable change in the FR that you hear, as compared to switching to the latest super luxury dac or amp.

anonknowmouse's picture

This nice guy has said what it took me years to figure out as a young 'audiophile'. The truth is that finding a headphone or speaker that sounds close to the average monitor in an average studio is not enough. Sound differences exist in every album, and these differences become more apparent with less colouration. Personally I would ignore amplifiers in sound colouration, as they make no audible difference to the sound unless they have electrical hum, low/high frequency roll off or are clipping.
I would say almost every time when you hear a significant problem with a song or album, it's a problem with the mixing and/or mastering. There is less consistency between albums than anyone would like, but sometimes minor flaws add to their character.

Other problems come from digital production. People tend to think more is better, but the more you push the more un-predictable the result. Digital clipping due to the level being too high (and low with video) is common. I'm sure this partly explains the growth of listening to vinyl made before the digital age, despite digital being technically superior.

castleofargh's picture

thanks for the vid, I'm also a big fan of floyd toole and most of the harman guys of that era.

every little thing adds up. of course I never expected a perfect DBT with loudness and impedance matched and hundreds of trials...
but using a switch, matching loudness, those are things you could do. and they would have a very direct impact on the relevance of the following observations.
enough to go from, mostly fantasy, to "hey I expect most observations to be at least in the right direction".

I trust your knowledge and seriousness, not your immunity to bias and placebo. because you're still human.

I probably shouldn't put the blame on you because all you plan to do is only as bad as headphonia and others. it's just that after getting used to get measurements even for 30$ Chinese IEMs, there really is something troubling with you going to tackle multi thousand gears with candid casual listening.
just saying.

Dreyka's picture

it's just that after getting used to get measurements even for 30$ Chinese IEMs, there really is something troubling with you going to tackle multi thousand gears with candid casual listening.
just saying.

You would think rigor would increase with price but it doesn't and that is really sad.

Tyll Hertsens's picture
Tell you what, you volunteer to come up her and spend a 3 months programming my tester so I can do balanced and tube amp testing; write routines for DAP and smart phone audio measurements; and write a DAC teasing routine from scratch, and then I'll measure everything that comes through the joint.

If you can't do that, then you'll just have to have a bit of softness and forgiveness in your being as you understand that I don't have unlimited amounts of time to work on these thing, and I'd appreciate some modicum of gratitude for those measurements and objective information I do provide.

FWIW, IMHO, there are a hell of a lot more cheap stuff sold than the expensive stuff, so the truth is measurements and rigor on cheap stuff is probably more important than on high-end stuff.

IOW, don't get your panties in a knot...enjoy the ride a bit.

John Grandberg's picture

Seriously, the sense of entitlement in some of these comments is staggering. Some thoughts come to mind:

A) These folks should demand their money back! Oh, wait....

B) Perhaps they should stick with one of the many other purely objective websites, where audio gear is reviewed by a figurehead of the industry on the same level as Tyll, but double blind procedures are always rigorously observed. Oh, wait...

C) We are not responsible for how people spend their money! If someone finds Tyll's cable comments "spurious" then let them skip that part. We trust our readers to have a mind of their own and go in whatever direction they choose - which won't always line up with the stuff we recommend or place on the Wall of Fame. We are just here to help illuminate some of the options and chronicle our experience with it, for better or worse. If a reader wants to use a green pen on their CDs, or Home Depot speaker cable, either way it's their own business.

D) What other prominent audio review site has published as many negative reviews as InnerFidelity? Who else has almost single-handedly ruined the reputation of a new high-end headphone (Ultrasone Edition 10)? I'd say there aren't many.

Come on people, this stuff is supposed to be enjoyable! Find a reviewer who you enjoy reading, and who has a taste that lines up with yours more often than not. Then take that reviewer for what they are, rather than make angry demands of them. "Hey Tyll, it's really about time you start rendering your charts in 3D. It's the responsible thing to do. But, you know, don't spend a lot of time on it...."

Beagle's picture

But I want condescending snobs and folks in lab coats to tell me how to enjoy my music and suck all the fun out of it! It's only right! How the hell would I know what I enjoy?

If you think something is better and you are enjoying it more, it IS and you ARE!

castleofargh's picture

I understand and respect the arguments from Tyll and John, and my remarks were only made from high(too high?)expectations, because Tyll got me used to have a lot of factual information to go with the more subjective parts. I am not trying to blame all the troubles of the world onto Tyll.

still the weaknesses of sighted evaluation are very well known, very well documented and nobody is immune to them. that does raise the question about the degree of reliability and purpose of those feedbacks when done without controls. that's all I've been saying.

but your comment, now that's the kind of silly hate bringer stuff that isn't even on topic at any given moment. you must be so proud.

lashto's picture

+100 for this comment and for Dreyka's above.

Some objective testing is better than none. Doesnt have to be perfect. A volume-matched SBT is already much, much, much better than any sighted laaalala and I surely want to know if 10K buys me better sound or just extra features and a nice case.

This is one of the top audio-review sites and rightly so but you do not stay on top with stuff like "it's too hard" or "we can't do it perfectly so we won't do it at all". Also, I do not understand that "nobody else does it" excuse mentioned by others ... does InnerFidelity want to be "nobody"?!

And by the way, I think the word "big" is already too small for that giant gear review. I'd go with "epic" or such ... or something like MotherOfAllReviewsEver. And dont forget to frame your epic power-bill at the end :)

Bobs Your Uncle's picture

These are times when strident, polarizing animus toward "The Other" is somehow deemed to be a positive character trait. "The Other" must be demeaned, discredited & dehumanized in the starkest possible terms.

Any feint to foster cooperation & compromise in service of a broadly shared common good is vehemently decried as a traitorous abandonment of perfect ideological purity; a purity that inexplicably materialized as the indisputable standard to be upheld.

These are times of unthinking sublimation to the monochromatic simplicity of black vs. white. It's an "Either-Or" proposition baby! Yer either Fer Us or Yer Agin Us!

Unless, of course, one happens to appreciate nuanced distinctions; subtle variability among factors; the rich interplay between discrete complimentary elements.

Of course, if one happens to embrace such perspectives, that in itself stands as an admission of disqualifying fallibility; an acknowledgement that one does not possess perfect knowledge & may even dispute its very existence. And such an imperfect person is little more than a wounded & bleeding target for anyone who does, in fact, Know It All.

But please remember Tyll ... Such folks live with a vexatious condition of Permanent Panty Knot. That oppressive bunching cannot be ignored & it prevents any enjoyment of any ride what-so-ever. Despite their self-declared confident superiority, they are to be pitied.

Bobs Your Uncle's picture

roskodan's picture

buy a M2...

i don't see any difference between subjective reviews, however biased they may be, and a graph, be it statistics or measurements, in the sense that in both cases one needs to know what it ultimately translates into in terms of one's personal preferences,

of course measurements are the compass to set the course in the right direction but are definitely not the goal, the ultimate destination, that is for the consumer to decide, which way to go,

that's why reviewers are needed, to explore the sea, to have a feeling of what's out there and of what are consumers' preferences, in base of that to steer for amazing places, and avoid the not so amazing ones,

the fact that the M2 measures perfectly will not make it sell better if it doesn't target a specific market segment that has better prospects,

no point in calling reviewers and telling them they are not doing it right and to come over for a lesson, if there isn’t a market segment, a consumer base, that will be interested in your products,

of course i can't wait to see the day when we get rid of all the reviewers and replace them with logical '0' and '1'... after that, one day or another, a 'hipster' will come down the road shouting 'we have the solution to all of your problems!!! ...a reviewer!!! ...in blood and flash!!!'

Theriverlethe's picture

I'm inclined to agree on issues where there can reasonably be expected to BE a difference in the first place - eg., headphones and speakers. However, if two devices measure equally within the reasonably known limits of human perception, it first needs to be established that there IS a difference.

Theriverlethe's picture

3 x 0% = 0%

anonknowmouse's picture

Floyd isn't saying don't enjoy music, collect equipment or opinions aren't valid. He's saying standardisation improves the technical performance and consistency of everything a standard defines.

Reviewers already use standardised language to describe characteristics. If you asked someone with no knowledge of audio what the difference between a 'warm' or 'cold' headphone is, they may think that they are designed for use in different seasons! However because both reader and reviewer use standardised language, we all hopefully know what is meant.
If standardised language and testing methodology already make reviews more accurate, it follows that making tests as practically standardised as possible, blind if possible, would improve accuracy a lot more.

However 'blind' testing is very difficult with experienced listeners, as earpads and headbands feel different.

Arguably your ears high frequency attenuation, and in-consistent mixing/mastering of a song, are the biggest variables with accurate headphones.

buckchester's picture

Well, at least when you attempt some degree of objective analysis (regardless of how many holes can be poked in it), it’s more valuable than the average reviewer who makes no attempt at objective analysis, where all the same holes, and many others, can be poked.

I think it would be very easy for professional reviewers to conduct volume level-matched blind tests. It might not be a perfect objective analysis, but would be closer than what we have now.

I am skeptical about the reason this not being done is due to costs. I would guess it's more to do with a worry or realization of not being able to reliably tell the differences between many products (DACs and amps in particular).

BTW, Tyll, I think you do a better job than most at trying to be as objective as you can. We need more objective analysis in this industry.

tony's picture

I imported the M2s,

These are Meridian Actives, from the early 1980s, they are excellent, I'd love to own a pair today, beautiful wood finish.

Of course there are Genelec and Emotiva and Focal versions that a person can own, fresh with generous warrantees ( very low failure rates ) priced affordably ( particularly the Emotiva ) , great re-sale capabilities ( buy with confidence ).

The Meridian stuff today is quite pricy, suited to the top 1% among us but superb if you can afford it.

My thanks to Tyll for giving us this Floyd Video, this man brings some clarity to my conundrums, I'll watch it again, making notes.

I like him dumping on the silly reviews, kind-of validates my own criticisms of Audio Industry Glossy Mags, ( trust me indeed! )

Nice stuff here, again.

Thank you

Tony in Michigan

ps. I'd love to be your assistant for the few months it will take to get this project of your's up and running smoothly but you'd be tossing me out into the snow with Sasquatch after a couple of days worth of my antics ( my wife would love to send me up there to Montana for a while, she'd pay my way and probably pay you something too. , would I get to ride your Scooters ? )

Willakan's picture

I was pondering whether to comment on this...figured what the hell, nobody really gives a crap either way.

First, I am enormously sympathetic as regards the difficulties of measuring things and the like. I don't think people realise the sheer amount of time, say, NwAvGuy had to spend to produce his ten-thousand-word-plus measurement articles. Considering the number of balls you have to keep in the air, it's great to see any progress on the measurement program.

That said...I'm confused. That's the bottom line: I won't even bother with my usual dance - you know the steps. As regards subjective evaluation holding some kind of cumulative value, I can link the relevant Sean Olive data showing a complete difference in speaker preferences under sighted conditions. I can parrot the now-defunct "Audio Critic," noting that much can be done with simple blind comparisons, with a switchbox and volume matching, without ludicrous expense or effort...but you know about this shit!

Sure, audiophilia is hardly the most important thing in the world - there's plenty of other things that are irredeemably crappified to rant about. That said, your comments on advertising revenue are horribly depressing. Is that the bottom line: without prolix nothings on grainy trebles and liquid backgrounds, analog tone-pictures and microdynamic detail, nobody would bother turning up? I'd like to think otherwise - in the other areas of consumer tech, there seems to be an insatiable appetite for extremely technical reviews - but you are doubtless better qualified to determine what sells than me.

Anyhow, that seems like a bitter note to end on. I've always enjoyed reading your articles and admired your earnest approach to untangling the world of audio. But I can't pretend to not be...well, not so much mystified by your conclusions as just disappointed. That sounds enormously patronising, and that's not my intention at all...I don't know, it just makes me sad.

Like Sean Olive says, most people don't get to experience high quality audio, and if the best "audiophilia" can manage to bring them into the fold is the laughable Pono - Monty, over at Xiph, talks of how the Pono people contacted him to talk about 24/192, ultimately agreeing that it was nonsense but needed to sell it to audiophiles - well, the results speak for themselves.

TheAudioGuild's picture

“The test of the machine is the satisfaction it gives you. There isn't any other test. If the machine produces tranquility it's right. If it disturbs you it's wrong until either the machine or your mind is changed.”

— Robert M. Pirsig, Zen And The Art of Motorcycle Maintenance

tony's picture

Availability and Reliability !

Not to put too fine a point on it but social acceptability also plays an important role.

Annnnnnnnddd don't forget Heroin

legality is a factor.

Geez, where are the lighting Bolts that used to fly off your keyboard?

Tony in Michigan

TheAudioGuild's picture

Um... Ok.

Smiling Kev's picture

He's being realistic and honest about what he has time to do and the equipment and motivation to do. I really value that kind of transparency!

I'm with Tyll 100% in regard to whether fully controlled scientific testing of consumer electronics will ever happen. Tyll says, "Nope!" I second that motion. I spent 30 years teaching grad students in social sciences how to read, critique and conduct well-designed lab and field experiments and analyze them using parametric and non-parametric multivariate statistical analysis routines. What we all learned is that research involving human subjects is notoriously messy, complex and - even at its very best - only quasi-scientific.

It's fine to be as systematic as possible about your research involving any aspect of human behavior, but you simply cannot realistically berate yourself or others when their procedures depart widely from those that can be applied in the physical sciences. Every hydrogen atom is exactly like every other hydrogen atom. Ditto for oxygen atoms. So, it's easy to study how H and O combine to make water. No two humans, however, are anything like each other in regard to any of hundreds of measurable traits and other features. Ditto for ONE person measured at one point in time and again just a few minutes later.

So, I don't expect Tyll to be a scientist. What he is is a well educated, highly experienced, intelligent, honest and open-minded expert who knows more about headphones than I could ever hope to learn. I have also discovered that his music preferences and the kind of sound he prefers from a headphone is pretty close to my own tastes. He also writes clearly, isn't a bit egotistical, and frequently makes me laugh. So I read and greatly enjoy his reviews and his many educational articles - and I trust his judgements as a very good starting point for doing my own research, evaluations and preference testing before I buy something. Once in a while we come to very different conclusion, as when the only reason that the Philips Fidelio in-ear monitor (which he gave Wall of Fame status) didn't deafen me with its shrieking treble was because they fell out each time I moved even a tiny bit. Much more often, I end up around the same place he concludes. For example, I am still blown away by just how good the Skullcandy Grind is and what a tremendous bargain it is (at $60 max) for use as an indestructible, general purpose headphone.

I can't wait to hear all about the nosebleedingly high end gear he's listening to this summer. I will NEVER be able to afford any of it - and, if I could, I still wouldn't spend that much money on audio gear. As a completely personal preference, I would want to very modestly upgrade my low-budget gear and then feed a whole bunch of hungry people in my county with the leftover money! Still, it's really fun to learn what kind of esoteric and exclusive sonic rewards await other purchasers of Big Sound!

How about if we just kick back and enjoy Tyll's summer project with him? Then, if you've got the dough and he's got the time, you could try to hire Tyll as a consultant and "personal buyer" to stock your audio shelves according to your specific criteria and preferences!

Gotta go! Long-suffering wife is gently suggesting that I need to stop ranting and get ready to go out to dinner. Enjoy your weekend!

Peace,
Kev

echineko's picture

I'm going to start off by saying I'm not an expert in any way, and merely got interested in this hobby/addiction relatively recently. The work you guys do here has been interesting, informative and fun (to me at least). Tyll even briefly gave me his feedback on something I was considering in the start.

But just to focus on one point, Tyll mentioned a bunch of stuff I don't understand which he doesn't have time to do (programming test routines and whatnot), just wondering, would it really be out of the realms of possibility to formally launch a campaign (here or on headfi and the like) to get qualified volunteers to help?

I believe that many knowledgeable and experienced people (not just random people who comment online) might be interested in helping out, has this ever been tried? Nothing to do with the current project, but for future use, I somehow don't think it's that far fetched. After all, no one can do everything by themselves

xnor's picture

Well, let me start first in saying that I disagree with many statements in this article. Others in the comments have already posted some things that match my sentiments.

But it sounds a bit like admitting defeat.
"We can't do science perfectly, so we stick to no-science". No! That's a bad attitude, really bad.
I'm not saying you are guilty of that Tyll, quite the opposite. It really shows that you often try putting as solid science as possible into your reviews.

I have no problem if it is not possible.. but this should be explained why that is the case and made clear from the beginning. Otherwise it is just an excuse.

On the "3 good reviewers get you maybe 70%", I have to strongly disagree.
Take a look at Bayes' Theorem. The problem is that people will hear differences even if there are none. Let's use a low probability of 90% for that. Even if the reviewers can hear actual differences with a probability of 99%, your final probability of there being an actual audible difference is only 7% better than where you started.
To just break even with the probability of a false positive you'd need like 30 of such reviewers that all came to the same conclusion independently.
---
tl/dr: 3 reviewers will get you nowhere.

Seth195208's picture

Weeeeeeeeeee!

bernardperu's picture

what floyd toole clearly states is that sighted tests are plain invalid. No,way around that. Do not perform any gear testing unless you are blind. How expensive it is to put a blindold around your eyes when testing speakers, cables especially!, and other gear.

Floyd toole regards the usual hi fi review as invalid and he has great points.

Tyll, just put on the blindold, lose the advertising revenue of cable companies, but gain a lot more popularity that will lead to more sources of advertisement. Tyll, you know expensive cables are bullshit for the most part, now be an honest man and show us.

cundare's picture

I thought about writing a lengthy response to this piece because:
i) Floyd is a hero of mine;
ii) Coincidentally, I just reread the entire run of "The Audio Critic" and gained a significantly different perspective on what Aczel wrought all those years ago;
iii) In a previous career, as a primary speaker/audio reviewer for large mainstream (not audio-industry) tech publications (I'm talking 500K-1M circulations), had to deal with the type of compromises that Tyll describes (that is, in my case, having to review a desktop speaker system on a $250 budget); and
iv) I recently rediscovered a 1980s MIT journal article in which the researchers found virtually no difference between results of single-blind testing and double-blind testing of audio equipment.

I find this last point especially relevant because, as I'm sure most of have discovered, perceiving, identifying, and comparing sounds is more -- far more! -- complex and nuanced than many other types of comparisons. Citing proof that double-blind testing provides more reliable results in other fields does not necessarily carry over to audio testing. Maybe it does, but I've never seen any published science that unequivocally shows the superiority of double-blind testing of audio components -- not even Floyd's anecdotal data.

At the risk of falling off a cliff, let me push this train of thought even further. I might even be convinced that eliminating the non-sonic sources of bias that Floyd describes so well is not desirable because when we listen in the real world, those biases still exist and still affect our perceptions and enjoyment. That is, if a nonblinded tester falsely ascribes more extended low-end to a first speaker because that first speaker is physically larger than a second speaker, an owner/listener's perceptions and satisfaction will be similarly affected by such factors. So don't we misrepresent what a listener will actually experience if we go to crazy lengths to eliminate those biases during testing? After all, we could get an even more accurate idea of a speaker's sound by testing the speaker in an anechoic chamber. But the speaker would sound so different in a real-world room (and, yes, I realize that many speakers are designed to be used in a reflective room) that anechoic testing might not be the best way to achieve a reviewer's goal.

Hey, I'm not saying that I buy into this line of thought 100%, but my point is that audio testing is not nearly as black-and-white a process as, say, comparing color swatches or mass-densities of other types of subject matter. On this point, Tyll is right and Aczel was clearly wrong. Even Floyd, whom I believe is one of our best minds in this field, approaches these issues from his own perspective, even if that perspective is based in empirical observations, rather than training in the scientific method. (I'm sure that he, like I, is greatly influenced by his background in physics.) But we can all learn from those who view subtle issues from different perspectives. "From the mouths of babes" & all that. Tyll is absolutely right about nobody having all the answers.

Anyway, look at that -- I did ramble on for thousands of words after all. Sigh. The real reason I wanted to reply is to thank Tyll for posting the Floyd video and for addressing the issues that Toole raises in a way that encourages the rest of us to brainstorm in a civil manner. In my opinion, the real weakness of people like Aczel, and the reason why The Audio Critic was a commercial failure for so many years, is that Aczel, regardless of whether he was right or wrong (and, in retrospect, it's obvious now that he was wrong a lot of the time), he was abusive to anybody he felt did not have the credentials to challenge whatever happened to be his current position at the time. (I gather that this curmudgeonly persona was deliberately constructed in order to make the magazine more "compelling.") Nobody learns from being dictated to, even if the dictator is parroting some of the best minds of the time. What Tyll is doing here is more intelligent, respectful, mature, and IMO successful. Good going, guy!

I could write thousands more words on each of the topics enumerated above, but jeezus, even if I had the time to do so, who would have the time or patience to read them? Maybe some day, Tyll, we'll talk on the phone.

place1234's picture

scorpionwatch.com seems to be just everything we hope to find at an online fake watches store: an elegant, professional and friendly shopping environment.

wormcycle's picture

The dilemma of doing a full scientific evaluation, or not at all, has its equivalent in the world of system testing, and it is resolved using what we call "risk based testing". Risk based testing attributes are:
1. It's time boxed 2. It has a fixed cost 3. It is based on risk evaluation 4. It attempts to address the highest risk first.

I am a music lover and like good equipment, and I know next to nothing about headphone design and acoustics, but the comments from cundare tell me that similar approach could be used here. If, for example, blind testing is almost as good as double blind testing I assume that sound engineers and experienced reviewers can come up with a "good enough" evaluation criteria that would eliminate the most obvious bias. That's still would be 100% better than what Audiostream is doing.

Irfan Ali's picture

I have seen many times that speak fluently is best way to make home in person's heart that's the thing when we see everyone who speak hardly with other we avoid the person means speaking sound become harmless by our flounce News Paper In Urdu.

SyedAli's picture

Along these lines Another chilly hard truth is that moving your earphone to an alternate position on your head as you listen(even in case you're simply modifying it) might have a critical and exceptionally quantifiable change in the FR that you listen Pakistan News Epaper.

X