I’ve long suspected that most classical music fans can’t tell the difference between CD-quality downloads and the compressed kind you get from mainstream music stores.
The Test
Last week, I devised a simple test to see if that was the case. If you haven’t already, you might like to take the Pepsi Challenge before you read on. It’ll take about four minutes.
Visitors to this site were asked to listen to an audio file containing three different versions of the same 1’15” excerpt of Mozart’s piano concerto K491. To keep it a blind trial, all three samples were joined together and up-converted into a single 56mb WAV file.
Participants were told that one was CD-quality, and the other two were compressed files like you might buy from iTunes or Amazon. They were asked to identify the CD-quality one. This is what the question looked like:
The Answers
The correct answer is the third one. The first was a 320kbps MP3 like you might get from a specialist classical download store, and the second was a 256kbps AAC file, like you might buy from iTunes.
If my hypothesis is wrong, and most people can tell the difference, you’d expect more than half of all respondents to answer correctly. To confirm the hypothesis, you’d expect fewer than half of all respondents to get it right.
If you got it wrong, don’t feel bad. This was a test of the technology, not a test of your ears.
The sample size was 100 respondents. In the time it took to get 100 responses (six days), the post about this test was viewed about 1,000 times and the file clicked on about 200 times, in part thanks to incoming links from Miss Mussel, Alex Ross, Daniel Stephen Johnson and AC Douglas. I think this represents a pretty good cross-section of classical consumers on the web.
What Happened?
Most people (57%) got it wrong. Another 16% said they didn’t know.
CD wasn’t even the most popular guess – almost a third of all respondents (32%) thought that the lowest bit-rate file (256kbps AAC, like you’d get form iTunes) was CD quality. 27% got it right. That’s slightly worse than if everybody that thought they knew had just had just guessed.
Respondents were much more likely (3.7 times as likely, in fact) to get it wrong than to admit that they didn’t know.
Conclusions:
- Most classical consumers can’t reliably tell the difference between compressed downloads and CD-quality audio.
- The majority of consumers who think they can tell the difference are, at best, mistaken.
- A much larger (or very different) sample would be required before you could conclude that anybody can tell the difference.
Commentary:
Compressed audio has a bad reputation, but that’s mostly undeserved. There must be audiophiles that genuinely can tell the difference, but they’re a small minority. If exaggerated complaints about compressed audio prevent people from even trying digital downloads and deciding for themselves, then those complaints are harmful to the entire classical record industry.
If labels want to sell music to hi-fi enthusiasts, they should keep peddling CDs. If they want to sell music to music enthusiasts, they should stop whining about bit-rates and encourage their customers to try the download stores that already exist, and that have the capacity to reach millions of consumers worldwide.
If you’re about to launch a classical download store, you’d better have a lot more up your sleeve than a clunky site offering lossless downloads. When they try them side-by-side, a lot of customers will prefer the convenience of Coke.
Addendum (4/12/10):
We’re up to 30-something comments now, with many of them focussed on what this experiment tells us about the importance of playback equipment. Let me be really clear: this experiment tells us nothing at all about playback equipment, nor is it intended to. The experiment asks “can people tell” and not “why can’t they tell”. If you’re interested in hardware, the “why” is important. I’m much more interested in selling recordings. When you’re selling recordings, it doesn’t matter anywhere near as much.
Proper Discord 
I think your conclusions a bit, um, premature. See the second update to the following S&F post for details:
http://www.soundsandfury.com/soundsandfury/2010/03/dont-forget-those-good-quality-headphones.html
ACD
Had the playback system been standardized for all 100 participants, then, and only then, would the test have been a valid one and the results meaningful.
That’s not true. This is market research. It’s not up to me to tell people how to use the product. Record stores don’t tell you that you’re not qualified to buy music based on your choice of CD player. Everybody was asked the same question.
It wouldn’t be very useful to know what people thought about music played back on equipment that they didn’t own, and it seems completely reasonable to assume that almost everybody bothering to take part in the experiment would listen on the same stuff they’d use for any other music download.
If we only asked people that own high-end stereo equipment, we’d only be asking people that already agreed with you. That would be like excluding sick people from a drug trial.
It’s not my fault that you got it wrong. If you’d spent two minutes going to fetch a pair of headphones instead of insulting me, you might have done better.
Oh dear.
See Update #3 to our above linked post for our answer.
ACD
We might have to agree to differ on this one, because I fear we are going around in circles.
Most classical music fans can’t tell the difference between CD-quality downloads and the compressed kind you get from mainstream music stores. That might be because they don’t own (or can’t be bothered to use) good enough equipment to make the differences clear, but they still can’t tell.
While I agree with most of this test, there are some things that people need to remember.
1. CD Quality, played through speakers that can’t handle CD Quality, sounds crappy
2. To achieve a real noticeable difference in sound quality from a computer you need the aid of expensive equipment, which most people don’t have just lying around.
3. People don’t know what they’re actually listening for when it comes to CD v. Compression
That being said, the fact that you have a really unnoticeable difference coming for 3x more space on our computers says a lot. Great test
ha! just call me super ears! that was a fun post. thanks.
Thanks for taking time to vote.
The equipment you use to hear it on is of course of importance. For this test I ran the sample through my hi-fi reciever, and I was rather confident in choosing the third sample as the best. I also thought the second one sounded slightly worse than the first. But by all means, the difference was not huge.
Had I listened to this through my laptop speakers or with an average headset, I’m not sure I would have had any idea.
What’s the moral of this story, though? Don’t buy better speakers or you’ll have to replace all your music? The sudden growth in the large-format HDTV market has done great things for Blu-Ray sales, but that’s mostly because huge TVs make all your DVDs look bad.
Anecdotal evidence is only of limited use. My wife got it right using the speakers in her 13″ MacBook Pro. Did she get lucky? Did you? Only a large sample will tell.
Try the Digital Cabernet Challenge at The Villa-Lobos Magazine.
http://villa-lobos.blogspot.com/2010/03/cheers.html
Can you _really_ tell the difference between real wine and digital wine?
Nice.
More than ten years ago I was working for CBC Radio. A new digital audio system had just been installed. It saved audio in some sort of lossy format. (I’ve forgotten since which one.)
At the time I produced a classical music program that we recorded to DAT (Digital Audio Tape). I was curious if this new system might be used for recording my program but had concerns about the quality of the playback.
A colleague and I took a well recorded CD and copied a track onto this new system and then did a playback switching between the lossy format and the original CD. We made sure that the volume was equal between the two playbacks.
The recording was of a violin concerto. After extended listening the only difference we could really hear was in the solo violin and then mostly when the violin got up very high and had little orchestral accompaniment. The solo violin sound would take on a kind of grainy, sandy quality … losing the sweetness in the sound. That’s the best way I can describe it. And it was a very subtle effect. As I say, only noticeable after extended listening. I’m sure that encoding software has improved a lot in the past decade or so.
Over the past five years or so I’ve begun purchasing more and more music on line …. mostly at emusic.com. Frankly, it’s wonderful to be able to sample music by so many different musicians and composers before making the decision to buy. However, I find that for classical music recordings, it frustrating to not have an actual CD with a booklet giving some background on the music and composer … or even a listing of the tracks on the recording. The tracks themselves are often badly titled so that looking at the ‘track playing’ on my Walkman doesn’t even tell me much.
As a composer sound has always been very important to me and the quality of the recordings on my mp3 player has been a sore point from day one. Unfortunately I wasn’t able to hear the sample you posted – it wouldn’t play – but I conducted the same experiment when I first began ripping tracks from CDs several years ago. In trying to decide whether or not to use the 192 Kpbs (the maximum level allowable by my software) or the lower settings that allowed ‘more songs’ to be stored on the device. For the reason of sound quality I had purchased an 8 gig device – so when I listened to my sample recordings I was impressed by the sound. There was a substantial difference between the 156 Kpbs, 192, and CD: the 156 Kpbs lacked the fullness of the sound that the CD had, it simply could not deliver what I was looking for in a recording (I was using Beethoven’s Op. 130 ‘Cavatina’ for the test piece and Mahler’s ‘Blumine’).
The difference between the 192 Kpbs and the CD were still noticeable, but the quality was markedly better than that of the lower quality file and, for the most part, filled in the sonic gaps that seemed to be present in the previous incarnation of the recording. Recently I purchased a 16 Gig Sony Walkman and, having moved my files to the new gizmo was pleasantly surprised to discover that the sound was even better than what I had previously experienced. The previous comments about the quality of the sound having to do with the equipment have great merit – and this includes the quality of the headphones/speakers that one is using to listen to the music. The Sony Walkman mp3 player has the most amazing sound that I’ve experienced since going to a live performance – and the headphones are absolutely unreal (considering they are the ‘in-ear’ buds that normally hurt my ears … I managed to find some foam coverings that fit them and they work great, but the sound is exceptional).
Suffice it to say, my old opinions regarding audio compression have been seriously revised – but I have added the caveat – in bold crimson letters – that the playing device is of paramount importance. I’ll still use the highest quality files for the simple reason that I have 7 Gigs of free space – I could add a movie or two if I wanted, but if I do, it’s a breeze to remove. But seriously, as far as things sound, the file size doesn’t seem to be an issue anymore and, quite frankly, that just makes things a lot easier; I’ve never been fully entrenched in the world of technology (composers are a backward bunch sometimes).
Are you sure your 3 samples aer at the exact same volume. Otherwise people would have tendency to pick the louder version as better, even if very slighlty louder.
That’s a good point, Bob. I did indeed check the volume (you can see the waveforms in the image above) and they’re all equally loud. I did think of putting a fourth, louder MP3 in there, but concluded that this was complicated enough already. It would have added a fun dimension to the conclusions, for sure.
Sounds like a lot of people had the same experience I did – the first file (mp3) was definitely compressed, but I had difficulty telling the difference between the second and third on my crappy headphones. MP3’s are really horrible, and have some almost signature distortions that make them sound really compressed. Some of it may be from it being an older format, I think they’ve fixed some of the issues with compression since then, but I wonder if other people would also be able to tell that one was definitely compressed? The results seem to lean that way…
Why do all these knowledgeable audiophiles have such bad headphones? It’s not as if good headphones are even expensive. For $85 you can have a pair of perfectly good cans that’ll tell you exactly how bad your music is.
Thanks for the interesting and funny test.
I actually got it all right. 🙂 … listening to the file with a good stereo system.
I believe that your conclusions of this test are premature at best. In your conclusions you write: “Most classical consumers can’t reliably tell the difference between compressed downloads and CD-quality audio.”
How do you know what percentage of your 100 samples ARE indeed classical consumers? I will state that most people in this sample never have been at a concert and don’t even listen to classical music. Therefore they don’t know what to listen for or how classical music CAN sound.
Of course the playback system is the deciding factor here. An iTunes download sounds good on a cheap stereo and there will be no difference between the three files. Such a systems is simply not capable of reproducing the subtle differences.
But especially with good classical recordings these difference become very glaring when the different bit rates are played on a good stereo system with good speakers. Sure, most people don’t have such systems but shouldn’t we nonetheless try to strive to for the best possible quality? The lowest common denominator is not necessarily the best.
With today’s download speeds it doesn’t matter a lot if the file is a little bit bigger. I would be so happy if I had ripped my first CD rips not with 192k but as flac or at least 320k. Now that I have a better stereo system I really feel that I am missing a lot. With the old system it didn’t matter. Now they sound crappy.
Let me just say without trying to sound like an audiophile (I am not): There are even big differences between CD and high res formats like SACD or 24bit/96k downloads. The latter ones simply have much more air… the acoustics of the recording hall are there… the subtle reverbs, the noises musicians make when playing their instruments. All this is part of the experience when listening to classical music. At least for me. With a 256k download these hair raising elements of the music are gone… they are deleted/omitted for the sake of a smaller file size.
Regarding point 3 of your conclusions: I am pretty sure a majority of people CAN tell the difference when they hear the samples through a decent stereo system. And I would argue that most serious lovers of classical music have a better music system and don’t listen to their operas on an iPod.
Downloaded music and music stores are definitely a good thing… and you have a point saying that they reach a big audience.
For people listening to mostly pop and rock it indeed doesn’t matter if it’s a compressed file or not. The original is highly compressed too. This kind of music doesn’t have a lot to loose and 256k or 320k mp3 sound as good as CD.
But with classical music with high dynamics and a lot of soundstage and imaging it is important to preserve these aspects of a performance.
My Pepsi challenge: let only fans of classical music sit in front of a good stereo and repeat the test. I am sure the results will be very different.
Cheers,
Bill
How do you know what percentage of your 100 samples ARE indeed classical consumers?
I have made the rash assumption that the people who read my classical music blog, subtitled “Trouble With Classical Music”, which is almost exclusively about classical music, are interested in classical music.
I am pretty sure a majority of people CAN tell the difference when they hear the samples through a decent stereo system.
This would be relevant if they had chosen to USE a decent stereo system, which, it seems, even most of the self-proclaimed audiophiles did not.
My Pepsi challenge: let only fans of classical music sit in front of a good stereo and repeat the test. I am sure the results will be very different.
I’m sure it would – and that might be representative of your world. It just isn’t representative of the world that everybody else lives in. People who sell music don’t get to control how it is consumed, and if we try to control the way music is heard in our research, we get a totally distorted idea of what is important to the majority of our customers.
Points taken. But because the challenge was linked widely I am pretty sure there are many many people in the sample who are not representative of your regular readers. I had never seen your blog before but found the challenge through an external link.
“People who sell music don’t get to control how it is consumed”. I think that is exactly the point. So I simply ask that these sellers also offer higher resolutions. For the listeners who want them. I don’t have a problem with lowres files. But there should be a choice offered and I am more than happy to pay a little more for better quality files.
I had never seen your blog before but found the challenge through an external link.
A lot of people that are interested in classical music find this blog that way. I wouldn’t characterize them all as experts, but that’s representative of the wider marketplace.
So I simply ask that these sellers also offer higher resolutions.
That’s a business decision for the sellers. Would it be a good one? These results suggest that the audiophile download market, while clearly vocal, does not represent a large proportion of the total download market. That might be why specialist classical/audiophile download stores aren’t doing so well.
I forgot: I am a little bit irritated about the my world – your world rhetoric. I think there is only one world and people’s perceptions change. A friend of mine was very happy with his Bose system and its crappy (sorry) sound. He was happy UNTIL he heard his favorite opera Carmen (Maria Callas recording, standard FLAC CD rip) on my system. Not even three weeks later he had upgraded his system. It sounds great… he’s happily ripping his CDs now in FLAC… buying higher resolution files from the few sellers offering them. His way of listening to music, his demands, have changed. Apple e.g. easily could offer classical recordings in flac as well… wouldn’t really matter to offer this choice to the music fans who demand a more satisfying experience.
Yes, Mr. Miller, but it’s a business decision on Apple’s part! And if customers are not clamoring for high-resolution downloads, it’s not Apple’s job to support them. The minority of audiophiles can be well-served by sites that cater to them, and that’s great. If they become a large enough market, you can bet that Apple and other sites for compressed audio will do everything it can to steal them away and get that market-share. It costs money to store music in larger files. It doesn’t make financial sense to spend that money if you won’t get a return on it. It makes altruistic/artistic sense, but that’s not one for the private sector.
Not for the private sector? Excuse me? For who else?
BMW sells many more middle class cars than their top line Rolls Royce… Just because only a few people are interested in a Rolls doesn’t mean that there’s no market for them. After all I am paying more for a better file too… who says they are not getting a return for the little more storage space needed?
Your argument doesn’t fly with me.
As long as Apple doesn’t try they will never find out… ah well, I don’t like the closed universe of Apple anyway. People always were complaining that Microsoft was the big bad company trying to control how we use media. Actually it’s Apple…
I digress…
Um, for a non-profit, which may be willing to take a financial loss in order to achieve an artistic goal?
I didn’t say there’s no market. I said it’s small. When it is likely to become big, Apple will be there. For now, it makes sense for Apple to sell one format, not confuse its customers, many of whom (believe me) don’t know a kilobit from a trilobite, and are happy with what they have/get. I prefer lossless, too, but it matters only to a minority, and minorities don’t govern markets.
There are people who want $300 jeans. Most people are perfectly happy with Levi’s, though, and so Levi’s gets its marketshare, and so does Diesel. What Levi’s doesn’t do is try to replicate Diesel styles and sell them for $40. Because most people just want 5 pockets and for them to be blue!
A lossless recording of a crappy cd is still crappy(not that your sample was.)
One comment to AC Douglas, whose blog I enjoy. If you are listening to ‘loss less’ methods like FLAC and Apple Lossless, then the file quality is very different from compressed MP3. Is it perfect…I’m not sure because even on my high-quality home system I’m not convinced. I digitized all my classical albums for backup purposes (having read about your fire horror story for starters) and because I travel for a living. When I compare a loss less track to CD, I’m not sure it’s equal, but the transports aren’t either (I use a Squeezebox Duet at home) and it’s close enough that I do have a difficult time spotting them.
As a weekly traveler something like an iPod is required. It’s almost impossible to live without and coupled with something like a nice pair of Shure or Etymotic headphones, I don’t think it’s at all like what you may be used to. Either way, it’s either nice phones and iPod or waiting till I get home each week…..convenience wins out over the sonic ideal.
FLAC should record and reproduce the exact same waveform as the CD, but what happens to it once it becomes an analog signal is subject to the foibles of the electronics in your Squeezebox and CD player, and their connection to your receiver.
They might sound a little bit different without one being substantially better than the other. If there’s one indisputable outcome of the Pepsi Challenge above, it’s that even when we can just about hear a subtle difference between recordings, we often can’t tell which is supposed to be better.
JestMe:
Yes sir. Using an iPod with a good pair of headphones as a convenient substitute for the Real Thing on the road or whatever is just dandy. For that it has no peer.
ACD
Hi!
From a purely “scientific” point of view there should be no difference between the flac file and the CD because it’s lossless. I use a Linn DS-I to play back my flac files and the ripped CDs actually sound a little bit better than the same thing played back with my $3,500 Linn Majik CD player. That’s subjective of course but I find the flac files to have more substance and musicality. Maybe the Squeezebox is the limiting factor?
Hey Proper Discord: If there’s one thing the Pepsi Challenge shows, it’s that audiophiles will gather to discuss hardware wherever they can. 😉
the only thing it shows is that hardware is an important factor when discussing file quality…
no reason to be snippy… good arguments are more helpful:-)
The audiophile folks came out of the woodwork for this!!!! Now, here’s my question – how much music do they buy a year? If the audiophile pholks buy a few things a year, and the casual listeners who can’t tell the difference buy more, then, um, what kind of store is going to succeed?
What kind of store is succeeding now?
I’m one of those audiophiles coming out of the woodwork. I’m also a recording engineer and composer and do a great deal of work with digital media.
At first, i listened to the samples on the general speaks on my Macbook Pro. and like so many others, i chose the 256kbp sample. Only after clicking on it did i actually think for a second “wait, if one of these is iTunes encoded, then of course it will sound good on these speakers. it’s meant to!”
the question of hardware is an issue, but i agree with proper discord. large amounts of consumer are not listening to their music on Genelec 8050a’s. they are listening on things like Skull Candy headphones, or Sony earbuds, and, more often than not, the headphones that come with their iPod. for general listening, a high bit-rate compressed file is more than good enough.
Once i put the recording through my home monitors (which are not high end. I have Samson Rubicon 6a’s. for high detail work, i go into the studio with Genelec 8050a’s) and i could then hear the difference. there was a certain amount of depth on the third sample that was missing on the others.
when i’m walking down the street, enjoying music, i am listening to my iPod with its 256kbp AAC mp4a’s. when i’m driving across the country in my beat up jeep with no proper stereo, i’m listening to live streaming music off Pandora via my BlackBerry, or listening to my iPod. And i listen to a large amount of classical. I just bought Pli Selon Pli by Boulez, Christine Schafer- Soprano, via iTunes.
The only time i stress to use uncompressed high quality files 100% is in recording sessions and when writing electronic music. in fact, i usually recorded at a higher bit depth and sampling rate. why? because i want to capture as much detail as possible and then get it down to standard levels myself. That way, i’m controlling a bit more of what content is done. When i’m finished with a piece, i send it to competitions on a playable CD at 44.1 16bit. When i send rehearsal music out to performers, i normally send a 256kbp AAC. that way they can just import on their iPod without any conversions.
And, really, that’s where the problems happen, in numerous conversions. a 256kbp AAC done by a studio to be sold on iTunes will be of high quality. that mp3 being shared around the internet that was ripped off a CD by a bloke that doesn’t really know what’s happening, then compressed to send to a friend, then burned on a CD and given to someone else, that turns it into a WMA file, then converts it to mp3…and so on and so on. At that point large amounts of artifacts are introduced…
but that’s not the point. If you buy a digital track from iTunes, you’re getting a high quality track. I really don’t think most people can hear the difference. I spend my days mixing and i can only hear the difference on high quality speakers. And i only diss digital rips when i’m teaching recording classes or electronic music classes. because if you’re going to mess with something, it needs to be of perfect quality to begin with.
Anyway, i’m done rambling. For general listening there is nothing wrong with compressed digital formats. the tech has improved and it definitely sounds cleaner than 10 years ago. That’s what happens when the tracks get EQed a certain way in the process of conversion (much like the compression/expansion process done on Vinyl).
You raise a number of really interesting points here.
The advent of compressed music (and the portable players that use it) have had a big influence on the way we consume music, many of them quite positive.
As near CD-quality audio replaced tape as the most popular portable audio format, there has been a huge increase in demand for good-quality headphones. The earphones that come with a modern MP3 player aren’t studio monitors, but they’re in a completely different league to the similarly-sized (and priced) headphones that came with Walkmans ten years ago.
Similarly, the advent of low-cost digital audio workstations led to the development of amplified speakers like your Samson Rubicon 6as which, while obviously not nearly as good as the $4000 monitors in your studio, are probably almost as good as the sort of hifi you could have bought for $4k in the 80s.
Really, though, it is becoming less and less relevant what music sounds like on good gear like yours. I have a pair of Mackie HR824MKIIs in my living room. They cost over $1,000 and sound magnificent, but perhaps 10% of my listening happens on them. The rest of the time, I’m using something smaller.
It’ll be interesting to see if this changes, though. In the last few years, the cost of extremely high-resolution in-ear headphones has plummeted, and the number of products in the marketplace has skyrocketed. Storage is getting cheaper and smaller too. Perhaps we’ll see an increase in demand for high-resolution downloads in the next few years. Technology got us this far, after all, and shows no sign of retiring.
I have my doubts, though. Increasing bit-rates is a game of diminishing returns. You double the size of all the files to halve the number of people that won’t shop with you. Even when bandwidth and storage are marginal costs, there comes a point where it just isn’t worth it.
I posted a fairly long critique of your methodology in the previous post. Won’t recap here, but do want to support several claims you made, while introducing some caveats about your conclusions.
First, you’re exactly right that the issue here has NOTHING to do with playback equipment. The proof of that is overwhelming, and goes back to the dawn of psychoacoustics research in 1927 when Harvey Fletcher discovered the Fletcher-Munson curve. Fletcher didn’t use super-ultra-whizzbang digital audio and he didn’t have to, because he used proper experimental design. When you ask whether people can hear an A-B difference, or better yet, and A-B-X difference, you eliminate issues of playback quality. Psychoacoustics researchers used reel tape recorders with significant wow and flutter, and about a 55 dB signal-to-noise ratio (yes, 55 dB *at best*), and they still got reliable reproducible results due to proper experimental design.
So the excuse “we need better equipment” doesn’t wash. That’s nonsense. If reel recorders running at 15 ips with audible background noise and audible wow and flutter sufficed to do proper A-B testing at Bell Labs and Eindhoven and Stanford in the 1960s, then sure as dammit your iPod with its 96 dB signal to noise ratio and nonexistent wow and flutter is orders of magnitude better as a sound source than anything required at the best psychoacoustic research labs in the world. In particular, modern audio amps and preamps have such absurdly low total harmonic distortion and such high signal to noise ratios that any complaints about your equipment are utterly absurd. Maybe if we were back in 1901 listening to Pohlson’s steel-tape telegraphone the equipment would be an issue, or if we were listening to Edison wax cylinders, but, folks, we’re talking about modern digital audio technology. I hate to break this to your alleged “golden ears,” but modern digital audio technology was designed by some of the finest minds in technology, and they knew their psychoacoustics backwards and forwards, and modern 16-bit dithering noise-shaped downsampling DACS and ADCs use technology that so far exceeds the requirements of human hearing, it’s just absurd. I’ve done a lot of A-B tests on myself with Foobar 2000’s A-B audio testing plugin, and I absolutely cannot hear the difference twixt 14-bit and 16-bit audio. I absolutely cannot hear the difference twixt 24-bit and 16-bit audio, and I absolutely cannot hear the difference twixt 96 khz or 192 khz sampling rates and standard 44.1 khz sampling rates. There’s no way in hell I can hear the stairstepping noise from modern DACS in a modern noise-shaped dithered DAC, it’s 110 dB down. You CANNOT hear a signal 110 dB down in musical program material, it’s humanly impossible. For reference, that’s like claiming you can hear a pin drop while standing next to an operating jet engine. It’s not humanly possible. No one can do it, period. Stairstepping noise and quantization noise and sample-and-hold jitter and all that stuff are all so far down in the noise floor in modern digitally-filtered oversampled noise-shaped dithered DACs that even talking about hearing that stuff is ridiculous, it just isn’t on. Humans can hear that kind of stuff 110 dB or more down in the noise floor, only test equipment can detect that kind of audio distortion.
Having said what I can’t hear, let me point out that I did get the test right. The difference seemed obvious and involved some sparkling distortion on the high piano notes. However, my statement means nothing because I could well be just fooling myself. And here’s where we get into the real issues with this kind of test.
What sort of response would you expect from a Monte Carlo run on this sort of test? Suppose you just spray out 100 random numbers as answers to the test. How many would you expect to get right?
Well, you have a 33% chance of getting each question right, so you’d expect a 33% correct response rate. The response rate you got is pretty close to that response rate. The problem now is: how do we decide whether the difference between the response rate you got and a truly random response rate was statistically significant?
We’d need better experimental design to make that kind of judgment, as I pointed out previously. So the best you can conclude is some fairly weak tea: you can conclude that IF your sample universe is a normal distribution and IF the intra-group variance is less than the inter-group variance between your sample set and a genuinely normally dsitributed sample universe, THEN your sample set was within less than a standard deviation of what you’d get from an ergodic distribution. The trouble is that all these IFs haven’t been accounted for. In particular, we can be damn sure your sample set isn’t normally distributed, as I pointed out earlier. So you cannot make the conclusion you did.
You need better experimental design.
WIth regard to all the IFs, I can conclude a bit more than that. I can conclude that the experiment turned up no evidence to support the idea that people could tell the difference, something you’d expect to have happened IF a significant proportion of the respondents could tell the difference.
It’s entirely conceivable that people are picking the AAC because that’s what they’re used to, or because the codec is designed to sound good on computers. It’s also possible that AAC’s lead is not statistically significant. What is clear, though, is that there’s no landslide in favor of the right answer.
A number of the audiophile commenters seem completely outraged about this, and I don’t really understand why. Don’t they want to think that they have superior sonic taste? Are they worried I’ll take their CDs away because I don’t think enough people like them? Do they think that SACD was killed by an amateurish bit of market research and not by the market itself?
Some further comments:
A respondent above says: If you are listening to ‘loss less’ methods like FLAC and Apple Lossless, then the file quality is very different from compressed MP3. Is it perfect…I’m not sure because even on my high-quality home system I’m not convinced.
Sorry, this is pure irrationality. Lossless audio is bit identical to the original PCM audio 16-bit 44.1 khz source. Let me repeat: Every digital bit in the original source is identical to each digital bit in a lossless audio recording. The bits are merely arranged differently in a FLAC or SHN or La or Apple Lossless format recording (yes, there are more lossless audio formats than Apple Lossless).
Claiming that any human can hear a difference between two digital recording which are bit-for-bit identical is…well…it’s goofy. At this point, we’re in the realm of ufology. It’s pure nonsense.
People above made a big deal about headphone quality. I clearly heard the difference twixt the third audio sample and the first two, and I used cheap sony $3 headphones that I bought used at a salvation army store. Guess what, folks? Even cheap headphones made within the last couple of ears are excellent. Techology has advanced by leaps and bounds. Cheapness of headphones is not much of an issue today. Hint: $30 technics porta-pro sound as good as anything you can buy for less than $200, and even my cheap crummy sony headphone sound *nearly*as good as Joe Grado’s $200 cans. I know because I own Grados and the Porta-pros as well as these cheapo Sonys I use with my laptop, and I’ve compared ’em. The differences are surprisingly small, and mainly involve the bass. Guess what? The distortion in compressed mp3 and AAC files appears in the mid-high range, not the bass, so all this headphone talk is irrelevant. Unless you’re using 1940s cans with paper diaphragms, trust me, the headphones are not a big issue. Modern headphones made within the last 10 years or so use extremely high tech high performance electret carbon-fiber plastics for the headphone diaphragms and their frequency response is absolutely excellent even on cheap earbuds. The difference between today’s headphones and those 70s cans is like night and day. Headphone technology has advanced so far in the last 20 years that it’s just silly to talk about poor quailty even on low-end headphones today.