Pragmatist
Member
- Joined
- Mar 30, 2025
- Messages
- 83
Thoughts on A/B testing
- Thread starter BowsAndArrows
- Start date
Steve Woodhouse
Major Contributor
- Joined
- Aug 11, 2023
- Messages
- 1,798
So here’s the thing, BTW I’m not accusing everyone of this, but it’s an issue every time.
How many people have you heard say they could tell an obvious, night and day difference between DACs. Or a subjectivist reviewer post about a DAC, and throw in “obviously, it’s not as good as/ better than…” insert name of other DAC.
And no one throws up all the objections to their comparison, which they’re happy to throw at blind A/B comparisons. Despite the very obvious difference that a sighted test is without doubt flawed next to a double blind one.
All too often discussions just become a list of fantastical, improbable excuses for not conducting them.
But the hippo in the corridor remains. If you don’t trust a double blind A/B test, then any form of sighted test has all the sane problems, and a whole lot more. So why do you trust them?
At some point someone pipes up “I trust my own ears”. Well, if you trust them, it really is time to put your money where your mouth is and prove, if only to yourself, just how great those ears are.
And if anyone is not prepared to do that, you have to question the validity of them coming to a public forum and asking others to place their trust in the content of their posts.
That may sound harsh. But in the cold light of day, we all know it’s right. If you’re not prepared to test your hearing yourself, why would you ask others to trust it?
How many people have you heard say they could tell an obvious, night and day difference between DACs. Or a subjectivist reviewer post about a DAC, and throw in “obviously, it’s not as good as/ better than…” insert name of other DAC.
And no one throws up all the objections to their comparison, which they’re happy to throw at blind A/B comparisons. Despite the very obvious difference that a sighted test is without doubt flawed next to a double blind one.
All too often discussions just become a list of fantastical, improbable excuses for not conducting them.
But the hippo in the corridor remains. If you don’t trust a double blind A/B test, then any form of sighted test has all the sane problems, and a whole lot more. So why do you trust them?
At some point someone pipes up “I trust my own ears”. Well, if you trust them, it really is time to put your money where your mouth is and prove, if only to yourself, just how great those ears are.
And if anyone is not prepared to do that, you have to question the validity of them coming to a public forum and asking others to place their trust in the content of their posts.
That may sound harsh. But in the cold light of day, we all know it’s right. If you’re not prepared to test your hearing yourself, why would you ask others to trust it?
A late friend of mine and I both had the Slimdevices Transporter back in the day. It was a nonplusultra streamer for local files, managed by Slimserver (now Lyrion Music Server) in 2007. Here are the Stereophile measurements:
www.stereophile.com
My friend found that the Transorter was AC-coupled and that it had non-audiophile electrolytic capacitors in series with it's outputs. He changed them for somethinng expensive from Mundorf (I think it was Mundord) and heard all the difference in the world. Very pleased he was.
He asked to borrow my unmodoified one to do the changes to mine too. But he did an A/B comparison first.
No difference. Absolutely none.
Then he found that the outputs were at +5V, so the serial capacitors input voltage would never cross 0 V. Suddenly no difference made sense.
My take:A/B testing works. It zeroes euphoria and lets your ears find audible differences. If your hearing's up to snuff. Mine was back then. Good times. Damn fine looking box, the Transporter. Lovely VU meters too.
Slim Devices Transporter network music player Measurements
Sidebar 3: Measurements
He asked to borrow my unmodoified one to do the changes to mine too. But he did an A/B comparison first.
No difference. Absolutely none.
Then he found that the outputs were at +5V, so the serial capacitors input voltage would never cross 0 V. Suddenly no difference made sense.
My take:A/B testing works. It zeroes euphoria and lets your ears find audible differences. If your hearing's up to snuff. Mine was back then. Good times. Damn fine looking box, the Transporter. Lovely VU meters too.
It is not necessarily a DAC issue. It can be caused by any of the electronics or by the speakers. The amp or speakers are the mostly likely cause.
My point was that
"That effect indicates that there is more to our hearing that can be determined in a few minutes of A/B testing."
Headache inducing speakers are the most easily sold. They sound engaging in the shop and when you come back home where the floor isn't carpet tiles and the ceiling not sound absorbers and you tend to listen for an extended period of time it's different.
My ears are older than me. Funnily enough my rather good Phonaks were set up with measurements and some twenty minutes of A/B testing. There's less to our hearing tham most audiophiles think. There's also more to how we feel a day or a moment than there is to our equipment.
My ears are older than me. Funnily enough my rather good Phonaks were set up with measurements and some twenty minutes of A/B testing. There's less to our hearing tham most audiophiles think. There's also more to how we feel a day or a moment than there is to our equipment.
slartibartfast
Major Contributor
- Joined
- Apr 18, 2024
- Messages
- 3,566
I am pretty sure if I played a treble leaning system at high volume I would get a headache
This video discusses why out hearing/perception is, in fact, very sensitive and not entirely understood.
Fair enough, Good video (except the infinites and the masking effects of cables and a lot of other stuff where he takes care not to mention electrical phase, or acoustical phase for that matter), at least if you want to become paranoid about your hifi.
Let me rephrase: Most people are barking up the wrong three/put the energy/money in the wrong place in their music systems, i.e. not where it makes a significant improvement in it's sound.
My system's peak response, measured with Umik-1 and RWE in the listening position. 10% is -20 dB:
Translated into phase response:
Let me rephrase: Most people are barking up the wrong three/put the energy/money in the wrong place in their music systems, i.e. not where it makes a significant improvement in it's sound.
My system's peak response, measured with Umik-1 and RWE in the listening position. 10% is -20 dB:
Translated into phase response:
Last edited:
Wouldn’t cut it for you……or your significant other?
The discussion in this video is based on a per reviewed paper published in the scientific publication Applied Acoustics. The author is a professor in the department of Physics and Astrophysics at the University of South Carolina. He specializes in the fields of audio and acoustics and in quantum phenomena at low temperatures. I would not dismiss his work so lightly.
Here is the abstract of the paper
Abstract
This work reviews the human auditory system, elucidating some of the specialized mechanisms and non-linear pathways along the chain of events between physical sound and its perception. Customary relationships between frequency, time, and phase—such as the uncertainty principle—that hold for linear systems, do not apply straightforwardly to the hearing process. Auditory temporal resolution for certain processes can be a hundredth of the period of the signal, and can extend down to the microseconds time scale. The astonishingly large number of variations that correspond to the neural excitation pattern of 30,000 auditory nerve fibers, originating from 3500 inner hair cells, explicates the vast capacity of the auditory system for the resolution of sonic detail. And the ear is sensitive enough to detect a basilar-membrane amplitude at the level of a picometer, or about a hundred times smaller than an atom. This article surveys and provide new insights into some of the impressive capabilities of the human auditory system and explores their relationship to fidelity in reproduced sound.
I don’t see the pertinence of that questionWouldn’t cut it for you……or your significant other?
^It’s not Kunchur in the video, right?
Correct. What the guy on the video is discussing is the contents of Kunchur's paper. The content he presented is published research from an expert in the field. That content should be taken seriously. The "for anyone to take him seriously" comment seemed to imply that the whole content of the video should not be taken seriously. Hence my comment.
Last edited:
With electronics it can be valid with A/B if you can be sure you listen to exact the same volume. For stereo speakers its impossible because they only sound great when properly installed +- 5 cm with the correct distance between them, distance to frontwall , opening angle and so on. A good constructor of loudspeakers in a stereosetup mention this in the setup manual. And yes - different speakers need very different installation positions in a listening room, so one cant just place them at the same spot when doing A/B listening.A late friend of mine and I both had the Slimdevices Transporter back in the day. It was a nonplusultra streamer for local files, managed by Slimserver (now Lyrion Music Server) in 2007. Here are the Stereophile measurements:
My friend found that the Transorter was AC-coupled and that it had non-audiophile electrolytic capacitors in series with it's outputs. He changed them for somethinng expensive from Mundorf (I think it was Mundord) and heard all the difference in the world. Very pleased he was.Slim Devices Transporter network music player Measurements
Sidebar 3: Measurements
He asked to borrow my unmodoified one to do the changes to mine too. But he did an A/B comparison first.
No difference. Absolutely none.
Then he found that the outputs were at +5V, so the serial capacitors input voltage would never cross 0 V. Suddenly no difference made sense.
My take:A/B testing works. It zeroes euphoria and lets your ears find audible differences. If your hearing's up to snuff. Mine was back then. Good times. Damn fine looking box, the Transporter. Lovely VU meters too.
View attachment 19899
View attachment 19900
One complicated but more valid way to compare 2 pair of loudspeakers is to optimise the setup of one pair at a time in the room , putting tape on the exact spot on the floor, and then carry out the speakers from the room when comparing to another pair of speakers ( also optimised with tape on the floor to find the best sounding spot ).
It will take some time to do comparisons this way, but the result will be more valid.
Last edited:
I absolutely didn't not want to imply that.
Our hearing is certainly most complex and research hasn't stopped yet for good reason. I don't expect a YouTube video to fulfill the quality standards of a scientific paper, of course. But I question the bold statements made in this exact video. It comes across as "we have scientific proof that everything matters, always". That's simply not the case.
I also don't think that everything we do in our homes in search for what we think is "better sound" must follow scientific standards. We should just be careful to not draw wrong conclusions from such tinkering. What sounds better to me, sounds better to me. But that's no proof that silver sounds better than copper or soft domes sound better than metal domes or class d amps lacking PFFB sound better than those implementing it (to name just a few popular examples).
Steve Woodhouse
Major Contributor
- Joined
- Aug 11, 2023
- Messages
- 1,798
Yes, I agree. But that doesn’t negate in any way the validity of A/B testing to determine whether a difference in sound quality can be detected.
As for this being about DACs, I think that’s just shorthand, though other items obviously bring their own issues in regards to conducting such tests.
Steve Woodhouse
Major Contributor
- Joined
- Aug 11, 2023
- Messages
- 1,798
Harman have a set up in place for this. Not really practical in our own homes, though.
slartibartfast
Major Contributor
- Joined
- Apr 18, 2024
- Messages
- 3,566
Peer reviewed papers are only as good as the peers who review them. I have first hand knowledge of a PhD student who wrote a thesis full of inaccuracies about work done for a company I worked for. His supervisor who I assume reviewed it seemed to have no issue with it. One of his main conclusions was that the toroidal core of one of our variable transformers could be optimised by reducing the cross sectional area to half the current size which was clearly BS but his supervisor clearly believed it
Similar threads
