Interesting.
Pity that the simple (and cheap) solution did not work for you, but it’s good that you managed to find a way to get it to work.
I have since extended the buffer idea and turned it into an I2S multiplexer (needed for my project) and still do not get any distortion (if you’re interested you can find the details here) .
My hypothesis is that the tolerances of the I2S signal drive current supplied by the Up2Stream and the input load capacitance of the WONDOM amp are causing the problem. A drive current at the low end of the tolerance from the Up2Stream and a load capacitance at the high end of the tolerance for the WONDOM amp could result in a mistiming of the I2S signals leading to distortion. In my case, it could be that the drive-current/load-capacitance ratio on both chanels was enough to cause the mistiming, but could be adjusted by using the SN74HC126. In your case the ratio was maybe “worse” on one channel only and the WM8804 had enough drive current to get round the problem. My reading of the specs for both devices didn’t really throw light on the matter.
But as we all know, “hypothesis” is just a fancy word for “guess”, and without more engineering skills from my side and test equpiment is still remains that - a guess. Secondly, if you’re like me, you’re just happy that you’ve got something that works.