I have managed to solve the problem.
Eventually I suspected that the I2S inputs on the Wondom amp were capacitively loading the output from the Arylic board too much (better said the output from the wifi module).
I inserted a 74HC126 (powered with 3.3 V) between the Arylic board and the Wondom amp to buffer the I2S signals (including MCLK) and I was able to stream undistorted audio.