I managed to solve my problem which may also be the solution for the problem you had. See Up2Stream I2S connection to Wondom amp - audio heavily distorted when in WiFi mode - #2 by adoble
Maybe this helps if you ever want to go back to your original plan.