Yes, then you'll be using the Denon's DAC again, but you seem to forget the important fact that your Denon is also digitising the analog RCA inputs. Seems to me that the best connection method to such AVR's is digital through optical, coaxial and HDMI. The RCA inputs are there for convenience in case you run out of options, but are best to be avoided.
The "Pure Direct" mode you refer to seems to free up a few resources used for video processing, eq etc, but it is unlikely this it'll provide a purely analog signal path.
In other words: if you have the WiiM connected with RCA, the audio will first be converted from analog to digital by the WiiM's (probably superior) DAC, then converted back from analog to digital by the Denon's ADC and eventually converted back from digital to analog by the Denon's DAC. So it makes total sense that the Denon sounds better if you play the files directly from there - you're skipping one stage of DA and AD conversion.
Theoretically, you should at least be able to match the sound quality by connecting the WiiM through optical - basically using it as a streaming transport - but the WiiM won't be able to show its full potential paired with your AVR unfortunately.
