No, absolutely not. To get a smooth transition from the WiiM's lowpass (for sub) and highpass (for mains) you want to make sure that the subwoofer's lowpass filter is fully out of the way. @Wiimer got it straight.It's set to 160hz on the sub and 80 on the wiim, perhaps they should match?
Isn't it smarter to verify if the WiiM Amp is working correctly in the first place, rather than looking for a (lesser) workaround?traditionally for music, we invite rel users to use connect to loudspeakers mode... perhaps just go back to this approach and the more classic internal adjustment modes of the rel and see if it doesn't just match that way...
Don't think of the subwoofer's gain knobs indicating its capability. This is all about optimizing the overall gain structure. What works well with one amp might not be perfect for a different amp. Usually (with high quality subwoofer outputs), best results are obtained with subwoofer level set somewhat roughly around 50% and then finetuning the level in the output settings. In your case this doesn't seem to make a difference, since you've played around with all possible volume settings.1. the output level position at which my REL works perfectly in combination with the Pioneer NC-50 is around 30% of full subwoofer capability.
The .1 / LFE input and the associated level knob should be used, ideally.
Did you try to adjust the crossover frequency between 30 Hz and 250 Hz? Even (or rather especially) with no subwoofer connected, thish should make a tremendous difference to the sound of your main speakers. If it does not, then somehow your amp does not respond to the app's settings.
As you already found out, there is no general problem with the WiiM Amp subwoofer output. Also, there is nothing special about your sub that would make it somehow incompatible with the WiiM Amp. If all connections are good and all settings are as advised, there might be a technical problem.