Firmware regression testing

[jamster]

I have noticed a tendency to introduce new bugs whilst trying to fix existing ones.

In the past few weeks

- Qobuz Connect was broken by a firmware update (phantom playing when using a USB Dac)

- HDMI Arc - which has worked perfectly since I bought the device - is now unstable to the point that it mostly doesn’t work on my LG G4. Ironically, that firmware update boasted “improved stability” for HDMI Arc.

This is a very poor user experience. I hope there will be a refund option if stability cannot be improved as it’s not possible to enjoy the devices if they keep getting broken by forced, automatic firmware updates.

 

[Krabin]

If you haven't, make a feedback ticket in the app ofc. Try turning off eArc in ur lg settings because I faintly remember seeing that reccomended before.

I haven't had any problems introduced by updates (*knocks on wood*), but I and most consumers who care agree that devices should have a manual update option.

My LG B4 oled tv has it, and I thank the Lord because those updates are scary and have to be researched before installing. The WIIM updates are less scary because there isn't something like an expensive oled panel involved, and nobody's device has been bricked yet (we probably would've heard of it), but it would still be nice to have the option to turn off auto updates and manually update.

 

[WiiM Support]

I have noticed a tendency to introduce new bugs whilst trying to fix existing ones.

In the past few weeks

- Qobuz Connect was broken by a firmware update (phantom playing when using a USB Dac)

- HDMI Arc - which has worked perfectly since I bought the device - is now unstable to the point that it mostly doesn’t work on my LG G4. Ironically, that firmware update boasted “improved stability” for HDMI Arc.

This is a very poor user experience. I hope there will be a refund option if stability cannot be improved as it’s not possible to enjoy the devices if they keep getting broken by forced, automatic firmware updates.

Hi jamster,

We're sorry for the inconvenience. Please send us a feedback ticket so we can troubleshoot these two issues for you immediately. We'll certainly improve our regression test procedure to minimize the problems caused by the App and firmware updates. Thank you again for your continued patience and support!

 

[dtc]

Hi jamster,

We're sorry for the inconvenience. Please send us a feedback ticket so we can troubleshoot these two issues for you immediately. We'll certainly improve our regression test procedure to minimize the problems caused by the App and firmware updates. Thank you again for your continued patience and support!

I think you should also consider an option for the user to choose if they want to upgrade and also to remove an upgrade if it causes problems. In my case, very few of the updates are relevant to me, so why should I have an automatic upgrade forced on me? I would like users to be able to update as they wish.

 

[jamster]

Agree, although removing this update would re-introduce the Qobuz Connect bug, which I didn't realise was a bug with WiiM anyway (I blamed Qobuz) as it wasn't reported on the forums. A "Known-Issues" page would also be useful and in-line with industry practice.

WiiM is fast becoming Sonos. Botched firmware pgrades on an unstable platform. Hope they fix it soon.

 

[Danny Linguini]

I understand that issues like this can be frustrating, but comparing it to the universally panned Sonos botch is a bit of a stretch. Do you have any idea what regression testing something like this entails? There are virtually unlimited combinations of gear brands and models that people will (try to) connect to this hardware in every different way, multiplied by every feature and function within the hardware itself. Considering the very few complaints about things breaking, at least that I’ve seen, we’ve got a good long way to go before we’re approaching Sonos territory.

 

[rubato]

Speaking as a former Sonos customer, I'm happy to say that there is really no comparison. But I used to write software for credit unions: we had a large suite of automated tests, and one of the rules was that whenever you fixed a problem, you couldn't close the ticket without adding a test to make sure it didn't come back. I think the WiiM software is probably complex enough to justify something similar.

 

[CliffyBC]

What would be a benefit to me would be a repository of firmware updates and a method by which a user could remove any buggy update and roll back to firmware that worked correctly without having to put in a ticket to Wiim each time. No idea if this would even be possible tho.

Also user initiated firmware updates only.

 

[hgo58]

What would be a benefit to me would be a repository of firmware updates and a method by which a user could remove any buggy update and roll back to firmware that worked correctly without having to put in a ticket to Wiim each time. No idea if this would even be possible tho.

@WiiM Support could just add the latest firmware to the list of beta tests, available from the WiiM Home App.

 

[CliffyBC]

@WiiM Support could just add the latest firmware to the list of beta tests, available from the WiiM Home App.

Brilliant idea..

 

[WiiM Support]

Brilliant idea..

Hi Team,

We'll do this immediately.

 

[hgo58]

Hi Team,

We'll do this immediately.

Thanks. I have suggested this a couple of times before but good it got attention now.

 

[Mr Ee]

Hi Team,

We'll do this immediately.

How is that different from opting out of a beta and the latest firmware being installed?
Seems to me that what some users want is two (or more) of the official releases available in the beta screen so they can roll back easily if the latest has issues.

So what exactly is it that you intend to do immediately?

 

[hgo58]

How is that different from opting out of a beta and the latest firmware being installed?
Seems to me that what some users want is two (or more) of the official releases available in the beta screen so they can roll back easily if the latest has issues.

So what exactly is it that you intend to do immediately?

You will have the opportunity to roll back to a previous version (as if was a beta test) and stay on it until your issue is resolved.

 

[Mr Ee]

You will have the opportunity to roll back to a previous version (as if was a beta test) and stay on it until your issue is resolved.

I get that, but the quote was "add the latest firmware to the list of beta tests" and the response was "We'll do this immediately.".
It's the previous firmware (at least one) that's required, not the latest, hence asking what wiim actually mean.

 

[Burnside]

Yeah, it’s not clear whether that will also address those who don’t want automatic updates at all, or to apply them at a time of their choosing. If they add the latest firmware to the beta list but don’t apply it, that would address that. If they put the most recent but one on the beta list, that could allow users to rollback public firmware themselves in the event of an issue. Or will they do both? That might make the universal “check for updates” page redundant as what would that do once it found newer versions?

 

[hgo58]

I get that, but the quote was "add the latest firmware to the list of beta tests" and the response was "We'll do this immediately.".
It's the previous firmware (at least one) that's required, not the latest, hence asking what wiim actually mean.

What I meant was to add latest (current) firmware to the list and let it be there for some time (e.g. until at least 5 new versions has been released).

This will allow uses to "freeze" on a given version for some time. WiiM can decide when they get too old.

 

[Burnside]

Five seems excessive to me - two or at a push three should suffice. Who’s going to roll back that far? If WiiM do implement something like that, I’d expect them only to accept tickets for the most recent version.

 

[Wiimer]

Five seems excessive to me - two or at a push three should suffice. Who’s going to roll back that far? If WiiM do implement something like that, I’d expect them only to accept tickets for the most recent version.

One is good enough for me. It is enough if I can go back to one and check it works when I encounter problems with the latest version.

As the FW and the app are updated in parallel, it is possible that some of the app will not work properly due to too old FW.

I guess this feature is a double-edged sword for us and for the team. I'm a bit worried about it..

 

[Burnside]

One is good enough for me. It is enough if I can go back to one and check it works when I encounter problems with the latest version.

As the FW and the app are updated in parallel, it is possible that some of the app will not work properly due to too old FW.

I guess this feature is a double-edged sword for us and for the team. I'm a bit worried about it..

Yeah, it could be a double edged sword…