Firmware Update Causing Issues with Omnilogic???

Hey all. I posted last month about my screen not working on my Omnilogic and thanks to your help was able to figure out it was the PSU assembly that was messed up. A week later, a local Hayward service tech replaced it and all was well. So we thought.

Exactly 2 weeks later, wifi won't connect, screen is gone, etc. Of course this happens when I am out of town so my hubby is left to deal with it. The local Hayward tech comes out and says this time the PSU assembly, main board, LED display, and the tcell board where it plugs in are all toast. Apparently there are black burn marks from it getting hot. Our lightning protection indicator is still green indicating no strike. There are no other marks anywhere on the unit. I do not know if there were any fuse issues. The tech says he has never seen damage this extensive. He called Hayward to try to get it covered by warranty but his rep told him if he submitted it, they would deny it and say it was lightning. He submitted it and sure enough, they denied it.

Now, I am not a big computer person so I contacted my brother who did networking, programming, and cyber security for the Air Force for 25 years and a friend who does programming and networking as his job and asked both of them if a firmware update could cause an issue with how things were running with the unit and cause a short or something to over heat or whatever the technical term is, as I don't speak that techy language. I also told them not to tell me what I want to hear but to be honest with me. Both said absolutely, that can happen.

Interestingly, the update that was installed effected all of the components that were fried. Our very large expansion board with it's many actuators and the components attached to it were not touched. Nothing in the update effected anything on that expansion board either. I find it rather odd that a lightning strike would pick and choose very specific areas to burn up in a large electrical unit yet not burn up electrical areas right next to ones it did burn. There were no burnt wires showing the path of the lightning, nothing other than the connections where things plugged in were burnt like it shorted out. 🤷‍♀️ I installed the update June 18. The first failure occured July 12. They replaced the PSU on July 24. Then everything went out on Aug 8. So first time 25 days between and then 15 days between failures.

Tomorrow I am going to call Hayward to talk to them about this situation and hopefully get some coverage. At least parts. However, my concern is what happens in a couple of weeks when this happens *again* and we are in the same situation as our warranty expires on the 22nd. You would think after having the same issue so close together that they would consider something was going on? Maybe if this same thing happens with all of these same components going out again in a few weeks? It won't be coincidence or lightning hitting the same place 3 times. It is over $2500 to replace the parts and $3500 to get a whole new unit (tech suggested possibly doing that since there is only a 1 year warranty on parts and our warranty is expiring and we would get 3 years with a new unit).

Has anyone heard of anything like this happening? Our Omnilogic was installed in 2017. The local Hayward tech said he was given a wink and nod and a short warning about the pool company that built our pool by the Hayward rep. Of course they are bankrupt and out of business already as well. So it wouldn't surprise me if something was shoddy with the unit itself given the cluster that the entire build was. Just par for the course. But we truly never had any issues with it (other than wifi connection) until this firmware update.

I'd like to think I am pretty savvy with things and the thought process of a lightning strike just doesn't add up for me here. Am I crazy? I know it isn't 100% out of the realm of possibilities but neither is the possibility of it not being lightning and there is no way to definitively prove one either way.
 
I spent 40 years in IT - started as a developer, became a lead systems architect, and finished my career as Director of Solution Architecture at a very large company. I've seen it all. With regards to it being the firmware, I have to say the probability is extremely low - nearly non-existent. Although anything is possible, this is unlikely for a number of reasons:
  1. Firmware is heavily tested at every company before it is ever released. It's tested using the proper steps AND it's tested for every conceivable incorrect installation as well.
  2. A user will never see a corrupted firmware file get installed. The first safeguard is a checksum that validates that the file that was downloaded survived the download process and is good to install. If one byte is different in the file it will show as corrupted and the software that installs the firmware won't read the file and process it. The 2nd safeguard is in the firmware install software itself. There are a number of checks in place to validate that the firmware is the correct file for the hardware it is being installed on as well as checking to see that the file does indeed match what a good file should look like.
  3. The most common issues with a firmware install is a user installing different versions on different components and power failure during the install. The first may make components either not work properly or not work at all, but can be easily corrected by updating the incorrect component. If a power failure occurs in the middle of a firmware update, it generally results in a bricked system (i.e. dead and not working). The latest firmware now has a failsafe to recover from that, but earlier versions did not. A Hayward tech with special equipment could get it running again. Neither of these issues cause physical damage.
  4. If, on the 0.1% chance, the firmware did have a specific flaw that could cause an issue, it wouldn't just be seen on a single OmniLogic - it would be seen on many in the field, and you would see posts all around concerning the issue.
Again, while anything is possible, it is highly unlikely that a firmware update caused the issue you are seeing.

With regards to a lightning strike... this could be caused by any type of surge in the system. A strike close to the home could come through a pipe, the ground, anything that feeds to the components. The home itself could have a wiring issue that causes a surge that would not be seen by a whole home protector. It could even be a surge coming back from connected equipment. Electricity is a sneaky thing. With regards to where you see the damage - I've seen instances where a single board is damaged in the middle of others, and other instances where the entire panel is blown. It depends the on the size and duration of the spike. A lower spike will damage the weakest component in it's path first - it may be only one board, but that one board may then leak it to others. Again, it's very unpredictable.

I would suspect improper installation, but you state it was running fine for years, so I don't see this as a strong candidate.

In my mind the most likely culprits here are:
  1. The new board that was installed (or the install of the board itself). I find it interesting that the issues occurred after the board install. It's not uncommon for a board to go bad, but to have multiple boards go bad is uncommon. Perhaps the new board is bad, but it would be more likely that the actual install would cause the issue. Was the system shut down properly, was everything grounded before replacement, were any wires moved? Those types of things. Again, uncommon, but possible.
  2. A faulty ground wire - perhaps a wire came loose somewhere or even got broken.
  3. A surge from the home or connected equipment.
I would talk to Hayward and see if you can actually get them to send a tech out to evaluate the issue. I would also get a good electrician to take a look at it and see if everything looks good to them. They can test for proper wiring and grounding.

Hope you get this resolved!
 
PoolBrews, Thank you SO much for taking the time to type out such a lengthy reply. Everything you said makes sense. I truly appreciate you explaining everything esp about the firmware. That is something I don't understand so reading all of that was so helpful.

In regards to the 3 things you suspect could be the possible causes and the questions CPS asked, I spoke to my husband and he showed me the components that were removed from the OmniLogic. The rep that came out from the local pool store is a certified Hayward tech. He replaced the various parts until he got the system working again. This included the main board, the display, the PSU, and the tcell connector (I hope I am calling that the right thing but you will see a pic of it in a second).

Now, when I was out of town, I was told "everything was burnt up". What I see is the connection from the tcell to the main board js burnt and that is it. Everything else appears fine but clearly wasn't working.

I did notice this "new" PSU that was just installed 2 weeks prior looks a bit banged up (maybe they all do?) and it has a sticker on it with a date code of 1735 which maybe means it was made or maybe refurbished March 5, 2017? Which would make me wonder if this "new" PSU is what caused the problem, as PoolBrews possibly suspected. Is that what this date code means? The sticker is from Contech (see below).

I took a bunch of photos of the parts that were removed. I hope these are helpful. As for the question "was the system shut down properly before the PSU install" etc, well, no. Everything was up and running and then the original PSU went out and it turned the entire system off. It would not turn on obviously since there was no way to convert the power to the system. I called the local rep and told them I troubleshot it and it was the PSU and they ordered it and came out and replaced it and booted the system back up. What he actually did when he got here, I have no idea, but he did confirm the PSU was bad. He also replaced one of our accent lights. That is all I know. With covid and everything, we haven't been hanging around while they do their work as I am high risk.

PSU
118216029_10221351220597197_5732610990972783633_o.jpg


118075251_10221351220997207_6788733115777407754_o.jpg


117826797_10221351221277214_8111045918285729330_o.jpg


Display
117950274_10221351221917230_1539267015617106163_o.jpg


Main Board
118242558_10221351222277239_8053660622471959927_o.jpg


118083166_10221351222637248_2027258848772176650_o.jpg


Tcell
118015613_10221351222917255_8512022534827289702_o.jpg


Main Board
117811868_10221351223357266_7025852609195363623_o.jpg


118067607_10221351223717275_498302411380245673_o.jpg


117707460_10221351224077284_2712585832553654595_o.jpg
 
We've seen a couple that were burnt exactly like that. When it happens, you need a new T-Cell board and main board. This issue and those two boards SHOULD be 100% covered by the warranty.

If it was just the main board and tcell board, they may have approved the warranty claim. However, since it was those 2 things PLUS the PSU and the display, Hayward is saying it has to be lightning for all of those things to go bad at once.

I am still not convinced it was lightning. I am also questioning this PSU assembly that looks like it is not a new part. :unsure: Esp if this all happens again in a few weeks for a 3rd time now that our warranty has expired. Nothing I can do on the weekend either.
 
Thread Status
Hello , This thread has been inactive for over 60 days. New postings here are unlikely to be seen or responded to by other members. For better visibility, consider Starting A New Thread.