All zeroes after TED5000 Beta R482/R261 upgrade

[altenbach]

Well, the story is a bit more complicated, but at the moment all reading are zero forever.

It took a few failed attempts to upgrade the firmware, it would always restart after about 60% using the CD tool, same when using the browser, but after a few attempts it went to 100% and everything seemed OK.

After watching it for half a day, I noticed that one MTU (solar) does not send any data.

Decided to start over from scratch. I cleared everything, including history and settings, and re-uploaded the firmware. Now the firmware upgrade proceeded without problems (both files). Then I restored my settings and utility backup.
Everything looks right, including the MTU IDs, etc.

That was last night.

This morning, the footprint readings were still all zero, even though a significant amount of packets were received according to the stats page. Powercycling the gateway did not help. Later, I went into the panel and cycled the MTUs.
This did not help either.

Physically, everything looks normal. The MTUs are flashing at the typical rate. The LED on the side of the gateway flashes also at a regular rate.

However, all footprint readings are zero and the stats page is all zeroes for "MTU Rec." and "MTU skp". "LP sample" is at 2147483647 for both MTUs. Uptime right now is ~12000 seconds,  MTU IDs are correct.

Note that my hardware setup worked reliably for a very long time. I also have a filter installed. Nothing changed except the firmware upgrade.

Any ideas on how to troubleshoot further? I can give the remote access URL via PM if needed (I don't want it public).

[rotus8]

Have you checked the MTU IDs in the System Setup Wizard Product ID tab? They may have been corrupted when you did the update.

[altenbach]

Yes, as I said above "...Everything looks right, including the MTU IDs..."

[TedDev]

You may want to open up a support ticket to have one of our techs walk you through a couple of things to see if the issue can be determined.

Just a couple of quick questions (for my own curiosity):
   - on your stats.htm page are you seeing changing values for LAST MTU ID (it shows the serial id of the MTU that has sent the most recent packet). 
   - Are you manually re-entering you system settings or restoring them from a backup? if a backup, have you tried manually entering the system settings back in after the upgrage/reset to factory defaults?

[altenbach]

Yes, the last MTU ID was zero, there was not a single packet received (checked even a few hours after reboot). However, about 7 hours later (at ~4pm in the afternoon according to the history), the thing started working again as if nothing happened. Without any intervention on my part.

Is there an explanation?

As you can see from the hour history the first value is at 4 pm, while I rebooted everything around 9am that day.

What was it doing during that time? As I said, the lights were flashing regularly from the start.

[TedDev]

If the last MTU ID was zero on the stats page, then that means a packet wasn't received at all by the TED5000. The fact that it recovered sounds very much like something was interfering on the line itself (i.e. noise).  That MTU ID is set as soon as a packet is received (before any other logic is executed).

[altenbach]

Yes, there is a serious communication problem, but is is very strange and only occurs during certain times of the day.

The first indication was that I received alerts from the monitoring site that the sensor is down (now for several days in a row). The time was always 9:30am (within ~10minutes), after which there was no (0.000%!!!) communication from the MTU for 2-8 hours. before 9:30 am and after it recovers, the communication is rock-solid.

Obviously, there must be some external noise that arrives like clockwork every morning at the same time. None of my appliances and electronic devices fit that pattern as far as I can tell.

I'll do some more research and collect more data for better statistics. Stay tuned.....

[gregzoll]

altenbach do you live anywhere near industry?  What router are you using, and switch, if connecting the gateway through it.

[altenbach]

The internet is perfectly OK. It is only the communication (over the powerline) between MTU and Ted5000 gateway that goes down. I can always reach the ted.

They are currently digging a deep hole at a nearby park for some underground water storage tank, maybe it has something to do with it?

[wje]

I have a very similar problem after the latest update. Like you, mine has been working reliably (well, as reliably as they ever work) until I applied the firmware update. Yes, I did them in the correct order, the power stayed on, etc. I could even reload the saved config and history. But.....

My symptoms are that the display light blinks periodically, but the values never change. The web server crashes within minutes of a restart. I've reflashed, no change. Now, the installation utility detects my gateway, but times out when trying to reflash. All the lights are fine. After a year of putting up with flakey hardware and buggy software, I'm about ready to just throw the thing out. Great idea, terrible execution.

[wje]

I've gone back to the current official version, everything's working again.
After a wasted morning of reflashing/reloading/testing (deja vu), I've at least narrowed down the issue. Basically, things more or less work with the beta. Eventually, the remote display does start updating. Well, most times. The browser comes up, and I can look at the dashboard. BUT, when navigating off the dashboard tab, I can look at maybe 1 or 2 other tabs, then the gateway crashes. Hard. All the lights are off. Unplug, replug, works until I try to browse more. Crash. Repeat.