The Energy Detective Forums

General Category => Developer Support => Topic started by: hnrsoftware on April 17, 2013, 06:44:26 PM

Title: HTTP Fetches - load on gateway?
Post by: hnrsoftware on April 17, 2013, 06:44:26 PM
This is probably aimed at someone from the Gateway software development area, but I welcome thoughts from anyone who has done Footprints-alternative software development.

1. Fundamentally, in order to do analysis/graphing of TED5000 data in any ways other than those provided by Footprints, it is necessary to export and process the readings data.  I am comfortable with a Windows Utility I have written (freely available to anyone who wishes it) which exports the minute by minute data and maintains it in single day files (1 per MTU) and a "composite" file of all the MTUs together.  I run mine with Windows Scheduler every 10 minutes and the impact is not noticeable.  It can be run as infrequently as once per 2 days and it will maintain complete data.  I then do my analysis and graphing from these files rather than accessing the gateway.

2. Second by second data has some different problems associated with it.  Since Footprints resides in the gateway, it has the second data right at hand to graph.  (on the other hand, the gateway has a smaller cpu than a desktop machine, so Footprints must use a relatively heavy amount of local resources).  I have adapted my data exporting utility to extract and maintain second by second data, but I'm not particularly happy with the results.  The HTTP/RAW interface is decently efficient, but it takes a minimum of 1/3 second to obtain just 1 second of data.  With my 3 MTUs, this would be a theoretical continuous accessing of the gateway to keep current, but I have no idea if this would be too much load on the gateway - I would not want to prevent it from keeping itself updated from the MTUs and performing its own summarizing functions.   A somewhat better compromise would be 3 half-second fetches of 10 seconds of data, every 5 seconds, but that still seems like a lot of load.  Maybe every 20 seconds or 1 minute would be better, but then the data stops looking like it is realtime.

3. The approach of maintaining an exported data file of the day's second by second data seems to work adequately (updating several times an hour), but an 86,400 record CSV file is unwieldy, and I'm not sure anyone really wants a whole day's worth of data for any reason.  Graphing this data is slow, mainly just skipping data not being plotted.  I would welcome thoughts as to how anyone wants to store and use second by second data in ways they can't do with the Footprints software.

4. My main worries at the moment are concerning how much external http load a gateway can tolerate.  Does it simply delay or ignore requests when it is busy?  Will it start missing readings?

-- In light study of exported second data, I am seeing 1-second gaps in data at apparently random intervals, but occurring on all three MTUs at the same points.  I can understand interference in the powerline communication, but I would expect the MTUs to be affected separately and I would expect shorter and longer than 1 second gaps.  I am wondering if the gateway is getting distracted by something.  I intend to look at this further, but nothing leaps out.  It is entirely possible that my algorithms for merging 3 MTU readings into a single record is failing - just don't have an answer yet.

Howard -- h_richoux "at" 
Title: Re: HTTP Fetches - load on gateway?
Post by: RussellH on April 17, 2013, 09:21:30 PM
Off the top of my head, isn't there a "RAW" format available?  Perhaps that puts less of a CPU load on the gateway to export that.

Title: Re: HTTP Fetches - load on gateway?
Post by: hnrsoftware on April 18, 2013, 02:34:40 AM
Yes, I use the RAW http access.  One trouble is that it takes 3 separate http fetches to retrieve the data for 3 MTUs.  Even though the amount of returned data is very small, the round trip through Windows http support is a minimum of 1/3 second each.  Fetches of 60 seconds or so of data are 1/3-1/2 second per MTU.  10 minutes (600 seconds) of data only takes just under 1 second each. 

The point is that, to emulate Footprints displaying 3 MTUs worth of second data in "real time", would have a Windows program running flat out constantly doing retrievals of 2 or 3 seconds worth of data in each retrieval (to avoid missing any).  I would guess that most of the time is spent in http protocol/message handling within Windows, but I'm trying to find out if hitting the gateway with constant retrievals would affect its ability to do its main job - handling the MTU data.