What to do with boat data....

[Hydrozoan]

Having pulled your leg (gently, I believe ) much earlier in this thread that the data ‘might come in handy’, I am pleased for you that your labours have proved successful.

I can see that spotting trends could indeed be useful, but how are you going to detect them in the longer term (especially in a multivariate situation)? I presume not simply by labour-intensive eyeballing, and wonder if you are thinking of adding statistical process control methodology – things like Shewart and Cusum charts, for example.

 

[[70521]]

Though do you really want to know or just want to infer that it's a waste of time

(note the post was to just for the one or two who might actually be interested, are you actually included in that?)

I actually wanted to know.

In the day job I deal with huge amounts of data, think trillions of data points per day and wanted to know the thinking behind collecting it, or was it just because I can.

In my sailing I collect very few data points, battery charge once a day, but monitor it frequently, the GPS data gets collected automatically and it is useful to analysis back on the PC. Temperature is closely monitored; do I put another jumper on or change into shorts that sort of stuff, but I don't record it.

 

[PaulRainbow]

(note the post was to just for the one or two who might actually be interested, are you actually included in that?)

I always read your posts and find them interesting, but i'm afraid i just don't have the time to do any of it.

 

[GHA]

I actually wanted to know.

Oops - apologies! Thought you were joining in with the "Why bother"...

TBH I think there's a good chance you won't know what it might show until you look at it , and people really are useless at seeing what's actually going on - a deep built in way the brain works is to see what you expect to see and ignore the gorilla on the basketball pitch
And this wasn't really that hard - the Pi has all the data going through it anyway so seems silly not to record it . Only a bit more node-red. Then big leap forward past couple of days getting python to make that lovely html page graph from it.

 

[[70521]]

Oops - apologies! Thought you were joining in with the "Why bother"...

No worries.

TBH I think there's a good chance you won't know what it might show until you look at it , and people really are useless at seeing what's actually going on - a deep built in way the brain works is to see what you expect to see and ignore the gorilla on the basketball pitch
And this wasn't really that hard - the Pi has all the data going through it anyway so seems silly not to record it . Only a bit more node-red. Then big leap forward past couple of days getting python to make that lovely html page graph from it.

It is a great video and a great training tool.

Have you seen the one where the car dives onto the "car park" that is actually a frozen pond.

 

[GHA]

Having pulled your leg (gently, I believe ) much earlier in this thread that the data ‘might come in handy’, I am pleased for you that your labours have proved successful.

I can see that spotting trends could indeed be useful, but how are you going to detect them in the longer term (especially in a multivariate situation)? I presume not simply by labour-intensive eyeballing, and wonder if you are thinking of adding statistical process control methodology – things like Shewart and Cusum charts, for example.

Need to google those big words!
Actually, eyeballing probably is pretty good for the more important bits like battery voltage. A lot of liveaboards must be pretty obsessed with batteries living away from shore power so having a record of it is really useful.
As for running some statistical magic - python must have all manner of goodies to do that but I'm not sure it would tell you that much, ambient temperature has a big effect on battery voltage , maybe you could factor that in somehow or just wait for it to get warm again! Could be fun to have a play and see what comes of it though.
Also, now that it all pretty much works a few other ideas have come up - node-red can record all of openweathermap data so it would be interesting to graph a 5 day forecast against actual. And should be pretty easy to make a little 4 channel voltage sensor attached to a wifi module spitting out at maybe 100 times a second to plot voltage drop when the engine starts & anything else which might come up. Cheap as chips all this stuff and the software is free

 

[Hydrozoan]

... Actually, eyeballing probably is pretty good for the more important bits like battery voltage. A lot of liveaboards must be pretty obsessed with batteries living away from shore power so having a record of it is really useful.

As for running some statistical magic - python must have all manner of goodies to do that but I'm not sure it would tell you that much, ambient temperature has a big effect on battery voltage , maybe you could factor that in somehow or just wait for it to get warm again! Could be fun to have a play and see what comes of it though. ...

Sure you can eyeball the battery voltage, for example. But I can do that quite easily – and I do.

I was thinking of spotting subtle trends or step changes, following your example of the engine running a bit hotter. If you can arrange for data obtained under broadly comparable conditions – e.g. engine revs, time of running, water temperature – to be picked out automatically from the mass of your data, you can look for subtle long term changes using a suitable statistical technique. Cusum (‘cumulative sum’) charts are good at spotting the onset of small changes because they use accumulated data, rather than just comparing the current value with some fixed ‘control limit’. For a discussion see for example:
http://www.itl.nist.gov/div898/handbook/pmc/section3/pmc323.htm and https://www.spcforexcel.com/knowledge/variable-control-charts/keeping-process-target-cusum-charts.

All a bit OTT some would say, but as that’s the way you’re thinking of using the data it might open up a new way to look at it, and help you avoid the ‘DRIP’ syndrome – ‘Data Rich, Information Poor’! But perhaps the maths/stats, though it's not rocket science, would not be as much fun as the data gathering and IT bits.

 

[GHA]

Also, now that it all pretty much works a few other ideas have come up - node-red can record all of openweathermap data so it would be interesting to graph a 5 day forecast against actual.

Well that was easy!
http://www.moondogmoving.co.uk/faro.html
Click on the wheel zoom third button down top left and put the curser over the x axis, wheel zooms x axis, same for Y axis.
Faro live wind since 3rd week of Jan. Just need to start getting the 5 day forecast recorded into the database to overlay. Lines for 1 day ahead, 2 day etc over what actually happened.

 

[AntarcticPilot]

Well that was easy!
http://www.moondogmoving.co.uk/faro.html
Click on the wheel zoom third button down top left and put the curser over the x axis, wheel zooms x axis, same for Y axis.
Faro live wind since 3rd week of Jan. Just need to start getting the 5 day forecast recorded into the database to overlay. Lines for 1 day ahead, 2 day etc over what actually happened.

Better if the display at the cursor position converted the time to something human interpretable instead of the internal format!

 

[GHA]

Better if the display at the cursor position converted the time to something human interpretable instead of the internal format!

One thing at a time!!


Edit - Better? Still loads to learn about the inner workings of the bokeh module.
http://www.moondogmoving.co.uk/faro.html