What to do with boat data....

G

GHA

Well-Known Member
Joined
26 Jun 2013
Messages
12,552
Location
Hopefully somewhere warm
Visit site
So do I need a database, what to do?
The story so far... not finished yet but it looks like an arduino mega will be a busy hub of data onboard. Probably including engine revs, temp, exhaust temp, wind speed and relative direction, barometer pressure, battery voltage, GPS data plus no doubt many more. So since this is all happening it seems silly not to record it all. Easy on the arduino, just write it to a sd card. And should be easy to transfer over to a raspberry pi which acts as the nav computer running openplotter.
Then what?
So far the data is basically a .Csv file with everything time stamped.
So it would be nice to do something with all that lovely data, Like graph revs against engine temp over a days passage. Or whatever. And get it up onto the cloud.
My knowledge of databases isn't that great, so...... create something like a mysql locally in a drop box and let drop box take care of the uploading when WiFi is available ?
Or append a Google sheet each time WiFi is available?
With the IoT taking off there seems so much choice already.... hmmm..
Instinctively mysql seems the way to point the learning curve........




Thoughts?
 
GHA said:
So do I need a database, what to do?
The story so far... not finished yet but it looks like an arduino mega will be a busy hub of data onboard. Probably including engine revs, temp, exhaust temp, wind speed and relative direction, barometer pressure, battery voltage, GPS data plus no doubt many more. So since this is all happening it seems silly not to record it all. Easy on the arduino, just write it to a sd card. And should be easy to transfer over to a raspberry pi which acts as the nav computer running openplotter.
Then what?
So far the data is basically a .Csv file with everything time stamped.
So it would be nice to do something with all that lovely data, Like graph revs against engine temp over a days passage. Or whatever. And get it up onto the cloud.
My knowledge of databases isn't that great, so...... create something like a mysql locally in a drop box and let drop box take care of the uploading when WiFi is available ?
Or append a Google sheet each time WiFi is available?
With the IoT taking off there seems so much choice already.... hmmm..
Instinctively mysql seems the way to point the learning curve........




Thoughts?
Click to expand...

I would think that a spreadsheet is more suitable, especially as you mention graphs. But thats more analysis, not storage, in which case why not stick with csv instead of something you arent really familiar with but just sounds like it might be right.
 
Classic IoT thinking - starting with the technology that's available instead of a problem that needs solving, resulting in a toaster that's connected to the Internet for no good reason :p

(I'm teasing really; if you enjoy hacking on this stuff then that's all the justification required. But it does perhaps point towards an answer - if the work is technology-driven rather than solving a problem, perhaps the answer is "whatever technology you think will be most fun to play with"?)

Pete
 
prv said:
. But it does perhaps point towards an answer - if the work is technology-driven rather than solving a problem, perhaps the answer is "whatever technology you think will be most fun to play with"?)

Pete
Click to expand...
But is solving a problem, the arduino is multiplexing loads of sensor data before it gets pumped into a raspberry pi to be displayed or alarms triggered. Since all that data is there for the taking it seems silly not to record it.

Anyway, mysql looks like it might be the way to go for now, with maybe gnuplot and some scripts to graph it. :cool:
 
I've been working on a similar project with a raspberry pi, connected to a nmea 2000 network using the Actisense NGT-1 cable. So far, it is reading all the network data (using the canboat software) and translating to nmea 0183 (my code) for wifi access by iPads and such like as well as feeding OpenCPN on the pi itself. Next step is an automated log book using mysql and some graphing tools, with alerts for falling barometer readings, xte and others I haven't thought of yet. But like Pete says, its a bit of fun - although I'm hoping that it comes in useful at some stage.
 
Last edited:
How and where you are going to display the data will have an input to how you store it. If you are going to display the data on the boat via a Web page served up by the Pi, then mysql on the Pi would work. Replicating mysql databases to another server can be a real pain though, so if you want the data off the boat then a csv file is probably simplest.

Also, have a look at Google Charts for displaying the data!
 
adwuk said:
I've been working on a similar project with a raspberry pi, connected to a nmea 2000 network using the Actisense NGT-1 cable. So far, it is reading all the network data (using the canboat software) and translating to nmea 0183 (my code) for wifi access by iPads and such like as well as feeding OpenCPN on the pi itself. Next step is an automated log book using mysql and some graphing tools, with alerts for falling barometer readings, xte and others I haven't thought of yet. But like Pete says, its a bit of fun - although I'm hoping that it comes in useful at some stage.
Click to expand...
Have you checked out openplotter yet?
It has a powerful trigger/alert system built in, plus signalk beta. Signalk seems to be moving forward as well..
http://demo.signalk.org/instrumentpanel
Plus some guys are working on another webpage based instrument panel which should find its way into openplotter..

https://youtu.be/scxcvnoXXbU
 
I play with such things in excel. You can do pretty subtle things, but it has to be admitted that it is hard to produce robust systems whose innards you'll understand when you revisit after a six month pause.

I recently used excel to analyse a hill walking track and plot my speed against steepness of gradient. I have also got bits for analysing sailing routes and tracks.
 
MySQL local web server, php and JavaScript - there's your solution for hours of fun learning.

Getting data in and out of MySQL is so easy it's untrue, take a look at navicat- available on Mac and Windows. Also if using a free lamp stack (google it) you will probably have phpmyadmin installed anyway which does almost the same job. It used to be difficult but these days it's trivial to play with data.

Another way is to sign up for a BAAS such as Parse, also free for starters which makes the usage of data trivial.

Reason for suggesting the web systems approach is that you'll be able to code in one hit for any device that can display a web page. Phones, TVs.. Toasters etc �� And it's easy and rewarding very quickly.
 
GHA said:
But is solving a problem, the arduino is multiplexing loads of sensor data before it gets pumped into a raspberry pi to be displayed or alarms triggered. Since all that data is there for the taking it seems silly not to record it.
Click to expand...

None of that sounds like an actual end-user problem :)

This is a bit of a pet issue for me; I work on an "IoT cloud service" although in the underlying infrastructure rather than the application-level stuff that developers using us would see, let alone their customers with devices. But it does mean I see an awful lot of IoT announcements and press releases, and a significant proportion of them seem to have started with the thought of "hey, we can connect <random home appliance> to the Internet! Now let's try to justify why anyone would want that..." . Rather than starting at "it's really annoying that I always have to <whatever>, I wonder how I can make that easier" which might happen to end up with a network connection as part of the answer. Incidentally, plenty of them seem to be scarily insecure as well...

As I say, things are different for a hobbyist project where playing with the technology is a legitimate part of the goal, so I'm not trying to take a pop at your very interesting work.

Pete
 
All we kept was a log book which was updated every two hours with position, heading, wind strength and direction, any ships seen, barometric pressure and engine hours if any with a running total of that every page. I can't see the point of things like engine revs, exhaust temp, batterry voltage etc. All you need is the relevant information for passage making for the person coming on watch.
 
KellysEye said:
All you need is the relevant information for passage making for the person coming on watch.
Click to expand...

My first thought on seeing GHA's post was "Put all the data on a USB stick and bury it in a drawer. Then go sailing."
Come to think of it, that's also my second thought.

But I don't think his issue is need so much as technical amusement. Or perhaps a metaphysical fear that if his engine isn't producing data, it might not actually exist.
 
prv said:
None of that sounds like an actual end-user problem :)
Click to expand...
Maybe not a problem, but it's much more convenient having all the data on one screen in the cockpit (or bunk single handed offshore) , using an arduino to do the multiplexing is easier and so much cheaper than going off the shelf. And it turns out nasa wind & compass doesn't do a proper nmea checksum, so that needs sorting as well.
So the bulk of it is there for a purpose, getting wind, engine temp, gps, ais batt voltage into the Pi for easy viewing at a glance from anywhere onboard. That's where it started and that side of it makes a big difference, though a lot on top of that core is for the fun of tinkering as well , a worthy end on its' own :cool:
 
I use MySQL and a web server.

My Arduino has a TCP/IP stack available due to the GSM card. This has embedded HTTP commands. Therefore the arduino simply makes a http call to my webserver with the values as parameters. Note: some details have been changed to protect my system.

So the Arduino makes this web call to the server...

Code: 
http://www.yourwebsite.com/yourscript.php?value0="15"&value1="54"&value2="13"&value3="13"&value4="1"

This is the code in the arduino

Code: 
String webstring = "AT+HTTPPARA=\"URL\",\"http://";
  webstring.concat("YOUR_URL");
  webstring.concat("/yourscript.php?value0=");
  
  webstring.concat(t);
  webstring.concat("&value1=");
  webstring.concat(h);
  webstring.concat("&value2=");
  webstring.concat(batt1);
  webstring.concat("&value3=");
  webstring.concat(batt2);
  webstring.concat("&value4=");
  webstring.concat("1");
  webstring.concat("\"\r\n");
  sprintf(webbuffer, "%s", webstring.c_str());
  printLCD(ALERT_ROW, F("HTTP GET Param"));

On the webserver I then have a php script as follows

Code: 
<?php

$value0=$_GET['value0'];
$value1=$_GET['value1'];
$value2=$_GET['value2'];
$value3=$_GET['value3'];
$value4=$_GET['value4'];


$opendb=mysql_connect("localhost","USERNAME","PASSWORD") or mysql_error("ERROR");
mysql_select_db("DATABASE",$opendb);
if ($opendb){
  $query = "INSERT INTO DATABASE VALUES( '', '$value0', '$value1',  '$value2', '$value3', '$value4', UNIX_TIMESTAMP());";
  /* Run the query */
  $result= MYSQL_QUERY($query) or die( "ERROR" );
  mysql_close($opendb);
  echo "DBOK";
} else {
echo "ERROR";

Once its into a database on a webserver then the "world is your lobster" :) :) - you can present and manipulate the data in a multitude of ways.

You can run a small server on a Raspberry Pi
 
macd said:
My first thought on seeing GHA's post was "Put all the data on a USB stick and bury it in a drawer. Then go sailing."
Come to think of it, that's also my second thought.

But I don't think his issue is need so much as technical amusement. Or perhaps a metaphysical fear that if his engine isn't producing data, it might not actually exist.
Click to expand...

See the post above, it's all there primarily for a reason with bells ontop. :cool:
 
Sounds interesting to me. Of course the prime goal of the data is to provide useful functionality such as engine temperature alarms etc but since it is not that difficult to record the data then why not. Once you start collecting data then comes the issue of analysing it. You may be able to relate engine faults to past data and thus be more able to predict problems in the future. Or, you may be able to predict local weather variations from your weather data. You don't know what functionality you might gain until you have the data to analyse. Then there is simply the fun and geekiness of looking at all you graphs and seeing that the diagonal steam trap continues to read zero.
 
Well, gnuplot seems to be more than capable of making pretty graphs.
This is engine block temp and exhaust temp from a little while back..


e1UAX2k.png
 
You must log in or register to reply here.

Other threads that may be of interest

F
Emtrak AIS transponder configured and working, not showing my position in Navionics unless
Replies
5
Views
288
FairweatherDave
F
EugeneR
Help with getting sold boat data
Replies
8
Views
514
Tranona
T
dgadee
Schaefer 650 Furler - what can I do with it?
Replies
1
Views
109
dgadee
dgadee

Share this page

Top