What to do with boat data....

[AntarcticPilot]

Well its me as an amateur playing around and coding on the extreme of my ability so it is bound to be completely full of bad practice and all advice is gratefully received! Not 100% sure what you mean by the concatenation part , is it this

$query = "INSERT INTO DATABASE VALUES( '', '$value0', '$value1', '$value2', '$value3', '$value4', UNIX_TIMESTAMP());";

rather than the concatenate statements on the Arduino?

Yes. What will happen if I submit to your webserver a $value4 consisting of foo'); DROP TABLE DATABASE; ? Work out what your $query string will end up being...

Is your webserver open to the whole Internet, or is there some kind of password that only your Arduino knows? There are thousands of machines out there patiently trying out automated attacks against every server they can find...

Pete

Yes, Pete has clarified what I meant. I'm not a PHP guru, so I don't know how PHP does it, but in other languages something like:


Preparedstatement mystmnt = connection.prepareStatement( "Insert into database values (?,?,?,?,?)");
mystment.insertvalue(1,value1);

etc.

mystmnt.execute;

Is what you do. The insertvalue methods ensure that the data is escaped and sanitized such that it cannot cause a problem.

As you can probably see, Java is my most recent language!

 

[prv]

Data feng Shui. That will be the next big thing.

i shall be available to line up your hard drive inner mystical energies for a harmonious boot up process.

I saw an article the other day interviewing a silly woman in the US who claims to do exactly this kind of thing. Burns special herbs to cast out viruses and suchlike.

Just goes to show that for many people computers are a kind of magic

Pete

 

[prv]

I'm not a PHP guru, so I don't know how PHP does it

Badly, apparently:

http://eev.ee/blog/2012/04/09/php-a-fractal-of-bad-design/#security

"The original built-in MySQL bindings, still widely-used, have no way to create prepared statements."

:hopeless:

Pete

 

[laika]

Burns special herbs to cast out viruses and suchlike.

Computer scientists have been burning special herbs whilst dealing with software issues for decades.

 

[Hydrozoan]

Ooops, I meant to put a smilie after the "it's not easy" to show i got the joke. ...:

Well not much of a joke - especially as I recall the late Charles Stock's totting up of his pence per nautical mile figure at the end of a particularly busy season, and thinking no way was I going to ...

 

[Amulet]

Lessons learned in IT (there are many more)

1. If you told the management something was impossible, don't tell them about the baroque, labour-intensive and error-prone method that you then invent after lying awake at night puzzling (enabling you to do it ONCE) . They'll just ask you to do it every Tuesday from then on.

2. If one of your IT staff comes in and tells you they've done something clever, fire them.

3. If one of your IT staff comes in and says "I found a pretty straight-forward solution", promote them.

My nav data processing methods are of class 1 - and I'd never share them!

 

[mlines]

Badly, apparently:

http://eev.ee/blog/2012/04/09/php-a-fractal-of-bad-design/#security

"The original built-in MySQL bindings, still widely-used, have no way to create prepared statements."

:hopeless:

Pete

A quick google appears to show that I probably have to use the same method to write to the database but I can apply some filter/sanitize php commands to ensure the input is clean

 

[smert]

Yes, Pete has clarified what I meant. I'm not a PHP guru, so I don't know how PHP does it, but in other languages something like:


Preparedstatement mystmnt = connection.prepareStatement( "Insert into database values (?,?,?,?,?)");
mystment.insertvalue(1,value1);

etc.

mystmnt.execute;

Is what you do. The insertvalue methods ensure that the data is escaped and sanitized such that it cannot cause a problem.

As you can probably see, Java is my most recent language!

Check out mysqli rather than plain mysql for connecting to your database. You can then use prepared statements like AntarticPilot suggests.
http://uk1.php.net/manual/en/mysqli.quickstart.prepared-statements.php

 

[prv]

I can apply some filter/sanitize php commands to ensure the input is clean

A couple of decades of experience (not mine) says that you can't

Pete

 

[SiteSurfer]

Yes you can Pete, people have been stripping slashes and sanitizing for ages.
However - mysqli is the way to go with prepared statements, or for even easier - try the Zend Framework Db class, its very very good.

You'll wind up with something like "Select foo FROM bar WHERE id = ?"; $data = $db->fetchAll('table',$value);
The second argument to fetchAll() is an array of values to substitute for parameter placeholders in the SQL statement.

http://framework.zend.com/manual/1.12/en/zend.db.adapter.html

 

[rgarside]

My data logger writes all the NMEA network activity to an SD card.

From the NMEA log file I generate a position, time and depth file that I send to TeamSurv [ http://www.teamsurv.eu ]. I also generate a .csv file. I have set up Python procedures to do all that.

Excel is not too good for plotting lots of datapoints, but I have found Veusz pretty good for quickly generating plots from the .csv files. See http://home.gna.org/veusz/

The data recorded includes NMEA 0183 RMC, RMB, DBT, MWV, VLW, VHW, and a couple of XDR sentences that carry temperature and pressure, and battery voltage. So there is sufficient to look at polar plots, track, battery, etc.

TeamSurv is an initiative to crowd-source hydrographic data to improve charts, especially in areas which are not frequently surveyed. If you have not heard of it, take a look at their website.

 

[adwuk]

I decided on RRDTool with/without Highcharts

Great tip, thanks. I now have my Raspberry Pi's barometer chip recording data and viewing live updates on my iPad. It was interesting watching it as the low passed through today.

 

[GHA]

Another area...

Another ebay purchase i just got round to hooking up..
https://www.adafruit.com/products/1085
4 channel ADC, or basically a very accurate volt meter.

this coming in from the arduino 3.3v pin..

Voltage: 3333.5625mV
Voltage: 3333.1875mV
Voltage: 3333.3750mV
Voltage: 3333.5625mV
Voltage: 3333.1875mV
Voltage: 3333.3750mV
Voltage: 3333.3750mV

Hard to know the accuracy but it tallies up well with the most accurate meter I have, Fluke 110.
That's in default more, there is an internal programmable gain amplifier as well so i wonder how accurately it could measure amps in/out by measuring the voltage drop across a battery monitor shunt..

very good site here about it all

http://henrysbench.capnfatz.com/henrys-bench/arduino-ads1115-module-getting-started-tutorial/

 

[vas]

Another area...

Another ebay purchase i just got round to hooking up..
https://www.adafruit.com/products/1085
4 channel ADC, or basically a very accurate volt meter.

this coming in from the arduino 3.3v pin..




Hard to know the accuracy but it tallies up well with the most accurate meter I have, Fluke 110.
That's in default more, there is an internal programmable gain amplifier as well so i wonder how accurately it could measure amps in/out by measuring the voltage drop across a battery monitor shunt..

very good site here about it all

http://henrysbench.capnfatz.com/henrys-bench/arduino-ads1115-module-getting-started-tutorial/

4 ADCs for 15USD, why not 5 20A current sensors for 7.5quid?
seriously, isn't it going to be too difficult and iffy to get any meaningful data out of 4th and 5th decimal of a Volt?

cheers

V.

 

[GHA]

4 ADCs for 15USD, why not 5 20A current sensors for 7.5quid?
seriously, isn't it going to be too difficult and iffy to get any meaningful data out of 4th and 5th decimal of a Volt?

cheers

V.

The accuracy required increases with the amount of wine consumed before clicking the ebay buy button

And the current sensors output a voltage, so it's still reliant on the arduino ADC & reference, though bang for buck it's maybe better to invest in an accurate voltage reference like a LM4040 instead of the internal arduino reference.
Or have another glass of wine and get something full on 16 bit

 

[vas]

The accuracy required increases with the amount of wine consumed before clicking the ebay buy button

And the current sensors output a voltage, so it's still reliant on the arduino ADC & reference, though bang for buck it's maybe better to invest in an accurate voltage reference like a LM4040 instead of the internal arduino reference.
Or have another glass of wine and get something full on 16 bit

all that wine's not going to help when you try programming the buggers!

I try to be sober when buying on ebay, as I've bought a few silly things over the years (still in a drawer in the garage...). However you cannot go that wrong with things costing 2, 3 or 5euro a piece. Granted you end up buying loads of them (checked my paypal and had bought around 120quid worth of arduino bits over the last month!) but still way too much fun to be had for the money

cheers

V.

 

[lpdsn]

seriously, isn't it going to be too difficult and iffy to get any meaningful data out of 4th and 5th decimal of a Volt?

Would make a good random number generator.

 

[AntarcticPilot]

Would make a good random number generator.

I don't think so - have you noticed that the resolution of the A/D converter seems to be 0.1875 mV? The decimal places only have a few possible values.

 

[GHA]

I don't think so - have you noticed that the resolution of the A/D converter seems to be 0.1875 mV? The decimal places only have a few possible values.

Might be a bit noisy down below a mV, even so, less than 2 quid if you don't mind waiting for one to turn up from China

 

[GHA]

I decided on RRDTool with/without Highcharts (depending on how fancy you want it to look)
example below of both.

Found it fairly simple too....which was important!

The beauty of these, is the size does not grow and grow, old data is averaged out for you.

View attachment 55532View attachment 55533

Well, I haven't spent anymore time at all for ages on this until having a quick hit at it this morning.

So rddtool is it!

What a handy powerful program, thanks

Now what is happening is that a Raspberry Pi running openplotter is the nav computer and wifi repeater, so it's on all the time basically, thankfully drawing little power with the monitor turned off.
From inside openplotter there's a powerful "actions" tab, a new action is now set up to repeat every 1 minute which calls a bash script passing some data to it, basically any nmea data which includes engine temp etc. The bash script then sends an update to the rrdtool database.

Simples!

The next release of openplotter promises to have a much better implementation of signalk so hopefully recording any data will be even easier.

http://oss.oetiker.ch/rrdtool/
https://sailoog.gitbooks.io/openplotter-documentation/content/en/index.html