Live boat instruments' data on internet via GSM - open source project

[GHA]

RealVNC is great, but you have to know where the target is located (e.g. Marina's network), and hope that they aren't blocking the VNC TCP ports (never mind tedious stuff like port forwarding and other boats connected to the network). Outbound is trivial, but initiating a connection from outside .. how does openplotter get around that?

No idea, I just click on connect in the vnc app on a laptop from anywhere and it does. Usually with the Pi connected to a mobile phone wifi network for web access. Port defaults to 5900 but is configurable on the Pi, but isn't that just local connections - not cloud connections?.
There are no port settings in the cloud setup on the viewer,
Why do you need to know where the Pi is located? Not needed for VNC , just need to log on to vnc on both the Pi & app and the app will find the Pi.

 

[AngusMcDoon]

Are you getting around this issue by only accessing the data indirectly through a remote server which hosts the data for you rather than talking to the boat system directly? If so could you point me in the direction of services that allow you to do this?

Yes, that's exactly what I'm doing, via MQTT. MQTT is a publish/subscribe means of communicating. All clients connect to a server (called a broker) and any client can subscribe to a message identified by its topic (like a directory but ending in a topic name... foo/bar/saunders etc.) and any client can publish to a topic. The data can be anything you like, from 1 byte to megabytes. It's often text (like I use in this project) or JSON, but can be binary.

The IP address of the broker is fixed and known in advance so dynamic IP addresses don't matter. There are a few public free brokers about, I use Hive. Only the insecure version is free but I don't care if anyone knows the windspeed at my boat. Everyone shares the same topic space but you are allowed 250 characters so you can make yours unique.

MQTT has 3 quality of service (QoS) which have higher levels of guaranteeing that the message got through. I use the lowest level - fire and forget. If a publish doesn't make it to the broker the next publish will. This is fine for non-critical data like boat monitoring.

MQTT was designed for satellite transfer which makes it very lean in bytes used. Some packets are only 2 bytes in length. Because of this it is also the protocol of choice for low bandwidth mobile connections. It works well with slow GPRS where other protocols with higher bandwidth needs like some of those suggested on this thread will not work so well over GPRS. Because of this it has become the protocol of choice for internet of things devices (IoT) that will often have only a low bandwidth connection or low data transfer requirement. Not all connected device in the future will have huge bandwidth. In 5G/LTE there are new mobile technologies coming along that are deliberately low bandwidth especially for IoT devices. They will have low power requirement and better range than the high speed technologies. The modems are starting to become available but not on cheap breakout boards yet. GPRS still has some life left in it. It can't be switched off yet because of all the devices that use it still (like many smart meters).

Hive have a webpage that lets you play here...

MQTT Websocket Client

Just click the connect button, leave all the other stuff as is.
Under subscriptions click Add New Topic Subscription button.
Enter foor/bar/saunders in Topic, choose QoS 0 and click Subscribe button.
Under Publish in Topic enter foor/bar/saunders, choose QoS 0, enter something under Message, click Publish button
The new message should arrive under Messages

If you are using something like a RPi then MQTT will already be available or easily downloaded as a package. It's available for Arduino and there are also free libraries for embedded devices like MQTT-C that can be ported. If you need a QoS 0 sub-set for very small devices there's now another free implementation available - mine .

 

[GHA]

. It seems like NodeRed in openplotter could be used to manage data distribution through the attached modem to an external site.

If you go for MQTT to send the data from the Pi then signalk which is preinstalled in openplotter has an app to do just that >
signalk-mqtt-gw
Or node-red has an mqtt node which would allow better control over what and when gets send plus allow some , node-red also is an app in signalk. Both extremely powerful programs
Openplotter promising to get more involved in IOT using telegram soon as well.
Telegram – a new era of messaging

 

[fifer]

Yes, that's exactly what I'm doing, via MQTT. MQTT is a publish/subscribe means of communicating. All clients connect to a server (called a broker) and any client can subscribe to a message identified by its topic (like a directory but ending in a topic name... foo/bar/saunders etc.) and any client can publish to a topic. The data can be anything you like, from 1 byte to megabytes. It's often text (like I use in this project) or JSON, but can be binary.

The IP address of the broker is fixed and known in advance so dynamic IP addresses don't matter. There are a few public free brokers about, I use Hive. Only the insecure version is free but I don't care if anyone knows the windspeed at my boat. Everyone shares the same topic space but you are allowed 250 characters so you can make yours unique.

MQTT has 3 quality of service (QoS) which have higher levels of guaranteeing that the message got through. I use the lowest level - fire and forget. If a publish doesn't make it to the broker the next publish will. This is fine for non-critical data like boat monitoring.

MQTT was designed for satellite transfer which makes it very lean in bytes used. Some packets are only 2 bytes in length. Because of this it is also the protocol of choice for low bandwidth mobile connections. It works well with slow GPRS where other protocols with higher bandwidth needs like some of those suggested on this thread will not work so well over GPRS. Because of this it has become the protocol of choice for internet of things devices (IoT) that will often have only a low bandwidth connection or low data transfer requirement. Not all connected device in the future will have huge bandwidth. In 5G/LTE there are new mobile technologies coming along that are deliberately low bandwidth especially for IoT devices. They will have low power requirement and better range than the high speed technologies. The modems are starting to become available but not on cheap breakout boards yet. GPRS still has some life left in it. It can't be switched off yet because of all the devices that use it still (like many smart meters).

Hive have a webpage that lets you play here...

MQTT Websocket Client

Just click the connect button, leave all the other stuff as is.
Under subscriptions click Add New Topic Subscription button.
Enter foor/bar/saunders in Topic, choose QoS 0 and click Subscribe button.
Under Publish in Topic enter foor/bar/saunders, choose QoS 0, enter something under Message, click Publish button
The new message should arrive under Messages

If you are using something like a RPi then MQTT will already be available or easily downloaded as a package. It's available for Arduino and there are also free libraries for embedded devices like MQTT-C that can be ported. If you need a QoS 0 sub-set for very small devices there's now another free implementation available - mine .

Thanks very much!

The key aspects of this for me are:
- price - how much does the kit cost
- power consumption - how much electrical power is wasted in capturing, processing and displaying/sending the data
- quality and quantity of data - priority data is battery voltage, then solar panel generation, wind speed after that.
- safety - the more bits of electronics I have daisy chained together, the greater the risk that there might be a fire hazard whilst the boat is unattended

Still trying to work out what the optimum solution is, but it definately appears to me that having a low power and low bandwidth data acquisition and export system on board is the way forward, as opposed to remote operation of something like a Pi via VNC, which will be more expensive in power and bandwidth. VNC does seem to have produced an excellent off the shelf tool however. Perhaps a Pi zero running openplotter just to handle data transfer might be the best compromise here.

I also like the idea of pulling data off the MQTT broker into an sql database, as well as building a website from scratch which will allow me to display both current values and time series historical data. A lot of this is overkill, but I see it as a good hook to help me learn all these skills! Bits and pieces of front and back end web programming involved here which could be quite interesting to do as a project.

 

[fifer]

If you go for MQTT to send the data from the Pi then signalk which is preinstalled in openplotter has an app to do just that >
signalk-mqtt-gw
Or node-red has an mqtt node which would allow better control over what and when gets send plus allow some , node-red also is an app in signalk. Both extremely powerful programs
Openplotter promising to get more involved in IOT using telegram soon as well.
Telegram – a new era of messaging

Good to know!

 

[GHA]

Good to know!

Host, run, and code Python in the cloud: PythonAnywhere might be worth a look for writing to a database as well and running some scripts to create plots, python does some pretty pictures . Big plus for Angus is low power & no web required. Rpi is quite simple and easy to add sensors but more power and web access needed.

 

[GHA]

Thanks very much!

The key aspects of this for me are:
- price - how much does the kit cost
- power consumption - how much electrical power is wasted in capturing, processing and displaying/sending the data
- quality and quantity of data - priority data is battery voltage, then solar panel generation, wind speed after that.
- safety - the more bits of electronics I have daisy chained together, the greater the risk that there might be a fire hazard whilst the boat is unattended

Still trying to work out what the optimum solution is, but it definately appears to me that having a low power and low bandwidth data acquisition and export system on board is the way forward, as opposed to remote operation of something like a Pi via VNC, which will be more expensive in power and bandwidth. VNC does seem to have produced an excellent off the shelf tool however. Perhaps a Pi zero running openplotter just to handle data transfer might be the best compromise here.

I also like the idea of pulling data off the MQTT broker into an sql database, as well as building a website from scratch which will allow me to display both current values and time series historical data. A lot of this is overkill, but I see it as a good hook to help me learn all these skills! Bits and pieces of front and back end web programming involved here which could be quite interesting to do as a project.

Just been having a bit of a play - an option is have something running node-red, might as well be signalk as it's so wonderful, on the boat. Pi zero should easily cope with it. In node=red send every nmea sentence to an online broker. Then at home more node red to read the mqtt and send it locally as udp which opencpn can read

Again a Pi could do the work at home, signalk/openplotter makes it easy to install influxdb time based database and signalk has an app to write everything you want to that for logging. Not as much net traffic as vnc but still all the data. Not as sexy as Angus's Pic though, but still quite pretty

 

[laika]

RealVNC is great, but you have to know where the target is located (e.g. Marina's network), and hope that they aren't blocking the VNC TCP ports (never mind tedious stuff like port forwarding and other boats connected to the network). Outbound is trivial, but initiating a connection from outside .. how does openplotter get around that?

To expand upon GHA's reply...seems realvnc (the company) have a cloud offering which is bundled with raspbian:
Raspberry Pi | VNC® Connect
A 10 minute wade through the marketing blurb on their website hasn't revealed the mechanics of how it works but presumably you create an account with their cloud service, server maintains an outbound connection to that from the pi, client connects to the cloud service with the same account and connection is made there. Hence no need for port forwarding etc. at the marina end. (but that's just a guess....)

 

[GHA]

To expand upon GHA's reply...seems realvnc (the company) have a cloud offering which is bundled with raspbian:
Raspberry Pi | VNC® Connect
A 10 minute wade through the marketing blurb on their website hasn't revealed the mechanics of how it works but presumably you create an account with their cloud service, server maintains an outbound connection to that from the pi, client connects to the cloud service with the same account and connection is made there. Hence no need for port forwarding etc. at the marina end. (but that's just a guess....)

Not sure how it works under the hood but that's pretty much it. Create an account on their website, on the Pi in VNC log on with the same details (and select unix password) , then on your vnc app on the laptop there's a tab called MyName team, and Pi that has logged on will show up on that, double click on that name and if it's online in you go. Old machines you've logged on with before but don't use anymore can be deleted on their web site so they don't show up anymore. Simple and very handy,