= IoT swarm implementation ! Apostolos rootApostolos@swarmlab.io // Metadata: :description: Intro and Install :keywords: IoT :data-uri: :toc: right :toc-title: Πίνακας περιεχομένων :toclevels: 4 :source-highlighter: highlight :icons: font :sectnums: {empty} + == Intro .Intro IoT Swarm **** We will be trying to create a swarm implementation that will allow communication between all of the members/nodes. [NOTE] ==== .Imaging a swarm image:./Swarming2.png[alt="Swarm"] To undertand this better lets look at the picture bellow and imagine that red dots are iot devices that can send and receive and black ones are clients that gather data. .Architecture of swarm communication image:./swarmlabn-1.png[alt="Swarm Gateway"] - Red Node: Sensor Node and Gateway Role - Black and Red Node: Sensor Node - Client ==== *To make our life easier at this task we will be using the following tools...* **** == Prepare installation === http://docs.swarmlab.io/SwarmLab-HowTos/labs/Howtos/docker/install.adoc.html[Install docker^] [TIP] ==== *Docker* is a set of platform as a service (PaaS) products that use OS-level virtualization to deliver software in packages called containers. Containers are isolated from one another and bundle their own software, libraries and configuration files; they can communicate with each other through well-defined channels ==== === Clone IoT swarm implementation example [source.bash] ---- git clone https://git.swarmlab.io:3000/zeus/iot-swarm-example.git cd iot-swarm-example ---- ==== Webclient tree .base dir Webclient [source,bash] ---- src/browser-client/src/ ├── App.vue // <1> ├── assets │   ├── css │   │   └── themify-icons.css │   ├── fonts │   │   ├── glyphicons-halflings-regular.448c34a.woff2 │   │   ├── glyphicons-halflings-regular.e18bbf6.ttf │   │   ├── glyphicons-halflings-regular.f4769f9.eot │   │   ├── glyphicons-halflings-regular.fa27723.woff │   │   ├── themify.eot │   │   ├── themify.svg │   │   ├── themify.ttf │   │   └── themify.woff │   └── logo.png ├── components // <4> │   ├── doclive │   │   ├── AdhocView.vue │   │   └── runLlo.vue │   └── DocLive.vue ├── main.js // <2> └── store ├── index.js └── modules └── create_pipelineLLO.js // <3> ---- <1> load App <2> App config <3> Vuex and Rest calls <4> Components ==== IoT server tree .base dir IoT server [source,bash] ---- src/IoT/llo/ ├── bclient.js ├── client.js ├── iotclient.js // <2> └── iotserver.js // <1> ---- <1> IoT server <2> IoT client === Control services ==== start IoT server [source.bash] ---- cd iot-swarm-example ./start-iotserver.sh ---- ==== stop IoT server [source.bash] ---- cd iot-swarm-example ./stop-iotserver.sh ---- ==== start IoT client [source.bash] ---- cd iot-swarm-example ./start-iotclient.sh ---- ==== stop IoT client [source.bash] ---- cd iot-swarm-example ./stop-iotclient.sh ---- ==== start IoT client-n [source.bash] ---- cd iot-swarm-example ./start-iotclient-n.sh ---- ==== stop IoT client-n [source.bash] ---- cd iot-swarm-example ./stop-iotclient-n.sh ---- ==== start IoT webclient [source.bash] ---- cd iot-swarm-example ./start-iotwebclient.sh ---- ==== stop IoT webclient [source.bash] ---- cd iot-swarm-example ./stop-iotwebclient.sh ---- ==== stop All services [source.bash] ---- cd iot-swarm-example ./stop-all.sh ---- === Use webclient open in Browser: http://localhost:8080 and open Web Developer with ctrl+shift+K ==== send data in request body with GET type text in _"Get iot Data"_ and klick on it See _"action"_ in - *Console* in browser - and in Linux *Terminal* ==== Using Sockets to send and receive data type text in editor box See _"action"_ in - *Console* in browser - and in Linux *Terminal* == Technologies === MVC A core principle of the MVC pattern is the view layer's ignorance with respect to the model layer. Views are dumb objects. They only know how to present data to the user. They don't know or understand what they are presenting. MVC: Division across three code components only: Model, View, and Controller. ... Microservices: An app is divided into a set of specialized classes that interact with each other using APIs. This model is being used by companies like Netflix, Spotify, and eBay. Model: This part manages the data on your site. Its role is to retrieve the raw information from the database, organize, and assemble it so that it can be processed by the controller. View: This part focuses on the display. It is here where the data recovered by the model will be presented to the user. Controller: This part manages the logic of the code and makes decisions. When the user interacts with the view, the request is processed by the controller. It waits for the user to interact with the view to retrieve the request. Thus, it is the controller that will define the display logic, and display the next view on the screen. === Microservices Architecture Microservices can be defined as an improvement, a kind of refinement, of what we know as service-oriented architecture (SOA). In this architecture, a large application is made in the form of small monofunctional modules. Each microservice is autonomous. Microservices do not share a data layer. Each has its own database and load balancer. So that each of these services can be deployed, adjusted, and redeployed individually without jeopardizing the integrity of an application. As a result, you will only need to change a couple self-contained services instead of having to redeploy the entire application. == Software === Client site (PC) ==== Vue .vuejs [NOTE] ==== Vue.js is an open-source, progressive JavaScript framework for building user interfaces (UIs) and single-page applications. ==== *Library modularization* using a framework is common in frontend development. What differentiates Vue.js from other alternatives is: - its *“high decoupling”*, how easy it is to extend functionalities, and how well all parts work once more modules are included. For example, if we want to organize and render small visual components, all we need is Vue.js’s ‘core’ library. It is not necessary to include additional libraries. As the application grows, - we have libraries to manage *routes* such as *‘vue-router’*, - libraries to manage the global state such as *‘vuex’* - and libraries to build responsive web applications such as *‘bootstrap-vue’*. - Additionally, if our application needs to be optimized or needs good SEO, we can include the *‘vue-server-rendering’* library. In the following figure, we can see how the libraries we just mentioned are progressively included, from a small SPA to multi-page applications (MPA). image::./intro.png[intro] TIP: The name of the framework – Vue – is the same phonetically in English as view, and it corresponds to the traditional Model-View-Controller (MVC) architecture React and Angular are other Frameworks similar to vuejs ==== Vuex Vuex is a state management pattern + library for Vue.js applications. - It serves as a centralized store for all the components in an application, with rules ensuring that the state can only be mutated in a predictable fashion. - It also integrates with Vue's official devtools extension to provide advanced features such as zero-config time-travel debugging and state snapshot export / import. .What is a "State Management Pattern"? [NOTE] ==== *Let's start with a simple Vue counter app:* [source,javascript] ---- new Vue({ // state data () { // <1> return { count: 0 } }, // view template: ` // <2>
{{ count }}
`, // actions methods: { // <3> increment () { this.count++ } } }) ---- <1> The state, the source of truth that drives our app; <2> The view, a declarative mapping of the state; <3> The actions, the possible ways the state could change in reaction to user inputs from the view. This is a simple representation of the concept of "one-way data flow": ==== image::flow.png[one-way data flow^] image::vuex.png[advanced] TIP: https://scrimba.com/learn/vuex[Online Vuex cources^] ==== Using Axios to Consume APIs TIP: Axios is a library for http communication, making ajax requests, and so on. https://github.com/axios/axios[See more^] ==== Using socket.io to Consume Websocket Socket.IO aims to make realtime apps possible in every browser and mobile device, blurring the differences between the different transport mechanisms. It supports multiple transports, such as WebSockets, Flash sockets, long polling and more, automatically falling back when a transport fails https://socket.io/[See more info here: Socket.io^] Vue Packages - https://www.npmjs.com/package/vue-socket.io-extended[socket.io client^] - https://www.npmjs.com/package/vue-socket.io[socket.io] === Client/Server site (IoT device) ==== Nodejs TIP: As an asynchronous event-driven JavaScript runtime, Node.js is designed to build scalable network applications. Almost no function in Node.js directly performs I/O, so the process never blocks. Thanks to this, scalable systems is very reasonable to be developed in Node.js. https://nodejs.org/en/docs/guides/blocking-vs-non-blocking/[Overview of Blocking vs Non-Blocking^] TIP: *Node.js* is similar in design to, and influenced by, systems like *Ruby's Event Machine* and *Python's Twisted.* Node.js takes the event model a bit further. It presents an *event loop as a runtime construct* instead of a library. *In other systems, there is always a blocking call to start the event-loop.* Typically, - behavior is defined through callbacks at the beginning of a script, - and at the end a server is started through a blocking call like *EventMachine::run().* [NOTE] ==== In Node.js, there is no such start-the-event-loop call. - *Node.js* simply *enters the event loop after executing the input script.* - *Node.js exits the event loop* when there are *no more callbacks to perform.* ==== Node.js being *designed without threads* doesn't mean you can't take advantage of multiple cores in your environment. Child processes can be spawned by using our *child_process.fork() API,* and are designed to be easy to communicate with. TIP: Built upon that same interface is the cluster module, which *allows you to share sockets between processes* to enable load balancing over your cores. ==== socket.io *Socket.IO* is a library that enables *real-time*, *bidirectional* and *event-based* communication between the browser and the server. It consists of: - a Node.js server: Source | API - a Javascript client library for the browser (which can be also run from Node.js): Source | API image::bidirectional-communication.png[] https://socket.io/[See more info here: Socket.io^] ==== express Express is a minimal and flexible Node.js web application framework that provides a robust set of features for web and mobile applications. *Fast, unopinionated, minimalist web framework for Node.js* ==== NoSQL A NoSQL (originally referring to "non-SQL" or "non-relational") database provides a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations used in relational databases. https://www.mongodb.com/nosql-explained[More info] MongoDB is a general purpose, document-based, distributed database built for modern application developers and for the cloud - https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/[Intro] https://docs.mongodb.com/drivers/node/[MongoDB and Nodejs ] == Example Application === Client site ==== main (config) [source,javascript] ---- import Vue from 'vue' import App from './App.vue' import store from '@/store/index' import BootstrapVue from 'bootstrap-vue' import 'bootstrap/dist/css/bootstrap.css' import 'bootstrap-vue/dist/bootstrap-vue.css' import VueSweetalert2 from 'vue-sweetalert2'; Vue.use(VueSweetalert2); import VueSocketIOExt from 'vue-socket.io-extended'; // <1> import io from 'socket.io-client'; // <2> const socket = io('http://localhost:8084', { // <3> autoConnect: false }); Vue.use(VueSocketIOExt, socket); Vue.use(BootstrapVue); Vue.config.productionTip = false new Vue({ el: '#app', store, render: h => h(App) }) ---- <1> import socket package <2> import socket package <3> Connect to socket server ==== App (load) [source,html] ---- ---- <1> your html <2> import module <3> Send Message to other components ==== Create compoment [source,html] ---- ---- <1> your html <2> import your components help files <3> Export your component <4> your component files <5> insert your component <6> your methods <7> method to controll window behavior <8> on action ==== Use your component [source,html] ---- ---- <1> use component <2> import components ==== Sockets [source,html] ---- ---- <1> import module <2> import module <3> import module <4> json examples coming from IoT device <5> open socket <6> close socket See also See https://www.digitalocean.com/community/tutorials/vuejs-component-lifecycle[Vue.js Lifecycle Hooks^] ==== create store [source,javascript] ---- import { mapState, mapActions, commit } from 'vuex' // <1> import store from '@/store/index' import axios from 'axios' // <2> export default { namespaced: true, // <3> state: { // <4> llo: {}, socketid:'' }, getters: { // <5> getllosrc (state, container) { //console.log("js1 get "+JSON.stringify(state.llo)) return state.llo }, getsocketid (state, container) { //console.log("js1 get "+JSON.stringify(state.llo)) return state.socketid } }, mutations: { // <6> setllo (state, data) { //console.log("js1 set "+JSON.stringify(data)) state.llo=data; }, setsocketid (state, data) { //console.log("js1 set "+JSON.stringify(data)) state.socketid = data; } }, actions: { // <7> async get_data({commit,rootGetters}, value) { try { let p = await axios.get("http://localhost:8084/run", { // <8> timeout: 45000, params: { code: value.code } }); //var p = value.code console.log("paramp "+JSON.stringify(p)) store.dispatch('pipelineLLO/setScriptllo', p) return p; } catch (e) { if(e.error == "invalid_token"){ window.location.href = 'https://api-login.swarmlab.io:8089'; }else{ var R = { ERROR_str: e, ERROR: 'yes' } return R; } } }, setScriptllo({commit}, value) { //console.log("container "+value) commit('setllo', value) }, setsocketllo({commit}, value) { //console.log("container "+value) commit('setsocketid', value) }, setScriptCodlogAction({commit}, value) { //console.log("container "+value) commit('setScriptCodelog', value) } } } ---- <1> import vuex https://vuex.vuejs.org/[More info] <2> import axios <3> enable namespace <4> create state (store) <5> getters <6> setters <7> actions sync/async <8> Rest call === Server site [source.javascript] ---- var path = require('path'); // <1> var app = require('express')(); // <1> var http = require('http').Server(app); // <1> var io = require('socket.io')(http); // <1> const socketAuth = require('socketio-auth'); // <1> const cors = require('cors') // <2> const whitelist = [ 'http://localhost:3080', 'http://localhost:3081', 'http://localhost:3082' ] const corsOptions = { credentials: true, methods: ['GET', 'PUT', 'POST', 'DELETE', 'OPTIONS'], optionsSuccessStatus: 200, // some legacy browsers (IE11, various SmartTVs) choke on 204 allowedHeaders: [ 'Content-Type', 'Authorization', 'X-Requested-With', 'device-remember-token', 'Access-Control-Allow-Origin', 'Access-Control-Allow-Headers', 'Origin', 'Accept' ], origin: function(origin, callback) { if (whitelist.indexOf(origin) !== -1) { callback(null, true) } else { callback(null, true) } } } app.get('/run', [ // <3> //check('access_token').isLength({ min: 40 }), //check('llo').isBase64() ], cors(corsOptions), (req, res, next) => { // <4> var RES = new Object(); RES.code = req.query["code"] // <5> console.error('socket get from client' + RES.code); RES.error = false RES.error_msg = "ok" io.emit("iotdata", RES) // <6> io.in('iot').emit('message', RES); // <7> res.json(RES) }); app.post('/run', [ // <8> //check('access_token').isLength({ min: 40 }), //check('llo').isBase64() ], cors(corsOptions), (req, res, next) => { console.error('socket post from client'); io.emit("customEmit", 'data') var RES = new Object(); RES.error = false RES.error_msg = "ok" res.json(RES) }); io.on('connection', s => { // <9> console.error('socket connection'); var id = s.id s.on('pingServerEmit', obj => { console.error('socket.io pingServer'); var data = obj+' testserver' io.emit("customEmit", data) io.in('iot').emit('message', data); console.error('from client '+ s.id + ' obj ' + obj); }); s.on('subscribe', function(room) { // <10> console.log('joining room', room); s.join(room); }) io.on('unsubscribe', function(room) { console.log('leaving room', room); io.leave(room); }) // when the client emits 'new message', this listens and executes s.on('log', (data, room) => { s.to('iot').emit('message', data); console.log('broadcast', data); }); }); http.listen(8084, () => console.error('listening on http://localhost:8084/')); console.error('socket.io example'); ---- <1> load modules <2> config cors https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS[More info^] <3> create Get <4> use cors <5> get data <6> send with socket to all <7> send with socket to room <8> post <9> handle socket connections <10> subscribe === IoT Device [source,javascript] ---- var path = require('path'); var app = require('express')(); var http = require('http').Server(app); var io = require('socket.io')(http); const socketAuth = require('socketio-auth'); const axios = require('axios'); axios.defaults.timeout = 30000 const helmet = require('helmet'); const cors = require('cors') const whitelist = [ 'http://localhost:3080', 'http://localhost:3081', 'http://localhost:3082' ] const corsOptions = { credentials: true, methods: ['GET', 'PUT', 'POST', 'DELETE', 'OPTIONS'], optionsSuccessStatus: 200, // some legacy browsers (IE11, various SmartTVs) choke on 204 allowedHeaders: [ 'Content-Type', 'Authorization', 'X-Requested-With', 'device-remember-token', 'Access-Control-Allow-Origin', 'Access-Control-Allow-Headers', 'Origin', 'Accept' ], origin: function(origin, callback) { if (whitelist.indexOf(origin) !== -1) { callback(null, true) } else { callback(null, true) //callback(new Error('Not allowed by CORS')) } } } app.use(helmet()); app.get('/run', [ //check('access_token').isLength({ min: 40 }), //check('llo').isBase64() ], cors(corsOptions), (req, res, next) => { var RES = new Object(); RES.tmp_token = req.query["access_token"] RES.datafile = req.query["datafile"] RES.service = req.query["service"] RES.action = req.query["action"] RES.error = false RES.error_msg = "ok" res.json(RES) }); app.post('/run', [ //check('access_token').isLength({ min: 40 }), //check('llo').isBase64() ], cors(corsOptions), (req, res, next) => { console.error('socket post'); io.emit("customEmit", 'data') var RES = new Object(); RES.error = false RES.error_msg = "ok" res.json(RES) }); socketoptions = { secure:true, reconnect: true, rejectUnauthorized : false }; // Client var io2 = require('socket.io-client'); var socket = io2.connect('http://localhost:8084', socketoptions); var global = {} socket.on('connection', s => { console.log(s) console.error('socket2 connection'); global.id = s.id s.emit('log', 'client'); }); var roomiot = 'iot' socket.emit('subscribe', roomiot); // <1> socket.emit('log', 'client1'); socket.on('message', function (data) { // <2> //console.log('from room ' + data); console.log("from room iot "+JSON.stringify(data)) //io.emit("customEmit", data) }); io.on('connection', s => { console.error('socket connection'); var roomiot = 'iot' var id = s.id s.on('pingServer', obj => { console.error('socket.io pingServer'); //console.error(s); console.error('fromclient '+obj); var data = obj+' testfromclient '+id socket.emit("pingServerEmit", data) socket.emit('send', { room: roomiot, message: 'message iot' }); }); s.on('customEmit', obj => { var data = obj+' test customeEdit '+id //console.error('from server ' + data); //s.emit("pingServerEmit", data) socket.emit("customEmit", data) }); }); http.listen(3081, () => console.error('listening on http://localhost:3081/')); console.error('socket.io example'); ---- <1> subscribe <2> event on message