= List! image::ROOT:swarmlab.png[swarmlab,150,float=right] == Why Docker https://www.cloudbees.com/blog/why-dockero[Why Docker^] == Intro docker https://www.compose-spec.io/[docker-compose^] https://github.com/veggiemonk/awesome-docker[github docker-compose^] https://twitter.com/awesome_docker[github awesome docker] === Learn to build and deploy your distributed applications easily to the cloud with Docker https://docker-curriculum.com/[HowTo] == Next Step DockerSwarm **** TIP: DockerSwarm video::Tm0Q5zr3FL4[youtube, start=0] **** https://www.simplilearn.com/tutorials/docker-tutorial/docker-swarm[docker-swarm tutorial^] === Getting started with swarm - official from docker https://docs.docker.com/engine/swarm/swarm-tutorial/https://k21academy.com/docker-kubernetes/docker-swarm/[docker-swarm^] === Replicability, use the same files as when developing locally. https://dockerswarm.rocks/[dockerswarm.rocks^] === Open Source Cloud Guide από την IBM https://open-cloud-guide.dev/[open-cloud huide^] === Alternatives https://www.nomadproject.io/[monad^] https://dev.to/stevenmcgown/kubernetes-for-dummies-5hmh[kubernetes^] === Raft consensus in swarm mode https://docs.docker.com/engine/swarm/raft/[raft^] https://raft.github.io/[raft github^] http://thesecretlivesofdata.com/raft/[raft demo^] == Next Step Storage **** TIP: Storage video::W2Z7fbCLSTw[youtube, start=0] **** === Generic https://docs.mongodb.com/manual/replication/[Mongo replica^] https://docs.mongodb.com/manual/core/capped-collections/[mongo capped^] === On memory - Key-value https://redis.io/topics/cluster-tutorial[Redis cluster^] https://redis.io/[redis server^] https://www.instaclustr.com/blog/redis-vs-memcached/[redis vs memcached^] https://etcd.io/[etcd^] **** TIP: etcd video::OmphHSaO1sE[youtube, start=0] **** **** TIP: redis video::G1rOthIU-uo[youtube, start=0] **** *etcd vs. Redis* Like etcd, Redis is an open source tool, but their basic functionalities are different. [cols="1,1"] |=== |Redis|etcd | Redis is an in-memory data store and can function as a database, cache, or message broker. Redis supports a wider variety of data types and structures than etcd and has much faster read/write performance. | etcd has superior fault tolerance, stronger failover and continuous data availability capabilities. |Redis is better suited for serving as a distributed memory caching system than for storing and distributed system configuration information. |etcd persists all stored data to disk. |=== === Sharing ==== NFS https://cloud.netapp.com/blog/azure-anf-blg-linux-nfs-server-how-to-set-up-server-and-client[NFS^] https://www.gluster.org/[GlusterFS^] == Deploy https://www.compose-spec.io/[docker-compose^] **** TIP: Deploy video::zLltQxVb3NI[youtube, start=0] **** https://12factor.net/[12factor^] == Development - From Design to Application === programming language ==== nodeJS **** TIP: NodeJS video::ENrzD9HAZK4[youtube, start=0] **** https://nodejs.org/en/[nodeJS^] https://p5js.org/[p5js^] https://threejs.org/[threejs^] ==== python https://www.python.org/[python^] === Big data and Scientific analysis https://numpy.org/[numpy^] https://scipy.org/[scipy^] https://jupyter.org/[jupyter^] === Rest API **** TIP: Rest api video::s7wmiS2mSXY[youtube, start=0] **** === Sockets **** TIP: Sockets video::ZBM28ZPlin8[youtube, start=0] **** https://socket.io/[socket.io^] === Swarm intelligence ==== TensorSwarm: A framework for reinforcement learning of robot swarms. https://github.com/TensorSwarm/TensorSwarm[TensorSwarm^] ==== ROS - Robot Operating System https://www.ros.org/[ROS] ==== Reinforcement Learning https://www.tensorflow.org/agents/tutorials/0_intro_rl[Introduction to RL^] https://www.tensorflow.org/js/guide/nodejs?hl=es[tensorflow.js^] https://github.com/karpathy/reinforcejs[common RL algorithms^] https://developer.ibm.com/tutorials/an-introduction-to-ai-in-nodejs/[An introduction to AI in Node.js^] https://pytorch.org/[pytorch^] https://reagent.ai/[pytorch - Reinforcement Learning Platform^] *PyTorch vs. TensorFlow* [cols="1,1"] |=== |PyTorch | TensorFlow |PyTorch is open source deep learning framework created by developers at Facebook and released in 2017. |TensorFlow is open source deep learning framework created by developers at Google and released in 2015. |*Top PyTorch Projects* * CheXNet: Radiologist-level pneumonia detection on chest X-rays with deep learning. https://stanfordmlgroup.github.io/projects/chexnet/[url^] * PYRO: Pyro is a universal probabilistic programming language (PPL) written in Python and supported by PyTorch on the backend. https://pyro.ai/[url^] * Horizon: A platform for applied reinforcement learning (Applied RL) https://horizonrl.com[url^] |*Top TensorFlow Projects* * Magenta: An open source research project exploring the role of machine learning as a tool in the creative process. https://magenta.tensorflow.org/[url^] * Sonnet: Sonnet is a library built on top of TensorFlow for building complex neural networks. https://sonnet.dev/[url^] * Ludwig: Ludwig is a toolbox to train and test deep learning models without the need to write code. https://uber.github.io/ludwig/[url^] | *Production Deployment* PyTorch doesn't provide any framework to deploy models directly on to the web. You'll have to use either Flask or Django as the backend server. | *Production Deployment* When it comes to deploying trained models to production, TensorFlow is the clear winner. We can directly deploy models in TensorFlow using TensorFlow serving which is a framework that uses REST Client API. |*Visualization* PyTorch developers use Visdom, the features provided by Visdom are very minimalistic and limited. |*Visualization* When it comes to visualization of the training process, TensorFlow takes the lead. Visualization helps the developer track the training process and debug in a more convenient way. TenforFlow’s visualization library is called TensorBoard. |Python |Tensorflow has APIs for various langages : * C (used to develop other APIs in practice, though) * C++ (Tensorflow is written in C++ and CUDA) * Java * GO * Rust * Haskell (yes, even Haskell) * C# * Julia * R * Python * And, of course, for JavaScript, since 2019. |*Pytorch Cons:* * Third-party needed for visualization. * API server needed for production. |*TensorFlow Cons:* * Debugging method. * Hard to make quick changes. |=== === Reasons to learn machine learning with JavaScript ==== *Private machine learning* Most machine learning applications rely on client-server architectures. Users must send their data where the machine learning models are running. Machine learning models on your device without sending any data to the cloud. And you don’t need to install any additional software. Powerful JavaScript machine learning libraries: * https://ml5js.org/[ml5js^] * http://caza.la/synaptic/#/[synaptic^] * https://brain.js.org/#/[brain^] *Easy integration of machine learning in web and mobile applications* Python support in mobile operating systems is still in the preliminary stages. Javascript has a rich set of cross-platform mobile app development tools such as Cordova and Ionic TIP: Machine learning with Node.js is fairly new, but it is fast evolving because there is growing interest in adding machine learning capabilities to web and mobile applications. ==== == Auth https://oauth.net/2/[auth2^] https://oauth.net/code/nodejs/[Nodejs auth library^] === Auth server https://www.keycloak.org/[auth2 server^] == Security VPN etc https://www.wireguard.com/[simply the best VPN out there^] == POC === netflix **** TIP: Netflix video::CZ3wIuvmHeM[youtube, start=0] **** 6:50 monolithic 8:24 minroservices 19:00 microservice see also 37:00 https://netflix.github.io/[netflix github^] === Robots and IoT https://nodebots.io/[nodebot^] http://johnny-five.io/[johny-five^] https://ardupilot.org/[eduplot^] https://github.com/ArduPilot/ardupilot[ardupilot^]