Swarmlab docs

Application development in a distributed system

Development of Distributed Systems from Design to Application


You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

377 lines
8.8 KiB

3 years ago
= List!
image::ROOT:swarmlab.png[swarmlab,150,float=right]
== Why Docker
3 years ago
* https://www.cloudbees.com/blog/why-docker[Why Docker^]
3 years ago
== Intro docker
3 years ago
3 years ago
* https://www.compose-spec.io/[docker-compose^]
3 years ago
3 years ago
* https://github.com/veggiemonk/awesome-docker[github docker-compose^]
3 years ago
3 years ago
* https://twitter.com/awesome_docker[github awesome docker]
3 years ago
=== Learn to build and deploy your distributed applications easily to the cloud with Docker
3 years ago
3 years ago
* https://docker-curriculum.com/[HowTo]
3 years ago
== Next Step DockerSwarm
3 years ago
****
TIP: DockerSwarm
video::Tm0Q5zr3FL4[youtube, start=0]
****
3 years ago
3 years ago
* https://www.simplilearn.com/tutorials/docker-tutorial/docker-swarm[docker-swarm tutorial^]
3 years ago
=== Getting started with swarm - official from docker
3 years ago
3 years ago
* https://docs.docker.com/engine/swarm/swarm-tutorial/https://k21academy.com/docker-kubernetes/docker-swarm/[docker-swarm^]
3 years ago
=== Replicability, use the same files as when developing locally.
3 years ago
3 years ago
* https://dockerswarm.rocks/[dockerswarm.rocks^]
3 years ago
3 years ago
=== Open Source Cloud Guide from IBM
3 years ago
3 years ago
* https://open-cloud-guide.dev/[open-cloud guide^]
3 years ago
=== Alternatives
3 years ago
3 years ago
* https://www.nomadproject.io/[monad^]
3 years ago
3 years ago
* https://dev.to/stevenmcgown/kubernetes-for-dummies-5hmh[kubernetes^]
3 years ago
=== Raft consensus in swarm mode
3 years ago
3 years ago
* https://docs.docker.com/engine/swarm/raft/[raft^]
3 years ago
3 years ago
* https://raft.github.io/[raft github^]
3 years ago
3 years ago
* http://thesecretlivesofdata.com/raft/[raft demo^]
3 years ago
== Next Step Storage
3 years ago
****
TIP: Storage
video::W2Z7fbCLSTw[youtube, start=0]
****
3 years ago
=== Generic
3 years ago
3 years ago
* https://docs.mongodb.com/manual/replication/[Mongo replica^]
3 years ago
3 years ago
* https://docs.mongodb.com/manual/core/capped-collections/[mongo capped^]
3 years ago
3 years ago
=== On memory - Key-value
3 years ago
3 years ago
* https://redis.io/topics/cluster-tutorial[Redis cluster^]
* https://redis.io/[redis server^]
* https://www.instaclustr.com/blog/redis-vs-memcached/[redis vs memcached^]
* https://etcd.io/[etcd^]
3 years ago
3 years ago
****
3 years ago
TIP: etcd
3 years ago
video::OmphHSaO1sE[youtube, start=0]
****
3 years ago
****
TIP: redis
video::G1rOthIU-uo[youtube, start=0]
****
3 years ago
3 years ago
*etcd vs. Redis*
3 years ago
3 years ago
Like etcd, Redis is an open source tool, but their basic functionalities are different.
3 years ago
[cols="1,1"]
|===
3 years ago
|Redis|etcd
3 years ago
| Redis is an in-memory data store and can function as a database, cache, or message broker. Redis supports a wider variety of data types and structures than etcd and has much faster read/write performance.
3 years ago
| etcd has superior fault tolerance, stronger failover and continuous data availability capabilities.
|Redis is better suited for serving as a distributed memory caching system than for storing and distributed system configuration information.
|etcd persists all stored data to disk.
3 years ago
|===
3 years ago
=== Sharing
==== NFS
3 years ago
3 years ago
* https://cloud.netapp.com/blog/azure-anf-blg-linux-nfs-server-how-to-set-up-server-and-client[NFS^]
3 years ago
3 years ago
* https://www.gluster.org/[GlusterFS^]
3 years ago
== Deploy
3 years ago
* https://www.compose-spec.io/[docker-compose^]
3 years ago
****
TIP: Deploy
video::zLltQxVb3NI[youtube, start=0]
****
3 years ago
https://12factor.net/[12factor^]
3 years ago
3 years ago
== Development - From Design to Application
3 years ago
=== programming language
==== nodeJS
3 years ago
****
TIP: NodeJS
video::ENrzD9HAZK4[youtube, start=0]
****
3 years ago
* https://nodejs.org/en/[nodeJS^]
3 years ago
3 years ago
* https://p5js.org/[p5js^]
3 years ago
3 years ago
* https://threejs.org/[threejs^]
3 years ago
==== python
3 years ago
* https://www.python.org/[python^]
3 years ago
=== Big data and Scientific analysis
3 years ago
3 years ago
* https://numpy.org/[numpy^]
3 years ago
3 years ago
* https://scipy.org/[scipy^]
3 years ago
3 years ago
* https://jupyter.org/[jupyter^]
3 years ago
=== Rest API
3 years ago
****
TIP: Rest api
video::s7wmiS2mSXY[youtube, start=0]
****
3 years ago
=== Sockets
3 years ago
****
TIP: Sockets
video::ZBM28ZPlin8[youtube, start=0]
****
3 years ago
* https://socket.io/[socket.io^]
3 years ago
3 years ago
=== Swarm intelligence
==== TensorSwarm: A framework for reinforcement learning of robot swarms.
3 years ago
* https://github.com/TensorSwarm/TensorSwarm[TensorSwarm^]
3 years ago
==== ROS - Robot Operating System
https://www.ros.org/[ROS]
==== Reinforcement Learning
3 years ago
* https://www.tensorflow.org/agents/tutorials/0_intro_rl[Introduction to RL^]
* https://www.tensorflow.org/js/guide/nodejs?hl=es[tensorflow.js^]
* https://github.com/karpathy/reinforcejs[common RL algorithms^]
* https://developer.ibm.com/tutorials/an-introduction-to-ai-in-nodejs/[An introduction to AI in Node.js^]
* https://pytorch.org/[pytorch^]
* https://reagent.ai/[pytorch - Reinforcement Learning Platform^]
3 years ago
*PyTorch vs. TensorFlow*
[cols="1,1"]
|===
|PyTorch | TensorFlow
|PyTorch is open source deep learning framework created by developers at Facebook and released in 2017.
|TensorFlow is open source deep learning framework created by developers at Google and released in 2015.
3 years ago
|*Top PyTorch Projects*
3 years ago
3 years ago
* CheXNet: Radiologist-level pneumonia detection on chest X-rays with deep learning. https://stanfordmlgroup.github.io/projects/chexnet/[url^]
3 years ago
3 years ago
* PYRO: Pyro is a universal probabilistic programming language (PPL) written in Python and supported by PyTorch on the backend. https://pyro.ai/[url^]
3 years ago
3 years ago
* Horizon: A platform for applied reinforcement learning (Applied RL) https://horizonrl.com[url^]
3 years ago
3 years ago
|*Top TensorFlow Projects*
3 years ago
3 years ago
* Magenta: An open source research project exploring the role of machine learning as a tool in the creative process. https://magenta.tensorflow.org/[url^]
3 years ago
3 years ago
* Sonnet: Sonnet is a library built on top of TensorFlow for building complex neural networks. https://sonnet.dev/[url^]
3 years ago
3 years ago
* Ludwig: Ludwig is a toolbox to train and test deep learning models without the need to write code. https://uber.github.io/ludwig/[url^]
3 years ago
3 years ago
| *Production Deployment*
PyTorch doesn't provide any framework to deploy models directly on to the web. You'll have to use either Flask or Django as the backend server.
| *Production Deployment*
When it comes to deploying trained models to production, TensorFlow is the clear winner. We can directly deploy models in TensorFlow using TensorFlow serving which is a framework that uses REST Client API.
|*Visualization*
PyTorch developers use Visdom, the features provided by Visdom are very minimalistic and limited.
|*Visualization*
When it comes to visualization of the training process, TensorFlow takes the lead. Visualization helps the developer track the training process and debug in a more convenient way.
TenforFlow’s visualization library is called TensorBoard.
|Python
|Tensorflow has APIs for various langages :
* C (used to develop other APIs in practice, though)
3 years ago
3 years ago
* C++ (Tensorflow is written in C++ and CUDA)
3 years ago
3 years ago
* Java
3 years ago
3 years ago
* GO
3 years ago
3 years ago
* Rust
3 years ago
3 years ago
* Haskell (yes, even Haskell)
3 years ago
3 years ago
* C#
3 years ago
3 years ago
* Julia
3 years ago
3 years ago
* R
3 years ago
3 years ago
* Python
3 years ago
3 years ago
* And, of course, for JavaScript, since 2019.
|*Pytorch Cons:*
* Third-party needed for visualization.
3 years ago
3 years ago
* API server needed for production.
|*TensorFlow Cons:*
* Debugging method.
3 years ago
3 years ago
* Hard to make quick changes.
3 years ago
|===
3 years ago
=== Reasons to learn machine learning with JavaScript
====
*Private machine learning*
Most machine learning applications rely on client-server architectures. Users must send their data where the machine learning models are running.
Machine learning models on your device without sending any data to the cloud.
And you don’t need to install any additional software.
Powerful JavaScript machine learning libraries:
* https://ml5js.org/[ml5js^]
* http://caza.la/synaptic/#/[synaptic^]
* https://brain.js.org/#/[brain^]
*Easy integration of machine learning in web and mobile applications*
Python support in mobile operating systems is still in the preliminary stages.
Javascript has a rich set of cross-platform mobile app development tools such as Cordova and Ionic
TIP: Machine learning with Node.js is fairly new, but it is fast evolving because there is growing interest in adding machine learning capabilities to web and mobile applications.
3 years ago
====
=== Reasons to learn machine learning with Python - PyTorch
====
*Python is better suited for server-side training of machine learning models*
It can scale and distribute its load on server clusters to accelerate the training process. Once the model is trained, you can compress it and deliver it on user devices for inference. Fortunately, machine learning libraries written in different languages are highly compatible.
3 years ago
3 years ago
For instance, if you train your deep learning model with TensorFlow or Keras for Python, you can save it in one of several language-independent formats such as JSON or HDF5. You can then send the saved model to the user’s device and load it with TensorFlow.js or another JavaScript deep learning library.
3 years ago
====
3 years ago
3 years ago
== Auth
3 years ago
3 years ago
* https://oauth.net/2/[auth2^]
3 years ago
3 years ago
* https://oauth.net/code/nodejs/[Nodejs auth library^]
3 years ago
3 years ago
=== Auth server
3 years ago
3 years ago
* https://www.keycloak.org/[auth2 server^]
3 years ago
== Security VPN etc
3 years ago
3 years ago
* https://www.wireguard.com/[simply the best VPN out there^]
3 years ago
== POC
=== netflix
3 years ago
****
TIP: Netflix
video::CZ3wIuvmHeM[youtube, start=0]
****
3 years ago
3 years ago
6:50 monolithic
3 years ago
3 years ago
8:24 minroservices
3 years ago
3 years ago
19:00 microservice
3 years ago
3 years ago
see also 37:00
3 years ago
3 years ago
https://netflix.github.io/[netflix github^]
3 years ago
=== Robots and IoT
3 years ago
* https://nodebots.io/[nodebot^]
3 years ago
3 years ago
* http://johnny-five.io/[johny-five^]
3 years ago
3 years ago
* https://ardupilot.org/[eduplot^]
3 years ago
3 years ago
* https://github.com/ArduPilot/ardupilot[ardupilot^]
3 years ago