You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
375 lines
8.8 KiB
375 lines
8.8 KiB
= List!
|
|
|
|
image::ROOT:swarmlab.png[swarmlab,150,float=right]
|
|
|
|
|
|
== Why Docker
|
|
|
|
* https://www.cloudbees.com/blog/why-docker[Why Docker^]
|
|
|
|
== Intro docker
|
|
|
|
* https://www.compose-spec.io/[docker-compose^]
|
|
|
|
* https://github.com/veggiemonk/awesome-docker[github docker-compose^]
|
|
|
|
* https://twitter.com/awesome_docker[github awesome docker]
|
|
|
|
=== Learn to build and deploy your distributed applications easily to the cloud with Docker
|
|
|
|
* https://docker-curriculum.com/[HowTo]
|
|
|
|
== Next Step DockerSwarm
|
|
|
|
****
|
|
TIP: DockerSwarm
|
|
|
|
video::Tm0Q5zr3FL4[youtube, start=0]
|
|
****
|
|
|
|
* https://www.simplilearn.com/tutorials/docker-tutorial/docker-swarm[docker-swarm tutorial^]
|
|
|
|
=== Getting started with swarm - official from docker
|
|
|
|
* https://docs.docker.com/engine/swarm/swarm-tutorial/https://k21academy.com/docker-kubernetes/docker-swarm/[docker-swarm^]
|
|
|
|
=== Replicability, use the same files as when developing locally.
|
|
|
|
* https://dockerswarm.rocks/[dockerswarm.rocks^]
|
|
|
|
=== Open Source Cloud Guide from IBM
|
|
|
|
* https://open-cloud-guide.dev/[open-cloud huide^]
|
|
|
|
=== Alternatives
|
|
|
|
* https://www.nomadproject.io/[monad^]
|
|
|
|
* https://dev.to/stevenmcgown/kubernetes-for-dummies-5hmh[kubernetes^]
|
|
|
|
=== Raft consensus in swarm mode
|
|
|
|
* https://docs.docker.com/engine/swarm/raft/[raft^]
|
|
|
|
* https://raft.github.io/[raft github^]
|
|
|
|
* http://thesecretlivesofdata.com/raft/[raft demo^]
|
|
|
|
== Next Step Storage
|
|
|
|
****
|
|
TIP: Storage
|
|
|
|
video::W2Z7fbCLSTw[youtube, start=0]
|
|
****
|
|
|
|
=== Generic
|
|
|
|
* https://docs.mongodb.com/manual/replication/[Mongo replica^]
|
|
|
|
* https://docs.mongodb.com/manual/core/capped-collections/[mongo capped^]
|
|
|
|
=== On memory - Key-value
|
|
|
|
* https://redis.io/topics/cluster-tutorial[Redis cluster^]
|
|
|
|
* https://redis.io/[redis server^]
|
|
|
|
* https://www.instaclustr.com/blog/redis-vs-memcached/[redis vs memcached^]
|
|
|
|
* https://etcd.io/[etcd^]
|
|
|
|
****
|
|
TIP: etcd
|
|
|
|
video::OmphHSaO1sE[youtube, start=0]
|
|
****
|
|
|
|
****
|
|
TIP: redis
|
|
|
|
video::G1rOthIU-uo[youtube, start=0]
|
|
****
|
|
|
|
*etcd vs. Redis*
|
|
|
|
Like etcd, Redis is an open source tool, but their basic functionalities are different.
|
|
|
|
[cols="1,1"]
|
|
|===
|
|
|Redis|etcd
|
|
|
|
| Redis is an in-memory data store and can function as a database, cache, or message broker. Redis supports a wider variety of data types and structures than etcd and has much faster read/write performance.
|
|
| etcd has superior fault tolerance, stronger failover and continuous data availability capabilities.
|
|
|
|
|Redis is better suited for serving as a distributed memory caching system than for storing and distributed system configuration information.
|
|
|etcd persists all stored data to disk.
|
|
|
|
|
|
|===
|
|
|
|
|
|
|
|
|
|
|
|
=== Sharing
|
|
|
|
==== NFS
|
|
|
|
* https://cloud.netapp.com/blog/azure-anf-blg-linux-nfs-server-how-to-set-up-server-and-client[NFS^]
|
|
|
|
* https://www.gluster.org/[GlusterFS^]
|
|
|
|
|
|
== Deploy
|
|
|
|
* https://www.compose-spec.io/[docker-compose^]
|
|
|
|
****
|
|
TIP: Deploy
|
|
|
|
video::zLltQxVb3NI[youtube, start=0]
|
|
****
|
|
|
|
https://12factor.net/[12factor^]
|
|
|
|
|
|
== Development - From Design to Application
|
|
|
|
=== programming language
|
|
==== nodeJS
|
|
****
|
|
TIP: NodeJS
|
|
|
|
video::ENrzD9HAZK4[youtube, start=0]
|
|
****
|
|
|
|
* https://nodejs.org/en/[nodeJS^]
|
|
|
|
* https://p5js.org/[p5js^]
|
|
|
|
* https://threejs.org/[threejs^]
|
|
|
|
==== python
|
|
https://www.python.org/[python^]
|
|
|
|
=== Big data and Scientific analysis
|
|
|
|
* https://numpy.org/[numpy^]
|
|
|
|
* https://scipy.org/[scipy^]
|
|
|
|
* https://jupyter.org/[jupyter^]
|
|
|
|
=== Rest API
|
|
****
|
|
TIP: Rest api
|
|
|
|
video::s7wmiS2mSXY[youtube, start=0]
|
|
****
|
|
|
|
=== Sockets
|
|
****
|
|
TIP: Sockets
|
|
|
|
video::ZBM28ZPlin8[youtube, start=0]
|
|
****
|
|
|
|
* https://socket.io/[socket.io^]
|
|
|
|
|
|
|
|
=== Swarm intelligence
|
|
|
|
==== TensorSwarm: A framework for reinforcement learning of robot swarms.
|
|
|
|
* https://github.com/TensorSwarm/TensorSwarm[TensorSwarm^]
|
|
|
|
|
|
==== ROS - Robot Operating System
|
|
|
|
https://www.ros.org/[ROS]
|
|
|
|
==== Reinforcement Learning
|
|
|
|
* https://www.tensorflow.org/agents/tutorials/0_intro_rl[Introduction to RL^]
|
|
|
|
* https://www.tensorflow.org/js/guide/nodejs?hl=es[tensorflow.js^]
|
|
|
|
* https://github.com/karpathy/reinforcejs[common RL algorithms^]
|
|
|
|
* https://developer.ibm.com/tutorials/an-introduction-to-ai-in-nodejs/[An introduction to AI in Node.js^]
|
|
|
|
* https://pytorch.org/[pytorch^]
|
|
|
|
* https://reagent.ai/[pytorch - Reinforcement Learning Platform^]
|
|
|
|
|
|
*PyTorch vs. TensorFlow*
|
|
|
|
[cols="1,1"]
|
|
|===
|
|
|PyTorch | TensorFlow
|
|
|
|
|PyTorch is open source deep learning framework created by developers at Facebook and released in 2017.
|
|
|TensorFlow is open source deep learning framework created by developers at Google and released in 2015.
|
|
|
|
|*Top PyTorch Projects*
|
|
|
|
* CheXNet: Radiologist-level pneumonia detection on chest X-rays with deep learning. https://stanfordmlgroup.github.io/projects/chexnet/[url^]
|
|
|
|
* PYRO: Pyro is a universal probabilistic programming language (PPL) written in Python and supported by PyTorch on the backend. https://pyro.ai/[url^]
|
|
|
|
* Horizon: A platform for applied reinforcement learning (Applied RL) https://horizonrl.com[url^]
|
|
|
|
|*Top TensorFlow Projects*
|
|
|
|
* Magenta: An open source research project exploring the role of machine learning as a tool in the creative process. https://magenta.tensorflow.org/[url^]
|
|
|
|
* Sonnet: Sonnet is a library built on top of TensorFlow for building complex neural networks. https://sonnet.dev/[url^]
|
|
|
|
* Ludwig: Ludwig is a toolbox to train and test deep learning models without the need to write code. https://uber.github.io/ludwig/[url^]
|
|
|
|
|
|
| *Production Deployment*
|
|
PyTorch doesn't provide any framework to deploy models directly on to the web. You'll have to use either Flask or Django as the backend server.
|
|
| *Production Deployment*
|
|
When it comes to deploying trained models to production, TensorFlow is the clear winner. We can directly deploy models in TensorFlow using TensorFlow serving which is a framework that uses REST Client API.
|
|
|
|
|
|
|*Visualization*
|
|
PyTorch developers use Visdom, the features provided by Visdom are very minimalistic and limited.
|
|
|*Visualization*
|
|
When it comes to visualization of the training process, TensorFlow takes the lead. Visualization helps the developer track the training process and debug in a more convenient way.
|
|
TenforFlow’s visualization library is called TensorBoard.
|
|
|
|
|Python
|
|
|Tensorflow has APIs for various langages :
|
|
|
|
* C (used to develop other APIs in practice, though)
|
|
|
|
* C++ (Tensorflow is written in C++ and CUDA)
|
|
|
|
* Java
|
|
|
|
* GO
|
|
|
|
* Rust
|
|
|
|
* Haskell (yes, even Haskell)
|
|
|
|
* C#
|
|
|
|
* Julia
|
|
|
|
* R
|
|
|
|
* Python
|
|
|
|
* And, of course, for JavaScript, since 2019.
|
|
|
|
|*Pytorch Cons:*
|
|
|
|
* Third-party needed for visualization.
|
|
|
|
* API server needed for production.
|
|
|
|
|*TensorFlow Cons:*
|
|
|
|
* Debugging method.
|
|
|
|
* Hard to make quick changes.
|
|
|
|
|
|
|===
|
|
|
|
|
|
=== Reasons to learn machine learning with JavaScript
|
|
|
|
====
|
|
*Private machine learning*
|
|
|
|
Most machine learning applications rely on client-server architectures. Users must send their data where the machine learning models are running.
|
|
|
|
Machine learning models on your device without sending any data to the cloud.
|
|
|
|
And you don’t need to install any additional software.
|
|
|
|
Powerful JavaScript machine learning libraries:
|
|
|
|
* https://ml5js.org/[ml5js^]
|
|
|
|
* http://caza.la/synaptic/#/[synaptic^]
|
|
|
|
* https://brain.js.org/#/[brain^]
|
|
|
|
|
|
*Easy integration of machine learning in web and mobile applications*
|
|
|
|
Python support in mobile operating systems is still in the preliminary stages.
|
|
|
|
Javascript has a rich set of cross-platform mobile app development tools such as Cordova and Ionic
|
|
|
|
|
|
TIP: Machine learning with Node.js is fairly new, but it is fast evolving because there is growing interest in adding machine learning capabilities to web and mobile applications.
|
|
|
|
====
|
|
|
|
=== Reasons to learn machine learning with Python - PyTorch
|
|
|
|
====
|
|
*Python is better suited for server-side training of machine learning models*
|
|
|
|
It can scale and distribute its load on server clusters to accelerate the training process. Once the model is trained, you can compress it and deliver it on user devices for inference. Fortunately, machine learning libraries written in different languages are highly compatible.
|
|
|
|
For instance, if you train your deep learning model with TensorFlow or Keras for Python, you can save it in one of several language-independent formats such as JSON or HDF5. You can then send the saved model to the user’s device and load it with TensorFlow.js or another JavaScript deep learning library.
|
|
====
|
|
|
|
|
|
== Auth
|
|
|
|
* https://oauth.net/2/[auth2^]
|
|
|
|
* https://oauth.net/code/nodejs/[Nodejs auth library^]
|
|
|
|
=== Auth server
|
|
|
|
* https://www.keycloak.org/[auth2 server^]
|
|
|
|
== Security VPN etc
|
|
|
|
* https://www.wireguard.com/[simply the best VPN out there^]
|
|
|
|
== POC
|
|
|
|
=== netflix
|
|
|
|
****
|
|
TIP: Netflix
|
|
|
|
video::CZ3wIuvmHeM[youtube, start=0]
|
|
****
|
|
|
|
6:50 monolithic
|
|
|
|
8:24 minroservices
|
|
|
|
19:00 microservice
|
|
|
|
see also 37:00
|
|
|
|
https://netflix.github.io/[netflix github^]
|
|
|
|
=== Robots and IoT
|
|
* https://nodebots.io/[nodebot^]
|
|
|
|
* http://johnny-five.io/[johny-five^]
|
|
|
|
* https://ardupilot.org/[eduplot^]
|
|
|
|
* https://github.com/ArduPilot/ardupilot[ardupilot^]
|
|
|
|
|
|
|
|
|
|
|
|
|