Swarmlab docs

Application development in a distributed system

Development of Distributed Systems from Design to Application


You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

233 lines
4.8 KiB

3 years ago
= Guide to open source development in the cloud!
3 years ago
3 years ago
An https://developer.ibm.com/blogs/oreilly-open-source-skill-survey-blog/[O’Reilly survey^], commissioned by IBM in Fall 2020, pointed out that, when choosing cloud providers, 70% of respondents prefer one based on open source. The survey went on to say:
TIP: Developers who want to develop cloud applications should work on honing their open source skills, which underly every major cloud platform, instead of only focusing on building skills for a proprietary cloud.
The most desirable open source skills are those related to *Linux (containers)*, *artificial intelligence* and *machine learning*, and *data storage.*
This goals of this guide(https://open-cloud-guide.dev/[Text^] by IBM / Apache Licensed, 2021) are to provide:
* Comprehensive learning for hybrid cloud and multicloud developers.
* A cloud-neutral view of how various clouds are using open source in their offerings.
https://open-cloud-guide.dev/about
3 years ago
== Docker
* https://www.cloudbees.com/blog/why-docker[Docker^]
== Intro docker
* https://www.compose-spec.io/[docker-compose^]
* https://github.com/veggiemonk/awesome-docker[github docker-compose^]
* https://twitter.com/awesome_docker[github awesome docker]
=== Learn to build and deploy your distributed applications easily to the cloud with Docker
* https://docker-curriculum.com/[HowTo]
== Next Step DockerSwarm
****
TIP: DockerSwarm
video::Tm0Q5zr3FL4[youtube, start=0]
****
* https://www.simplilearn.com/tutorials/docker-tutorial/docker-swarm[docker-swarm tutorial^]
=== Getting started with swarm - official from docker
* https://docs.docker.com/engine/swarm/swarm-tutorial/https://k21academy.com/docker-kubernetes/docker-swarm/[docker-swarm^]
=== Replicability, use the same files as when developing locally.
* https://dockerswarm.rocks/[dockerswarm.rocks^]
=== Open Source Cloud Guide from IBM
* https://open-cloud-guide.dev/[open-cloud guide^]
=== Alternatives
* https://www.nomadproject.io/[monad^]
* https://dev.to/stevenmcgown/kubernetes-for-dummies-5hmh[kubernetes^]
=== Raft consensus in swarm mode
* https://docs.docker.com/engine/swarm/raft/[raft^]
* https://raft.github.io/[raft github^]
* http://thesecretlivesofdata.com/raft/[raft demo^]
== Next Step Storage
****
TIP: Storage
video::W2Z7fbCLSTw[youtube, start=0]
****
=== Generic
* https://docs.mongodb.com/manual/replication/[Mongo replica^]
* https://docs.mongodb.com/manual/core/capped-collections/[mongo capped^]
=== On memory - Key-value
* https://redis.io/topics/cluster-tutorial[Redis cluster^]
* https://redis.io/[redis server^]
* https://www.instaclustr.com/blog/redis-vs-memcached/[redis vs memcached^]
* https://etcd.io/[etcd^]
****
TIP: etcd
video::OmphHSaO1sE[youtube, start=0]
****
****
TIP: redis
video::G1rOthIU-uo[youtube, start=0]
****
*etcd vs. Redis*
Like etcd, Redis is an open source tool, but their basic functionalities are different.
[cols="1,1"]
|===
|Redis|etcd
| Redis is an in-memory data store and can function as a database, cache, or message broker. Redis supports a wider variety of data types and structures than etcd and has much faster read/write performance.
| etcd has superior fault tolerance, stronger failover and continuous data availability capabilities.
|Redis is better suited for serving as a distributed memory caching system than for storing and distributed system configuration information.
|etcd persists all stored data to disk.
|===
=== Sharing
==== NFS
* https://cloud.netapp.com/blog/azure-anf-blg-linux-nfs-server-how-to-set-up-server-and-client[NFS^]
* https://www.gluster.org/[GlusterFS^]
== Deploy
* https://www.compose-spec.io/[docker-compose^]
****
TIP: Deploy
video::zLltQxVb3NI[youtube, start=0]
****
https://12factor.net/[12factor^]
== Development - From Design to Application
=== programming language
==== nodeJS
****
TIP: NodeJS
video::ENrzD9HAZK4[youtube, start=0]
****
* https://nodejs.org/en/[nodeJS^]
* https://p5js.org/[p5js^]
* https://threejs.org/[threejs^]
==== python
* https://www.python.org/[python^]
=== Big data and Scientific analysis
* https://numpy.org/[numpy^]
* https://scipy.org/[scipy^]
* https://jupyter.org/[jupyter^]
=== Rest API
****
TIP: Rest api
video::s7wmiS2mSXY[youtube, start=0]
****
=== Sockets
****
TIP: Sockets
video::ZBM28ZPlin8[youtube, start=0]
****
* https://socket.io/[socket.io^]
== Auth
* https://oauth.net/2/[auth2^]
* https://oauth.net/code/nodejs/[Nodejs auth library^]
=== Auth server
* https://www.keycloak.org/[auth2 server^]
== Security VPN etc
* https://www.wireguard.com/[simply the best VPN out there^]
== POC
=== netflix
****
TIP: Netflix
video::CZ3wIuvmHeM[youtube, start=0]
****
6:50 monolithic
8:24 minroservices
19:00 microservice
see also 37:00
https://netflix.github.io/[netflix github^]