Swarmlab docs

Application development in a distributed system

Development of Distributed Systems from Design to Application


You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

278 lines
6.7 KiB

3 years ago
= Guide to open source development in the cloud!
3 years ago
3 years ago
An https://developer.ibm.com/blogs/oreilly-open-source-skill-survey-blog/[O’Reilly survey^], commissioned by IBM in Fall 2020, pointed out that, when choosing cloud providers, 70% of respondents prefer one based on open source. The survey went on to say:
TIP: Developers who want to develop cloud applications should work on honing their open source skills, which underly every major cloud platform, instead of only focusing on building skills for a proprietary cloud.
The most desirable open source skills are those related to *Linux (containers)*, *artificial intelligence* and *machine learning*, and *data storage.*
This goals of this guide(https://open-cloud-guide.dev/[Text^] by IBM / Apache Licensed, 2021) are to provide:
* Comprehensive learning for hybrid cloud and multicloud developers.
* A cloud-neutral view of how various clouds are using open source in their offerings.
3 years ago
====
****
TIP: Why Open Source!
video::a8fHgx9mE5U[youtube, start=0]
****
3 years ago
Open source refers to an *openly distributed* program code (including codes that are *free of charge*) that can be *utilized* and *modified* by the end user *without restriction.*
3 years ago
====
3 years ago
== Docker
* https://www.cloudbees.com/blog/why-docker[Docker^]
== Intro docker
* https://www.compose-spec.io/[docker-compose^]
* https://github.com/veggiemonk/awesome-docker[github docker-compose^]
* https://twitter.com/awesome_docker[github awesome docker]
=== Learn to build and deploy your distributed applications easily to the cloud with Docker
* https://docker-curriculum.com/[HowTo]
3 years ago
== Alternatives to Docker
3 years ago
TIP: https://apptainer.org/docs/admin/main/[Apptainer/Singularity] - *Simply the best* Container system for HPC
3 years ago
====
Singularity can be used with Docker images. This feature was included because developers use and really like using Docker and scientists have already put much resources into creating Docker images.
Features:
* Use a Docker image instantly as a Singularity image
* Pull a Docker image
* Build images with bases from assembled Docker layers
====
3 years ago
== Next Step DockerSwarm
****
TIP: DockerSwarm
video::Tm0Q5zr3FL4[youtube, start=0]
****
* https://www.simplilearn.com/tutorials/docker-tutorial/docker-swarm[docker-swarm tutorial^]
=== Getting started with swarm - official from docker
* https://docs.docker.com/engine/swarm/swarm-tutorial/https://k21academy.com/docker-kubernetes/docker-swarm/[docker-swarm^]
=== Replicability, use the same files as when developing locally.
* https://dockerswarm.rocks/[dockerswarm.rocks^]
=== Alternatives
* https://www.nomadproject.io/[monad^]
* https://dev.to/stevenmcgown/kubernetes-for-dummies-5hmh[kubernetes^]
=== Raft consensus in swarm mode
* https://docs.docker.com/engine/swarm/raft/[raft^]
* https://raft.github.io/[raft github^]
* http://thesecretlivesofdata.com/raft/[raft demo^]
3 years ago
== Code as a Service (CaaS)
TIP: Automation with Code as a Service. CaaS’s primary service or function is automation. It uses software to automate repetitive practices to hasten and simplify implementations and processes.
=== https://docs.ansible.com/ansible/latest/user_guide/playbooks_intro.html[Ansible^]
Ansible is an open source automation software promoted by Redhat Corporation. In addition, to cloud provisioning, it assists in application development, intra-service orchestration, and configuration.
Ansible uses the simple programming language YAML to create playbooks for automation.
Ansible has many modules that integrate with the most common cloud solutions such as AWS, Google Cloud Platform (GCP) and VMware.
https://www.terraform.io/cli[Teraform^]
Terraform is an infrastructure as code software by Hashi Corporation.
It primarily focuses on creating data center infrastructure that is provided by large public clouds. Teraform utilizes JSON language to define infrastructure templates with integrations such as AWS, Azure, GCP, and IBM cloud.
====
You can find a ready-to-use example for Ansible in swarmlab.io-->Labroom-->Linux in examples directory
====
3 years ago
== Next Step Storage
****
TIP: Storage
video::W2Z7fbCLSTw[youtube, start=0]
****
=== Generic
* https://docs.mongodb.com/manual/replication/[Mongo replica^]
* https://docs.mongodb.com/manual/core/capped-collections/[mongo capped^]
=== On memory - Key-value
* https://redis.io/topics/cluster-tutorial[Redis cluster^]
* https://redis.io/[redis server^]
* https://www.instaclustr.com/blog/redis-vs-memcached/[redis vs memcached^]
* https://etcd.io/[etcd^]
****
TIP: etcd
video::OmphHSaO1sE[youtube, start=0]
****
****
TIP: redis
video::G1rOthIU-uo[youtube, start=0]
****
*etcd vs. Redis*
Like etcd, Redis is an open source tool, but their basic functionalities are different.
[cols="1,1"]
|===
|Redis|etcd
| Redis is an in-memory data store and can function as a database, cache, or message broker. Redis supports a wider variety of data types and structures than etcd and has much faster read/write performance.
| etcd has superior fault tolerance, stronger failover and continuous data availability capabilities.
|Redis is better suited for serving as a distributed memory caching system than for storing and distributed system configuration information.
|etcd persists all stored data to disk.
|===
=== Sharing
==== NFS
* https://cloud.netapp.com/blog/azure-anf-blg-linux-nfs-server-how-to-set-up-server-and-client[NFS^]
* https://www.gluster.org/[GlusterFS^]
3 years ago
* https://github.com/minio/minio[Minio^]
3 years ago
== Deploy
* https://www.compose-spec.io/[docker-compose^]
****
TIP: Deploy
video::zLltQxVb3NI[youtube, start=0]
****
https://12factor.net/[12factor^]
== Development - From Design to Application
=== programming language
==== nodeJS
****
TIP: NodeJS
video::ENrzD9HAZK4[youtube, start=0]
****
* https://nodejs.org/en/[nodeJS^]
==== python
* https://www.python.org/[python^]
=== Big data and Scientific analysis
* https://numpy.org/[numpy^]
* https://scipy.org/[scipy^]
* https://jupyter.org/[jupyter^]
3 years ago
=== Html5 graphical librarys
* https://p5js.org/[p5js^]
* https://threejs.org/[threejs^]
3 years ago
=== Rest API
****
TIP: Rest api
video::s7wmiS2mSXY[youtube, start=0]
****
=== Sockets
****
TIP: Sockets
video::ZBM28ZPlin8[youtube, start=0]
****
* https://socket.io/[socket.io^]
== Auth
* https://oauth.net/2/[auth2^]
* https://oauth.net/code/nodejs/[Nodejs auth library^]
=== Auth server
* https://www.keycloak.org/[auth2 server^]
== Security VPN etc
* https://www.wireguard.com/[simply the best VPN out there^]
== POC
=== netflix
****
TIP: Netflix
video::CZ3wIuvmHeM[youtube, start=0]
****
6:50 monolithic
8:24 minroservices
19:00 microservice
see also 37:00
https://netflix.github.io/[netflix github^]
3 years ago
== Open Source Cloud Guide from IBM
* https://open-cloud-guide.dev/[open-cloud guide^]
3 years ago