You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
384 lines
13 KiB
384 lines
13 KiB
= Guide to open source development in the cloud!
|
|
|
|
|
|
An https://developer.ibm.com/blogs/oreilly-open-source-skill-survey-blog/[O’Reilly survey^], commissioned by IBM in Fall 2020, pointed out that, when choosing cloud providers, 70% of respondents prefer one based on open source. The survey went on to say:
|
|
|
|
TIP: Developers who want to develop cloud applications should work on honing their open source skills, which underly every major cloud platform, instead of only focusing on building skills for a proprietary cloud.
|
|
|
|
The most desirable open source skills are those related to *Linux (containers)*, *artificial intelligence* and *machine learning*, and *data storage.*
|
|
|
|
This goals of this guide(https://open-cloud-guide.dev/[Text^] by IBM / Apache Licensed, 2021) are to provide:
|
|
|
|
* Comprehensive learning for hybrid cloud and multicloud developers.
|
|
* A cloud-neutral view of how various clouds are using open source in their offerings.
|
|
* The https://machalliance.org/[MACH^] technology: **M**icroservices based, **A**PI-first, **C**loud-native SaaS and **H**eadless.
|
|
|
|
====
|
|
****
|
|
TIP: Why Open Source! (must-see - Highly Recommended)
|
|
|
|
video::a8fHgx9mE5U[youtube, start=0]
|
|
****
|
|
Open source refers to an *openly distributed* program code (including codes that are *free of charge*) that can be *utilized* and *modified* by the end user *without restriction.*
|
|
|
|
====
|
|
|
|
[NOTE]
|
|
====
|
|
|
|
There is also now solid metrics data from Google's https://googleprojectzero.blogspot.com/p/about-project-zero.html[Project Zero^], Google's security research team, that https://googleprojectzero.blogspot.com/2022/02/a-walk-through-project-zero-metrics.html[Linux developers are doing a faster job of fixing security bugs than anyone else^], including Google.
|
|
|
|
The researchers found that open source developers *fixed problems on Linux in just 25 days* on average (Google 44, Mozilla 46, Apple 69 and Microsoft 83!). In addition, Linux developers have improved their speed in repairing security vulnerabilities from 32 days in 2019 to *just 15 in 2021.*
|
|
|
|
See *Bug fix time* https://googleprojectzero.blogspot.com/2022/02/a-walk-through-project-zero-metrics.html[here^]
|
|
|
|
====
|
|
|
|
|
|
== Docker
|
|
|
|
* https://www.cloudbees.com/blog/why-docker[Docker^]
|
|
|
|
== Intro docker
|
|
|
|
* https://www.compose-spec.io/[docker-compose^]
|
|
|
|
* https://github.com/veggiemonk/awesome-docker[github docker-compose^]
|
|
|
|
* https://twitter.com/awesome_docker[github awesome docker]
|
|
|
|
* https://developers.redhat.com/blog/2014/05/15/practical-introduction-to-docker-containers#search_private_registry[A practical introduction to Docker containers ]
|
|
|
|
* https://developers.redhat.com/blog/2016/01/13/a-practical-introduction-to-docker-container-terminology?utm_campaign=containers&intcmp=70160000000h1s6AAA#[A Practical Introduction to Docker Container Terminology^]
|
|
|
|
=== Learn to build and deploy your distributed applications easily to the cloud with Docker
|
|
|
|
* https://docker-curriculum.com/[HowTo]
|
|
|
|
|
|
|
|
=== Alternatives to Docker
|
|
|
|
TIP: https://apptainer.org/docs/admin/main/[Apptainer/Singularity] - *Simply the best* Container system for HPC
|
|
|
|
====
|
|
Singularity can be used with Docker images. This feature was included because developers use and really like using Docker and scientists have already put much resources into creating Docker images.
|
|
|
|
Features:
|
|
|
|
* Use a Docker image instantly as a Singularity image
|
|
* Pull a Docker image
|
|
* Build images with bases from assembled Docker layers
|
|
|
|
====
|
|
|
|
=== Linux containers (must-read - Highly Recommended)
|
|
|
|
* https://www.redhat.com/en/topics/containers/whats-a-linux-container?utm_campaign=containers&intcmp=70160000000h1s6AAA#history-of-containers[A brief history of containers^]
|
|
|
|
* https://www.redhat.com/en/topics/containers/whats-a-linux-container?utm_campaign=containers&intcmp=70160000000h1s6AAA#just-virtualization[Isn’t this just virtualization?^]
|
|
|
|
* https://opensource.com/article/18/8/sysadmins-guide-containers[A sysadmin's guide to containers^]
|
|
|
|
== Next Step DockerSwarm
|
|
|
|
****
|
|
TIP: DockerSwarm
|
|
|
|
video::Tm0Q5zr3FL4[youtube, start=0]
|
|
****
|
|
|
|
* https://www.simplilearn.com/tutorials/docker-tutorial/docker-swarm[docker-swarm tutorial^]
|
|
|
|
=== Getting started with swarm - official from docker
|
|
|
|
* https://docs.docker.com/engine/swarm/swarm-tutorial/https://k21academy.com/docker-kubernetes/docker-swarm/[docker-swarm^]
|
|
|
|
=== Replicability, use the same files as when developing locally.
|
|
|
|
* https://dockerswarm.rocks/[dockerswarm.rocks^]
|
|
|
|
|
|
=== Alternatives
|
|
|
|
* https://www.nomadproject.io/[monad^]
|
|
|
|
* https://dev.to/stevenmcgown/kubernetes-for-dummies-5hmh[kubernetes^]
|
|
|
|
** https://opensource.com/sites/default/files/gated-content/getting_started_with_kubernetes.pdf[Getting started with Kubernetes^]
|
|
|
|
** https://opensource.com/sites/default/files/gated-content/a_guide_to_kubernetes_for_sres_and_sysadmin.pdf[Kubernetes for sysadmins^]
|
|
|
|
=== Raft consensus in swarm mode
|
|
|
|
* https://docs.docker.com/engine/swarm/raft/[raft^]
|
|
|
|
* https://raft.github.io/[raft github^]
|
|
|
|
* http://thesecretlivesofdata.com/raft/[raft demo^]
|
|
|
|
|
|
|
|
|
|
== Next Step Storage
|
|
|
|
****
|
|
TIP: Storage
|
|
|
|
video::W2Z7fbCLSTw[youtube, start=0]
|
|
****
|
|
|
|
=== Generic
|
|
|
|
* https://docs.mongodb.com/manual/replication/[Mongo replica^]
|
|
|
|
* https://docs.mongodb.com/manual/core/capped-collections/[mongo capped^]
|
|
|
|
=== On memory - Key-value
|
|
|
|
* https://redis.io/topics/cluster-tutorial[Redis cluster^]
|
|
|
|
* https://redis.io/[redis server^]
|
|
|
|
* https://www.instaclustr.com/blog/redis-vs-memcached/[redis vs memcached^]
|
|
|
|
* https://etcd.io/[etcd^]
|
|
|
|
****
|
|
TIP: etcd
|
|
|
|
video::OmphHSaO1sE[youtube, start=0]
|
|
****
|
|
|
|
****
|
|
TIP: redis
|
|
|
|
video::G1rOthIU-uo[youtube, start=0]
|
|
****
|
|
|
|
*etcd vs. Redis*
|
|
|
|
Like etcd, Redis is an open source tool, but their basic functionalities are different.
|
|
|
|
[cols="1,1"]
|
|
|===
|
|
|Redis|etcd
|
|
|
|
| Redis is an in-memory data store and can function as a database, cache, or message broker. Redis supports a wider variety of data types and structures than etcd and has much faster read/write performance.
|
|
| etcd has superior fault tolerance, stronger failover and continuous data availability capabilities.
|
|
|
|
|Redis is better suited for serving as a distributed memory caching system than for storing and distributed system configuration information.
|
|
|etcd persists all stored data to disk.
|
|
|
|
|
|
|===
|
|
|
|
|
|
=== Sharing
|
|
|
|
==== NFS
|
|
|
|
* https://cloud.netapp.com/blog/azure-anf-blg-linux-nfs-server-how-to-set-up-server-and-client[NFS^]
|
|
|
|
* https://www.gluster.org/[GlusterFS^]
|
|
|
|
* https://github.com/minio/minio[Minio^]
|
|
|
|
== Next step Deploy
|
|
|
|
* https://www.compose-spec.io/[docker-compose^]
|
|
|
|
****
|
|
TIP: Deploy
|
|
|
|
video::zLltQxVb3NI[youtube, start=0]
|
|
****
|
|
|
|
https://12factor.net/[12factor^]
|
|
|
|
== Development - From Design to Application
|
|
|
|
=== programming language
|
|
|
|
==== nodeJS
|
|
****
|
|
TIP: NodeJS
|
|
|
|
video::ENrzD9HAZK4[youtube, start=0]
|
|
****
|
|
|
|
* https://nodejs.org/en/[nodeJS^]
|
|
|
|
==== python
|
|
|
|
* https://www.python.org/[python^]
|
|
|
|
=== Big data and Scientific analysis
|
|
|
|
* https://numpy.org/[numpy^]
|
|
|
|
* https://scipy.org/[scipy^]
|
|
|
|
* https://jupyter.org/[jupyter^]
|
|
|
|
=== Html5 graphical librarys
|
|
|
|
* https://p5js.org/[p5js^]
|
|
|
|
* https://threejs.org/[threejs^]
|
|
|
|
|
|
=== Rest API
|
|
****
|
|
TIP: Rest api
|
|
|
|
video::s7wmiS2mSXY[youtube, start=0]
|
|
****
|
|
|
|
=== Sockets
|
|
****
|
|
TIP: Sockets
|
|
|
|
video::ZBM28ZPlin8[youtube, start=0]
|
|
****
|
|
|
|
* https://socket.io/[socket.io^]
|
|
|
|
|
|
|
|
== Code as a Service (CaaS)
|
|
|
|
****
|
|
TIP: Automation with Code as a Service. CaaS’s primary service or function is automation. It uses software to automate repetitive practices to hasten and simplify implementations and processes.
|
|
|
|
video::VRoQLVHdNHE[youtube, start=0]
|
|
****
|
|
|
|
|
|
=== https://docs.ansible.com/ansible/latest/user_guide/playbooks_intro.html[Ansible^]
|
|
|
|
Ansible is an open source automation software promoted by Redhat Corporation. In addition, to cloud provisioning, it assists in application development, intra-service orchestration, and configuration.
|
|
Ansible uses the simple programming language YAML to create playbooks for automation.
|
|
Ansible has many modules that integrate with the most common cloud solutions such as AWS, Google Cloud Platform (GCP) and VMware.
|
|
|
|
|
|
=== https://www.terraform.io/cli[Teraform^]
|
|
|
|
Terraform is an infrastructure as code software by Hashi Corporation.
|
|
It primarily focuses on creating data center infrastructure that is provided by large public clouds. Teraform utilizes JSON language to define infrastructure templates with integrations such as AWS, Azure, GCP, and IBM cloud.
|
|
|
|
====
|
|
* You can find a ready-to-use example for Ansible in swarmlab.io-->Labroom-->Linux in examples directory
|
|
|
|
* https://opensource.com/sites/default/files/gated-content/ansible_automation_for_sysadmins_v2.pdf[Ansible Automation for SysAdmins^]
|
|
|
|
* https://opensource.com/sites/default/files/gated-content/starting_a_devops_transformation.pdf[Analyzing the DNA of DevOps^]
|
|
|
|
====
|
|
|
|
|
|
== Auth
|
|
|
|
* https://oauth.net/2/[auth2^]
|
|
|
|
* https://oauth.net/code/nodejs/[Nodejs auth library^]
|
|
|
|
=== Auth server
|
|
|
|
* https://www.keycloak.org/[auth2 server^]
|
|
|
|
== Security VPN etc
|
|
|
|
* https://www.wireguard.com/[simply the best VPN out there^]
|
|
|
|
== POC
|
|
|
|
=== netflix
|
|
|
|
****
|
|
TIP: Netflix
|
|
|
|
video::CZ3wIuvmHeM[youtube, start=0]
|
|
****
|
|
|
|
6:50 monolithic
|
|
|
|
8:24 minroservices
|
|
|
|
19:00 microservice
|
|
|
|
see also 37:00
|
|
|
|
https://netflix.github.io/[netflix github^]
|
|
|
|
|
|
== Open Source Cloud Guide from IBM
|
|
|
|
* https://open-cloud-guide.dev/[open-cloud guide^]
|
|
|
|
|
|
== Et cetera
|
|
|
|
|
|
****
|
|
|
|
TIP: The Cathedral and the Bazaar
|
|
|
|
* https://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar[The Cathedral and the Bazaar^]
|
|
** https://monoskop.org/images/e/e0/Raymond_Eric_S_The_Cathedral_and_the_Bazaar_rev_ed.pdf[The Cathedral and the Bazaar-English^]
|
|
* http://www.catb.org/~esr/writings/cathedral-bazaar/[The Cathedral and the Bazaar - Translations^] (must-read - Highly Recommended)
|
|
|
|
====
|
|
"Sociologists years ago discovered that *the averaged opinion of a mass of equally expert (or equally ignorant) observers is quite a bit more reliable a predictor than the opinion of a single randomly chosen observer.*
|
|
They called this *the Delphi effect [https://en.wikipedia.org/wiki/The_Wisdom_of_Crowds[Wikipedia^]][https://en.wikipedia.org/wiki/Collective_intelligence[Wikipedia^]].* +
|
|
It appears that what Linus has shown is that this applies even to debugging an operating system—that *the Delphi effect can tame development complexity even at the complexity level of an OS kernel*"
|
|
|
|
====
|
|
|
|
****
|
|
|
|
* http://www.catb.org/~esr/faqs/hacker-howto.html[How To Become A Hacker^]
|
|
* https://www.gutenberg.org/[Project Gutenberg^] - >60,000 free eBooks)
|
|
* https://www.openbook.gr[openbooks library^] - >11.458 ελληνικά e-books
|
|
* https://www.gnu.org/doc/other-free-books.en.html#navigation[Free Books from Other Publishers^]
|
|
* https://opensource.com/resources/organizations[Open source organizations^]
|
|
|
|
=== Cheat Sheet
|
|
|
|
* https://opensource.com/sites/default/files/gated-content/cheat_sheet_microservices.pdf[Microservices^]
|
|
|
|
* https://opensource.com/sites/default/files/gated-content/osdc_cheatsheet-ansible.pdf[Ansible^]
|
|
|
|
* https://opensource.com/sites/default/files/gated-content/cheat_sheet_ansible-k8s.pdf[Ansible k8s^]
|
|
|
|
* https://opensource.com/sites/default/files/gated-content/curl-cheat-sheet.pdf[curl^]
|
|
|
|
* https://opensource.com/sites/default/files/gated-content/osdc_cheatsheet-networking-2021.4.8.pdf[network^]
|
|
|
|
|
|
== Last but not least
|
|
|
|
=== KISS
|
|
|
|
https://en.wikipedia.org/wiki/KISS_principle[KISS^] [http://principles-wiki.net/principles:keep_it_simple_stupid[Wiki]] [http://principles-wiki.net/principles:you_ain_t_gonna_need_it[Wiki]] is related to a fair number of other famous quotes, phrases and principles. Among them:
|
|
|
|
* *Occam’s Razor* - “Among competing hypotheses, the one with the fewest assumptions should be selected” (but often restated as “The simplest solution is most likely the correct solution” which is not quite the same thing).
|
|
* *Albert Einstein’s* – “Make everything as simple as possible but not simpler” (it is possible that Einstein never actually said this and it was actually a paraphrase of something he said during a lecture but the principle remains sound).
|
|
* *Leonardo Da Vinci’s* – “Simplicity is the ultimate sophistication” (when, perhaps, the greatest designer in history offers this advice, it’s almost certainly good advice).
|
|
* *Ludwig Mies Van Der Rohe’s* – “Less is more” (Mies was a highly respected architect and peer of the better known Frank Lloyd Wright)
|
|
* *Bjarne Stroustrup’s* “Make Simple Tasks Simple!” (Stroustrup is a Danish computer scientist and highly regarded academic).
|
|
* *Antoin Marie Jean-Baptiste Roger, comte de Saint-Exupery’s* - “It seems that perfection is not reached when there is nothing left to add but when there is nothing left to take away.”
|
|
|
|
|
|
****
|
|
TIP: Examples of "non-KISS" solutions!
|
|
|
|
video::350486709[vimeo,start=39,end=104]
|
|
|
|
|
|
image:Heath_Robinson_WWI.png[width=500,float="right"]
|
|
|
|
|
|
|
|
****
|
|
|
|
|