Intros

• Hello! We are:

  • 👷🏻‍♀️ AJ (@s0ulshake, EphemeraSearch)

  • 🐳 Jérôme (@jpetazzo, Enix SAS)

• The workshop will run from 10:00 to 15:00 (EEST timezone, GMT+3)

• We will have short breaks at 11:10 and 13:50 (approximately!)

• And a longer lunch break at 12:20 (about 30 minutes)

• Feel free to interrupt for questions at any time

• Especially when you see full screen container pictures!

• Live feedback, questions, help: Gitter

logistics.md

A brief introduction

• This was initially written by Jérôme Petazzoni to support in-person, instructor-led workshops and tutorials

• Credit is also due to multiple contributors — thank you!

• You can also follow along on your own, at your own pace

• We included as much information as possible in these slides

• We recommend having a mentor to help you ...

• ... Or be comfortable spending some time reading the Kubernetes documentation ...

• ... And looking for answers on StackOverflow and other outlets

k8s/intro.md

Accessing these slides now

• We recommend that you open these slides in your browser:

  https://2020-09-fwdays.container.training/

• Use arrows to move to next/previous slide

  (up, down, left, right, page up, page down)

• Type a slide number + ENTER to go to that slide

• The slide number is also visible in the URL bar

  (e.g. .../#123 for slide 123)

shared/about-slides.md

Accessing these slides later

• Slides will remain online so you can review them later if needed

  (let's say we'll keep them online at least 1 year, how about that?)

• You can download the slides using that URL:

  https://2020-09-fwdays.container.training/slides.zip

  (then open the file kube-fullday.yml.html)

• You will find new versions of these slides on:

  https://container.training/

shared/about-slides.md

These slides are open source

• You are welcome to use, re-use, share these slides

• These slides are written in markdown

• The sources of these slides are available in a public GitHub repository:

  https://github.com/jpetazzo/container.training

• Typos? Mistakes? Questions? Feel free to hover over the bottom of the slide ...

👇 Try it! The source file will be shown and you can view it on GitHub and fork and edit it.

shared/about-slides.md

Chat room

• We've set up a chat room that we will monitor during the workshop

• Don't hesitate to use it to ask questions, or get help, or share feedback

• The chat room will also be available after the workshop

• Join the chat room: Gitter

• Say hi in the chat room!

shared/chat-room-im.md

Module 1

• Pre-requirements

• Our sample application

• Kubernetes concepts

• First contact with kubectl

• Running our first containers on Kubernetes

(auto-generated TOC)

Module 2

• Labels and annotations

• Revisiting kubectl logs

• Accessing logs from the CLI

• Declarative vs imperative

• Kubernetes network model

• Exposing containers

• Shipping images with a registry

• Running our application on Kubernetes

(auto-generated TOC)

Module 3

• Deploying with YAML

• Scaling our demo app

• Daemon sets

• Labels and selectors

• Rolling updates

(auto-generated TOC)

Module 4

• Namespaces

• Exposing HTTP services with Ingress resources

• Volumes

• Managing configuration

• Setting up Kubernetes

• Running a local development cluster

• Links and resources

(auto-generated TOC)

Module 5

• (Extra content)

• Healthchecks

• Executing batch jobs

• Network policies

• Authentication and authorization

(auto-generated TOC)

Module 6

• (More extra content)

• Stateful sets

• Running a Consul cluster

• Persistent Volumes Claims

• Local Persistent Volumes

• Highly available Persistent Volumes

(auto-generated TOC)

shared/toc.md

Pre-requirements

• Be comfortable with the UNIX command line

  • navigating directories

  • editing files

  • a little bit of bash-fu (environment variables, loops)

• Some Docker knowledge

  • docker run, docker ps, docker build

  • ideally, you know how to write a Dockerfile and build it
    (even if it's a FROM line and a couple of RUN commands)

• It's totally OK if you are not a Docker expert!

shared/prereqs.md

Hands-on sections

• The whole workshop is hands-on

• We are going to build, ship, and run containers!

• You are invited to reproduce all the demos

• All hands-on sections are clearly identified, like the gray rectangle below

shared/prereqs.md

Doing or re-doing the workshop on your own?

• Use something like Play-With-Docker or Play-With-Kubernetes

  Zero setup effort; but environment are short-lived and might have limited resources

• Create your own cluster (local or cloud VMs)

  Small setup effort; small cost; flexible environments

• Create a bunch of clusters for you and your friends (instructions)

  Bigger setup effort; ideal for group training

shared/connecting.md

For a consistent Kubernetes experience ...

• If you are using your own Kubernetes cluster, you can use shpod

• shpod provides a shell running in a pod on your own cluster

• It comes with many tools pre-installed (helm, stern...)

• These tools are used in many exercises in these slides

• shpod also gives you completion and a fancy prompt

shared/connecting.md

We will (mostly) interact with node1 only

These remarks apply only when using multiple nodes, of course.

• Unless instructed, all commands must be run from the first VM, node1

• We will only check out/copy the code on node1

• During normal operations, we do not need access to the other nodes

• If we had to troubleshoot issues, we would use a combination of:

  • SSH (to access system logs, daemon status...)

  • Docker API (to check running containers and container engine status)

shared/connecting.md

Terminals

Once in a while, the instructions will say:
"Open a new terminal."

There are multiple ways to do this:

• create a new window or tab on your machine, and SSH into the VM;

• use screen or tmux on the VM and open a new window from there.

You are welcome to use the method that you feel the most comfortable with.

shared/connecting.md

Tmux cheatsheet

Tmux is a terminal multiplexer like screen.

You don't have to use it or even know about it to follow along.
But some of us like to use it to switch between terminals.
It has been preinstalled on your workshop nodes.

• Ctrl-b c → creates a new window
• Ctrl-b n → go to next window
• Ctrl-b p → go to previous window
• Ctrl-b " → split window top/bottom
• Ctrl-b % → split window left/right
• Ctrl-b Alt-1 → rearrange windows in columns
• Ctrl-b Alt-2 → rearrange windows in rows
• Ctrl-b arrows → navigate to other windows
• Ctrl-b d → detach session
• tmux attach → reattach to session

shared/connecting.md

Our sample application

• We will clone the GitHub repository onto our node1

• The repository also contains scripts and tools that we will use through the workshop

(You can also fork the repository on GitHub and clone your fork if you prefer that.)

shared/sampleapp.md

Downloading and running the application

Let's start this before we look around, as downloading will take a little time...

Compose tells Docker to build all container images (pulling the corresponding base images), then starts all containers, and displays aggregated logs.

shared/sampleapp.md

What's this application?

DockerCoins in the microservices era

• DockerCoins is made of 5 services:

  • rng = web service generating random bytes

  • hasher = web service computing hash of POSTed data

  • worker = background process calling rng and hasher

  • webui = web interface to watch progress

  • redis = data store (holds a counter updated by worker)

• These 5 services are visible in the application's Compose file, docker-compose.yml

shared/sampleapp.md

How DockerCoins works

• worker invokes web service rng to generate random bytes

• worker invokes web service hasher to hash these bytes

• worker does this in an infinite loop

• every second, worker updates redis to indicate how many loops were done

• webui queries redis, and computes and exposes "hashing speed" in our browser

(See diagram on next slide!)

shared/sampleapp.md

Service discovery in container-land

How does each service find out the address of the other ones?

Example in worker/worker.py

redis = Redis("redis")
def get_random_bytes():
    r = requests.get("http://rng/32")
    return r.content
def hash_bytes(data):
    r = requests.post("http://hasher/",
                      data=data,
                      headers={"Content-Type": "application/octet-stream"})

(Full source code available here)

shared/sampleapp.md

Show me the code!

• You can check the GitHub repository with all the materials of this workshop:
  https://github.com/jpetazzo/container.training

• The application is in the dockercoins subdirectory

• The Compose file (docker-compose.yml) lists all 5 services

• redis is using an official image from the Docker Hub

• hasher, rng, worker, webui are each built from a Dockerfile

• Each service's Dockerfile and source code is in its own directory

  (hasher is in the hasher directory, rng is in the rng directory, etc.)

shared/sampleapp.md

Our application at work

• On the left-hand side, the "rainbow strip" shows the container names

• On the right-hand side, we see the output of our containers

• We can see the worker service making requests to rng and hasher

• For rng and hasher, we see HTTP access logs

shared/sampleapp.md

Connecting to the web UI

• "Logs are exciting and fun!" (No-one, ever)

• The webui container exposes a web dashboard; let's view it

A drawing area should show up, and after a few seconds, a blue graph will appear.

shared/sampleapp.md

Stopping the application

• If we interrupt Compose (with ^C), it will politely ask the Docker Engine to stop the app

• The Docker Engine will send a TERM signal to the containers

• If the containers do not exit in a timely manner, the Engine sends a KILL signal

Clean up

• Before moving on, let's remove those containers

shared/composedown.md

Kubernetes concepts

• Kubernetes is a container management system

• It runs and manages containerized applications on a cluster

What can we do with Kubernetes?

• Let's imagine that we have a 3-tier e-commerce app:

  • web frontend

  • API backend

  • database (that we will keep out of Kubernetes for now)

• We have built images for our frontend and backend components

  (e.g. with Dockerfiles and docker build)

• We are running them successfully with a local environment

  (e.g. with Docker Compose)

• Let's see how we would deploy our app on Kubernetes!

k8s/concepts-k8s.md

Basic things we can ask Kubernetes to do

Other things that Kubernetes can do for us

• Autoscaling

  (straightforward on CPU; more complex on other metrics)

• Resource management and scheduling

  (reserve CPU/RAM for containers; placement constraints)

• Advanced rollout patterns

  (blue/green deployment, canary deployment)

k8s/concepts-k8s.md

More things that Kubernetes can do for us

• Batch jobs

  (one-off; parallel; also cron-style periodic execution)

• Fine-grained access control

  (defining what can be done by whom on which resources)

• Stateful services

  (databases, message queues, etc.)

• Automating complex tasks with operators

  (e.g. database replication, failover, etc.)

k8s/concepts-k8s.md

Kubernetes architecture

k8s/concepts-k8s.md

Kubernetes architecture

• Ha ha ha ha

• OK, I was trying to scare you, it's much simpler than that ❤️

k8s/concepts-k8s.md

Credits

• The first schema is a Kubernetes cluster with storage backed by multi-path iSCSI

  (Courtesy of Yongbok Kim)

• The second one is a simplified representation of a Kubernetes cluster

  (Courtesy of Imesh Gunaratne)

k8s/concepts-k8s.md

Kubernetes architecture: the nodes

• The nodes executing our containers run a collection of services:

  • a container Engine (typically Docker)

  • kubelet (the "node agent")

  • kube-proxy (a necessary but not sufficient network component)

• Nodes were formerly called "minions"

  (You might see that word in older articles or documentation)

k8s/concepts-k8s.md

Kubernetes architecture: the control plane

• The Kubernetes logic (its "brains") is a collection of services:

  • the API server (our point of entry to everything!)

  • core services like the scheduler and controller manager

  • etcd (a highly available key/value store; the "database" of Kubernetes)

• Together, these services form the control plane of our cluster

• The control plane is also called the "master"

k8s/concepts-k8s.md

Interacting with Kubernetes

• We will interact with our Kubernetes cluster through the Kubernetes API

• The Kubernetes API is (mostly) RESTful

• It allows us to create, read, update, delete resources

• A few common resource types are:

  • node (a machine — physical or virtual — in our cluster)

  • pod (group of containers running together on a node)

  • service (stable network endpoint to connect to one or multiple containers)

k8s/concepts-k8s.md

Scaling

• How would we scale the pod shown on the previous slide?

• Do create additional pods

  • each pod can be on a different node

  • each pod will have its own IP address

• Do not add more NGINX containers in the pod

  • all the NGINX containers would be on the same node

  • they would all have the same IP address
    (resulting in Address alreading in use errors)

k8s/concepts-k8s.md

Together or separate

• Should we put e.g. a web application server and a cache together?
  ("cache" being something like e.g. Memcached or Redis)

• Putting them in the same pod means:

  • they have to be scaled together

  • they can communicate very efficiently over localhost

• Putting them in different pods means:

  • they can be scaled separately

  • they must communicate over remote IP addresses
    (incurring more latency, lower performance)

• Both scenarios can make sense, depending on our goals

k8s/concepts-k8s.md

Credits

• The first diagram is courtesy of Lucas Käldström, in this presentation

  • it's one of the best Kubernetes architecture diagrams available!

• The second diagram is courtesy of Weave Works

  • a pod can have multiple containers working together

  • IP addresses are associated with pods, not with individual containers

Both diagrams used with permission.

First contact with kubectl

• kubectl is (almost) the only tool we'll need to talk to Kubernetes

• It is a rich CLI tool around the Kubernetes API

  (Everything you can do with kubectl, you can do directly with the API)

• On our machines, there is a ~/.kube/config file with:

  • the Kubernetes API address

  • the path to our TLS certificates used to authenticate

• You can also use the --kubeconfig flag to pass a config file

• Or directly --server, --user, etc.

• kubectl can be pronounced "Cube C T L", "Cube cuttle", "Cube cuddle"...

k8s/kubectlget.md

kubectl get

• Let's look at our Node resources with kubectl get!

k8s/kubectlget.md

Obtaining machine-readable output

• kubectl get can output JSON, YAML, or be directly formatted

k8s/kubectlget.md

(Ab)using kubectl and jq

• It's super easy to build custom reports

k8s/kubectlget.md

Type names

• The most common resource names have three forms:

  • singular (e.g. node, service, deployment)

  • plural (e.g. nodes, services, deployments)

  • short (e.g. no, svc, deploy)

• Some resources do not have a short name

• Endpoints only have a plural form

  (because even a single Endpoints resource is actually a list of endpoints)

k8s/kubectlget.md

Viewing details

• We can use kubectl get -o yaml to see all available details

• However, YAML output is often simultaneously too much and not enough

• For instance, kubectl get node node1 -o yaml is:

  • too much information (e.g.: list of images available on this node)

  • not enough information (e.g.: doesn't show pods running on this node)

  • difficult to read for a human operator

• For a comprehensive overview, we can use kubectl describe instead

k8s/kubectlget.md

kubectl describe

• kubectl describe needs a resource type and (optionally) a resource name

• It is possible to provide a resource name prefix

  (all matching objects will be displayed)

• kubectl describe will retrieve some extra information about the resource

(We should notice a bunch of control plane pods.)

k8s/kubectlget.md

Listing running containers

• Containers are manipulated through pods

• A pod is a group of containers:

  • running together (on the same node)

  • sharing resources (RAM, CPU; but also network, volumes)

Namespaces

• Namespaces allow us to segregate resources

Accessing namespaces

• By default, kubectl uses the default namespace

• We can see resources in all namespaces with --all-namespaces

Here are our system pods!

k8s/kubectlget.md

What are all these control plane pods?

• etcd is our etcd server

• kube-apiserver is the API server

• kube-controller-manager and kube-scheduler are other control plane components

• coredns provides DNS-based service discovery (replacing kube-dns as of 1.11)

• kube-proxy is the (per-node) component managing port mappings and such

• weave is the (per-node) component managing the network overlay

• the READY column indicates the number of containers in each pod

  (1 for most pods, but weave has 2, for instance)

k8s/kubectlget.md

Scoping another namespace

• We can also look at a different namespace (other than default)

k8s/kubectlget.md

Namespaces and other kubectl commands

• We can use -n/--namespace with almost every kubectl command

• Example:

  • kubectl create --namespace=X to create something in namespace X

• We can use -A/--all-namespaces with most commands that manipulate multiple objects

• Examples:

  • kubectl delete can delete resources across multiple namespaces

  • kubectl label can add/remove/update labels across multiple namespaces

k8s/kubectlget.md

Services

• A service is a stable endpoint to connect to "something"

  (In the initial proposal, they were called "portals")

ClusterIP services

• A ClusterIP service is internal, available from the cluster only

• This is useful for introspection from within containers

The command above should either time out, or show an authentication error. Why?

k8s/kubectlget.md

Time out

• Connections to ClusterIP services only work from within the cluster

• If we are outside the cluster, the curl command will probably time out

  (Because the IP address, e.g. 10.96.0.1, isn't routed properly outside the cluster)

• This is the case with most "real" Kubernetes clusters

• To try the connection from within the cluster, we can use shpod

k8s/kubectlget.md

Authentication error

This is what we should see when connecting from within the cluster:

$ curl -k https://10.96.0.1
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {
  },
  "code": 403
}

k8s/kubectlget.md

Explanations

• We can see kind, apiVersion, metadata

• These are typical of a Kubernetes API reply

• Because we are talking to the Kubernetes API

• The Kubernetes API tells us "Forbidden"

  (because it requires authentication)

• The Kubernetes API is reachable from within the cluster

  (many apps integrating with Kubernetes will use this)

k8s/kubectlget.md

DNS integration

• Each service also gets a DNS record

• The Kubernetes DNS resolver is available from within pods

  (and sometimes, from within nodes, depending on configuration)

• Code running in pods can connect to services using their name

  (e.g. https://kubernetes/...)

Running our first containers on Kubernetes

• First things first: we cannot run a container

Choose your own adventure

• First, let's check which version of Kubernetes we're running

• In the following slides, we will talk about 1.17- or 1.18+

  (to indicate "up to Kubernetes 1.17" and "from Kubernetes 1.18")

k8s/kubectl-run.md

Starting a simple pod with kubectl run

• kubectl run is convenient to start a single pod

• We need to specify at least a name and the image we want to use

• Optionally, we can specify the command to run in the pod

k8s/kubectl-run.md

What do we see?

• In Kubernetes 1.18+, the output tells us that a Pod is created:

  pod/pingpong created

• In Kubernetes 1.17-, the output is much more verbose:

  kubectl run --generator=deployment/apps.v1 is DEPRECATED 
  and will be removed in a future version. Use kubectl run 
  --generator=run-pod/v1 or kubectl create instead.
  deployment.apps/pingpong created

• There is a deprecation warning ...

• ... And a Deployment was created instead of a Pod

🤔 What does that mean?

k8s/kubectl-run.md

Show me all you got!

• What resources were created by kubectl run?

Note: kubectl get all is a lie. It doesn't show everything.

(But it shows a lot of "usual suspects", i.e. commonly used resources.)

k8s/kubectl-run.md

The situation with Kubernetes 1.18+

NAME           READY   STATUS    RESTARTS   AGE
pod/pingpong   1/1     Running   0          9s
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   3h30m

We wanted a pod, we got a pod, named pingpong. Great!

(We can ignore service/kubernetes, it was already there before.)

k8s/kubectl-run.md

The situation with Kubernetes 1.17-

NAME                            READY   STATUS        RESTARTS   AGE
pod/pingpong-6ccbc77f68-kmgfn   1/1     Running       0          11s
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   3h45
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/pingpong   1/1     1            1           11s
NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/pingpong-6ccbc77f68   1         1         1       11s

Our pod is not named pingpong, but pingpong-xxxxxxxxxxx-yyyyy.

We have a Deployment named pingpong, and an extra Replica Set, too. What's going on?

k8s/kubectl-run.md

From Deployment to Pod

We have the following resources:

• deployment.apps/pingpong

  This is the Deployment that we just created.

• replicaset.apps/pingpong-xxxxxxxxxx

  This is a Replica Set created by this Deployment.

• pod/pingpong-xxxxxxxxxx-yyyyy

  This is a pod created by the Replica Set.

Let's explain what these things are.

k8s/kubectl-run.md

Pod

• Can have one or multiple containers

• Runs on a single node

  (Pod cannot "straddle" multiple nodes)

• Pods cannot be moved

  (e.g. in case of node outage)

• Pods cannot be scaled

  (except by manually creating more Pods)

k8s/kubectl-run.md

Replica Set

• Set of identical (replicated) Pods

• Defined by a pod template + number of desired replicas

• If there are not enough Pods, the Replica Set creates more

  (e.g. in case of node outage; or simply when scaling up)

• If there are too many Pods, the Replica Set deletes some

  (e.g. if a node was disconnected and comes back; or when scaling down)

• We can scale up/down a Replica Set

  • we update the manifest of the Replica Set

  • as a consequence, the Replica Set controller creates/deletes Pods

k8s/kubectl-run.md

Deployment

• Replica Sets control identical Pods

• Deployments are used to roll out different Pods

  (different image, command, environment variables, ...)

• When we update a Deployment with a new Pod definition:

  • a new Replica Set is created with the new Pod definition

  • that new Replica Set is progressively scaled up

  • meanwhile, the old Replica Set(s) is(are) scaled down

• This is a rolling update, minimizing application downtime

• When we scale up/down a Deployment, it scales up/down its Replica Set

k8s/kubectl-run.md

kubectl run through the ages

• When we want to run an app on Kubernetes, we generally want a Deployment

• Up to Kubernetes 1.17, kubectl run created a Deployment

  • it could also create other things, by using special flags

  • this was powerful, but potentially confusing

  • creating a single Pod was done with kubectl run --restart=Never

  • other resources could also be created with kubectl create ...

• From Kubernetes 1.18, kubectl run creates a Pod

  • other kinds of resources can still be created with kubectl create

k8s/kubectl-run.md

Creating a Deployment the proper way

• Let's destroy that pingpong app that we created

• Then we will use kubectl create deployment to re-create it

k8s/kubectl-run.md

Running ping in a Deployment

• When using kubectl create deployment, we cannot indicate the command to execute

  (at least, not in Kubernetes 1.18; but that changed in Kubernetes 1.19)

• We can:

  • write a custom YAML manifest for our Deployment

Creating a Deployment running ping

• Let's create a Deployment named pingpong

• It will use the image jpetazzo/ping

k8s/kubectl-run.md

Viewing container output

• Let's use the kubectl logs command

• We will pass either a pod name, or a type/name

  (E.g. if we specify a deployment or replica set, it will get the first pod in it)

• Unless specified otherwise, it will only show logs of the first container in the pod

  (Good thing there's only one in ours!)

k8s/kubectl-run.md

Streaming logs in real time

• Just like docker logs, kubectl logs supports convenient options:

  • -f/--follow to stream logs in real time (à la tail -f)

  • --tail to indicate how many lines you want to see (from the end)

  • --since to get logs only after a given timestamp

k8s/kubectl-run.md

Scaling our application

• We can create additional copies of our container (I mean, our pod) with kubectl scale

k8s/kubectl-run.md

Streaming logs of multiple pods

• What happens if we try kubectl logs now that we have multiple pods?

  kubectl logs deploy/pingpong --tail 3

kubectl logs will warn us that multiple pods were found.

It is showing us only one of them.

We'll see later how to address that shortcoming.

k8s/kubectl-run.md

Resilience

• The deployment pingpong watches its replica set

• The replica set ensures that the right number of pods are running

• What happens if pods disappear?

k8s/kubectl-run.md

What happened?

• kubectl delete pod terminates the pod gracefully

  (sending it the TERM signal and waiting for it to shutdown)

• As soon as the pod is in "Terminating" state, the Replica Set replaces it

• But we can still see the output of the "Terminating" pod in kubectl logs

• Until 30 seconds later, when the grace period expires

• The pod is then killed, and kubectl logs exits

Labels and annotations

• Most Kubernetes resources can have labels and annotations

• Both labels and annotations are arbitrary strings

  (with some limitations that we'll explain in a minute)

• Both labels and annotations can be added, removed, changed, dynamically

• This can be done with:

  • the kubectl edit command

  • the kubectl label and kubectl annotate

  • ... many other ways! (kubectl apply -f, kubectl patch, ...)

k8s/labels-annotations.md

Viewing labels and annotations

• Let's see what we get when we create a Deployment

So, what do we get?

k8s/labels-annotations.md

Labels and annotations for our Deployment

• We see one label:

  Labels: app=clock

• This is added by kubectl create deployment

• And one annotation:

  Annotations: deployment.kubernetes.io/revision: 1

• This is to keep track of successive versions when doing rolling updates

k8s/labels-annotations.md

And for the related Pod?

• Let's look up the Pod that was created and check it too

So, what do we get?

k8s/labels-annotations.md

Labels and annotations for our Pod

• We see two labels:

  Labels: app=clock
          pod-template-hash=xxxxxxxxxx

• app=clock comes from kubectl create deployment too

• pod-template-hash was assigned by the Replica Set

  (when we will do rolling updates, each set of Pods will have a different hash)

• There are no annotations:

  Annotations: <none>

k8s/labels-annotations.md

Selectors

• A selector is an expression matching labels

• It will restrict a command to the objects matching at least all these labels

k8s/labels-annotations.md

Settings labels and annotations

• The easiest method is to use kubectl label and kubectl annotate

k8s/labels-annotations.md

Other ways to view labels

• kubectl get gives us a couple of useful flags to check labels

• kubectl get --show-labels shows all labels

• kubectl get -L xyz shows the value of label xyz

k8s/labels-annotations.md

Other ways to view labels

• We can use the --show-labels flag with kubectl get

k8s/labels-annotations.md

Differences between labels and annotations

• The key for both labels and annotations:

  • must start and end with a letter or digit

  • can also have . - _ (but not in first or last position)

  • can be up to 63 characters, or 253 + / + 63

• Label values are up to 63 characters, with the same restrictions

• Annotations values can have arbitrary characeters (yes, even binary)

• Maximum length isn't defined

  (dozens of kilobytes is fine, hundreds maybe not so much)

Revisiting kubectl logs

• In this section, we assume that we have a Deployment with multiple Pods

  (e.g. pingpong that we scaled to at least 3 pods)

• We will highlights some of the limitations of kubectl logs

k8s/kubectl-logs.md

Streaming logs of multiple pods

• By default, kubectl logs shows us the output of a single Pod

kubectl logs only shows us the logs of one of the Pods.

k8s/kubectl-logs.md

Viewing logs of multiple pods

• When we specify a deployment name, only one single pod's logs are shown

• We can view the logs of multiple pods by specifying a selector

• If we check the pods created by the deployment, they all have the label app=pingpong

  (this is just a default label that gets added when using kubectl create deployment)

k8s/kubectl-logs.md

Streaming logs of multiple pods

• Can we stream the logs of all our pingpong pods?

Note: combining -l and -f is only possible since Kubernetes 1.14!

Let's try to understand why ...

k8s/kubectl-logs.md

Shortcomings of kubectl logs

• We don't see which pod sent which log line

• If pods are restarted / replaced, the log stream stops

• If new pods are added, we don't see their logs

• To stream the logs of multiple pods, we need to write a selector

• There are external tools to address these shortcomings

  (e.g.: Stern)

k8s/kubectl-logs.md

Accessing logs from the CLI

• The kubectl logs command has limitations:

  • it cannot stream logs from multiple pods at a time

  • when showing logs from multiple pods, it mixes them all together

• We are going to see how to do it better

k8s/logs-cli.md

Doing it manually

• We could (if we were so inclined) write a program or script that would:

  • take a selector as an argument

  • enumerate all pods matching that selector (with kubectl get -l ...)

  • fork one kubectl logs --follow ... command per container

  • annotate the logs (the output of each kubectl logs ... process) with their origin

  • preserve ordering by using kubectl logs --timestamps ... and merge the output

Stern

Stern is an open source project by Wercker.

From the README:

Stern allows you to tail multiple pods on Kubernetes and multiple containers within the pod. Each result is color coded for quicker debugging.

The query is a regular expression so the pod name can easily be filtered and you don't need to specify the exact id (for instance omitting the deployment id). If a pod is deleted it gets removed from tail and if a new pod is added it automatically gets tailed.

Exactly what we need!

k8s/logs-cli.md

Checking if Stern is installed

• Run stern (without arguments) to check if it's installed:

  $ stern
  Tail multiple pods and containers from Kubernetes
  Usage:
  stern pod-query [flags]

• If it's missing, let's see how to install it

k8s/logs-cli.md

Installing Stern

• Stern is written in Go, and Go programs are usually shipped as a single binary

• We just need to download that binary and put it in our PATH!

• Binary releases are available here on GitHub

• The following commands will install Stern on a Linux Intel 64 bit machine:

  sudo curl -L -o /usr/local/bin/stern \
     https://github.com/wercker/stern/releases/download/1.11.0/stern_linux_amd64
  sudo chmod +x /usr/local/bin/stern

• On macOS, we can also brew install stern or sudo port install stern

k8s/logs-cli.md

Using Stern

• There are two ways to specify the pods whose logs we want to see:

  • -l followed by a selector expression (like with many kubectl commands)

  • with a "pod query," i.e. a regex used to match pod names

• These two ways can be combined if necessary

k8s/logs-cli.md

Stern convenient options

• The --tail N flag shows the last N lines for each container

  (Instead of showing the logs since the creation of the container)

• The -t / --timestamps flag shows timestamps

• The --all-namespaces flag is self-explanatory

k8s/logs-cli.md

Using Stern with a selector

• When specifying a selector, we can omit the value for a label

• This will match all objects having that label (regardless of the value)

• Everything created with kubectl run has a label run

• Everything created with kubectl create deployment has a label app

• We can use that property to view the logs of all the pods created with kubectl create deployment

Declarative vs imperative

• Our container orchestrator puts a very strong emphasis on being declarative

• Declarative:

  I would like a cup of tea.

• Imperative:

  Boil some water. Pour it in a teapot. Add tea leaves. Steep for a while. Serve in a cup.

Declarative vs imperative

• What declarative would really be:

  I want a cup of tea, obtained by pouring an infusion¹ of tea leaves in a cup.

Declarative vs imperative

• Imperative systems:

  • simpler

  • if a task is interrupted, we have to restart from scratch

• Declarative systems:

  • if a task is interrupted (or if we show up to the party half-way through), we can figure out what's missing and do only what's necessary

  • we need to be able to observe the system

  • ... and compute a "diff" between what we have and what we want

shared/declarative.md

Declarative vs imperative in Kubernetes

• With Kubernetes, we cannot say: "run this container"

• All we can do is write a spec and push it to the API server

  (by creating a resource like e.g. a Pod or a Deployment)

• The API server will validate that spec (and reject it if it's invalid)

• Then it will store it in etcd

• A controller will "notice" that spec and act upon it

k8s/declarative.md

Reconciling state

• Watch for the spec fields in the YAML files later!

• The spec describes how we want the thing to be

• Kubernetes will reconcile the current state with the spec
  (technically, this is done by a number of controllers)

• When we want to change some resource, we update the spec

• Kubernetes will then converge that resource

19,000 words

They say, "a picture is worth one thousand words."

The following 19 slides show what really happens when we run:

kubectl create deployment web --image=nginx

k8s/deploymentslideshow.md

Kubernetes network model

• TL,DR:

  Our cluster (nodes and pods) is one big flat IP network.

Kubernetes network model: the good

• Everything can reach everything

• No address translation

• No port translation

• No new protocol

• The network implementation can decide how to allocate addresses

• IP addresses don't have to be "portable" from a node to another

  (We can use e.g. a subnet per node and use a simple routed topology)

• The specification is simple enough to allow many various implementations

k8s/kubenet.md

Kubernetes network model: the less good

• Everything can reach everything

  • if you want security, you need to add network policies

  • the network implementation that you use needs to support them

• There are literally dozens of implementations out there

  (15 are listed in the Kubernetes documentation)

• Pods have level 3 (IP) connectivity, but services are level 4 (TCP or UDP)

  (Services map to a single UDP or TCP port; no port ranges or arbitrary IP packets)

• kube-proxy is on the data path when connecting to a pod or container,
  and it's not particularly fast (relies on userland proxying or iptables)

k8s/kubenet.md

Kubernetes network model: in practice

• The nodes that we are using have been set up to use Weave

• We don't endorse Weave in a particular way, it just Works For Us

• Don't worry about the warning about kube-proxy performance

• Unless you:

  • routinely saturate 10G network interfaces
  • count packet rates in millions per second
  • run high-traffic VOIP or gaming platforms
  • do weird things that involve millions of simultaneous connections
    (in which case you're already familiar with kernel tuning)

• If necessary, there are alternatives to kube-proxy; e.g. kube-router

k8s/kubenet.md

Exposing containers

• We can connect to our pods using their IP address

• Then we need to figure out a lot of things:

  • how do we look up the IP address of the pod(s)?

  • how do we connect from outside the cluster?

  • how do we load balance traffic?

  • what if a pod fails?

• Kubernetes has a resource type named Service

• Services address all these questions!

k8s/kubectlexpose.md

Services in a nutshell

• Services give us a stable endpoint to connect to a pod or a group of pods

• An easy way to create a service is to use kubectl expose

• If we have a deployment named my-little-deploy, we can run:

  kubectl expose deployment my-little-deploy --port=80

  ... and this will create a service with the same name (my-little-deploy)

• Services are automatically added to an internal DNS zone

  (in the example above, our code can now connect to http://my-little-deploy/)

k8s/kubectlexpose.md

Advantages of services

• We don't need to look up the IP address of the pod(s)

  (we resolve the IP address of the service using DNS)

• There are multiple service types; some of them allow external traffic

  (e.g. LoadBalancer and NodePort)

• Services provide load balancing

  (for both internal and external traffic)

• Service addresses are independent from pods' addresses

  (when a pod fails, the service seamlessly sends traffic to its replacement)

k8s/kubectlexpose.md

Many kinds and flavors of service

• There are different types of services:

  ClusterIP, NodePort, LoadBalancer, ExternalName

• There are also headless services

• Services can also have optional external IPs

• There is also another resource type called Ingress

  (specifically for HTTP services)

• Wow, that's a lot! Let's start with the basics ...

k8s/kubectlexpose.md

ClusterIP

• It's the default service type

• A virtual IP address is allocated for the service

  (in an internal, private range; e.g. 10.96.0.0/12)

• This IP address is reachable only from within the cluster (nodes and pods)

• Our code can connect to the service using the original port number

• Perfect for internal communication, within the cluster

k8s/kubectlexpose.md

LoadBalancer

• An external load balancer is allocated for the service

  (typically a cloud load balancer, e.g. ELB on AWS, GLB on GCE ...)

• This is available only when the underlying infrastructure provides some kind of "load balancer as a service"

• Each service of that type will typically cost a little bit of money

  (e.g. a few cents per hour on AWS or GCE)

• Ideally, traffic would flow directly from the load balancer to the pods

• In practice, it will often flow through a NodePort first

k8s/kubectlexpose.md

NodePort

• A port number is allocated for the service

  (by default, in the 30000-32767 range)

• That port is made available on all our nodes and anybody can connect to it

  (we can connect to any node on that port to reach the service)

• Our code needs to be changed to connect to that new port number

• Under the hood: kube-proxy sets up a bunch of iptables rules on our nodes

• Sometimes, it's the only available option for external traffic

  (e.g. most clusters deployed with kubeadm or on-premises)

k8s/kubectlexpose.md

Running containers with open ports

• Since ping doesn't have anything to connect to, we'll have to run something else

• We could use the nginx official image, but ...

  ... we wouldn't be able to tell the backends from each other!

• We are going to use jpetazzo/httpenv, a tiny HTTP server written in Go

• jpetazzo/httpenv listens on port 8888

• It serves its environment variables in JSON format

• The environment variables will include HOSTNAME, which will be the pod name

  (and therefore, will be different on each backend)

k8s/kubectlexpose.md

Creating a deployment for our HTTP server

• We will create a deployment with kubectl create deployment

• Then we will scale it with kubectl scale

k8s/kubectlexpose.md

Exposing our deployment

• We'll create a default ClusterIP service

k8s/kubectlexpose.md

Services are layer 4 constructs

• You can assign IP addresses to services, but they are still layer 4

  (i.e. a service is not an IP address; it's an IP address + protocol + port)

• This is caused by the current implementation of kube-proxy

  (it relies on mechanisms that don't support layer 3)

• As a result: you have to indicate the port number for your service

  (with some exceptions, like ExternalName or headless services, covered later)

k8s/kubectlexpose.md

Testing our service

• We will now send a few HTTP requests to our pods

Shipping images with a registry

• Initially, our app was running on a single node

• We could build and run in the same place

• Therefore, we did not need to ship anything

• Now that we want to run on a cluster, things are different

• The easiest way to ship container images is to use a registry

k8s/shippingimages.md

How Docker registries work (a reminder)

• What happens when we execute docker run alpine ?

• If the Engine needs to pull the alpine image, it expands it into library/alpine

• library/alpine is expanded into index.docker.io/library/alpine

• The Engine communicates with index.docker.io to retrieve library/alpine:latest

• To use something else than index.docker.io, we specify it in the image name

• Examples:

  docker pull gcr.io/google-containers/alpine-with-bash:1.0
  docker build -t registry.mycompany.io:5000/myimage:awesome .
  docker push registry.mycompany.io:5000/myimage:awesome

k8s/shippingimages.md

Running DockerCoins on Kubernetes

• Create one deployment for each component

  (hasher, redis, rng, webui, worker)

• Expose deployments that need to accept connections

  (hasher, redis, rng, webui)

• For redis, we can use the official redis image

• For the 4 others, we need to build images and push them to some registry

k8s/shippingimages.md

Building and shipping images

• There are many options!

• Manually:

  • build locally (with docker build or otherwise)

  • push to the registry

• Automatically:

  • build and test locally

  • when ready, commit and push a code repository

  • the code repository notifies an automated build system

  • that system gets the code, builds it, pushes the image to the registry

k8s/shippingimages.md

Which registry do we want to use?

• There are SAAS products like Docker Hub, Quay ...

• Each major cloud provider has an option as well

  (ACR on Azure, ECR on AWS, GCR on Google Cloud...)

• There are also commercial products to run our own registry

  (Docker EE, Quay...)

• And open source options, too!

• When picking a registry, pay attention to its build system

  (when it has one)

k8s/shippingimages.md

Building on the fly

• Some services can build images on the fly from a repository

• Example: ctr.run

There might be a long pause before the first layer is pulled, because the API behind docker pull doesn't allow to stream build logs, and there is no feedback during the build.

It is possible to view the build logs by setting up an account on ctr.run.

Using images from the Docker Hub

• For everyone's convenience, we took care of building DockerCoins images

• We pushed these images to the DockerHub, under the dockercoins user

• These images are tagged with a version number, v0.1

• The full image names are therefore:

  • dockercoins/hasher:v0.1

  • dockercoins/rng:v0.1

  • dockercoins/webui:v0.1

  • dockercoins/worker:v0.1

k8s/buildshiprun-dockerhub.md

Running our application on Kubernetes

• We can now deploy our code (as well as a redis instance)

k8s/ourapponkube.md

Is this working?

• After waiting for the deployment to complete, let's look at the logs!

  (Hint: use kubectl get deploy -w to watch deployment events)

Connecting containers together

• Three deployments need to be reachable by others: hasher, redis, rng

• worker doesn't need to be exposed

• webui will be dealt with later

k8s/ourapponkube.md

Is this working yet?

• The worker has an infinite loop, that retries 10 seconds after an error

Exposing services for external access

• Now we would like to access the Web UI

• We will expose it with a NodePort

  (just like we did for the registry)

k8s/ourapponkube.md

Accessing the web UI

• We can now connect to any node, on the allocated node port, to view the web UI

Deploying with YAML

• So far, we created resources with the following commands:

  • kubectl run

  • kubectl create deployment

  • kubectl expose

• We can also create resources directly with YAML manifests

k8s/yamldeploy.md

kubectl apply vs create

• kubectl create -f whatever.yaml

  • creates resources if they don't exist

  • if resources already exist, don't alter them
    (and display error message)

• kubectl apply -f whatever.yaml

  • creates resources if they don't exist

  • if resources already exist, update them
    (to match the definition provided by the YAML file)

  • stores the manifest as an annotation in the resource

k8s/yamldeploy.md

Creating multiple resources

• The manifest can contain multiple resources separated by ---

 kind: ...
 apiVersion: ...
 metadata: ...
   name: ...
 ...
 ---
 kind: ...
 apiVersion: ...
 metadata: ...
   name: ...
 ...

k8s/yamldeploy.md

Creating multiple resources

• The manifest can also contain a list of resources

 apiVersion: v1
 kind: List
 items:
 - kind: ...
   apiVersion: ...
   ...
 - kind: ...
   apiVersion: ...
   ...

k8s/yamldeploy.md

Deploying dockercoins with YAML

• We provide a YAML manifest with all the resources for Dockercoins

  (Deployments and Services)

• We can use it if we need to deploy or redeploy Dockercoins

(If we deployed Dockercoins earlier, we will see warning messages, because the resources that we created lack the necessary annotation. We can safely ignore them.)

k8s/yamldeploy.md

Deleting resources

• We can also use a YAML file to delete resources

• kubectl delete -f ... will delete all the resources mentioned in a YAML file

  (useful to clean up everything that was created by kubectl apply -f ...)

• The definitions of the resources don't matter

  (just their kind, apiVersion, and name)

k8s/yamldeploy.md

Pruning¹ resources

• We can also tell kubectl to remove old resources

• This is done with kubectl apply -f ... --prune

• It will remove resources that don't exist in the YAML file(s)

• But only if they were created with kubectl apply in the first place

  (technically, if they have an annotation kubectl.kubernetes.io/last-applied-configuration)

¹If English is not your first language: to prune means to remove dead or overgrown branches in a tree, to help it to grow.

k8s/yamldeploy.md

YAML as source of truth

• Imagine the following workflow:

  • do not use kubectl run, kubectl create deployment, kubectl expose ...

  • define everything with YAML

  • kubectl apply -f ... --prune --all that YAML

  • keep that YAML under version control

  • enforce all changes to go through that YAML (e.g. with pull requests)

• Our version control system now has a full history of what we deploy

• Compares to "Infrastructure-as-Code", but for app deployments

k8s/yamldeploy.md

Scaling our demo app

• Our ultimate goal is to get more DockerCoins

  (i.e. increase the number of loops per second shown on the web UI)

• Let's look at the architecture again:

  

• The loop is done in the worker; perhaps we could try adding more workers?

k8s/scalingdockercoins.md

Adding another worker

• All we have to do is scale the worker Deployment

After a few seconds, the graph in the web UI should show up.

k8s/scalingdockercoins.md

Adding more workers

• If 2 workers give us 2x speed, what about 3 workers?

The graph in the web UI should go up again.

(This is looking great! We're gonna be RICH!)

k8s/scalingdockercoins.md

Adding even more workers

• Let's see if 10 workers give us 10x speed!

Why are we stuck at 10 hashes per second?

• If this was high-quality, production code, we would have instrumentation

  (Datadog, Honeycomb, New Relic, statsd, Sumologic, ...)

• It's not!

• Perhaps we could benchmark our web services?

  (with tools like ab, or even simpler, httping)

k8s/scalingdockercoins.md

Benchmarking our web services

• We want to check hasher and rng

• We are going to use httping

• It's just like ping, but using HTTP GET requests

  (it measures how long it takes to perform one GET request)

• It's used like this:

  httping [-c count] http://host:port/path

• Or even simpler:

  httping ip.ad.dr.ess

• We will use httping on the ClusterIP addresses of our services

k8s/scalingdockercoins.md