Installing a Kubernetes Cluster from Scratch

This particular tutorial will look at how to install a kubernetes cluster on digitalOcean droplets running ubuntu 18.04 (bionic). While there are various hosted kubernetes services available so that you don’t need to set up the cluster yourself, I found it to be very useful in understanding k8s. I would not have learned nearly as much, which I turned into some other writeup on kubernetes Kuberenetes howtos.

Ansible will be the primary tool used -

I will demonstrate installing and configuring the cluster with Ansible because it creates a repeatable process that is self-documenting. For some more information about using ansible, check out that writeup: Ansible Basics, Project Structure.

Setup; need to create droplets and get ssh setup

First we need the digitalocean droplets created, and then ssh connections configured so that Ansible can connect and do the cluster building work.

I recommend generating an ssh key, adding it to DO account

This will be secure enough and easier to work with than the password option for digitalocean droplets. You need to generate your own key, then add it to DO account. Once you do this, when you create a droplet you will have the option to add the key to it and access it with that key right from the start.

## Example key generation using the 
ssh-keygen -o -t ed25519 -f ~/.ssh/id_doaccess_ed25519

Add key to account by copy/pasting public key into DO -

Account > Security should be where to find the option to add ssh keys. You paste the public key contents, and then you will have the option to add it to any droplets you create.

Create Ubuntu 18.04 droplets

You will need a digitalocean account. Create two Ubuntu 18.04 droplets. This process may work with other versions of ubuntu, but I have not tested to see.

This cluster assumes two droplets or more -

One will act as the master, the other as a node. Specs depend on what you plan to run, but 2 CPU/ 2GB memory should be enough for the master.

Add the ssh key by checking the option under Authentication -

You should be able to select the SSH keys option and then key you want added if you have uploaded a key to your account.

Set up the ssh connections for the droplets -

Add this to your ansible.cfg; it sets up the ssh command and then points it to a file (ssh_config) that will be used to configure the hosts.

[ssh_connection]
ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s -F ./ssh_config 
pipelining = True

And the add entries to ./ssh_config for your remote servers -

Each server needs an entry like this. You can get the server ip from the DO interface.

Host 133.44.555.66
  User root
  Port 22
  PasswordAuthentication no
  IdentityFile /home/user/.ssh/id_yourkey_ed25519
  IdentitiesOnly yes
  LogLevel FATAL

The General steps for the install:

Run 1-5 on master/nodes

  1. Get some packages that will be used during the install process
  2. Install docker on master/nodes
  3. Add docker configuration, restart docker
  4. Install Kubernetes components (kubelet, kubectl, kubeadm)
  5. Configure kubelet

Run 6-7 on the master only
6. Initialize Cluster using kubeadm
7. Enable kubernetes networking by installing calico

Run 8-10 only on nodes
8. Get the join command and enable autocomplete
9. Configure nodes to deal with running containers
10. Join node to cluster

1. Some prerequisite packages -

Mostly to allow downloading/installing other things.

  - name: Install packages that allow apt to be used over HTTPS
    apt:
      name: "{{ packages }}"
      state: present
      update_cache: yes
    vars:
      packages:
      - apt-transport-https
      - ca-certificates
      - curl
      - gnupg-agent
      - software-properties-common

2. Installing Docker on master/nodes -

All the droplets need some container runtime. I am using docker 18.09 here. I think containerd is an alternative; I used docker since my kubernetes cluster grew from docker projects so I knew how to use it already.

  - name: Add an apt signing key for Docker
    apt_key:
      url: https://download.docker.com/linux/ubuntu/gpg
      state: present

  - name: Add apt repository for stable version
    apt_repository:
      repo: deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable
      state: present

  - name: Install docker and its dependecies
    apt:
      name: "{{ packages }}"
      state: present
      update_cache: yes
    vars:
      packages:
      - docker-ce=5:18.09*
    register: dockerinstall

3. Add docker config, restart docker -

This will copy a config over, restart docker, and do some swap disabling that is required for virtualization to not complain/fail.

  - name: Deploy Docker daemon.json
    copy:
      src: dockerdaemon.json
      dest: /etc/docker/daemon.json
    register: dockerdaemon

  - name: Restart docker with new settings
    when: dockerinstall.changed or dockerdaemon.changed
    service:
      daemon_reload: yes
      name: docker
      state: restarted
      enabled: true

  - name: Remove swapfile from /etc/fstab
    mount:
      name: "{{ item }}"
      fstype: swap
      state: absent
    with_items:
      - swap
      - none

  - name: Disable swap
    command: swapoff -a

The config is simple, mainly just need the native.cgroup driver opt -

The other options like the log driver and opts are to my preference, but setting the native.cgroupdriver is needed to avoid complaints regarding the docker install.

  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "5m"
  },
  "storage-driver": "overlay2"
}

4. Install Kubernetes components -

Kubelet and kubectl should both get installed to all droplets. Kubeadm seems only to be needed on the master, but I have been adding it to all of them in case there is some use having it there I am not aware of.

  - name: Add an apt signing key for Kubernetes
    apt_key:
      url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
      state: present

  - name: Adding apt repository for Kubernetes
    apt_repository:
      repo: deb https://apt.kubernetes.io/ kubernetes-xenial main
      state: present
      filename: kubernetes.list

  - name: Install Kubernetes binaries
    apt:
      name: "{{ packages }}"
      state: present
      update_cache: yes
      force: yes
    vars:
      packages:
        - kubelet=1.15*
        - kubeadm=1.15*
        - kubectl=1.15*

5. Configure Kubelet -

You may want a different kubelet config, but this is how I have it set up. The feature gates I add to /etc/default/kubelet are specifically to support the DO csi driver. I have attempted to configure these feature gates with the kubeadm config, but it does not seem to work, so I do it here and restart kubelet to make sure the config takes affect.

  - name: check if kubelet file exists
    stat:
      path: /etc/default/kubelet
    register: kubelet_default

  - name: add kubelet etc default file
    file:
      path: /etc/default/kubelet
      mode: '0644'
      state: touch
    when: kubelet_default.stat.exists == false

  - name: Configure /etc/default/kubelet with needed feature gates
    lineinfile:
      path: /etc/default/kubelet
      line: KUBELET_EXTRA_ARGS=--feature-gates="VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true,CSINodeInfo=true,CSIDriverRegistry=true"
      state: present
    register: kubeletconfig

  - name: Restart kubelet
    service:
      name: kubelet
      daemon_reload: yes
      state: restarted

6. Initialize the cluster using kubeadm -

This will use kubeadm to do the actual cluster setup.

  - name: Initialize the Kubernetes cluster using kubeadm and a --config file
    shell: |
      kubeadm init --config "{{dl_dir}}/kubeadm_config.yml"
    register: init_cluster

  - name: Setup kubeconfig for root, since that is the main user
    when: init_cluster is succeeded
    command: "{{ item }}"
    with_items:
     - mkdir -p $HOME/.kube
     - cp -f /etc/kubernetes/admin.conf $HOME/.kube/config

  - name: Grab kubeconfig for use with my local machine/user
    fetch:
      src: /etc/kubernetes/admin.conf
      dest: /home/jad/.kube/config
      flat: yes

This is the config file -

The network config is the important part, I am just setting them to match what calico wants. I try to set featuregates but it does not appear to work.

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
apiServer:
  extraArgs:
    feature-gates: "VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true,CSINodeInfo=true,CSIDriverRegistry=true"
    allow-privileged: "true"

networking:
  dnsDomain: cluster.local
  serviceSubnet: "10.96.0.0/12"
  podSubnet: "192.168.0.0/16"

---

# this appears not to work, so I set an etc/default/kubelet file too
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
FeatureGates: 
  VolumeSnapshotDataSource: true
  KubeletPluginsWatcher: true
  CSINodeInfo: true
  CSIDriverRegistry: true

---

7. Enable Kubernetes networking by installing calico

Although the calico manifest will add many objects, it should all work without intervention as long as you set up the feature gates and cluster correctly to this point. The calico install failing is a good indication that something somewhere is not right.

# the default pod IP pool for calico is 192.168.0.0/16
  - name: download calico object file
    get_url:
      url: https://docs.projectcalico.org/v3.8/manifests/calico.yaml
      dest: "{{ dl_dir }}/calico.yaml"

  - name: Install calico pod network
    command: kubectl create -f "{{ dl_dir }}/calico.yaml"

8. Get the join command and enable autocomplete -

The join command is going to be needed, autocomplete is something I add because typing out full kubernetes commands gets old fast.

  - name: Generate join command
    command: kubeadm token create --print-join-command
    register: join_command

  - name: Copy join command to local file
    local_action: copy content="{{ join_command.stdout_lines[0] }}" dest="./join-command"

  - name: Add autocomplete for kubectl command for root user.
    lineinfile:
      state: present
      line: source <(kubectl completion bash)
      path: /root/.bashrc

9. Configure nodes, mostly for running containers -

Disabling transparent huge pages and setting the vm.max_map_count end up being needed for virtualization to work, so I handle that here.

  - name: Disable Transparent Huge pages
    shell: |
         echo never > /sys/kernel/mm/transparent_hugepage/enabled
         echo never > /sys/kernel/mm/transparent_hugepage/defrag

  ## create the pvc to be used by elasticsearch
  - name: Add a service to disable transparent huge pages
    copy:
      src: disable-thp.service.yml
      force: yes
      dest: "/etc/systemd/system/disable-thp.service.yml"

  - name: Add a service to disable transparent huge pages
    copy:
      src: 10-nettcp.conf
      force: yes
      dest: "/etc/sysctl.d/10-nettcp.conf"

  - name: Disable Transparent Huge pages
    shell: |
        sysctl --load=/etc/sysctl.d/10-nodekube.conf

  - name: force systemd to reread configs after disable thp added
    systemd:
      daemon_reload: yes

  - name: alter vm.max_map_count in case elasticsearch ends up on node
    shell: "sysctl -w vm.max_map_count=262144"
    tags: cloud

  - name: Configure sysctl so max_map_count change persists after reboot
    lineinfile:
      path: /etc/sysctl.d/10-nodekube.conf
      line: vm.max_map_count=262144

10. Join nodes to the cluster

The Join command was copied to our local machine, so copy it over to the node and then run the join command, and the node should be successfully added to the cluster.

  - name: Copy the join command to server location
    copy: src=join-command dest=/tmp/join-command.sh mode=0777
    tags: cloud

  - name: Join the node to cluster
    when: reset_cluster is succeeded
    command: sh /tmp/join-command.sh
    tags: cloud

To see if it worked, you can run:

kubectl get nodes