Installing a Kubernetes Cluster from Scratch
This particular tutorial will look at how to install a kubernetes cluster on digitalOcean droplets running ubuntu 18.04 (bionic). While there are various hosted kubernetes services available so that you don’t need to set up the cluster yourself, I found it to be very useful in understanding k8s. I would not have learned nearly as much, which I turned into some other writeup on kubernetes Kuberenetes howtos.
Ansible will be the primary tool used -
I will demonstrate installing and configuring the cluster with Ansible because it creates a repeatable process that is self-documenting. For some more information about using ansible, check out that writeup: Ansible Basics, Project Structure.
Setup; need to create droplets and get ssh setup
First we need the digitalocean droplets created, and then ssh connections configured so that Ansible can connect and do the cluster building work.
I recommend generating an ssh key, adding it to DO account
This will be secure enough and easier to work with than the password option for digitalocean droplets. You need to generate your own key, then add it to DO account. Once you do this, when you create a droplet you will have the option to add the key to it and access it with that key right from the start.
## Example key generation using the
ssh-keygen -o -t ed25519 -f ~/.ssh/id_doaccess_ed25519
Add key to account by copy/pasting public key into DO -
Account > Security should be where to find the option to add ssh keys. You paste the public key contents, and then you will have the option to add it to any droplets you create.
Create Ubuntu 18.04 droplets
You will need a digitalocean account. Create two Ubuntu 18.04 droplets. This process may work with other versions of ubuntu, but I have not tested to see.
This cluster assumes two droplets or more -
One will act as the master, the other as a node. Specs depend on what you plan to run, but 2 CPU/ 2GB memory should be enough for the master.
Add the ssh key by checking the option under Authentication -
You should be able to select the SSH keys option and then key you want added if you have uploaded a key to your account.
Set up the ssh connections for the droplets -
Add this to your ansible.cfg; it sets up the ssh command and then points it to a file (ssh_config) that will be used to configure the hosts.
[ssh_connection]
ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s -F ./ssh_config
pipelining = True
And the add entries to ./ssh_config for your remote servers -
Each server needs an entry like this. You can get the server ip from the DO interface.
Host 133.44.555.66
User root
Port 22
PasswordAuthentication no
IdentityFile /home/user/.ssh/id_yourkey_ed25519
IdentitiesOnly yes
LogLevel FATAL
The General steps for the install:
Run 1-5 on master/nodes
- Get some packages that will be used during the install process
- Install docker on master/nodes
- Add docker configuration, restart docker
- Install Kubernetes components (kubelet, kubectl, kubeadm)
- Configure kubelet
Run 6-7 on the master only
6. Initialize Cluster using kubeadm
7. Enable kubernetes networking by installing calico
Run 8-10 only on nodes
8. Get the join command and enable autocomplete
9. Configure nodes to deal with running containers
10. Join node to cluster
1. Some prerequisite packages -
Mostly to allow downloading/installing other things.
- name: Install packages that allow apt to be used over HTTPS
apt:
name: "{{ packages }}"
state: present
update_cache: yes
vars:
packages:
- apt-transport-https
- ca-certificates
- curl
- gnupg-agent
- software-properties-common
2. Installing Docker on master/nodes -
All the droplets need some container runtime. I am using docker 18.09 here. I think containerd is an alternative; I used docker since my kubernetes cluster grew from docker projects so I knew how to use it already.
- name: Add an apt signing key for Docker
apt_key:
url: https://download.docker.com/linux/ubuntu/gpg
state: present
- name: Add apt repository for stable version
apt_repository:
repo: deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable
state: present
- name: Install docker and its dependecies
apt:
name: "{{ packages }}"
state: present
update_cache: yes
vars:
packages:
- docker-ce=5:18.09*
register: dockerinstall
3. Add docker config, restart docker -
This will copy a config over, restart docker, and do some swap disabling that is required for virtualization to not complain/fail.
- name: Deploy Docker daemon.json
copy:
src: dockerdaemon.json
dest: /etc/docker/daemon.json
register: dockerdaemon
- name: Restart docker with new settings
when: dockerinstall.changed or dockerdaemon.changed
service:
daemon_reload: yes
name: docker
state: restarted
enabled: true
- name: Remove swapfile from /etc/fstab
mount:
name: "{{ item }}"
fstype: swap
state: absent
with_items:
- swap
- none
- name: Disable swap
command: swapoff -a
The config is simple, mainly just need the native.cgroup driver opt -
The other options like the log driver and opts are to my preference, but setting the native.cgroupdriver is needed to avoid complaints regarding the docker install.
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "5m"
},
"storage-driver": "overlay2"
}
4. Install Kubernetes components -
Kubelet and kubectl should both get installed to all droplets. Kubeadm seems only to be needed on the master, but I have been adding it to all of them in case there is some use having it there I am not aware of.
- name: Add an apt signing key for Kubernetes
apt_key:
url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
state: present
- name: Adding apt repository for Kubernetes
apt_repository:
repo: deb https://apt.kubernetes.io/ kubernetes-xenial main
state: present
filename: kubernetes.list
- name: Install Kubernetes binaries
apt:
name: "{{ packages }}"
state: present
update_cache: yes
force: yes
vars:
packages:
- kubelet=1.15*
- kubeadm=1.15*
- kubectl=1.15*
5. Configure Kubelet -
You may want a different kubelet config, but this is how I have it set up. The feature gates I add to /etc/default/kubelet are specifically to support the DO csi driver. I have attempted to configure these feature gates with the kubeadm config, but it does not seem to work, so I do it here and restart kubelet to make sure the config takes affect.
- name: check if kubelet file exists
stat:
path: /etc/default/kubelet
register: kubelet_default
- name: add kubelet etc default file
file:
path: /etc/default/kubelet
mode: '0644'
state: touch
when: kubelet_default.stat.exists == false
- name: Configure /etc/default/kubelet with needed feature gates
lineinfile:
path: /etc/default/kubelet
line: KUBELET_EXTRA_ARGS=--feature-gates="VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true,CSINodeInfo=true,CSIDriverRegistry=true"
state: present
register: kubeletconfig
- name: Restart kubelet
service:
name: kubelet
daemon_reload: yes
state: restarted
6. Initialize the cluster using kubeadm -
This will use kubeadm to do the actual cluster setup.
- name: Initialize the Kubernetes cluster using kubeadm and a --config file
shell: |
kubeadm init --config "{{dl_dir}}/kubeadm_config.yml"
register: init_cluster
- name: Setup kubeconfig for root, since that is the main user
when: init_cluster is succeeded
command: "{{ item }}"
with_items:
- mkdir -p $HOME/.kube
- cp -f /etc/kubernetes/admin.conf $HOME/.kube/config
- name: Grab kubeconfig for use with my local machine/user
fetch:
src: /etc/kubernetes/admin.conf
dest: /home/jad/.kube/config
flat: yes
This is the config file -
The network config is the important part, I am just setting them to match what calico wants. I try to set featuregates but it does not appear to work.
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
apiServer:
extraArgs:
feature-gates: "VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true,CSINodeInfo=true,CSIDriverRegistry=true"
allow-privileged: "true"
networking:
dnsDomain: cluster.local
serviceSubnet: "10.96.0.0/12"
podSubnet: "192.168.0.0/16"
---
# this appears not to work, so I set an etc/default/kubelet file too
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
FeatureGates:
VolumeSnapshotDataSource: true
KubeletPluginsWatcher: true
CSINodeInfo: true
CSIDriverRegistry: true
---
7. Enable Kubernetes networking by installing calico
Although the calico manifest will add many objects, it should all work without intervention as long as you set up the feature gates and cluster correctly to this point. The calico install failing is a good indication that something somewhere is not right.
# the default pod IP pool for calico is 192.168.0.0/16
- name: download calico object file
get_url:
url: https://docs.projectcalico.org/v3.8/manifests/calico.yaml
dest: "{{ dl_dir }}/calico.yaml"
- name: Install calico pod network
command: kubectl create -f "{{ dl_dir }}/calico.yaml"
8. Get the join command and enable autocomplete -
The join command is going to be needed, autocomplete is something I add because typing out full kubernetes commands gets old fast.
- name: Generate join command
command: kubeadm token create --print-join-command
register: join_command
- name: Copy join command to local file
local_action: copy content="{{ join_command.stdout_lines[0] }}" dest="./join-command"
- name: Add autocomplete for kubectl command for root user.
lineinfile:
state: present
line: source <(kubectl completion bash)
path: /root/.bashrc
9. Configure nodes, mostly for running containers -
Disabling transparent huge pages and setting the vm.max_map_count end up being needed for virtualization to work, so I handle that here.
- name: Disable Transparent Huge pages
shell: |
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
## create the pvc to be used by elasticsearch
- name: Add a service to disable transparent huge pages
copy:
src: disable-thp.service.yml
force: yes
dest: "/etc/systemd/system/disable-thp.service.yml"
- name: Add a service to disable transparent huge pages
copy:
src: 10-nettcp.conf
force: yes
dest: "/etc/sysctl.d/10-nettcp.conf"
- name: Disable Transparent Huge pages
shell: |
sysctl --load=/etc/sysctl.d/10-nodekube.conf
- name: force systemd to reread configs after disable thp added
systemd:
daemon_reload: yes
- name: alter vm.max_map_count in case elasticsearch ends up on node
shell: "sysctl -w vm.max_map_count=262144"
tags: cloud
- name: Configure sysctl so max_map_count change persists after reboot
lineinfile:
path: /etc/sysctl.d/10-nodekube.conf
line: vm.max_map_count=262144
10. Join nodes to the cluster
The Join command was copied to our local machine, so copy it over to the node and then run the join command, and the node should be successfully added to the cluster.
- name: Copy the join command to server location
copy: src=join-command dest=/tmp/join-command.sh mode=0777
tags: cloud
- name: Join the node to cluster
when: reset_cluster is succeeded
command: sh /tmp/join-command.sh
tags: cloud
To see if it worked, you can run:
kubectl get nodes