2017-09-18 06:40:33 +02:00
|
|
|
---
|
|
|
|
systemd:
|
|
|
|
units:
|
|
|
|
- name: docker.service
|
2020-06-10 07:38:32 +02:00
|
|
|
enabled: true
|
2017-09-18 06:40:33 +02:00
|
|
|
- name: locksmithd.service
|
|
|
|
mask: true
|
|
|
|
- name: wait-for-dns.service
|
2020-06-10 07:38:32 +02:00
|
|
|
enabled: true
|
2017-09-18 06:40:33 +02:00
|
|
|
contents: |
|
|
|
|
[Unit]
|
|
|
|
Description=Wait for DNS entries
|
|
|
|
Wants=systemd-resolved.service
|
|
|
|
Before=kubelet.service
|
|
|
|
[Service]
|
|
|
|
Type=oneshot
|
|
|
|
RemainAfterExit=true
|
|
|
|
ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
|
|
|
|
[Install]
|
|
|
|
RequiredBy=kubelet.service
|
|
|
|
- name: kubelet.service
|
2020-06-10 07:38:32 +02:00
|
|
|
enabled: true
|
2017-09-18 06:40:33 +02:00
|
|
|
contents: |
|
|
|
|
[Unit]
|
2020-04-26 01:50:51 +02:00
|
|
|
Description=Kubelet
|
Change Flatcar kubelet.service container from rkt to docker
* Use docker to run the `kubelet.service` container
* Update Kubelet mounts to match Fedora CoreOS
* Remove unused `/etc/ssl/certs` mount (see
https://github.com/poseidon/typhoon/pull/810)
* Remove unused `/usr/share/ca-certificates` mount
* Remove `/etc/resolv.conf` mount, Docker default is ok
* Change `delete-node.service` to use docker instead of rkt
and inline ExecStart, as was done on Fedora CoreOS
* Fix permission denied on shutdown `delete-node`, caused
by the kubeconfig mount changing with the introduction of
node TLS bootstrap
Background
* podmand, rkt, and runc daemonless container process runners
provide advantages over the docker daemon for system containers.
Docker requires workarounds for use in systemd units where the
ExecStart must tail logs so systemd can monitor the daemonized
container. https://github.com/moby/moby/issues/6791
* Why switch then? On Flatcar Linux, podman isn't shipped. rkt
works, but isn't developing while container standards continue
to move forward. Typhoon has used runc for the Kubelet runner
before in Fedora Atomic, but its more low-level. So we're left
with Docker, which is less than ideal, but shipped in Flatcar
* Flatcar Linux appears to be shifting system components to
use docker, which does provide some limited guards against
breakages (e.g. Flatcar cannot enable docker live restore)
2020-10-19 07:51:25 +02:00
|
|
|
Requires=docker.service
|
|
|
|
After=docker.service
|
2017-09-25 03:04:48 +02:00
|
|
|
Wants=rpc-statd.service
|
2017-09-18 06:40:33 +02:00
|
|
|
[Service]
|
2022-04-21 03:57:03 +02:00
|
|
|
Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.6
|
2021-04-02 08:45:19 +02:00
|
|
|
ExecStartPre=/bin/mkdir -p /etc/cni/net.d
|
Inline Container Linux kubelet.service, deprecate kubelet-wrapper
* Change kubelet.service on Container Linux nodes to ExecStart Kubelet
inline to replace the use of the host OS kubelet-wrapper script
* Express rkt run flags and volume mounts in a clear, uniform way to
make the Kubelet service easier to audit, manage, and understand
* Eliminate reliance on a Container Linux kubelet-wrapper script
* Typhoon for Fedora CoreOS developed a kubelet.service that similarly
uses an inline ExecStart (except with podman instead of rkt) and a
more minimal set of volume mounts. Adopt the volume improvements:
* Change Kubelet /etc/kubernetes volume to read-only
* Change Kubelet /etc/resolv.conf volume to read-only
* Remove unneeded /var/lib/cni volume mount
Background:
* kubelet-wrapper was added in CoreOS around the time of Kubernetes v1.0
to simplify running a CoreOS-built hyperkube ACI image via rkt-fly. The
script defaults are no longer ideal (e.g. rkt's notion of trust dates
back to quay.io ACI image serving and signing, which informed the OCI
standard images we use today, though they still lack rkt's signing ideas).
* Shipping kubelet-wrapper was regretted at CoreOS, but remains in the
distro for compatibility. The script is not updated to track hyperkube
changes, but it is stable and kubelet.env overrides bridge most gaps
* Typhoon Container Linux nodes have used kubelet-wrapper to rkt/rkt-fly
run the Kubelet via the official k8s.gcr.io hyperkube image using overrides
(new image registry, new image format, restart handling, new mounts, new
entrypoint in v1.17).
* Observation: Most of what it takes to run a Kubelet container is defined
in Typhoon, not in kubelet-wrapper. The wrapper's value is now undermined
by having to workaround its dated defaults. Typhoon may be better served
defining Kubelet.service explicitly
* Typhoon for Fedora CoreOS developed a kubelet.service without the use
of a host OS kubelet-wrapper which is both clearer and eliminated some
volume mounts
2019-12-29 20:17:26 +01:00
|
|
|
ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
|
2019-09-19 09:15:39 +02:00
|
|
|
ExecStartPre=/bin/mkdir -p /opt/cni/bin
|
2018-04-22 00:13:38 +02:00
|
|
|
ExecStartPre=/bin/mkdir -p /var/lib/calico
|
2018-02-23 07:00:51 +01:00
|
|
|
ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
|
2017-09-18 06:40:33 +02:00
|
|
|
ExecStartPre=/usr/bin/bash -c "grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt"
|
Change Flatcar kubelet.service container from rkt to docker
* Use docker to run the `kubelet.service` container
* Update Kubelet mounts to match Fedora CoreOS
* Remove unused `/etc/ssl/certs` mount (see
https://github.com/poseidon/typhoon/pull/810)
* Remove unused `/usr/share/ca-certificates` mount
* Remove `/etc/resolv.conf` mount, Docker default is ok
* Change `delete-node.service` to use docker instead of rkt
and inline ExecStart, as was done on Fedora CoreOS
* Fix permission denied on shutdown `delete-node`, caused
by the kubeconfig mount changing with the introduction of
node TLS bootstrap
Background
* podmand, rkt, and runc daemonless container process runners
provide advantages over the docker daemon for system containers.
Docker requires workarounds for use in systemd units where the
ExecStart must tail logs so systemd can monitor the daemonized
container. https://github.com/moby/moby/issues/6791
* Why switch then? On Flatcar Linux, podman isn't shipped. rkt
works, but isn't developing while container standards continue
to move forward. Typhoon has used runc for the Kubelet runner
before in Fedora Atomic, but its more low-level. So we're left
with Docker, which is less than ideal, but shipped in Flatcar
* Flatcar Linux appears to be shifting system components to
use docker, which does provide some limited guards against
breakages (e.g. Flatcar cannot enable docker live restore)
2020-10-19 07:51:25 +02:00
|
|
|
# Podman, rkt, or runc run container processes, whereas docker run
|
|
|
|
# is a client to a daemon and requires workarounds to use within a
|
|
|
|
# systemd unit. https://github.com/moby/moby/issues/6791
|
|
|
|
ExecStartPre=/usr/bin/docker run -d \
|
|
|
|
--name kubelet \
|
|
|
|
--privileged \
|
|
|
|
--pid host \
|
|
|
|
--network host \
|
2021-04-02 08:45:19 +02:00
|
|
|
-v /etc/cni/net.d:/etc/cni/net.d:ro \
|
Change Flatcar kubelet.service container from rkt to docker
* Use docker to run the `kubelet.service` container
* Update Kubelet mounts to match Fedora CoreOS
* Remove unused `/etc/ssl/certs` mount (see
https://github.com/poseidon/typhoon/pull/810)
* Remove unused `/usr/share/ca-certificates` mount
* Remove `/etc/resolv.conf` mount, Docker default is ok
* Change `delete-node.service` to use docker instead of rkt
and inline ExecStart, as was done on Fedora CoreOS
* Fix permission denied on shutdown `delete-node`, caused
by the kubeconfig mount changing with the introduction of
node TLS bootstrap
Background
* podmand, rkt, and runc daemonless container process runners
provide advantages over the docker daemon for system containers.
Docker requires workarounds for use in systemd units where the
ExecStart must tail logs so systemd can monitor the daemonized
container. https://github.com/moby/moby/issues/6791
* Why switch then? On Flatcar Linux, podman isn't shipped. rkt
works, but isn't developing while container standards continue
to move forward. Typhoon has used runc for the Kubelet runner
before in Fedora Atomic, but its more low-level. So we're left
with Docker, which is less than ideal, but shipped in Flatcar
* Flatcar Linux appears to be shifting system components to
use docker, which does provide some limited guards against
breakages (e.g. Flatcar cannot enable docker live restore)
2020-10-19 07:51:25 +02:00
|
|
|
-v /etc/kubernetes:/etc/kubernetes:ro \
|
|
|
|
-v /etc/machine-id:/etc/machine-id:ro \
|
|
|
|
-v /usr/lib/os-release:/etc/os-release:ro \
|
|
|
|
-v /lib/modules:/lib/modules:ro \
|
|
|
|
-v /run:/run \
|
2021-11-13 06:03:48 +01:00
|
|
|
-v /sys/fs/cgroup:/sys/fs/cgroup \
|
Change Flatcar kubelet.service container from rkt to docker
* Use docker to run the `kubelet.service` container
* Update Kubelet mounts to match Fedora CoreOS
* Remove unused `/etc/ssl/certs` mount (see
https://github.com/poseidon/typhoon/pull/810)
* Remove unused `/usr/share/ca-certificates` mount
* Remove `/etc/resolv.conf` mount, Docker default is ok
* Change `delete-node.service` to use docker instead of rkt
and inline ExecStart, as was done on Fedora CoreOS
* Fix permission denied on shutdown `delete-node`, caused
by the kubeconfig mount changing with the introduction of
node TLS bootstrap
Background
* podmand, rkt, and runc daemonless container process runners
provide advantages over the docker daemon for system containers.
Docker requires workarounds for use in systemd units where the
ExecStart must tail logs so systemd can monitor the daemonized
container. https://github.com/moby/moby/issues/6791
* Why switch then? On Flatcar Linux, podman isn't shipped. rkt
works, but isn't developing while container standards continue
to move forward. Typhoon has used runc for the Kubelet runner
before in Fedora Atomic, but its more low-level. So we're left
with Docker, which is less than ideal, but shipped in Flatcar
* Flatcar Linux appears to be shifting system components to
use docker, which does provide some limited guards against
breakages (e.g. Flatcar cannot enable docker live restore)
2020-10-19 07:51:25 +02:00
|
|
|
-v /var/lib/calico:/var/lib/calico:ro \
|
2021-12-15 04:37:43 +01:00
|
|
|
-v /var/lib/containerd:/var/lib/containerd \
|
Change Flatcar kubelet.service container from rkt to docker
* Use docker to run the `kubelet.service` container
* Update Kubelet mounts to match Fedora CoreOS
* Remove unused `/etc/ssl/certs` mount (see
https://github.com/poseidon/typhoon/pull/810)
* Remove unused `/usr/share/ca-certificates` mount
* Remove `/etc/resolv.conf` mount, Docker default is ok
* Change `delete-node.service` to use docker instead of rkt
and inline ExecStart, as was done on Fedora CoreOS
* Fix permission denied on shutdown `delete-node`, caused
by the kubeconfig mount changing with the introduction of
node TLS bootstrap
Background
* podmand, rkt, and runc daemonless container process runners
provide advantages over the docker daemon for system containers.
Docker requires workarounds for use in systemd units where the
ExecStart must tail logs so systemd can monitor the daemonized
container. https://github.com/moby/moby/issues/6791
* Why switch then? On Flatcar Linux, podman isn't shipped. rkt
works, but isn't developing while container standards continue
to move forward. Typhoon has used runc for the Kubelet runner
before in Fedora Atomic, but its more low-level. So we're left
with Docker, which is less than ideal, but shipped in Flatcar
* Flatcar Linux appears to be shifting system components to
use docker, which does provide some limited guards against
breakages (e.g. Flatcar cannot enable docker live restore)
2020-10-19 07:51:25 +02:00
|
|
|
-v /var/lib/kubelet:/var/lib/kubelet:rshared \
|
|
|
|
-v /var/log:/var/log \
|
|
|
|
-v /opt/cni/bin:/opt/cni/bin \
|
|
|
|
$${KUBELET_IMAGE} \
|
2017-09-18 06:40:33 +02:00
|
|
|
--anonymous-auth=false \
|
2018-05-14 08:20:42 +02:00
|
|
|
--authentication-token-webhook \
|
|
|
|
--authorization-mode=Webhook \
|
2020-04-26 01:50:51 +02:00
|
|
|
--bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
|
2021-11-13 06:03:48 +01:00
|
|
|
--cgroup-driver=systemd \
|
2021-12-15 04:37:43 +01:00
|
|
|
--container-runtime=remote \
|
|
|
|
--container-runtime-endpoint=unix:///run/containerd/containerd.sock \
|
2017-09-18 06:40:33 +02:00
|
|
|
--client-ca-file=/etc/kubernetes/ca.crt \
|
2019-01-05 22:32:03 +01:00
|
|
|
--cluster_dns=${cluster_dns_service_ip} \
|
2017-12-09 22:36:59 +01:00
|
|
|
--cluster_domain=${cluster_domain_suffix} \
|
2019-12-29 20:20:59 +01:00
|
|
|
--healthz-port=0 \
|
2020-04-26 01:50:51 +02:00
|
|
|
--kubeconfig=/var/lib/kubelet/kubeconfig \
|
2019-09-18 06:24:30 +02:00
|
|
|
--node-labels=node.kubernetes.io/node \
|
2020-03-17 05:21:41 +01:00
|
|
|
%{~ for label in split(",", node_labels) ~}
|
2019-09-28 23:59:24 +02:00
|
|
|
--node-labels=${label} \
|
2020-03-17 05:21:41 +01:00
|
|
|
%{~ endfor ~}
|
2021-04-11 21:08:56 +02:00
|
|
|
%{~ for taint in split(",", node_taints) ~}
|
|
|
|
--register-with-taints=${taint} \
|
|
|
|
%{~ endfor ~}
|
2018-02-23 07:00:51 +01:00
|
|
|
--pod-manifest-path=/etc/kubernetes/manifests \
|
2018-05-14 03:16:10 +02:00
|
|
|
--read-only-port=0 \
|
2021-12-10 17:22:30 +01:00
|
|
|
--resolv-conf=/run/systemd/resolve/resolv.conf \
|
2020-04-26 01:50:51 +02:00
|
|
|
--rotate-certificates \
|
2018-02-23 07:00:51 +01:00
|
|
|
--volume-plugin-dir=/var/lib/kubelet/volumeplugins
|
Change Flatcar kubelet.service container from rkt to docker
* Use docker to run the `kubelet.service` container
* Update Kubelet mounts to match Fedora CoreOS
* Remove unused `/etc/ssl/certs` mount (see
https://github.com/poseidon/typhoon/pull/810)
* Remove unused `/usr/share/ca-certificates` mount
* Remove `/etc/resolv.conf` mount, Docker default is ok
* Change `delete-node.service` to use docker instead of rkt
and inline ExecStart, as was done on Fedora CoreOS
* Fix permission denied on shutdown `delete-node`, caused
by the kubeconfig mount changing with the introduction of
node TLS bootstrap
Background
* podmand, rkt, and runc daemonless container process runners
provide advantages over the docker daemon for system containers.
Docker requires workarounds for use in systemd units where the
ExecStart must tail logs so systemd can monitor the daemonized
container. https://github.com/moby/moby/issues/6791
* Why switch then? On Flatcar Linux, podman isn't shipped. rkt
works, but isn't developing while container standards continue
to move forward. Typhoon has used runc for the Kubelet runner
before in Fedora Atomic, but its more low-level. So we're left
with Docker, which is less than ideal, but shipped in Flatcar
* Flatcar Linux appears to be shifting system components to
use docker, which does provide some limited guards against
breakages (e.g. Flatcar cannot enable docker live restore)
2020-10-19 07:51:25 +02:00
|
|
|
ExecStart=docker logs -f kubelet
|
|
|
|
ExecStop=docker stop kubelet
|
|
|
|
ExecStopPost=docker rm kubelet
|
2017-09-18 06:40:33 +02:00
|
|
|
Restart=always
|
|
|
|
RestartSec=5
|
|
|
|
[Install]
|
|
|
|
WantedBy=multi-user.target
|
|
|
|
- name: delete-node.service
|
Change Flatcar kubelet.service container from rkt to docker
* Use docker to run the `kubelet.service` container
* Update Kubelet mounts to match Fedora CoreOS
* Remove unused `/etc/ssl/certs` mount (see
https://github.com/poseidon/typhoon/pull/810)
* Remove unused `/usr/share/ca-certificates` mount
* Remove `/etc/resolv.conf` mount, Docker default is ok
* Change `delete-node.service` to use docker instead of rkt
and inline ExecStart, as was done on Fedora CoreOS
* Fix permission denied on shutdown `delete-node`, caused
by the kubeconfig mount changing with the introduction of
node TLS bootstrap
Background
* podmand, rkt, and runc daemonless container process runners
provide advantages over the docker daemon for system containers.
Docker requires workarounds for use in systemd units where the
ExecStart must tail logs so systemd can monitor the daemonized
container. https://github.com/moby/moby/issues/6791
* Why switch then? On Flatcar Linux, podman isn't shipped. rkt
works, but isn't developing while container standards continue
to move forward. Typhoon has used runc for the Kubelet runner
before in Fedora Atomic, but its more low-level. So we're left
with Docker, which is less than ideal, but shipped in Flatcar
* Flatcar Linux appears to be shifting system components to
use docker, which does provide some limited guards against
breakages (e.g. Flatcar cannot enable docker live restore)
2020-10-19 07:51:25 +02:00
|
|
|
enabled: true
|
2017-09-18 06:40:33 +02:00
|
|
|
contents: |
|
|
|
|
[Unit]
|
Change Flatcar kubelet.service container from rkt to docker
* Use docker to run the `kubelet.service` container
* Update Kubelet mounts to match Fedora CoreOS
* Remove unused `/etc/ssl/certs` mount (see
https://github.com/poseidon/typhoon/pull/810)
* Remove unused `/usr/share/ca-certificates` mount
* Remove `/etc/resolv.conf` mount, Docker default is ok
* Change `delete-node.service` to use docker instead of rkt
and inline ExecStart, as was done on Fedora CoreOS
* Fix permission denied on shutdown `delete-node`, caused
by the kubeconfig mount changing with the introduction of
node TLS bootstrap
Background
* podmand, rkt, and runc daemonless container process runners
provide advantages over the docker daemon for system containers.
Docker requires workarounds for use in systemd units where the
ExecStart must tail logs so systemd can monitor the daemonized
container. https://github.com/moby/moby/issues/6791
* Why switch then? On Flatcar Linux, podman isn't shipped. rkt
works, but isn't developing while container standards continue
to move forward. Typhoon has used runc for the Kubelet runner
before in Fedora Atomic, but its more low-level. So we're left
with Docker, which is less than ideal, but shipped in Flatcar
* Flatcar Linux appears to be shifting system components to
use docker, which does provide some limited guards against
breakages (e.g. Flatcar cannot enable docker live restore)
2020-10-19 07:51:25 +02:00
|
|
|
Description=Delete Kubernetes node on shutdown
|
2017-09-18 06:40:33 +02:00
|
|
|
[Service]
|
2022-04-21 03:57:03 +02:00
|
|
|
Environment=KUBELET_IMAGE=quay.io/poseidon/kubelet:v1.23.6
|
2017-09-18 06:40:33 +02:00
|
|
|
Type=oneshot
|
|
|
|
RemainAfterExit=true
|
|
|
|
ExecStart=/bin/true
|
Change Flatcar kubelet.service container from rkt to docker
* Use docker to run the `kubelet.service` container
* Update Kubelet mounts to match Fedora CoreOS
* Remove unused `/etc/ssl/certs` mount (see
https://github.com/poseidon/typhoon/pull/810)
* Remove unused `/usr/share/ca-certificates` mount
* Remove `/etc/resolv.conf` mount, Docker default is ok
* Change `delete-node.service` to use docker instead of rkt
and inline ExecStart, as was done on Fedora CoreOS
* Fix permission denied on shutdown `delete-node`, caused
by the kubeconfig mount changing with the introduction of
node TLS bootstrap
Background
* podmand, rkt, and runc daemonless container process runners
provide advantages over the docker daemon for system containers.
Docker requires workarounds for use in systemd units where the
ExecStart must tail logs so systemd can monitor the daemonized
container. https://github.com/moby/moby/issues/6791
* Why switch then? On Flatcar Linux, podman isn't shipped. rkt
works, but isn't developing while container standards continue
to move forward. Typhoon has used runc for the Kubelet runner
before in Fedora Atomic, but its more low-level. So we're left
with Docker, which is less than ideal, but shipped in Flatcar
* Flatcar Linux appears to be shifting system components to
use docker, which does provide some limited guards against
breakages (e.g. Flatcar cannot enable docker live restore)
2020-10-19 07:51:25 +02:00
|
|
|
ExecStop=/bin/bash -c '/usr/bin/docker run -v /var/lib/kubelet:/var/lib/kubelet:ro --entrypoint /usr/local/bin/kubectl $${KUBELET_IMAGE} --kubeconfig=/var/lib/kubelet/kubeconfig delete node $HOSTNAME'
|
2017-09-18 06:40:33 +02:00
|
|
|
[Install]
|
|
|
|
WantedBy=multi-user.target
|
|
|
|
storage:
|
|
|
|
files:
|
|
|
|
- path: /etc/kubernetes/kubeconfig
|
|
|
|
filesystem: root
|
|
|
|
mode: 0644
|
|
|
|
contents:
|
|
|
|
inline: |
|
2018-02-26 21:17:42 +01:00
|
|
|
${kubeconfig}
|
2017-09-18 06:40:33 +02:00
|
|
|
- path: /etc/sysctl.d/max-user-watches.conf
|
|
|
|
filesystem: root
|
2020-06-10 07:38:32 +02:00
|
|
|
mode: 0644
|
2017-09-18 06:40:33 +02:00
|
|
|
contents:
|
|
|
|
inline: |
|
|
|
|
fs.inotify.max_user_watches=16184
|
|
|
|
passwd:
|
|
|
|
users:
|
|
|
|
- name: core
|
|
|
|
ssh_authorized_keys:
|
|
|
|
- "${ssh_authorized_key}"
|