typhoon/docs/advanced/customization.md

Customization

Typhoon provides Kubernetes clusters with defaults recommended for production. Terraform variables expose supported customization options. Advanced options are available for customizing the architecture or hosts as well.

Variables

Typhoon modules accept Terraform input variables for customizing clusters in meritorious ways (e.g. worker_count, etc). Variables are carefully considered to provide essentials, while limiting complexity and test matrix burden. See each platform's tutorial for options.

Addons

Clusters are kept to a minimal Kubernetes control plane by offering components like Nginx Ingress Controller, Prometheus, and Grafana as optional post-install addons. Customize addons by modifying a copy of our addon manifests.

Hosts

Background

Typhoon uses the Ignition system of Fedora CoreOS and Flatcar Linux to immutably declare a system via first-boot disk provisioning. Human-friendly Butane Configs define disk partitions, filesystems, systemd units, dropins, config files, mount units, raid arrays, users, and more before being converted to Ignition.

Controller and worker instances form a minimal and secure Kubernetes cluster on each platform. Typhoon provides the snippets feature to accept custom Butane Configs that are merged with instance declarations. This allows advanced host customization and experimentation.

!!! note Snippets cannot be used to modify an already existing instance, the antithesis of immutable provisioning. Ignition fully declares a system on first boot only.

!!! danger Snippets provide the powerful host customization abilities of Ignition. You are responsible for additional units, configs, files, and conflicts.

!!! danger Edits to snippets for controller instances can (correctly) cause Terraform to observe a diff (if not otherwise suppressed) and propose destroying and recreating controller(s). Recognize that this is destructive since controllers run etcd and are stateful. See blue/green clusters.

Usage

Define a Butane Config (docs, config) in version control near your Terraform workspace directory (e.g. perhaps in a snippets subdirectory). You may organize snippets into multiple files, if desired.

For example, ensure an /opt/hello file is created with permissions 0644 before boot.

=== "Fedora CoreOS"

```yaml
# custom-files.yaml
variant: fcos
version: 1.5.0
storage:
  files:
    - path: /opt/hello
      contents:
        inline: |
          Hello World
      mode: 0644
```

=== "Flatcar Linux"

```yaml
# custom-files.yaml
variant: flatcar
version: 1.0.0
storage:
  files:
    - path: /opt/hello
      contents:
        inline: |
          Hello World
      mode: 0644
```

Or ensure a systemd unit hello.service is created.

=== "Fedora CoreOS"

```yaml
# custom-units.yaml
variant: fcos
version: 1.5.0
systemd:
  units:
    - name: hello.service
      enabled: true
      contents: |
        [Unit]
        Description=Hello World
        [Service]
        Type=oneshot
        ExecStart=/usr/bin/echo Hello World!
        [Install]
        WantedBy=multi-user.target
```

=== "Flatcar Linux"

```yaml
# custom-units.yaml
variant: flatcar
version: 1.0.0
systemd:
  units:
    - name: hello.service
      enabled: true
      contents: |
        [Unit]
        Description=Hello World
        [Service]
        Type=oneshot
        ExecStart=/usr/bin/echo Hello World!
        [Install]
        WantedBy=multi-user.target
```

Reference the Butane contents by location (e.g. file("./custom-units.yaml")). On AWS, Azure, DigitalOcean, or Google Cloud extend the controller_snippets or worker_snippets list variables.

module "nemo" {
  ...
  worker_count            = 2
  controller_snippets = [
    file("./custom-files.yaml"),
    file("./custom-units.yaml"),
  ]
  worker_snippets = [
    file("./custom-files.yaml"),
    file("./custom-units.yaml")",
  ]
  ...
}

On Bare-Metal, different Butane configs may be used for each node (since hardware may be heterogeneous). Extend the snippets map variable by mapping a controller or worker name key to a list of snippets.

module "mercury" {
  ...
  snippets = {
    "node2" = [file("./units/hello.yaml")]
    "node3" = [
      file("./units/world.yaml"),
      file("./units/hello.yaml"),
    ]
  }
  ...
}

Architecture

Typhoon chooses variables to expose with purpose. If you must customize clusters in ways that aren't supported by input variables, fork Typhoon and maintain a repository with customizations. Reference the repository by changing the username.

module "nemo" {
  source = "git::https://github.com/USERNAME/typhoon//digital-ocean/flatcar-linux/kubernetes?ref=myspecialcase"
  ...
}

To customize low-level Kubernetes control plane bootstrapping, see the poseidon/terraform-render-bootstrap Terraform module.

System Images

Typhoon publishes Kubelet container images to Quay.io (default) and to Dockerhub (in case of a Quay outage or breach). Quay automated builds also provide the option for fully verifiable tagged images (build-{short_sha}).

To set an alternative etcd image or Kubelet image, use a snippet to set a systemd dropin.

=== "Kubelet"

```yaml
# kubelet-image-override.yaml
variant: fcos           <- remove for Flatcar Linux
version: 1.5.0          <- remove for Flatcar Linux
systemd:
  units:
    - name: kubelet.service
      dropins:
        - name: 10-image-override.conf
          contents: |
            [Service]
            Environment=KUBELET_IMAGE=docker.io/psdn/kubelet:v1.18.3
```

=== "etcd"

```yaml
# etcd-image-override.yaml
variant: fcos           <- remove for Flatcar Linux
version: 1.5.0          <- remove for Flatcar Linux
systemd:
  units:
    - name: etcd-member.service
      dropins:
        - name: 10-image-override.conf
          contents: |
            [Service]
            Environment=ETCD_IMAGE=quay.io/mymirror/etcd:v3.4.12
```

Then reference the snippet in the cluster or worker pool definition.

module "nemo" {
  ...

  worker_snippets = [
    file("./snippets/kubelet-image-override.yaml")
  ]
  ...
}