Kubernetes TLS Secret¶

The Agent can put each issued and renewed certificate into a Kubernetes cluster as a kubernetes.io/tls Secret. An Ingress or Gateway that names the Secret then serves the new certificate after every renewal. Nothing is installed in the cluster - no controller, custom resource or in-cluster component: the Agent runs outside the cluster, on a Windows or Linux machine, and talks to the cluster's API server with a kubeconfig file on that machine.

The private key is generated on the Agent and goes from the Agent straight to the cluster's API server, over TLS checked against the CA in the kubeconfig. It never goes to CertIntel. The kubeconfig and the token or client key in it stay on the Agent machine.

How it works¶

The bundled template kubernetes-tls-secret (shown as Kubernetes TLS Secret, in the Kubernetes group) has two steps:

1. Obtain certificate from the certificate's ACME account.
2. Write Kubernetes TLS Secret: one server-side apply request (field manager certintel-agent) creates the Secret <namespace>/<name> or replaces what the Agent manages in it - the type, tls.crt (the certificate and its chain), tls.key and the labels and annotations the workflow sets. Fields other tools set are kept; labels and annotations the Agent set before and no longer sets are removed. The Agent then reads the Secret back and checks that the first certificate in tls.crt is the one it issued (by SHA-256). A failed step is retried once after 30 seconds.

The job log names the step from its first line (Workflow step deploy — write Kubernetes TLS Secret started, then … completed: Secret web/shop-tls in the cluster at k8s.example.com; verified), and the renewal report shown in CertIntel says Certificate delivered to Kubernetes TLS Secret web/shop-tls in the cluster at k8s.example.com. The cluster is named by the host of the kubeconfig's server.

The cluster needs Kubernetes 1.22 or later (server-side apply). Ingress controllers that watch Secrets, such as Traefik and ingress-nginx, serve the new certificate on their own. A controller that caches certificates needs its own reload; the Agent does not do that.

Prepare the cluster¶

Give the Agent its own account with only the rights it needs: get, create and patch on Secrets in the Secret's namespace. For example, for namespace web:

kubectl -n web create serviceaccount certintel-agent
kubectl -n web create role certintel-agent-secrets --verb=get,create,patch --resource=secrets
kubectl -n web create rolebinding certintel-agent-secrets \
  --role=certintel-agent-secrets --serviceaccount=web:certintel-agent
kubectl -n web apply -f - <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: certintel-agent-token
  annotations:
    kubernetes.io/service-account.name: certintel-agent
type: kubernetes.io/service-account-token
EOF
kubectl -n web get secret certintel-agent-token -o jsonpath='{.data.token}' | base64 -d

Write a kubeconfig with that token and the cluster's CA:

apiVersion: v1
kind: Config
clusters:
- name: prod
  cluster:
    server: https://k8s.example.com:6443
    certificate-authority-data: <the cluster CA, base64>
users:
- name: certintel-agent
  user:
    token: <the ServiceAccount token>
contexts:
- name: prod-certintel
  context:
    cluster: prod
    user: certintel-agent
current-context: prod-certintel

Without a ServiceAccount of your own¶

If you may not create a ServiceAccount or a Role (for example you operate only the namespace), use a kubeconfig you already have for that namespace, as long as its credential may get, create and patch Secrets there and is a token or a client certificate (not an exec plugin, see below). Write its current context as one file with the credential embedded, wherever kubectl uses that kubeconfig:

(umask 077; kubectl config view --raw --minify --flatten > cluster.kubeconfig)

Then put the file in the Agent's folder as described below. Check its rights first with kubectl -n web auth can-i create secrets (and get, patch).

What the kubeconfig may use¶

• Cluster: an https:// server; its CA from certificate-authority-data or a certificate-authority file (without one, the machine's trusted roots); tls-server-name. insecure-skip-tls-verify and proxy-url are refused: the key goes only to a verified API server, directly. The Agent does not use a proxy from the environment or follow redirects, and needs TLS 1.2 or later.
• Credential: token or tokenFile, or a client certificate (client-certificate / client-certificate-data with client-key / client-key-data). exec plugins, auth-provider, username/password and impersonation (as) are refused. Kubeconfigs from EKS, GKE or AKS with Entra ID sign in with an exec plugin: use a ServiceAccount token kubeconfig as above instead.
• Context: the workflow's Kubeconfig context, else the file's current-context, else its only context.
• Files: paths in the kubeconfig are relative to its own folder, as with kubectl. Each file must be a regular file of at most 1 MiB; on Linux, one that every user may change is refused.

Put the kubeconfig on the Agent machine¶

A workflow created by a dashboard assignment reads its kubeconfig only from the Agent's own folder, and not through a link, so a proposal cannot name another kubeconfig the service could read:

• Linux: /etc/certintel/kubernetes. The service runs as certintel-agent and cannot see home folders, so ~/.kube/config does not work there either:

sudo install -d -m 0750 -o root -g certintel-agent /etc/certintel/kubernetes
sudo install -m 0640 -o root -g certintel-agent cluster.kubeconfig /etc/certintel/kubernetes/cluster.kubeconfig

• Windows: %ProgramData%\CertIntelAgent\kubernetes (usually C:\ProgramData\CertIntelAgent\kubernetes), inside the Agent's own folder, which only SYSTEM and Administrators can read. In an elevated PowerShell:

New-Item -ItemType Directory -Force "$env:ProgramData\CertIntelAgent\kubernetes"
Copy-Item .\cluster.kubeconfig "$env:ProgramData\CertIntelAgent\kubernetes\cluster.kubeconfig"

A workflow created on the machine itself may name any absolute path the service can read; the Linux folder above is still the place to use.

Fields¶

Create the workflow¶

• From the dashboard, guided: Get a certificate for a server on the dashboard's Overview. Choose Kubernetes TLS Secret under What it runs: Kubeconfig file, Namespace and Secret name appear at once. The kubeconfig field says the folder rule (below) with an example path, /etc/certintel/kubernetes/cluster.kubeconfig. The DNS step says that the names must point at the cluster's Ingress controller (the address of its load balancer, or of the nodes it listens on), not at the Agent's server, which only writes the Secret. Before the proposal, the page says what the cluster must already have - the namespace and, to serve the names, an Ingress there whose spec.tls[].secretName is the Secret - with kubectl -n NAMESPACE get ingress to look, because the Agent does not report what is in the cluster. It also gives the commands that put the kubeconfig where the Agent reads it, to run on the Agent machine where kubectl uses the cluster and the account the Agent should use:

  sudo install -d -m 0750 -o root -g certintel-agent /etc/certintel/kubernetes
  kubectl config view --raw --minify --flatten | sudo tee /etc/certintel/kubernetes/cluster.kubeconfig >/dev/null
  sudo chown root:certintel-agent /etc/certintel/kubernetes/cluster.kubeconfig && sudo chmod 0640 /etc/certintel/kubernetes/cluster.kubeconfig
  

  These copy the kubeconfig kubectl uses now, usually your own with all of its rights, so under them (and under the Windows line) the page recommends instead a dedicated account for the Agent that may only get, create and patch Secrets in the Secret's namespace, with a link to Prepare the cluster. It says there is nothing to configure or reload on the server (on a Windows Agent: copy the kubeconfig into C:\ProgramData\CertIntelAgent\kubernetes). After the proposal the page has no approval step for a server that applies proposals by itself (the default of a new installation); otherwise Approve the proposal on the server comes first. Then The agent gets the certificate and Use it in the cluster; there is no separate confirmation step, because the Agent writes the Secret and reads it back. Once the certificate is issued, the step Use it in the cluster names the Secret, for example web/shop-tls, with the command that prints the certificate in it, its subject and dates (run it where kubectl uses that cluster, with base64 and openssl):

  kubectl -n web get secret shop-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -dates
  

  It also says that renewals replace the Secret and the Ingress controller picks them up. - From the dashboard, by hand: on Workflow Assignments, New workflow, tick the Agent, then choose the template Kubernetes TLS Secret. The dashboard checks the values with the same rules as the Agent and refuses a bad namespace, Secret name, label or annotation list, or a kubeconfig outside the Agent's folder, with HTTP 422 naming the setting by its label, for example Setting 'Kubeconfig file' is required by Kubernetes TLS Secret: fill it in. The device applies the proposal by itself, or a local administrator approves it there, as for every dashboard-assigned workflow. - Windows desktop app: the Kubernetes group of the template list. - Linux terminal interface: T on the Workflows screen. - Command line (elevated):

sudo certintel workflows create --template kubernetes-tls-secret \
  --certificate-id shop --domains shop.example.com --account ACCOUNT-ID \
  --set kubeconfig=/etc/certintel/kubernetes/cluster.kubeconfig \
  --set namespace=web --set secret_name=shop-tls --issue

A dry run (workflow.preflight, certintel workflows preflight ID) loads and checks the kubeconfig without contacting the cluster. The certificate the Secret receives exists only in a real run, so the dry run passes and lists it as not checked: deploy not checked until the workflow runs: certificate (${steps.issue.certificate}).

When it fails¶

The step fails with the cluster's reason, and the failed step output reaches CertIntel without the token or client key. Until the Secret is written, Get a certificate for a server says Issued; the Secret is not written yet: writing the certificate into the Secret, or reading it back, failed, with the Agent's message. Fix what it names; the Agent tries again by itself, every 30 seconds for about 15 minutes for the certificate's first installation, then less often, and the page says Done once the Agent reports that it wrote the Secret.

Security¶

• Anyone who can read Secrets in that namespace can read the private key, as for any TLS Secret. Keep the Agent's account to get, create and patch on Secrets in one namespace.
• Anyone with root or administrator rights on the Agent machine can use the kubeconfig. The token and client key are kept out of logs, reports and the failed step output; the Agent clears them from its own buffers after use as far as the runtime allows.

Related pages¶

• Dashboard-assigned workflows
• Appliance deployment (remote) - the same deployment step for appliances
• Linux Agent