On-Demand Workers
Besides static workers that run all the time, InfraKitchen can start task workers on demand when tasks are waiting and remove them once they are idle.
On-demand workers are defined by worker templates (Admin → Worker Templates):
- Provider – Kubernetes (a pod per worker).
- Configuration – a Pod manifest and optional namespace.
- Max workers – how many workers of this template may run at once.
- Idle timeout – a worker exits after being idle this long.
How a Template Is Chosen
Each task is routed to one template when it is queued:
- The template selected on the resource or executor
- The global default template
- No template: the task runs on a static worker
Other tasks (syncs, tool downloads, workflows, scheduler jobs) use the default template, or static workers when no default is set.
Routing is strict: a template’s workers take only its tasks, and static workers take only tasks with no template. A selected template that is disabled keeps its tasks: they wait until it is enabled again instead of running on workers the resource did not choose. Queued tasks are re-routed whenever the default or a selection changes.
Upgrading: if you run on-demand workers only, set a default template, otherwise tasks without a selection wait for a static worker.
Global default
Set it with the Global default picker on Admin → Worker Templates or Set as default on a template page. Choosing Static workers clears it. The default template cannot be disabled.
Selecting a template
Resources and executors have a Worker Template field; leave it empty to use the default. Templates differ in what their workers can reach (e.g. a ServiceAccount), so choosing one needs access to worker templates; other users always get the default. For resources, changing it needs approval like other fields. A template cannot be deleted while resources or executors select it.
Why a task is waiting
The task panel of a resource or executor shows the template a queued task waits for and why:
| Reason | Meaning |
|---|---|
| A worker is being started | The autoscaler is spawning a worker |
| Workers keep failing | The template is backed off, see its last error |
| Template is disabled | The template was disabled while the task waited |
| All workers are busy | The template is at max workers |
| No static workers are online | The task has no template and no static worker runs |
How It Works
The autoscaler runs in the scheduler. For each enabled template it spawns workers for waiting tasks, up to max workers. Workers exit after the idle timeout and are then deleted. The Instances tab of a template lists its workers.
A worker that fails or does not start within WORKER_START_TIMEOUT_SECONDS is reported in the template’s
last error and kept for WORKER_FAILED_RETENTION_SECONDS so its logs can be read. The template is then backed
off (30 seconds, doubling up to 10 minutes) until one of its workers registers.
Kubernetes Provider
The scheduler and backend must run in the cluster where workers are started; they use their pod’s ServiceAccount.
Grant access to pods
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: infrakitchen-worker-manager
namespace: infrakitchen
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "create", "delete"]
# Worker logs on the template and Workers pages
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
# Optional: lets Validate check that inherited Secrets and ConfigMaps exist in the workers' namespace
- apiGroups: [""]
resources: ["secrets", "configmaps"]
verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: infrakitchen-worker-manager
namespace: infrakitchen
subjects:
- kind: ServiceAccount
name: <SERVICE_ACCOUNT_NAME>
namespace: infrakitchen
roleRef:
kind: Role
name: infrakitchen-worker-manager
apiGroup: rbac.authorization.k8s.ioCreate it in the InfraKitchen namespace and in every namespace templates spawn workers into.
Create a worker template
Open Admin → Worker Templates → Create. The pod manifest is prefilled with a default worker pod. Use Validate on the template page to dry-run it.
The worker container inherits the image (if not set) and the env/envFrom of the InfraKitchen pod, so database
credentials never go into templates. Secrets are copied as references, so they must also exist in other namespaces
workers run in; Validate reports missing ones. Variables set in the template take precedence.
InfraKitchen finds its own pod by the POD_NAME variable, or by its hostname when it is not set. Set it when the
pod uses hostNetwork or a custom hostname:
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
Settings
| Variable | Default | Description |
|---|---|---|
WORKER_AUTOSCALER_INTERVAL_SECONDS |
15 |
How often the autoscaler runs |
WORKER_START_TIMEOUT_SECONDS |
600 |
Delete workers still pending after this long |
WORKER_FAILED_RETENTION_SECONDS |
3600 |
Keep failed workers this long for their logs |
To remove a template, disable it first: its queued tasks move to the default and its workers exit after their current task. It can be deleted once no workers run and nothing selects it.