Autoscale CI onto any cloud with a TypeScript workflow

Autoscaling CI usually waits on a driver for your cloud

You want your CI to burst onto cloud compute when the queue backs up, then give it back when it drains. The common autoscalers get there with a driver per cloud. GitLab Runner’s autoscaler talks to each provider through a separate fleeting plugin: AWS, Google Cloud and Azure officially, and others from the community. Buildkite ships an Elastic CI Stack per cloud. Each driver is provider SDK code that somebody has to write, test and keep current as the provider’s API moves.

If your cloud isn’t on the list, you wait for someone to build the driver. If it is, you inherit whatever the driver’s authors decided about instance types, images and lifecycle, and you configure around it. Your image, your network and your budget all sit behind another team’s release cycle.

The event scaler hands provisioning to your workflow

KiCI’s scaler has an event backend that performs no cloud calls of its own. When demand rises, its spawn() emits a reserved kici.scaler.scale-up event; when an instance should go away, destroy() emits kici.scaler.scale-down. Those are ordinary KiCI custom events, and you consume them with the same kiciEvent() trigger you’d use for anything else, in a provisioning workflow you write in TypeScript. The workflow-driven autoscaling guide covers the whole contract.

So “cloud autoscaling” stops being “which drivers ship this quarter” and becomes “write a workflow.” KiCI keeps the hard, generic parts: demand detection, capacity caps, reservations, dispatching the pending job to the new machine, spawn timeouts, idle and heartbeat teardown. It leaves one thing open on purpose: how a machine comes up. That’s the extension point. Plug in any cloud’s API there and it works, with no driver required, so KiCI ships zero cloud SDK code and you’re never limited to a supported list.

The credential handoff is built for this. The scale-up event carries a single-use claim code, not a token. Your workflow forwards that code into the instance’s cloud-init, and the agent exchanges it for its own short-lived token once it boots. The token is minted inside the machine that uses it, and never transits provisioning or the persisted event log at all.

Writing the provisioning workflow

Here’s the reference provisioning workflow. It subscribes to the scale-up event, forwards the single-use claim code into a cloud-init, and boots a labelled instance whose agent claims its own token and registers under the scaler-chosen agentId. This one targets Hetzner; for another cloud, swap the boot call for that cloud’s API and keep the shape.

import {
  workflow,
  job,
  kiciEvent,
  buildAgentCloudInit,
  SCALER_EVENT_NAMES,
  ScalerScaleUpPayload,
} from '@kici-dev/sdk';

const SCALER_NAME = 'hetzner';
/** Hard lifetime cap for a provisioned instance (teardown layer L2). */
const MAX_LIFETIME_MINUTES = 30;

export default workflow('hetzner-autoscale-provision', {
  on: [kiciEvent({ name: SCALER_EVENT_NAMES.scaleUp, match: { '$.scalerName': SCALER_NAME } })],
  jobs: [
    job('provision', {
      runsOn: ['default'],
      // Bind the context whose scope holds the credential this job reads.
      // Without it `ctx.secrets` resolves nothing: the job option is `context`,
      // and an unrecognised key is dropped at compile time rather than
      // rejected, so the step would fail on a missing secret at run time.
      context: 'hetzner-autoscale',
      run: async (ctx) => {
        const payload = ScalerScaleUpPayload.parse(ctx.rawPayload);

        // Forward the single-use claim code into cloud-init; the agent claims
        // its own token in-instance, so the token never transits provisioning.
        const userData = buildAgentCloudInit(
          {
            claimCode: payload.claimCode,
            agentId: payload.agentId,
            orchestratorUrl: payload.orchestratorUrl,
            labels: payload.labels,
          },
          {
            maxLifetimeMinutes: MAX_LIFETIME_MINUTES,
            // Container delivery starts the agent with `docker run`, and the
            // stock `debian-12` image below ships no Docker — so install it.
            // Drop this line when your image already carries Docker.
            deliveryMode: 'container',
            packages: ['docker.io'],
            agentEnv: e2eAgentEnv(ctx.env),
          },
        );

        const token = await ctx.secrets.get('HETZNER_API_TOKEN');
        const client = new HetznerClient(token);
        const { id } = await client.createServer({
          name: `kici-agent-${payload.agentId}`,
          // A current-generation shared-vCPU type available in the region below.
          server_type: 'cpx12',
          image: 'debian-12',
          user_data: userData,
          // Every teardown layer keys off these labels.
          labels: {
            'kici-managed': 'hetzner-autoscale',
            'kici-agent-id': payload.agentId,
            'kici-scaler': SCALER_NAME,
            ...e2eServerLabels(ctx.env),
          },
        });

        ctx.log.info(`Provisioned Hetzner server ${id} for agent ${payload.agentId}`);
      },
    }),
  ],
});

Everything provider-specific lives in the run body: the instance type, the image, the labels, the API call. HetznerClient here is a thin wrapper over Hetzner’s REST API; the copy in the guide makes the same calls with fetch, so it needs nothing installed. The KiCI-specific parts are the kiciEvent trigger, the payload parse, and the buildAgentCloudInit call. There’s no backend to register and no plugin to install. Supporting a new cloud isn’t a feature request; it’s a new workflow you write today.

Where the provisioning workflows live

The provisioning and teardown workflows are ordinary workflows, not global ones. They live in one repo you pick, and the scaler’s provisioningTargets names it. KiCI delivers the scale-up and scale-down events only to workflows registered in that repo, so a kiciEvent subscription somewhere else never boots a server.

A dedicated infra repo is the tidy choice: the provisioning code and its reviews stay in one place, and your application repos don’t change at all. A job that wants a Hetzner machine says runsOn: ['hetzner'], and that’s it. Here’s the whole infra repo:

myorg/infra/
└── .kici/
    ├── package.json            # declares @kici-dev/sdk (kici init writes it)
    ├── workflows/
    │   ├── hetzner-provision.ts
    │   └── hetzner-teardown.ts
    └── kici.lock.json          # written by kici compile

Two things make the repo live. KiCI has to receive its pushes, and the two workflows have to reach its default branch, because event triggers register only from the default branch’s lock file. The setup steps below cover both.

Teardown that can’t leak a machine

The scary part of “boot a cloud instance on demand” is the one you don’t see: the instance that doesn’t get cleaned up when something crashes mid-run, quietly billing you for a week. So teardown doesn’t rely on any single path succeeding.

The scale-down event fires the mirror-image teardown workflow, which deletes the servers for the scaled-down agent by label:

import {
  workflow,
  job,
  kiciEvent,
  SCALER_EVENT_NAMES,
  ScalerScaleDownPayload,
} from '@kici-dev/sdk';

const SCALER_NAME = 'hetzner';

export default workflow('hetzner-autoscale-teardown', {
  on: [kiciEvent({ name: SCALER_EVENT_NAMES.scaleDown, match: { '$.scalerName': SCALER_NAME } })],
  jobs: [
    job('teardown', {
      runsOn: ['default'],
      // Bind the context whose scope holds the credential this job reads.
      // Without it `ctx.secrets` resolves nothing: the job option is `context`,
      // and an unrecognised key is dropped at compile time rather than
      // rejected, so the step would fail on a missing secret at run time.
      context: 'hetzner-autoscale',
      run: async (ctx) => {
        const payload = ScalerScaleDownPayload.parse(ctx.rawPayload);

        const token = await ctx.secrets.get('HETZNER_API_TOKEN');
        const client = new HetznerClient(token);

        const servers = await client.listByLabel(`kici-agent-id==${payload.agentId}`);
        if (servers.length === 0) {
          ctx.log.info(
            `No Hetzner server found for agent ${payload.agentId}; nothing to tear down`,
          );
          return;
        }
        for (const server of servers) {
          await client.deleteServer(server.id);
          ctx.log.info(`Deleted Hetzner server ${server.id} for agent ${payload.agentId}`);
        }
      },
    }),
  ],
});

That’s the happy path. Underneath it, the instance also shuts itself down: the cloud-init bakes in a hard lifetime cap that powers the machine off once it expires, so a machine that loses contact with the orchestrator still stops on its own. And every resource is labelled at creation (kici-managed=hetzner-autoscale), which lets a small reaper run on a timer and delete every labelled machine older than its TTL. The reaper is the backstop for the cases nothing else covers (a hard kill, a crashed orchestrator, a rebooted host) because it depends on nothing but the label and the clock. A leaked instance is measured in minutes, not invoices.

The reaper is a dependency-free script plus the thin API client it calls, published as an example you can copy. Its core lists the labelled servers and deletes the ones past the TTL:

/**
 * Delete servers older than the TTL. A delete 404 (already gone) counts as a
 * success. A delete that fails is recorded and the run moves on, so one server
 * Hetzner refuses to delete never shields the servers listed after it.
 */
export async function reap(client: ReapableClient, opts: ReapOptions): Promise<ReapResult> {
  // An empty selector lists every server in the project, so only an explicit
  // whole-project sweep may send one.
  if (!opts.sweepWholeProject && opts.managedLabel.trim() === '') {
    throw new Error('reap: managedLabel is empty; set sweepWholeProject to sweep every server');
  }
  const selector = opts.sweepWholeProject ? '' : opts.managedLabel;
  const servers = await client.listByLabel(selector);
  const cutoff = opts.now() - opts.olderThanMs;

  const result: ReapResult = { deleted: 0, failed: [] };
  for (const server of servers) {
    const createdMs = Date.parse(server.created);
    // A server whose timestamp cannot be parsed is left alone (fail-safe: never
    // delete something we cannot age).
    if (Number.isNaN(createdMs) || createdMs >= cutoff) continue;
    try {
      await client.deleteServer(server.id);
      result.deleted += 1;
    } catch (err) {
      const error = err instanceof Error ? err.message : String(err);
      result.failed.push({ id: server.id, name: server.name, error });
    }
  }
  return result;
}

When you don’t need a provisioning workflow

• One host, or a fixed pool of machines. The built-in container scaler starts a one-shot agent container when a job arrives and removes it when the job ends. There’s no workflow to write, and it’s the backend most setups start with.
• You’d rather not own the provisioning code. With the event scaler, the provisioning workflow, the teardown workflow and the reaper are yours. When your cloud’s API changes, you update them.
• Jobs that can’t wait for a machine to boot. A cloud instance has to boot and start the agent before the job runs, so a queued job waits for that. A warm pool keeps ready agents for a label set, at the cost of paying for them while they sit idle.

Set it up

Let your coding agent do it. Every step below is a file to write or a command to run, which is the kind of work a coding agent is good at. Give yours this post’s URL and ask it to set up a Hetzner event scaler (or one for your own cloud) by following these steps. For the SDK details, point it at llms.txt or pipe kici docs llm sdk into its context.

The copy-ready provisioning and teardown workflows are in the workflow-driven autoscaling guide; they read their token as HETZNER_API_TOKEN from a hetzner-autoscale context. Another cloud follows the same steps with its own API calls.

1. Sign up and bring up an orchestrator. Create a free account at app.kici.dev and follow the 5-minute quickstart through Part 2, so pushes from your GitHub App reach the orchestrator. You also need a Hetzner Cloud project.

2. Create the repo that holds the provisioning workflows. Create a repo such as myorg/infra and install the quickstart’s GitHub App on it, so KiCI receives its pushes. Clone it and run kici init in the clone. That creates .kici/, with the package.json that declares @kici-dev/sdk. Delete any starter workflow it adds that you don’t want to run.

3. Make the orchestrator reachable from your instances. Each instance’s agent dials the orchestrator’s agent endpoint (/ws) from the cloud network, so expose that endpoint where the instances can reach it. If the address differs from the orchestrator’s own KICI_ORCHESTRATOR_URL, set it as orchestratorUrl on the scaler entry in step 6.

4. Pick the agent that runs the provisioning workflow. The provisioning workflow runs on an agent too, and it can’t wait for the pool it’s growing. The quickstart’s scaler already spawns one: in both workflows, you’ll change runsOn: ['default'] to that scaler’s labels, ['linux', 'container'] for the Docker quickstart or ['linux', 'bare-metal'] for bare metal. Those agents keep their default roles, so they also pack the workflow’s source before the job runs.

5. Store the cloud token in a context. Create a Read & Write API token in the Hetzner project. Then, where you run kici-admin against your orchestrator, create the context, store the token in its scope, and bind the two:

   kici-admin context create --org <orgId> --name hetzner-autoscale
   kici-admin secret set <orgId> hetzner-autoscale HETZNER_API_TOKEN --prompt
   kici-admin context bind --org <orgId> --env hetzner-autoscale --scope hetzner-autoscale

   Leave the context’s protection rules off. An approval hold on it would hold every scale-up, and the agents would never appear.

6. Declare the event scaler. Open the scalers.yaml the quickstart created: next to docker-compose.yaml for Docker, or ~/.config/kici/scalers.yaml for bare metal. Add a hetzner entry under scalers:, next to the quickstart’s own. provisioningTargets is the repo from step 2, written as owner/repo, and the label set is what your jobs will ask for:

   scalers:
     # ... the quickstart's scaler stays here ...
     - name: hetzner
       type: event
       maxAgents: 10
       provisioningTargets:
         - myorg/infra
       labelSets:
         - labels: [hetzner]

   Load it with kici-admin scaler reload. It checks the file and applies all of it or none of it, on the orchestrator it points at and every orchestrator connected to it, and prints what changed or why it refused the file. Here it prints added: hetzner.

7. Add the two workflows to the repo. In the clone from step 2, save the guide’s provisioning workflow as .kici/workflows/hetzner-provision.ts and its teardown workflow as .kici/workflows/hetzner-teardown.ts. Make the runsOn change from step 4 in both. The provisioning workflow starts the agent with docker run and installs docker.io on the stock debian-12 image; drop that packages line if you boot an image that already has Docker.

8. Compile and push to the default branch. From the root of the clone:

   kici compile
   git add .kici/
   git commit -m "ci: add Hetzner autoscaling workflows"
   git push origin main

   KiCI registers the scale-up and scale-down subscriptions only from the default branch’s lock file, so push to that branch (main here). Until that push, a scale-up reaches no one. Confirm both workflows registered with kici-admin registration list --repo myorg/infra: it lists hetzner-autoscale-provision and hetzner-autoscale-teardown.

9. Schedule the reaper. Copy reap.ts and hetzner-client.ts from examples/hetzner-autoscale/ into one directory on a host that isn’t one of the pool’s servers and has Node.js 22.18 or later. Put a project token in HCLOUD_TOKEN, set KICI_HETZNER_REAP_TTL_MIN above your longest job, and run node reap.ts from a timer every few minutes. The example’s README has a systemd service and timer to copy, and the teardown and reaper runbook covers the TTL to pick and the alerts to set on it.

10. Send a job to the pool and watch it. In any repo KiCI runs, give a job runsOn: ['hetzner'] and push it. That repo needs no other change. A kici-agent-<agent-id> server appears in the Hetzner console, its agent registers and runs the job, and the teardown workflow deletes the server once the agent goes idle. If no agent registers, the provisioning workflow’s own run shows the API error, and kici-admin diagnose names the failure on the scaler:hetzner row.

The guide also covers the event contract, the buildAgentCloudInit options, and the AWS variant of the same workflow.