Autoscale onto any cloud by writing a workflow, not an integration

The problem: autoscaling means adopting an integration

You want your CI to burst onto cloud compute when the queue backs up, then give it back when it drains. Every tool solves this by shipping a driver per cloud — an AWS backend, a GCP backend, a Hetzner backend — each a lump of provider SDK code the tool has to write, test, and keep current as the provider’s API moves. If your cloud isn’t on the list, you wait. If it is, you inherit whatever the driver’s authors decided about instance types, images, and lifecycle, and you configure around it.

The mismatch is that your cloud only works if someone else already built the driver for it. Your image, your network, your budget — all gated behind another team’s release cycle and roadmap.

The KiCI model: the scaler emits events, a workflow does the work

KiCI’s scaler has an event backend that performs no cloud calls of its own. When demand rises, its spawn() emits a reserved kici.scaler.scale-up event; when an instance should go away, destroy() emits kici.scaler.scale-down. Those are ordinary KiCI custom events, and you consume them with the same kiciEvent() trigger you’d use for anything else — in a provisioning workflow you write in TypeScript.

So “cloud autoscaling” stops being “which drivers ship this quarter” and becomes “write a workflow.” KiCI keeps the parts that are genuinely hard and generic — demand detection, capacity caps, reservations, dispatching the pending job to the new machine, spawn timeouts, idle and heartbeat teardown — and leaves one thing open on purpose: how a machine comes up. That’s the extension point. Plug in any cloud’s API there and it works — no driver required, so KiCI ships zero cloud SDK code and you’re never limited to a supported list.

The credential handoff is built for this. The scale-up event carries a single-use claim code, not a token. Your workflow forwards that code into the instance’s cloud-init, and the agent exchanges it for its own short-lived token once it boots — so the token is minted inside the machine that uses it, and never transits provisioning or the persisted event log at all.

Writing the provisioning workflow

Here’s the reference provisioning workflow. It subscribes to the scale-up event, forwards the single-use claim code into a cloud-init, and boots a labelled instance whose agent self-claims and registers under the scaler-chosen agentId. This one targets Hetzner; a customer swaps the boot call for their own cloud’s API and keeps the shape.

import {
  workflow,
  job,
  kiciEvent,
  buildAgentCloudInit,
  SCALER_EVENT_NAMES,
  ScalerScaleUpPayload,
} from '@kici-dev/sdk';

const SCALER_NAME = 'hetzner';
/** Hard lifetime cap for a provisioned instance (teardown layer L2). */
const MAX_LIFETIME_MINUTES = 30;

export default workflow('hetzner-autoscale-provision', {
  on: [kiciEvent({ name: SCALER_EVENT_NAMES.scaleUp, match: { '$.scalerName': SCALER_NAME } })],
  jobs: [
    job('provision', {
      runsOn: ['default'],
      // Bind the context whose scope holds the credential this job reads.
      // Without it `ctx.secrets` resolves nothing: the job option is `context`,
      // and an unrecognised key is dropped at compile time rather than
      // rejected, so the step would fail on a missing secret at run time.
      context: 'hetzner-autoscale',
      run: async (ctx) => {
        const payload = ScalerScaleUpPayload.parse(ctx.rawPayload);

        // Forward the single-use claim code into cloud-init; the agent claims
        // its own token in-instance, so the token never transits provisioning.
        const userData = buildAgentCloudInit(
          {
            claimCode: payload.claimCode,
            agentId: payload.agentId,
            orchestratorUrl: payload.orchestratorUrl,
            labels: payload.labels,
          },
          {
            maxLifetimeMinutes: MAX_LIFETIME_MINUTES,
            deliveryMode: 'payload',
            agentEnv: e2eAgentEnv(ctx.env),
          },
        );

        const token = await ctx.secrets.get('KICI_HETZNER_E2E_SCALER_API_TOKEN');
        const client = new HetznerClient(token);
        const { id } = await client.createServer({
          name: `kici-agent-${payload.agentId}`,
          // A current-generation shared-vCPU type available in the region below.
          server_type: 'cpx12',
          image: 'debian-12',
          user_data: userData,
          // Every teardown layer keys off these labels.
          labels: {
            'kici-managed': 'hetzner-autoscale',
            'kici-agent-id': payload.agentId,
            'kici-scaler': SCALER_NAME,
            ...e2eServerLabels(ctx.env),
          },
        });

        ctx.log.info(`Provisioned Hetzner server ${id} for agent ${payload.agentId}`);
      },
    }),
  ],
});

Everything provider-specific lives in the run body — that’s the whole extension surface: the instance type, the image, the labels, the API call. The KiCI-specific parts are the kiciEvent trigger, the payload parse, and the buildAgentCloudInit call. There’s no backend to register, no plugin to install. Supporting a new cloud isn’t a feature request; it’s a new workflow you write today.

Teardown you can’t leak

The scary part of “boot a cloud instance on demand” is the one you don’t see: the instance that doesn’t get cleaned up when something crashes mid-run, quietly billing you for a week. So teardown doesn’t rely on any single path succeeding.

The scale-down event fires the mirror-image teardown workflow, which deletes the servers for the scaled-down agent by label:

import {
  workflow,
  job,
  kiciEvent,
  SCALER_EVENT_NAMES,
  ScalerScaleDownPayload,
} from '@kici-dev/sdk';

const SCALER_NAME = 'hetzner';

export default workflow('hetzner-autoscale-teardown', {
  on: [kiciEvent({ name: SCALER_EVENT_NAMES.scaleDown, match: { '$.scalerName': SCALER_NAME } })],
  jobs: [
    job('teardown', {
      runsOn: ['default'],
      // Bind the context whose scope holds the credential this job reads.
      // Without it `ctx.secrets` resolves nothing: the job option is `context`,
      // and an unrecognised key is dropped at compile time rather than
      // rejected, so the step would fail on a missing secret at run time.
      context: 'hetzner-autoscale',
      run: async (ctx) => {
        const payload = ScalerScaleDownPayload.parse(ctx.rawPayload);

        const token = await ctx.secrets.get('KICI_HETZNER_E2E_SCALER_API_TOKEN');
        const client = new HetznerClient(token);

        const servers = await client.listByLabel(`kici-agent-id==${payload.agentId}`);
        if (servers.length === 0) {
          ctx.log.info(
            `No Hetzner server found for agent ${payload.agentId}; nothing to tear down`,
          );
          return;
        }
        for (const server of servers) {
          await client.deleteServer(server.id);
          ctx.log.info(`Deleted Hetzner server ${server.id} for agent ${payload.agentId}`);
        }
      },
    }),
  ],
});

That’s the happy path. Underneath it, the instance also shuts itself down — the cloud-init bakes in a hard lifetime cap that powers the machine off once it expires, so a machine that loses contact with the orchestrator still goes away on its own. And every resource is labelled at creation (kici-managed=hetzner-autoscale), which lets a small reaper run on a timer and delete every labelled machine older than its TTL. The reaper is the backstop that survives the cases nothing else does — a hard kill, a crashed orchestrator, a rebooted host — because it depends on nothing but the label and the clock. A leaked instance is measured in minutes, not invoices.

Try it

The full pattern — the event scaler config, the reserved event contract, and the provisioning and teardown workflows — is in the workflow-driven autoscaling guide. It’s the same idea as the rest of KiCI: the thing that used to be a driver you waited for is TypeScript you can extend to any cloud you like.