Cloudflare Workflows

Durable execution for multi-step processes that must survive failures.

Prerequisites: PRINCIPLES.md, PRIMITIVES.md, queues.md

---

Overview

Cloudflare Workflows provide durable execution for multi-step processes. Unlike ephemeral Worker requests (30s timeout), Workflows can run for hours or days, automatically persisting state between steps and surviving restarts.

Property	Workers	Queues	Workflows
Execution model	Request/response	Fire-and-forget	Step-by-step
Duration	30s max	N/A	Hours/days
State persistence	None	None	Automatic
Failure handling	Caller retries	DLQ	Per-step retry
Best for	Fast paths	Async fanout	Multi-step processes

Key Insight: Workflows are not for the RTB hot path. They are for processes that span time and must complete reliably.

---

Core Concepts

Steps

A Workflow is a sequence of named steps. Each step:

• Executes exactly once (at-least-once with deduplication)
• Persists its return value before proceeding
• Can be retried independently on failure
• Is addressable by name for debugging

workflow.step('validate_payment', async () => {
  const result = await validateWithStripe(payment);
  return result;  // Persisted before next step runs
});

workflow.step('record_fact', async () => {
  await ledger.appendFact({
    type: 'payment_received',
    // ...
  });
});

Sleep

Workflows can sleep for arbitrary durations without consuming resources:

workflow.step('wait_for_settlement', async () => {
  await workflow.sleep('24 hours');
});

Use cases:

• Wait for payment settlement (T+1, T+2)
• Delay before retry
• Scheduled follow-up actions
• Grace periods before escalation

State Persistence

Each step’s return value is automatically persisted. On restart:

1. Completed steps are skipped
2. Their persisted return values are available
3. Execution resumes from the interrupted step

// If this workflow restarts after step 2...
const invoice = workflow.step('create_invoice', async () => {
  return await createInvoice(account);  // Skipped, returns persisted value
});

const sent = workflow.step('send_invoice', async () => {
  return await sendEmail(invoice);  // Skipped, returns persisted value
});

workflow.step('await_payment', async () => {
  await workflow.sleep('7 days');  // Resumes here
});

Retry Semantics

Each step has configurable retry behavior:

workflow.step('call_external_api', async () => {
  return await externalApi.call();
}, {
  retries: {
    limit: 5,
    delay: '1 second',
    backoff: 'exponential'
  }
});

Default retry behavior:

• 3 retries with exponential backoff
• Starting delay: 1 second
• Max delay: 5 minutes

After exhausting retries:

• Workflow enters failed state
• Can be manually retried or abandoned
• Error details preserved for debugging

---

How z0 Uses Workflows

1. Multi-Step Billing Processes

Invoice generation → delivery → payment tracking → reconciliation.

// Invoice workflow for tenant billing
workflow.step('gather_charges', async () => {
  const charges = await ledger.queryFacts({
    type: 'charge',
    tenant_id,
    invoice_id: null,  // Unbilled charges
    timestamp: { lt: period_end }
  });
  return charges;
});

workflow.step('create_invoice_fact', async () => {
  return await ledger.appendFact({
    type: 'invoice_created',
    tenant_id,
    data: { charges, total, period }
  });
});

workflow.step('send_to_billing_provider', async () => {
  return await stripe.createInvoice(invoice);
});

workflow.step('record_issued', async () => {
  return await ledger.appendFact({
    type: 'invoice_issued',
    source_id: invoice.id
  });
});

// Wait for payment or timeout
workflow.step('await_payment', async () => {
  await workflow.sleep('30 days');
});

workflow.step('check_payment_status', async () => {
  const status = await stripe.getInvoiceStatus(invoice.external_id);
  if (status === 'paid') {
    await ledger.appendFact({ type: 'payment_received', ... });
  } else {
    await ledger.appendFact({ type: 'payment_overdue', ... });
    // Trigger escalation
  }
});

2. Payment Reconciliation

Match incoming payments to outstanding invoices.

// Payment reconciliation workflow
workflow.step('fetch_bank_transactions', async () => {
  return await bankApi.getTransactions({ since: lastSync });
});

workflow.step('match_to_invoices', async () => {
  const matches = [];
  for (const tx of transactions) {
    const invoice = await findMatchingInvoice(tx);
    if (invoice) {
      matches.push({ tx, invoice });
    }
  }
  return matches;
});

workflow.step('record_payments', async () => {
  for (const { tx, invoice } of matches) {
    await ledger.appendFact({
      type: 'payment_received',
      source_id: invoice.id,
      amount: tx.amount,
      data: { bank_reference: tx.id }
    });
  }
});

workflow.step('flag_unmatched', async () => {
  const unmatched = transactions.filter(tx => !matches.find(m => m.tx.id === tx.id));
  if (unmatched.length > 0) {
    await alert.send('reconciliation_manual_review', { unmatched });
  }
});

3. Long-Running Integrations

CRM sync, data exports, bulk operations.

// CRM deal sync workflow
workflow.step('fetch_deals_batch', async () => {
  return await hubspot.getDeals({
    modified_since: lastSync,
    limit: 100
  });
});

workflow.step('upsert_entities', async () => {
  for (const deal of deals) {
    await entityStore.upsert({
      type: 'deal',
      external_source: 'hubspot',
      external_id: deal.id,
      contact_id: await resolveContact(deal.contact),
      status: mapDealStatus(deal.stage),
      metadata: deal.properties
    });
  }
});

workflow.step('record_outcomes', async () => {
  for (const deal of deals.filter(d => d.stage === 'closed_won')) {
    await ledger.appendFact({
      type: 'outcome',
      subtype: 'deal_won',
      deal_id: deal.z0_id,
      contact_id: deal.contact_id
    });
  }
});

// If more pages, spawn continuation
workflow.step('continue_if_needed', async () => {
  if (deals.length === 100) {
    await workflows.create('crm_sync', { cursor: deals[99].id });
  }
});

4. Saga Patterns

Distributed transactions with compensation on failure.

// Transfer funds between accounts (saga pattern)
workflow.step('debit_source', async () => {
  const debit = await ledger.appendFact({
    type: 'charge',
    from_entity: source_account,
    amount: transfer_amount
  });
  return debit;
});

workflow.step('credit_destination', async () => {
  try {
    const credit = await ledger.appendFact({
      type: 'deposit',
      to_entity: destination_account,
      amount: transfer_amount
    });
    return { success: true, credit };
  } catch (error) {
    // Compensation: reverse the debit
    await ledger.appendFact({
      type: 'credit_issued',
      to_entity: source_account,
      amount: transfer_amount,
      data: { reason: 'transfer_failed', original_debit: debit.id }
    });
    return { success: false, error };
  }
});

workflow.step('record_transfer', async () => {
  if (credit_result.success) {
    await ledger.appendFact({
      type: 'lifecycle',
      subtype: 'transfer_completed',
      data: { debit: debit.id, credit: credit_result.credit.id }
    });
  }
});

---

When to Use What

Scenario	Use	Why
RTB routing decision	Worker	Must complete in <50ms
Fact write	Durable Object	Single-threaded consistency
Async Fact replication	Queue	Fire-and-forget, high throughput
Daily budget reset	DO Alarm	Scheduled, single entity
Invoice generation	Workflow	Multi-step, must complete
Payment reconciliation	Workflow	External APIs, needs retry
CRM sync batch	Workflow	Long-running, paginated
Distributed transaction	Workflow	Needs compensation

Decision Tree

Is this a single atomic operation?
├─ Yes → Durable Object
└─ No, multiple steps
   ├─ Must all complete together? → Workflow (saga pattern)
   └─ Independent steps?
      ├─ Time-sensitive (<1s)? → Worker + parallel DO calls
      └─ Can be async?
         ├─ Simple fanout? → Queue
         └─ Orchestrated sequence? → Workflow

DO Alarms vs Workflows

Feature	DO Alarms	Workflows
Scope	Single entity	Cross-entity
Trigger	Scheduled time	On-demand or scheduled
State	DO’s SQLite	Workflow engine
Duration	Single execution	Multi-step, hours/days
Use case	”Wake me at midnight"	"Process this invoice”

Use DO Alarms for: Daily budget reset, cache expiration, scheduled entity maintenance.

Use Workflows for: Multi-step processes, external API orchestration, saga patterns.

Webhooks: Queue or Workflow?

Scenario	Use
Single webhook delivery	Queue (simpler)
Webhook with retry + compensation	Queue (built-in retry)
Multi-webhook orchestration	Workflow
Webhook requiring external confirmation	Workflow (sleep until confirmed)

---

Workflow Patterns

Idempotency

Steps should be idempotent. The same step may execute multiple times on retry.

// Bad: Creates duplicate Facts on retry
workflow.step('bad_record', async () => {
  await ledger.appendFact({ ... });  // No idempotency key
});

// Good: Idempotent with deduplication key
workflow.step('good_record', async () => {
  // DO ledger checks data.idempotency_key before appending
  // If a Fact with this key exists, returns existing Fact without creating duplicate
  await ledger.appendFact({
    ...
    data: { idempotency_key: `${workflow_id}_${step_name}` }
  });
});

Error Handling

Distinguish between retryable and terminal errors:

workflow.step('external_call', async () => {
  try {
    return await externalApi.call();
  } catch (error) {
    if (error.code === 'RATE_LIMITED') {
      throw error;  // Retryable, workflow will retry
    }
    if (error.code === 'INVALID_INPUT') {
      // Terminal error, record and continue
      await ledger.appendFact({
        type: 'lifecycle',
        subtype: 'integration_failed',
        data: { error: error.message }
      });
      return { failed: true, reason: error.message };
    }
    throw error;  // Unknown, let workflow handle
  }
});

Workflow Observability

Every workflow execution should be traceable:

workflow.step('instrumented_step', async () => {
  const span = tracer.startSpan('z0.workflow.billing.create_invoice');
  try {
    const result = await createInvoice();
    span.setStatus('ok');
    return result;
  } catch (error) {
    span.setStatus('error', error.message);
    throw error;
  } finally {
    span.end();
  }
});

---

Limitations

Limitation	Constraint	Workaround
Step payload size	1MB	Store large data in R2, pass reference
Total steps	1000	Break into child workflows
Execution time	30 days	Design for completion or timeout
Concurrent workflows	Per-account limits	Queue workflow creation

---

Implementation Checklist

Before using Workflows in production:

• Each step is idempotent
• Retry limits configured appropriately
• Terminal vs retryable errors distinguished
• Compensation logic for saga patterns
• Workflow observability instrumented
• Alerting on workflow failures
• Manual retry/abandon procedures documented

---

Summary

Concept	z0 Application
Steps	Each Fact write is a step, ensuring persistence
Sleep	Payment settlement windows, grace periods
Retry	External API resilience
State	Invoice context across multi-day processes
Saga	Distributed transactions with compensation

Workflows are z0’s tool for processes that span time and must complete. They are not for the hot path. They are for the billing, reconciliation, and integration processes that happen in the background and must never be lost.