Webhook Delivery System

Reliable delivery of events to external systems with at-least-once guarantees, payload signing, and full audit trails.

Prerequisites: PRIMITIVES.md, PRINCIPLES.md, error-handling.md, queues.md

---

Overview

The z0 webhook delivery system bridges internal Facts to external systems (CRMs, notification services, analytics platforms). Every webhook delivery is tracked, signed, and retryable. Failures never silently disappear—they’re recorded as Facts and queued for retry or manual intervention.

Principle	Webhook Implementation
Principle 2: Facts Are Immutable	Every delivery attempt creates an immutable Fact
Principle 6: Configs Are Versioned	Webhook Configs are versioned; Facts reference config_version
Principle 8: Errors Are First-Class	Failed deliveries are Facts with full error context

Design Philosophy:

1. Deliver or explain why not - Never silently drop events
2. Sign everything - Recipients verify authenticity via HMAC
3. Idempotent by design - Same event delivered multiple times produces same result
4. Assume hostility - External systems will timeout, return errors, change endpoints
5. Track everything - Full audit trail from trigger to delivery to response

---

Architecture

Flow

Fact Recorded (e.g., outcome.deal_won)
    │
    ▼
Webhook Matcher: Query active Webhook Configs
    │ WHERE trigger IN (fact.type.fact.subtype)
    │ AND applies_to IN (fact.campaign_id, fact.tenant_id)
    │
    ▼
For each matched Webhook Config:
    │
    ├─ Record webhook_triggered Fact (Principle 2)
    │
    ├─ Insert into local webhook_outbox table (status: 'pending')
    │
    ├─ Fast Path: Invoke processOutboxItem() via ctx.waitUntil()
    │
    └─ Return (Non-blocking response to caller)

Async Delivery (Fast Path or Alarm):
    │
    ├─ Build payload from triggering Fact
    │
    ├─ Sign payload with HMAC-SHA256
    │
    ├─ POST to webhook URL with signature headers
    │
    ├─ If HTTP 200:
    │   └─ Update outbox status to 'sent'
    │
    └─ If HTTP Error/Network Error:
        ├─ Increment attempt count
        ├─ Calculate exponential backoff
        ├─ Update next_retry_at
        └─ If max retries hit: Mark as 'dead'

Components

Component	Role	Technology
Webhook Matcher	Finds matching Webhook Configs for Facts	Durable Object (local configs)
Webhook Outbox	Durable storage for pending deliveries	Durable Object (SQLite table)
Delivery Runner	Executes HTTP POST and handles retries	Durable Object (waitUntil + Alarm)
Signature Generator	HMAC-SHA256 signing	Web Crypto API
Retry Manager	Exponential backoff logic	DO logic + backoff.ts
Fact Recorder	Tracks triggered/sent status	Durable Object (Fact ledger)

---

Webhook Config Schema

Webhooks are Configs (type: webhook, category: logic). They define trigger conditions, endpoint details, retry policies, and payload formatting.

Schema

Config:
  id: webhook_001
  type: webhook
  category: logic

  name: "HubSpot Deal Won"
  applies_to: campaign_001    # Or account ID for account-wide webhooks
  scope: campaign             # campaign, account, asset
  tenant_id: ten_abc123

  version: 2
  effective_at: 2026-01-15T00:00:00Z
  superseded_at: null         # null = current version

  settings:
    # Endpoint
    url: "https://customer.com/webhooks/z0-events"
    secret_ref: "webhook_secret_001"  # Reference to secret in KV/Secrets Manager

    # Triggers (OR logic: any match fires webhook)
    triggers:
      - outcome.deal_won
      - outcome.call_qualified
      - dispute.opened
      - dispute.resolved

    # Retry Policy
    retry:
      enabled: true
      max_attempts: 5
      backoff_base_seconds: 60
      backoff_multiplier: 2
      backoff_max_seconds: 3600
      timeout_seconds: 30

    # Payload Configuration
    payload:
      format: json              # json (future: xml, form)
      include_fields:
        - fact_id
        - fact_type
        - fact_subtype
        - timestamp

      # Optional: custom field mapping
      field_mapping:
        fact_type: event_type

      # Optional: include full triggering Fact
      include_full_fact: false

    # Headers (optional custom headers)
    headers:
      X-Client-ID: "client_abc123"
      X-Environment: "production"

Field Definitions

Field	Type	Required	Description
url	string	Yes	HTTPS endpoint (must be https://)
secret_ref	string	Yes	Reference to signing secret (never inline secret)
triggers	string[]	Yes	Fact type.subtype patterns to match
retry.enabled	boolean	No	Default: true
retry.max_attempts	integer	No	Default: 5, max: 20
retry.backoff_base_seconds	integer	No	Default: 60
retry.backoff_multiplier	number	No	Default: 2
retry.backoff_max_seconds	integer	No	Default: 3600 (1 hour)
retry.timeout_seconds	integer	No	Default: 30, max: 300
payload.format	enum	No	Default: json
payload.include_fields	string[]	No	Default: all Fact fields
payload.field_mapping	object	No	Rename fields in payload
payload.include_full_fact	boolean	No	Default: false
headers	object	No	Custom HTTP headers

Validation Rules

interface WebhookConfigValidation {
  // URL must be HTTPS
  url: /^https:\/\/.+/,

  // Secret reference format
  secret_ref: /^webhook_secret_[a-z0-9]+$/,

  // Triggers must match pattern
  triggers: /^[a-z_]+\.[a-z_]+$/,

  // Retry constraints
  max_attempts: { min: 1, max: 20 },
  backoff_base_seconds: { min: 1, max: 3600 },
  timeout_seconds: { min: 5, max: 300 }
}

Security Rules:

1. URL must be HTTPS (reject HTTP)
2. Secret must be referenced, never inlined
3. Timeout must be bounded (prevent hanging connections)
4. Max attempts must be bounded (prevent infinite retries)

---

Webhook Facts

Per Principle 2 (Facts Are Immutable) and Principle 8 (Errors Are First-Class), every webhook trigger and delivery attempt is recorded as a Fact.

webhook_triggered

Records when a Fact matches a Webhook Config and delivery is queued.

Fact:
  id: fact_wht_001
  type: webhook
  subtype: triggered
  timestamp: 2026-01-17T10:30:00.123Z

  # Links
  source_id: fact_outcome_001   # Triggering Fact ID
  tenant_id: ten_abc123

  # Config Audit (Principle 6)
  config_id: webhook_001
  config_version: 2

  data:
    url: "https://customer.com/webhooks/z0-events"
    trigger: "outcome.deal_won"
    delivery_id: "del_001"      # Unique delivery identifier
    queued_at: 2026-01-17T10:30:00.123Z

Purpose: Proves webhook was triggered. Enables debugging “why didn’t this fire?” questions.

webhook_sent (Success)

Records successful delivery to external endpoint.

Fact:
  id: fact_whs_001
  type: webhook
  subtype: sent
  timestamp: 2026-01-17T10:30:01.234Z

  # Links
  source_id: fact_wht_001       # References triggered Fact
  tenant_id: ten_abc123

  # Config Audit
  config_id: webhook_001
  config_version: 2

  data:
    delivery_id: "del_001"
    url: "https://customer.com/webhooks/z0-events"

    # Delivery Result
    success: true
    status_code: 200
    response_time_ms: 234
    response_headers:
      Content-Type: "application/json"
    response_body: '{"received": true}'

    # Retry Context
    attempt: 1
    max_attempts: 5

    # Signature
    signature_algorithm: "sha256"
    timestamp_header: "1705500600"

webhook_sent (Failure)

Records failed delivery with full error context.

Fact:
  id: fact_whs_002
  type: webhook
  subtype: sent
  timestamp: 2026-01-17T10:31:00.456Z

  # Links
  source_id: fact_wht_001
  tenant_id: ten_abc123
  campaign_id: campaign_001

  # Config Audit
  config_id: webhook_001
  config_version: 2

  data:
    delivery_id: "del_001"
    url: "https://customer.com/webhooks/z0-events"

    # Delivery Result
    success: false
    status_code: 503
    error_type: "transient"     # From error taxonomy
    error_code: "SERVICE_UNAVAILABLE"
    error_message: "Service temporarily unavailable"
    response_time_ms: 30000     # Timed out

    # Retry Context
    attempt: 2
    max_attempts: 5
    will_retry: true
    next_retry_at: 2026-01-17T10:33:00.456Z
    backoff_seconds: 120

    # Signature (same signature for all retries)
    signature_algorithm: "sha256"
    timestamp_header: "1705500600"

webhook_sent (Max Retries Exceeded)

Records when delivery fails after all retry attempts.

Fact:
  id: fact_whs_006
  type: webhook
  subtype: sent
  timestamp: 2026-01-17T11:30:00.789Z

  source_id: fact_wht_001
  tenant_id: ten_abc123
  config_id: webhook_001
  config_version: 2

  data:
    delivery_id: "del_001"
    url: "https://customer.com/webhooks/z0-events"

    success: false
    status_code: 503
    error_type: "terminal"
    error_code: "MAX_RETRIES_EXCEEDED"
    error_message: "Failed after 5 attempts"

    attempt: 5
    max_attempts: 5
    will_retry: false

    # Dead Letter Queue
    sent_to_dlq: true
    dlq_message_id: "dlq_del_001"
    dlq_timestamp: 2026-01-17T11:30:00.789Z

    # Audit Trail
    first_attempt_at: 2026-01-17T10:30:01.234Z
    total_duration_ms: 3599555

Fact Invariants

// Every webhook_sent traces to webhook_triggered
∀ Fact(webhook.sent) → ∃ Fact(webhook.triggered)
  WHERE sent.source_id = triggered.id

// Same delivery_id across all attempts
∀ Fact(webhook.sent WHERE source_id = X) →
  data.delivery_id is constant

// Config version locked for all attempts
∀ Fact(webhook.sent WHERE source_id = X) →
  config_id AND config_version are constant

// Attempt numbers are sequential
∀ Fact(webhook.sent WHERE source_id = X) →
  attempt ∈ [1, max_attempts]

// DLQ only on max retries
∀ Fact(webhook.sent WHERE sent_to_dlq = true) →
  attempt = max_attempts

---

Payload Signing

All webhook payloads are signed with HMAC-SHA256 to enable recipient verification. This prevents spoofing and proves authenticity.

Signature Algorithm

function signPayload(
  payload: string,
  timestamp: number,
  secret: string
): string {
  // Message = timestamp + "." + payload
  const message = `${timestamp}.${payload}`;

  // HMAC-SHA256(secret, message)
  const signature = crypto
    .subtle
    .sign(
      { name: 'HMAC', hash: 'SHA-256' },
      await importKey(secret),
      new TextEncoder().encode(message)
    );

  // Return hex-encoded signature
  return 'sha256=' + bufferToHex(signature);
}

async function importKey(secret: string): Promise<CryptoKey> {
  return await crypto.subtle.importKey(
    'raw',
    new TextEncoder().encode(secret),
    { name: 'HMAC', hash: 'SHA-256' },
    false,
    ['sign']
  );
}

function bufferToHex(buffer: ArrayBuffer): string {
  return Array.from(new Uint8Array(buffer))
    .map(b => b.toString(16).padStart(2, '0'))
    .join('');
}

HTTP Headers

POST /webhooks/z0-events HTTP/1.1
Host: customer.com
Content-Type: application/json
X-Z0-Signature: sha256=abc123def456...
X-Z0-Timestamp: 1705500600
X-Z0-Delivery-Id: del_001
X-Z0-Config-Version: 2
X-Z0-Attempt: 1
User-Agent: z0-webhooks/1.0

Header	Purpose
X-Z0-Signature	HMAC-SHA256 signature for verification
X-Z0-Timestamp	Unix timestamp (seconds) used in signature
X-Z0-Delivery-Id	Unique delivery identifier (idempotency)
X-Z0-Config-Version	Webhook Config version (Principle 6)
X-Z0-Attempt	Retry attempt number (1-indexed)

Signature Verification (Recipient)

// Recipient implementation
async function verifyWebhook(
  payload: string,
  signature: string,
  timestamp: string,
  secret: string
): Promise<boolean> {
  // 1. Check timestamp freshness (prevent replay attacks)
  const now = Math.floor(Date.now() / 1000);
  const requestTime = parseInt(timestamp, 10);

  if (now - requestTime > 300) {  // 5 minute tolerance
    throw new Error('Webhook timestamp too old (replay attack?)');
  }

  // 2. Compute expected signature
  const expected = await signPayload(payload, requestTime, secret);

  // 3. Constant-time comparison (prevent timing attacks)
  return constantTimeCompare(signature, expected);
}

function constantTimeCompare(a: string, b: string): boolean {
  if (a.length !== b.length) return false;

  let result = 0;
  for (let i = 0; i < a.length; i++) {
    result |= a.charCodeAt(i) ^ b.charCodeAt(i);
  }

  return result === 0;
}

Recipient Verification Steps

1. Extract headers: X-Z0-Signature, X-Z0-Timestamp
2. Check timestamp: Reject if > 5 minutes old (replay attack)
3. Compute expected signature: HMAC-SHA256(secret, timestamp + ”.” + payload)
4. Constant-time compare: Prevent timing attacks
5. Check delivery ID: Deduplicate using X-Z0-Delivery-Id

Example Payload

{
  "delivery_id": "del_001",
  "timestamp": "2026-01-17T10:30:00.123Z",
  "trigger": "outcome.deal_won",
  "config_version": 2,

  "fact": {
    "id": "fact_outcome_001",
    "type": "outcome",
    "subtype": "deal_won",
    "timestamp": "2026-01-17T10:29:58.000Z",
    "data": {
      "campaign_id": "campaign_001",
      "contact_id": "contact_123",
      "deal_id": "deal_456",
      "amount": 5000.00,
      "currency": "USD"
    }
  }
}

---

Retry Policy

Per error-handling.md, webhooks use exponential backoff with jitter for transient failures.

Default Policy

retry:
  enabled: true
  max_attempts: 5
  backoff_base_seconds: 60      # Start at 1 minute
  backoff_multiplier: 2         # Double each time
  backoff_max_seconds: 3600     # Cap at 1 hour
  timeout_seconds: 30

Retry Schedule

Attempt	Delay (no jitter)	Delay (with jitter)	Cumulative Time
1	0s	0s	0s
2	60s	60-90s	~1 min
3	120s	120-180s	~3 min
4	240s	240-360s	~8 min
5	480s	480-720s	~20 min

After attempt 5 fails: Send to Dead Letter Queue.

Retryable vs Terminal Errors

function isRetryable(response: Response): boolean {
  // Transient errors: retry
  if ([429, 503, 504].includes(response.status)) {
    return true;
  }

  // Server errors: retry (assuming idempotency)
  if ([500, 502].includes(response.status)) {
    return true;
  }

  // Client errors: don't retry (permanent failure)
  if (response.status >= 400 && response.status < 500) {
    return false;
  }

  // Timeout: retry
  if (response.status === 0 && response.error === 'ETIMEDOUT') {
    return true;
  }

  return false;
}

Error Type	Status	Retry?	Reason
Rate Limited	429	Yes	Temporary capacity limit
Bad Request	400	No	Invalid payload (won’t change on retry)
Unauthorized	401	No	Invalid credentials
Not Found	404	No	Endpoint doesn’t exist
Timeout	-	Yes	Network transient
Service Unavailable	503	Yes	Temporary outage
Gateway Timeout	504	Yes	Upstream timeout
Internal Server Error	500	Yes	May be transient

Backoff Calculation

function calculateBackoff(
  attempt: number,
  config: RetryConfig
): number {
  // Exponential: delay = base * multiplier^(attempt - 1)
  const exponential = config.backoff_base_seconds * Math.pow(
    config.backoff_multiplier,
    attempt - 1
  );

  // Cap at max
  const capped = Math.min(exponential, config.backoff_max_seconds);

  // Add jitter (0-50% of delay)
  const jitter = capped * 0.5 * Math.random();

  return capped + jitter;
}

---

Delivery Worker Implementation

Queue Message Schema

interface WebhookDeliveryMessage {
  delivery_id: string;          // Unique delivery identifier
  triggered_fact_id: string;    // Fact that triggered webhook
  webhook_config_id: string;
  webhook_config_version: number;

  attempt: number;              // Current attempt (1-indexed)
  max_attempts: number;

  payload: object;              // Pre-built payload
  url: string;
  secret_ref: string;
  timeout_seconds: number;

  timestamp: number;            // Unix timestamp for signature

  tenant_id: string;
  campaign_id?: string;
}

Queue Consumer

export default {
  async queue(
    batch: MessageBatch<WebhookDeliveryMessage>,
    env: Env
  ): Promise<void> {
    for (const message of batch.messages) {
      try {
        await deliverWebhook(message.body, env);
        message.ack();

      } catch (error) {
        // Determine if retryable
        if (isRetryable(error) && message.body.attempt < message.body.max_attempts) {
          // Calculate backoff
          const backoffSeconds = calculateBackoff(
            message.body.attempt,
            await getRetryConfig(message.body.webhook_config_id, env)
          );

          // Record failure Fact
          await recordWebhookSent(message.body, error, false, env);

          // Requeue with delay
          await env.WEBHOOK_QUEUE.send(
            {
              ...message.body,
              attempt: message.body.attempt + 1
            },
            {
              delaySeconds: Math.floor(backoffSeconds)
            }
          );

          message.ack();

        } else {
          // Terminal error or max retries exceeded
          await recordWebhookSent(message.body, error, true, env);
          await sendToDLQ(message.body, error, env);
          message.ack();
        }
      }
    }
  }
};

Delivery Function

async function deliverWebhook(
  msg: WebhookDeliveryMessage,
  env: Env
): Promise<void> {
  // 1. Fetch secret
  const secret = await env.SECRETS.get(msg.secret_ref);
  if (!secret) {
    throw new Error(`Secret ${msg.secret_ref} not found`);
  }

  // 2. Serialize payload
  const payloadString = JSON.stringify(msg.payload);

  // 3. Sign payload
  const signature = await signPayload(
    payloadString,
    msg.timestamp,
    secret
  );

  // 4. Execute HTTP POST
  const controller = new AbortController();
  const timeoutId = setTimeout(
    () => controller.abort(),
    msg.timeout_seconds * 1000
  );

  try {
    const response = await fetch(msg.url, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'x-z0-signature': signature,
        'x-z0-timestamp': msg.timestamp.toString(),
        'x-z0-delivery-id': msg.delivery_id,
        'x-z0-config-version': msg.webhook_config_version.toString(),
        'x-z0-attempt': msg.attempt.toString(),
        'User-Agent': 'z0-webhooks/1.0'
      },
      body: payloadString,
      signal: controller.signal
    });

    clearTimeout(timeoutId);

    // 5. Check response
    if (!response.ok) {
      const error = new HTTPError(
        `Webhook delivery failed: ${response.status}`,
        response.status,
        await response.text()
      );

      throw error;
    }

    // 6. Record success
    await recordWebhookSent(msg, response, false, env);

  } catch (error) {
    clearTimeout(timeoutId);

    if (error.name === 'AbortError') {
      throw new TimeoutError(`Webhook timeout after ${msg.timeout_seconds}s`);
    }

    throw error;
  }
}

Fact Recording

async function recordWebhookSent(
  msg: WebhookDeliveryMessage,
  result: Response | Error,
  terminal: boolean,
  env: Env
): Promise<void> {
  const isSuccess = result instanceof Response && result.ok;
  const isError = result instanceof Error;

  const fact: Fact = {
    type: 'webhook',
    subtype: 'sent',
    timestamp: Date.now(),

    source_id: msg.triggered_fact_id,
    tenant_id: msg.tenant_id,
    campaign_id: msg.campaign_id,

    config_id: msg.webhook_config_id,
    config_version: msg.webhook_config_version,

    data: {
      delivery_id: msg.delivery_id,
      url: msg.url,

      success: isSuccess,
      status_code: isError ? 0 : result.status,

      ...(isSuccess && {
        response_time_ms: result.headers.get('X-Response-Time'),
        response_headers: Object.fromEntries(result.headers.entries()),
        response_body: await result.text()
      }),

      ...(isError && {
        error_type: isRetryable(result) ? 'transient' : 'terminal',
        error_code: result.code || 'UNKNOWN',
        error_message: result.message
      }),

      attempt: msg.attempt,
      max_attempts: msg.max_attempts,
      will_retry: !terminal && msg.attempt < msg.max_attempts,

      ...(terminal && {
        sent_to_dlq: true,
        dlq_message_id: `dlq_${msg.delivery_id}`,
        dlq_timestamp: Date.now()
      })
    }
  };

  await env.FACT_LEDGER.appendFact(fact);
}

---

Dead Letter Queue

When all retry attempts fail, webhook deliveries are sent to the Dead Letter Queue for manual intervention.

DLQ Message Schema

interface WebhookDLQMessage {
  delivery_id: string;
  triggered_fact_id: string;
  webhook_config_id: string;
  webhook_config_version: number;

  url: string;
  payload: object;

  error: {
    code: string;
    message: string;
    stack?: string;
    last_status_code?: number;
  };

  attempts: number;
  first_attempt_at: number;
  last_attempt_at: number;
  sent_to_dlq_at: number;

  metadata: {
    tenant_id: string;
    campaign_id?: string;
    contact_id?: string;
    deal_id?: string;
  };
}

DLQ Processing

async function sendToDLQ(
  msg: WebhookDeliveryMessage,
  error: Error,
  env: Env
): Promise<void> {
  const dlqMessage: WebhookDLQMessage = {
    delivery_id: msg.delivery_id,
    triggered_fact_id: msg.triggered_fact_id,
    webhook_config_id: msg.webhook_config_id,
    webhook_config_version: msg.webhook_config_version,

    url: msg.url,
    payload: msg.payload,

    error: {
      code: error.code || 'UNKNOWN',
      message: error.message,
      stack: error.stack,
      last_status_code: error.status
    },

    attempts: msg.max_attempts,
    first_attempt_at: msg.timestamp,
    last_attempt_at: Date.now(),
    sent_to_dlq_at: Date.now(),

    metadata: {
      tenant_id: msg.tenant_id,
      campaign_id: msg.campaign_id
    }
  };

  // Send to DLQ
  await env.WEBHOOK_DLQ.send(dlqMessage);

  // Alert ops team
  await env.ALERTS.send({
    severity: 'warning',
    title: 'Webhook delivery failed after max retries',
    message: `Delivery ${msg.delivery_id} to ${msg.url} failed after ${msg.max_attempts} attempts`,
    metadata: dlqMessage.metadata
  });
}

Manual Replay

Operators can replay failed webhooks from the DLQ:

async function replayFromDLQ(
  dlqMessageId: string,
  env: Env
): Promise<void> {
  // 1. Fetch DLQ message
  const dlqMessage = await env.WEBHOOK_DLQ.getMessage(dlqMessageId);

  // 2. Record replay decision
  await env.FACT_LEDGER.appendFact({
    type: 'lifecycle',
    subtype: 'webhook_replayed',
    timestamp: Date.now(),
    data: {
      delivery_id: dlqMessage.delivery_id,
      dlq_message_id: dlqMessageId,
      reason: 'manual_replay'
    }
  });

  // 3. Re-enqueue (reset attempt to 1)
  await env.WEBHOOK_QUEUE.send({
    ...dlqMessage,
    attempt: 1,
    timestamp: Math.floor(Date.now() / 1000)  // New timestamp for signature
  });
}

---

Idempotency

Recipients must handle duplicate deliveries (same event sent multiple times).

Delivery ID

Each webhook delivery has a unique delivery_id that remains constant across all retry attempts:

delivery_id = del_{triggered_fact_id}_{webhook_config_id}_{timestamp}

Example: del_fact_outcome_001_webhook_001_1705500600

Recipient Deduplication

// Recipient implementation
const processedDeliveries = new Set<string>();

async function handleWebhook(req: Request): Promise<Response> {
  const deliveryId = req.headers.get('x-z0-delivery-id');

  // Check if already processed
  if (processedDeliveries.has(deliveryId)) {
    // Return 200 (idempotent)
    return new Response('Already processed', { status: 200 });
  }

  // Verify signature
  const isValid = await verifyWebhook(
    await req.text(),
    req.headers.get('x-z0-signature'),
    req.headers.get('x-z0-timestamp'),
    process.env.z0_WEBHOOK_SECRET
  );

  if (!isValid) {
    return new Response('Invalid signature', { status: 401 });
  }

  // Process webhook
  await processEvent(await req.json());

  // Mark as processed
  processedDeliveries.add(deliveryId);

  return new Response('OK', { status: 200 });
}

Storage Options:

• In-memory Set (ephemeral, fast)
• Redis/KV (persistent, distributed)
• Database table (permanent audit trail)

TTL: Delivery IDs can be expired after 7 days (webhooks won’t retry beyond that).

---

Webhook Matching

When a Fact is recorded, the system queries for matching Webhook Configs.

Matching Logic

async function findMatchingWebhooks(
  fact: Fact,
  env: Env
): Promise<Config[]> {
  // Query Webhook Configs
  const webhooks = await env.CONFIG_SHARD.query({
    type: 'webhook',
    applies_to: [fact.campaign_id, fact.tenant_id],  // Check both scopes
    superseded_at: null  // Only active configs
  });

  // Filter by trigger
  return webhooks.filter(webhook => {
    const factPattern = `${fact.type}.${fact.subtype}`;
    return webhook.settings.triggers.includes(factPattern);
  });
}

Trigger Patterns

Triggers use exact matching on {type}.{subtype}:

Trigger Pattern	Matches
outcome.deal_won	Fact(outcome, deal_won)
outcome.call_qualified	Fact(outcome, call_qualified)
dispute.opened	Fact(dispute, opened)
dispute.resolved	Fact(dispute, resolved)

Future: Support wildcards (outcome.*, *.opened)

Scope Precedence

Webhook Configs follow standard Config scope precedence (per PRIMITIVES.md):

asset > campaign > account

Example:

• Fact: { campaign_id: campaign_001, tenant_id: ten_abc123 }
• Webhook A: applies_to: campaign_001 (campaign scope)
• Webhook B: applies_to: ten_abc123 (account scope)
• Both fire (no exclusion; all matches trigger)

---

Security

Secret Management

Webhook secrets are never stored in Config settings. They’re referenced by ID and stored in Cloudflare Secrets Manager.

# WRONG: Secret inline
settings:
  secret: "sk_abc123..."  # ❌ Never do this

# RIGHT: Secret reference
settings:
  secret_ref: "webhook_secret_001"  # ✅ Reference only

Secret Storage

// Write secret (admin operation)
await env.SECRETS.put('webhook_secret_001', 'whsec_abc123...');

// Read secret (delivery worker)
const secret = await env.SECRETS.get('webhook_secret_001');

Secret Rotation

async function rotateWebhookSecret(
  webhookId: string,
  env: Env
): Promise<void> {
  // 1. Generate new secret
  const newSecret = generateSecret();
  const newSecretRef = `webhook_secret_${Date.now()}`;

  // 2. Store new secret
  await env.SECRETS.put(newSecretRef, newSecret);

  // 3. Update Webhook Config (creates new version)
  await updateWebhookConfig(webhookId, {
    secret_ref: newSecretRef
  }, env);

  // 4. Return new secret to customer (show once)
  return newSecret;
}

Rotation Process:

1. Generate new secret
2. Store with new reference ID
3. Update Webhook Config (increments version)
4. Old deliveries (in-flight) use old secret (config_version locked)
5. New deliveries use new secret
6. Old secret expires after 7 days

HTTPS Requirement

All webhook URLs must use HTTPS. HTTP is rejected at Config creation:

function validateWebhookURL(url: string): void {
  if (!url.startsWith('https://')) {
    throw new ValidationError('Webhook URL must use HTTPS');
  }

  // Optional: validate domain format
  const urlObj = new URL(url);
  if (!urlObj.hostname.includes('.')) {
    throw new ValidationError('Invalid webhook URL');
  }
}

IP Allowlisting

Recipients can restrict webhook delivery to z0 IP ranges (published at https://web1.co/.well-known/webhook-ips.json):

{
  "webhook_ips": [
    "192.0.2.0/24",
    "198.51.100.0/24"
  ],
  "updated_at": "2026-01-17T00:00:00Z"
}

---

Observability

Metrics

interface WebhookMetrics {
  // Counters
  webhooks_triggered_total: Counter;
  webhooks_sent_total: Counter;
  webhooks_succeeded_total: Counter;
  webhooks_failed_total: Counter;
  webhooks_dlq_total: Counter;

  // Histograms
  webhook_delivery_duration_ms: Histogram;
  webhook_retry_attempts: Histogram;
  webhook_payload_size_bytes: Histogram;

  // Gauges
  webhook_queue_depth: Gauge;
  webhook_dlq_depth: Gauge;
}

Alerts

const webhookAlerts = [
  {
    name: 'high_failure_rate',
    condition: 'webhooks_failed_total / webhooks_sent_total > 0.05',
    severity: 'warning',
    message: 'Webhook failure rate exceeds 5%'
  },
  {
    name: 'dlq_growing',
    condition: 'rate(webhook_dlq_depth[5m]) > 0',
    severity: 'warning',
    message: 'Webhook DLQ is growing'
  },
  {
    name: 'delivery_timeout',
    condition: 'webhook_delivery_duration_ms{quantile="0.95"} > 25000',
    severity: 'info',
    message: 'P95 webhook delivery time exceeds 25s'
  }
];

Dashboard Queries

-- Webhook delivery success rate by config
SELECT
  config_id,
  COUNT(*) AS total,
  SUM(CASE WHEN data.success THEN 1 ELSE 0 END) AS succeeded,
  AVG(data.response_time_ms) AS avg_response_time_ms
FROM facts
WHERE type = 'webhook' AND subtype = 'sent'
  AND timestamp >= NOW() - INTERVAL '24 hours'
GROUP BY config_id;

-- Failed deliveries awaiting retry
SELECT
  config_id,
  data.url,
  data.attempt,
  data.next_retry_at,
  data.error_message
FROM facts
WHERE type = 'webhook' AND subtype = 'sent'
  AND data.success = false
  AND data.will_retry = true
ORDER BY data.next_retry_at;

-- DLQ messages
SELECT
  config_id,
  data.url,
  data.delivery_id,
  data.attempts,
  data.error_message
FROM facts
WHERE type = 'webhook' AND subtype = 'sent'
  AND data.sent_to_dlq = true
ORDER BY timestamp DESC;

---

Anti-Patterns

1. Inline Secrets

Wrong:

settings:
  secret: "whsec_abc123..."  # ❌ Secret in Config

Right:

settings:
  secret_ref: "webhook_secret_001"  # ✅ Reference only

2. HTTP URLs

Wrong:

settings:
  url: "http://example.com/webhook"  # ❌ Insecure

Right:

settings:
  url: "https://example.com/webhook"  # ✅ HTTPS required

3. Unbounded Retries

Wrong:

retry:
  max_attempts: 999  # ❌ Will hammer endpoint

Right:

retry:
  max_attempts: 5    # ✅ Bounded retries
  backoff_max_seconds: 3600

4. Silent Failures

Wrong:

// ❌ Don't swallow errors
try {
  await deliverWebhook(msg);
} catch (error) {
  console.error('Webhook failed');  // Silent failure
}

Right:

// ✅ Record failures as Facts
try {
  await deliverWebhook(msg);
} catch (error) {
  await recordWebhookSent(msg, error, false, env);
  throw error;
}

5. Changing Signature on Retry

Wrong:

// ❌ New timestamp on each retry
const timestamp = Math.floor(Date.now() / 1000);
const signature = await signPayload(payload, timestamp, secret);

Right:

// ✅ Same timestamp for all retries (from triggered Fact)
const timestamp = msg.timestamp;  // Locked at trigger time
const signature = await signPayload(payload, timestamp, secret);

---

Summary

Component	Implementation
Config	Webhook Config (type: webhook, category: logic)
Facts	webhook_triggered, webhook_sent (success/failure)
Queue	Cloudflare Queue for async delivery
Signing	HMAC-SHA256 with timestamp
Retry	Exponential backoff, max 5 attempts (configurable)
DLQ	Dead letter queue for max retries exceeded
Idempotency	Delivery ID in header, recipient deduplication
Security	HTTPS only, secret references, signature verification

The webhook delivery system ensures reliable event delivery to external systems while maintaining full auditability through Facts (Principle 2), Config versioning (Principle 6), and error tracking (Principle 8). Every trigger, attempt, success, and failure is recorded—never silently dropped.

---

References

• PRIMITIVES.md - Entity, Fact, Config schemas
• PRINCIPLES.md - z0 design principles
• error-handling.md - Retry patterns, circuit breakers, DLQ
• queues.md - Cloudflare Queue usage
• notification-delivery.md - Internal notification patterns