SDK Dogfooding (Self-Observability)

Use z0 primitives to observe z0 itself.

Prerequisites: core-concepts.md, SystemLedger

---

Overview

The z0 SDK can use its own primitives to track internal behavior. This is called “dogfooding” - the SDK observes itself using the same patterns developers use for their domains.

When an observabilityStub is provided, SDK components automatically emit Facts about their own behavior:

• CachedStateManager: Cache hit/miss rates
• SchemaManager: Schema initialization events (future)
• HydrationManager: Replay completion events (future)

This provides platform-wide observability without special-case code. Observability is enabled automatically when you wire the stub - no mode toggle required.

---

Architecture

Single-Tenant Deployment

                 SystemLedger (id: 'system')
                 ├── Receives: SDK observability facts
                 └── Uses: appendFact(), same as any ledger
                            │
              ┌─────────────┼─────────────┐
              ▼             ▼             ▼
        EntityLedger   EntityLedger   EntityLedger
        (your domain)  (your domain)  (your domain)
              │
              └── emitSdkObservability() ──→ SystemLedger

Each EntityLedger can emit observability facts upstream to SystemLedger. This uses the same ParentDOClient pattern as config inheritance - fire-and-forget with circuit breaker protection.

Workers for Platforms (Multi-Tenant SaaS)

For multi-tenant deployments using Cloudflare Workers for Platforms, the architecture extends to support isolated customer namespaces:

┌─────────────────────────────────────────────────────────────────┐
│                    Platform Provider Account                     │
│                                                                  │
│   SystemLedger (platform-wide observability)                    │
│        │                                                         │
│        ├── Aggregates SDK facts from all tenants                │
│        └── Platform-level dashboards & alerts                   │
│                    │                                             │
├────────────────────┼────────────────────────────────────────────┤
│                    │  Workers for Platforms (Dispatch Namespace) │
│                    ▼                                             │
│   ┌────────────────────────────────────────────────────────┐    │
│   │  Customer A Namespace                                   │    │
│   │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐    │    │
│   │  │EntityLedger │  │EntityLedger │  │EntityLedger │    │    │
│   │  │  (account)  │  │  (project)  │  │  (invoice)  │    │    │
│   │  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘    │    │
│   │         └────────────────┴─────────────────┘           │    │
│   │                          │                              │    │
│   │                          ▼                              │    │
│   │               TenantSystemLedger                        │    │
│   │         (customer's observability facts)               │    │
│   └────────────────────────────────────────────────────────┘    │
│                                                                  │
│   ┌────────────────────────────────────────────────────────┐    │
│   │  Customer B Namespace                                   │    │
│   │  ... (same structure)                                   │    │
│   └────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────┘

Key patterns:

• Each customer gets their own Durable Object namespace (isolation)
• TenantSystemLedger aggregates observability within a customer
• Platform SystemLedger aggregates across all customers
• Facts flow up the hierarchy: Entity → Tenant → Platform

TenantSystemLedger

TenantSystemLedger extends SystemLedger with tenant-specific aggregation:

import { TenantSystemLedger, TenantSystemLedgerEnv } from '@z0-app/sdk';

export class MyTenantSystemLedger extends TenantSystemLedger {
  constructor(ctx: DurableObjectState, env: TenantSystemLedgerEnv) {
    super(ctx, env, {
      aggregationIntervalMs: 60_000 // Flush every minute (default)
    });
  }
}

Features:

• Receives SDK observability facts from tenant’s EntityLedgers
• Aggregates stats over time (reduces write amplification)
• Flushes to platform via PLATFORM_METRICS_QUEUE or PLATFORM_SYSTEM_LEDGER
• Uses DO alarms for periodic flushing

Recursive Entity Hierarchy

Entities can contain child entities, forming a recursive tree structure. Observability facts propagate up this hierarchy:

Platform (entity)
├── Org (entity)
│   ├── Tenant (entity)
│   │   ├── Project (entity)
│   │   │   ├── Asset (entity)
│   │   │   └── Asset (entity)
│   │   └── Project (entity)
│   └── Tenant (entity)
└── Org (entity)

Each entity can:

1. Emit facts about its own state changes
2. Receive facts from child entities (rollup aggregation)
3. Forward facts to parent entities (bubbling)

// Example: Project entity receives Asset facts and aggregates
class ProjectLedger extends EntityLedger<Env> {
  protected async onChildFact(childId: string, fact: Fact): Promise<void> {
    if (fact.type === 'asset' && fact.subtype === 'processed') {
      // Aggregate asset processing into project metrics
      await this.appendFact({
        type: 'project',
        subtype: 'asset_rollup',
        data: {
          asset_id: childId,
          processing_time_ms: fact.data.duration_ms
        }
      });
    }
  }
}

Hierarchy benefits:

• Natural isolation boundaries (tenant can’t see other tenant’s facts)
• Automatic aggregation (platform sees total, tenant sees their total)
• Consistent patterns at every level (same SDK, same primitives)

---

Enabling Observability

Observability is automatically enabled when you provide an observabilityStub. No mode toggle required.

Wire the Observability Stub

Pass the stub when creating EntityLedger instances:

import { EntityLedger, BootstrapConfig } from '@z0-app/sdk';

// Optional: Configure fallback behavior
const bootstrapConfig: BootstrapConfig = {
  fallback_on_failure: true, // Continue if observability target unavailable (default: true)
  warn_on_fallback: true     // Log warnings on fallback (default: true)
};

// In your DO class
export class AccountLedger extends EntityLedger<Env> {
  constructor(ctx: DurableObjectState, env: Env) {
    super(ctx, env, {
      // Wire the observability target - this enables SDK observability automatically
      observabilityStub: env.SYSTEM_LEDGER.get(env.SYSTEM_LEDGER.idFromName('system')),
      bootstrapConfig // Optional
    });
  }
}

wrangler.toml Configuration

[[durable_objects.bindings]]
name = "SYSTEM_LEDGER"
class_name = "SystemLedger"

[[durable_objects.bindings]]
name = "ACCOUNT_LEDGER"
class_name = "AccountLedger"

worker.ts

export { SystemLedger, AccountLedger } from './ledgers';

export default {
  async fetch(request: Request, env: Env) {
    // Your routing logic...
  }
};

Disabling Observability

Simply omit the observabilityStub option. The SDK operates normally without emitting observability facts.

---

SDK Observability Facts

Facts emitted by SDK components use type: 'sdk' with these subtypes:

cache_stats

Emitted by CachedStateManager on interval (default: 60s):

{
  type: 'sdk',
  subtype: 'cache_stats',
  tenant_id: 'entity_tenant_id',
  data: {
    entity_id: 'account_123',
    entity_type: 'account',
    period_start: 1700000000000,
    period_end: 1700000060000,
    hits: 42,
    misses: 8
  }
}

schema_init (future)

Emitted when SchemaManager initializes tables:

{
  type: 'sdk',
  subtype: 'schema_init',
  data: {
    entity_id: 'account_123',
    entity_type: 'account',
    tables_created: ['facts', 'entities', 'configs'],
    duration_ms: 15
  }
}

hydration_done (future)

Emitted when HydrationManager completes replay:

{
  type: 'sdk',
  subtype: 'hydration_done',
  data: {
    entity_id: 'account_123',
    entity_type: 'account',
    facts_replayed: 1523,
    duration_ms: 450,
    source: 'r2'
  }
}

---

Querying SDK Facts

Use SystemLedger methods to query observability data:

const systemLedger = await getSystemLedger();

// Get all SDK facts for an entity
const facts = systemLedger.getSdkFacts('account_123');

// Get cache stats specifically
const cacheStats = systemLedger.getCacheStats('account_123');
// Returns: CacheStatsData[]

// Get platform-wide stats
const stats = systemLedger.getStats();
// Returns: { tenant_count, entity_count, sdk_facts_count }

---

Fallback Behavior

When fallback_on_failure: true (default):

• If the observability target is unavailable, SDK continues operating
• Observability facts are dropped silently (or with warning if warn_on_fallback: true)
• No impact on primary functionality

When fallback_on_failure: false:

• Errors propagate if observability target fails
• Use only when observability is critical

---

Aggregation

To prevent write amplification, CachedStateManager aggregates stats:

Config	Default	Description
min_interval_ms	60000 (1 min)	Minimum time between flushes

Stats accumulate in memory and flush when:

1. Interval expires (checked on each cache operation)
2. flushStats() called explicitly
3. Stats tracking disabled

// Custom interval
this.cachedStateManager.enableStatsTracking((stats) => {
  this.emitSdkObservability('cache_stats', stats);
}, { min_interval_ms: 30_000 }); // 30 seconds

---

Performance Considerations

Overhead

Operation	Overhead
Cache get/set	~1 counter increment (in-memory)
Stats flush	1 HTTP call to SystemLedger (async, fire-and-forget)
Circuit breaker	Prevents thundering herd on SystemLedger failure

Scaling

SystemLedger receives facts from all EntityLedgers. At scale:

1. Aggregation - Stats batched to 1 fact per minute per entity
2. Fire-and-forget - EntityLedgers don’t wait for response
3. Circuit breaker - Protects SystemLedger from overload

For very high scale, consider:

• Longer aggregation intervals
• Sampling (emit stats for subset of entities)
• Sharding SystemLedger by tenant (future)

---

Example: Full Setup

Single-Tenant Setup

ledgers/system.ts

import { SystemLedger } from '@z0-app/sdk';
export { SystemLedger };

// ledgers/account.ts
import { EntityLedger, LedgerOptions } from '@z0-app/sdk';

interface AccountEnv {
  SYSTEM_LEDGER: DurableObjectNamespace;
}

export class AccountLedger extends EntityLedger<AccountEnv> {
  constructor(ctx: DurableObjectState, env: AccountEnv) {
    const options: LedgerOptions = {
      // This enables SDK observability automatically
      observabilityStub: env.SYSTEM_LEDGER.get(
        env.SYSTEM_LEDGER.idFromName('system')
      ),
      bootstrapConfig: {
        fallback_on_failure: true,
        warn_on_fallback: true
      }
    };
    super(ctx, env, options);
  }

  protected async updateCachedState(fact: Fact): Promise<void> {
    // Your domain logic...
    // CachedStateManager automatically tracks hits/misses
  }
}

Multi-Tenant Setup (Workers for Platforms)

For multi-tenant deployments, use TenantSystemLedger to aggregate per-tenant:

ledgers/tenant-system.ts

import { TenantSystemLedger, TenantSystemLedgerEnv } from '@z0-app/sdk';
export { TenantSystemLedger };

// ledgers/account.ts
import { EntityLedger, LedgerOptions } from '@z0-app/sdk';

interface TenantEnv extends TenantSystemLedgerEnv {
  TENANT_SYSTEM_LEDGER: DurableObjectNamespace;
  TENANT_ID: string;
}

export class AccountLedger extends EntityLedger<TenantEnv> {
  constructor(ctx: DurableObjectState, env: TenantEnv) {
    const options: LedgerOptions = {
      // Point to TenantSystemLedger instead of SystemLedger
      observabilityStub: env.TENANT_SYSTEM_LEDGER.get(
        env.TENANT_SYSTEM_LEDGER.idFromName(env.TENANT_ID)
      ),
    };
    super(ctx, env, options);
  }
}

# wrangler.toml (tenant worker)
[[durable_objects.bindings]]
name = "TENANT_SYSTEM_LEDGER"
class_name = "TenantSystemLedger"

[[durable_objects.bindings]]
name = "ACCOUNT_LEDGER"
class_name = "AccountLedger"

# Environment variable set per-tenant by Workers for Platforms
[vars]
TENANT_ID = ""  # Set dynamically via dispatch binding

---

Types Reference

// Bootstrap configuration (optional)
interface BootstrapConfig {
  fallback_on_failure: boolean;  // Default: true
  warn_on_fallback?: boolean;    // Default: true
}

// SDK observability fact
interface SdkObservabilityFact {
  type: 'sdk';
  subtype: SdkObservabilitySubtype;
  entity_id: string;
  entity_type: string;
  tenant_id: string;
  data: SdkObservabilityData;
}

type SdkObservabilitySubtype =
  | 'cache_stats'
  | 'fact_appended'
  | 'schema_init'
  | 'hydration_done';

interface CacheStatsData {
  period_start: number;
  period_end: number;
  hits: number;
  misses: number;
}

// Aggregated tenant stats (from TenantSystemLedger)
interface TenantStatsPayload {
  tenant_id: string;
  period_start: number;
  period_end: number;
  cache_hits: number;
  cache_misses: number;
  facts_count: number;
  entities_count: number;
}

---

Summary

Concept	Implementation
What it is	SDK observing itself via z0 primitives
Why	Battle-test the SDK, platform observability
How to enable	Pass observabilityStub option (auto-enables)
Current support	CachedStateManager hit/miss tracking
Future support	SchemaManager, HydrationManager, FactManager
Default	Disabled (no stub = no observability)

The SDK dogfooding architecture demonstrates a key principle: the same patterns that work for your domain work for the platform itself.