Reorganize docs: move specs and plans to doc/ subdirectories
Move doc/specs/ui.md to doc/spec/ui.md. Move plans/module-system.md to doc/plans/. Add doc/spec/agents-runtime.md and docs/ reference specs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Forgotten
685
doc/plans/module-system.md
Normal file
685
doc/plans/module-system.md
Normal file
@@ -0,0 +1,685 @@
|
||||
# Paperclip Module System
|
||||
|
||||
## Overview
|
||||
|
||||
Paperclip's module system lets you extend the control plane with new capabilities — revenue tracking, observability, notifications, dashboards — without forking core. Modules are self-contained packages that register routes, UI pages, database tables, and lifecycle hooks.
|
||||
|
||||
Separately, **Company Templates** are code-free data packages (agent teams, org charts, goal hierarchies) that you can import to bootstrap a new company.
|
||||
|
||||
Both are discoverable through the **Company Store**.
|
||||
|
||||
---
|
||||
|
||||
## Concepts
|
||||
|
||||
| Concept | What it is | Contains code? |
|
||||
|---------|-----------|----------------|
|
||||
| **Module** | A package that extends Paperclip's API, UI, and data model | Yes |
|
||||
| **Company Template** | A data snapshot — agents, projects, goals, org structure | No (JSON only) |
|
||||
| **Company Store** | Registry for browsing/installing modules and templates | — |
|
||||
| **Hook** | A named event in the core that modules can subscribe to | — |
|
||||
| **Slot** | An exclusive category where only one module can be active (e.g., `observability`) | — |
|
||||
|
||||
---
|
||||
|
||||
## Module Architecture
|
||||
|
||||
### File Structure
|
||||
|
||||
```
|
||||
modules/
|
||||
observability/
|
||||
paperclip.module.json # manifest (required)
|
||||
src/
|
||||
index.ts # entry point — exports register function
|
||||
routes.ts # Express router
|
||||
hooks.ts # hook handlers
|
||||
schema.ts # Drizzle table definitions
|
||||
migrations/ # SQL migrations (generated by drizzle-kit)
|
||||
ui/ # React components (lazy-loaded by the shell)
|
||||
index.ts # exports page/widget definitions
|
||||
TokenDashboard.tsx
|
||||
```
|
||||
|
||||
Modules live in a top-level `modules/` directory. Each module is a pnpm workspace package.
|
||||
|
||||
### Manifest (`paperclip.module.json`)
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "observability",
|
||||
"name": "Observability",
|
||||
"description": "Token tracking, cost metrics, and agent performance instrumentation",
|
||||
"version": "0.1.0",
|
||||
"author": "paperclip",
|
||||
|
||||
"slot": "observability",
|
||||
|
||||
"hooks": [
|
||||
"agent:heartbeat",
|
||||
"agent:created",
|
||||
"issue:status_changed",
|
||||
"budget:threshold_crossed"
|
||||
],
|
||||
|
||||
"routes": {
|
||||
"prefix": "/observability",
|
||||
"entry": "./src/routes.ts"
|
||||
},
|
||||
|
||||
"ui": {
|
||||
"pages": [
|
||||
{
|
||||
"path": "/observability",
|
||||
"label": "Observability",
|
||||
"entry": "./src/ui/index.ts"
|
||||
}
|
||||
],
|
||||
"widgets": [
|
||||
{
|
||||
"id": "token-burn-rate",
|
||||
"label": "Token Burn Rate",
|
||||
"placement": "dashboard",
|
||||
"entry": "./src/ui/index.ts"
|
||||
}
|
||||
]
|
||||
},
|
||||
|
||||
"schema": "./src/schema.ts",
|
||||
|
||||
"configSchema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"retentionDays": { "type": "number", "default": 30 },
|
||||
"enablePrometheus": { "type": "boolean", "default": false },
|
||||
"prometheusPort": { "type": "number", "default": 9090 }
|
||||
}
|
||||
},
|
||||
|
||||
"requires": {
|
||||
"core": ">=0.1.0"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Key fields:
|
||||
|
||||
- **`id`**: Unique identifier, used as the npm package name suffix (`@paperclip/mod-observability`)
|
||||
- **`slot`**: Optional exclusive category. If set, only one module with this slot can be active. Omit for modules that can coexist freely.
|
||||
- **`hooks`**: Which core events this module subscribes to. Declared upfront so the core knows what to emit.
|
||||
- **`routes.prefix`**: Mounted under `/api/modules/<prefix>`. The module owns this namespace.
|
||||
- **`ui.pages`**: Adds entries to the sidebar. Lazy-loaded React components.
|
||||
- **`ui.widgets`**: Injects components into existing pages (e.g., dashboard cards).
|
||||
- **`schema`**: Drizzle table definitions for module-owned tables. Prefixed with `mod_<id>_` to avoid collisions.
|
||||
- **`configSchema`**: JSON Schema for module configuration. Validated before the module loads.
|
||||
|
||||
### Entry Point
|
||||
|
||||
The module's `src/index.ts` exports a `register` function that receives the module API:
|
||||
|
||||
```typescript
|
||||
import type { ModuleAPI } from "@paperclip/core";
|
||||
import { createRouter } from "./routes.js";
|
||||
import { onHeartbeat, onBudgetThreshold } from "./hooks.js";
|
||||
|
||||
export default function register(api: ModuleAPI) {
|
||||
// Register route handler
|
||||
api.registerRoutes(createRouter(api.db, api.config));
|
||||
|
||||
// Subscribe to hooks
|
||||
api.on("agent:heartbeat", onHeartbeat);
|
||||
api.on("budget:threshold_crossed", onBudgetThreshold);
|
||||
|
||||
// Register a background service (optional)
|
||||
api.registerService({
|
||||
name: "metrics-aggregator",
|
||||
interval: 60_000, // run every 60s
|
||||
async run(ctx) {
|
||||
await aggregateMetrics(ctx.db);
|
||||
},
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### Module API Surface
|
||||
|
||||
```typescript
|
||||
interface ModuleAPI {
|
||||
// Identity
|
||||
moduleId: string;
|
||||
config: Record<string, unknown>; // validated against configSchema
|
||||
|
||||
// Database
|
||||
db: Db; // shared Drizzle client
|
||||
|
||||
// Routes
|
||||
registerRoutes(router: Router): void;
|
||||
|
||||
// Hooks
|
||||
on(event: HookEvent, handler: HookHandler): void;
|
||||
|
||||
// Background services
|
||||
registerService(service: ServiceDef): void;
|
||||
|
||||
// Logging (scoped to module)
|
||||
logger: Logger;
|
||||
|
||||
// Access core services (read-only helpers)
|
||||
core: {
|
||||
agents: AgentService;
|
||||
issues: IssueService;
|
||||
projects: ProjectService;
|
||||
goals: GoalService;
|
||||
activity: ActivityService;
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
Modules get a scoped logger, access to the shared database, and read access to core services. They register their own routes and hook handlers. They do NOT monkey-patch core — they extend through defined interfaces.
|
||||
|
||||
---
|
||||
|
||||
## Hook System
|
||||
|
||||
### Core Hook Points
|
||||
|
||||
Hooks are the primary integration point. The core emits events at well-defined moments. Modules subscribe in their `register` function.
|
||||
|
||||
| Hook | Payload | When |
|
||||
|------|---------|------|
|
||||
| `server:started` | `{ port }` | After the Express server begins listening |
|
||||
| `agent:created` | `{ agent }` | After a new agent is inserted |
|
||||
| `agent:updated` | `{ agent, changes }` | After an agent record is modified |
|
||||
| `agent:deleted` | `{ agent }` | After an agent is removed |
|
||||
| `agent:heartbeat` | `{ agentId, timestamp, meta }` | When an agent checks in. `meta` carries tokens_used, cost, latency, etc. |
|
||||
| `agent:status_changed` | `{ agent, from, to }` | When agent status transitions (idle→active, active→error, etc.) |
|
||||
| `issue:created` | `{ issue }` | After a new issue is inserted |
|
||||
| `issue:status_changed` | `{ issue, from, to }` | When issue moves between statuses |
|
||||
| `issue:assigned` | `{ issue, agent }` | When an issue is assigned to an agent |
|
||||
| `goal:created` | `{ goal }` | After a new goal is inserted |
|
||||
| `goal:completed` | `{ goal }` | When a goal's status becomes complete |
|
||||
| `budget:spend_recorded` | `{ agentId, amount, total }` | After spend is incremented |
|
||||
| `budget:threshold_crossed` | `{ agentId, budget, spent, percent }` | When an agent crosses 80%, 90%, or 100% of budget |
|
||||
|
||||
### Hook Execution Model
|
||||
|
||||
```typescript
|
||||
// In the core — hook emitter
|
||||
class HookBus {
|
||||
private handlers = new Map<string, HookHandler[]>();
|
||||
|
||||
register(event: string, handler: HookHandler) {
|
||||
const list = this.handlers.get(event) ?? [];
|
||||
list.push(handler);
|
||||
this.handlers.set(event, list);
|
||||
}
|
||||
|
||||
async emit(event: string, payload: unknown) {
|
||||
const handlers = this.handlers.get(event) ?? [];
|
||||
// Run all handlers concurrently. Failures are logged, never block core.
|
||||
await Promise.allSettled(
|
||||
handlers.map(h => h(payload))
|
||||
);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Design rules:
|
||||
- **Hooks are fire-and-forget.** A failing hook handler never crashes or blocks the core operation.
|
||||
- **Hooks are concurrent.** All handlers for an event run in parallel via `Promise.allSettled`.
|
||||
- **Hooks are post-commit.** They fire after the database write succeeds, not before. No vetoing.
|
||||
- **Hooks receive immutable snapshots.** Handlers get a copy of the data, not a mutable reference.
|
||||
|
||||
This keeps the core fast and resilient. If you need pre-commit validation (e.g., "reject this budget change"), that's a different mechanism (middleware/interceptor) we can add later if needed.
|
||||
|
||||
### Observability Hook Example
|
||||
|
||||
```typescript
|
||||
// modules/observability/src/hooks.ts
|
||||
import type { Db } from "@paperclip/db";
|
||||
import { tokenMetrics } from "./schema.js";
|
||||
|
||||
export function createHeartbeatHandler(db: Db) {
|
||||
return async (payload: {
|
||||
agentId: string;
|
||||
timestamp: Date;
|
||||
meta: { tokensUsed?: number; costCents?: number; model?: string };
|
||||
}) => {
|
||||
const { agentId, timestamp, meta } = payload;
|
||||
|
||||
if (meta.tokensUsed != null) {
|
||||
await db.insert(tokenMetrics).values({
|
||||
agentId,
|
||||
tokensUsed: meta.tokensUsed,
|
||||
costCents: meta.costCents ?? 0,
|
||||
model: meta.model ?? "unknown",
|
||||
recordedAt: timestamp,
|
||||
});
|
||||
}
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
Every heartbeat, the observability module records token usage into its own `mod_observability_token_metrics` table. The core doesn't know or care about this table — it just emits the hook.
|
||||
|
||||
---
|
||||
|
||||
## Database Strategy for Modules
|
||||
|
||||
### Table Namespacing
|
||||
|
||||
Module tables are prefixed with `mod_<moduleId>_` to avoid collisions with core tables and other modules:
|
||||
|
||||
```typescript
|
||||
// modules/observability/src/schema.ts
|
||||
import { pgTable, uuid, integer, text, timestamp } from "drizzle-orm/pg-core";
|
||||
|
||||
export const tokenMetrics = pgTable("mod_observability_token_metrics", {
|
||||
id: uuid("id").primaryKey().defaultRandom(),
|
||||
agentId: uuid("agent_id").notNull(),
|
||||
tokensUsed: integer("tokens_used").notNull(),
|
||||
costCents: integer("cost_cents").notNull().default(0),
|
||||
model: text("model").notNull(),
|
||||
recordedAt: timestamp("recorded_at", { withTimezone: true }).notNull().defaultNow(),
|
||||
});
|
||||
|
||||
export const alertRules = pgTable("mod_observability_alert_rules", {
|
||||
id: uuid("id").primaryKey().defaultRandom(),
|
||||
agentId: uuid("agent_id"),
|
||||
metricName: text("metric_name").notNull(),
|
||||
threshold: integer("threshold").notNull(),
|
||||
enabled: boolean("enabled").notNull().default(true),
|
||||
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
|
||||
});
|
||||
```
|
||||
|
||||
### Migration Strategy
|
||||
|
||||
Each module manages its own migrations in `src/migrations/`. The core migration runner discovers and applies them:
|
||||
|
||||
1. Core migrations run first (always)
|
||||
2. Module migrations run in dependency order
|
||||
3. Each module's migrations are tracked in a `mod_migrations` table with the module ID
|
||||
4. `pnpm db:migrate` runs everything. `pnpm db:migrate --module observability` runs one.
|
||||
|
||||
Modules can reference core tables via foreign keys (e.g., `agent_id → agents.id`) but core tables never reference module tables. This is a strict one-way dependency.
|
||||
|
||||
---
|
||||
|
||||
## Module Loading & Lifecycle
|
||||
|
||||
### Discovery
|
||||
|
||||
On server startup:
|
||||
|
||||
```
|
||||
1. Scan modules/ directory for paperclip.module.json manifests
|
||||
2. Validate each manifest (JSON Schema check on configSchema, required fields)
|
||||
3. Check slot conflicts (error if two active modules claim the same slot)
|
||||
4. Topological sort by dependencies (if module A requires module B)
|
||||
5. For each module in order:
|
||||
a. Validate module config against configSchema
|
||||
b. Run pending migrations
|
||||
c. Import entry point and call register(api)
|
||||
d. Mount routes at /api/modules/<prefix>
|
||||
e. Start background services
|
||||
6. Emit server:started hook
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Module config lives in the server's environment or a config file:
|
||||
|
||||
```jsonc
|
||||
// paperclip.config.json (or env vars)
|
||||
{
|
||||
"modules": {
|
||||
"enabled": ["observability", "revenue", "notifications"],
|
||||
"config": {
|
||||
"observability": {
|
||||
"retentionDays": 90,
|
||||
"enablePrometheus": true
|
||||
},
|
||||
"revenue": {
|
||||
"stripeSecretKey": "$STRIPE_SECRET_KEY"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`$ENV_VAR` references are resolved at load time. Secrets never go in the config file directly.
|
||||
|
||||
### Disabling a Module
|
||||
|
||||
Setting a module's enabled state to false:
|
||||
1. Stops its background services
|
||||
2. Unmounts its routes (returns 404)
|
||||
3. Unsubscribes its hook handlers
|
||||
4. Does NOT drop its database tables (data is preserved)
|
||||
|
||||
---
|
||||
|
||||
## UI Integration
|
||||
|
||||
### How Module UI Works
|
||||
|
||||
The core UI shell provides:
|
||||
- A sidebar with slots for module-contributed nav items
|
||||
- A dashboard with widget mount points
|
||||
- A module settings page
|
||||
|
||||
Modules declare pages and widgets in the manifest. The shell lazy-loads them:
|
||||
|
||||
```typescript
|
||||
// ui/src/modules/loader.ts
|
||||
// At build time or runtime, discover module UI entries and create lazy routes
|
||||
|
||||
import { lazy } from "react";
|
||||
|
||||
// Generated from manifests
|
||||
export const modulePages = [
|
||||
{
|
||||
path: "/observability",
|
||||
label: "Observability",
|
||||
component: lazy(() => import("@paperclip/mod-observability/ui")),
|
||||
},
|
||||
];
|
||||
|
||||
export const dashboardWidgets = [
|
||||
{
|
||||
id: "token-burn-rate",
|
||||
label: "Token Burn Rate",
|
||||
placement: "dashboard",
|
||||
component: lazy(() => import("@paperclip/mod-observability/ui").then(m => ({ default: m.TokenBurnRateWidget }))),
|
||||
},
|
||||
];
|
||||
```
|
||||
|
||||
### Module UI Contract
|
||||
|
||||
A module's UI entry exports named components:
|
||||
|
||||
```typescript
|
||||
// modules/observability/src/ui/index.ts
|
||||
export { default } from "./ObservabilityPage";
|
||||
export { TokenBurnRateWidget } from "./TokenBurnRateWidget";
|
||||
```
|
||||
|
||||
Module UI components receive a standard props interface:
|
||||
|
||||
```typescript
|
||||
interface ModulePageProps {
|
||||
moduleId: string;
|
||||
config: Record<string, unknown>;
|
||||
}
|
||||
|
||||
interface ModuleWidgetProps {
|
||||
moduleId: string;
|
||||
config: Record<string, unknown>;
|
||||
className?: string;
|
||||
}
|
||||
```
|
||||
|
||||
Module UI hits the module's own API routes (`/api/modules/observability/*`) for data.
|
||||
|
||||
---
|
||||
|
||||
## Company Templates
|
||||
|
||||
### Format
|
||||
|
||||
A company template is a JSON file describing a full company structure:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "startup-in-a-box",
|
||||
"name": "Startup in a Box",
|
||||
"description": "A 5-agent startup team with engineering, product, and ops",
|
||||
"version": "1.0.0",
|
||||
"author": "paperclip",
|
||||
|
||||
"agents": [
|
||||
{
|
||||
"ref": "ceo",
|
||||
"name": "CEO Agent",
|
||||
"role": "pm",
|
||||
"budgetCents": 100000,
|
||||
"metadata": { "responsibilities": "Strategy, fundraising, hiring" }
|
||||
},
|
||||
{
|
||||
"ref": "eng-lead",
|
||||
"name": "Engineering Lead",
|
||||
"role": "engineer",
|
||||
"reportsTo": "ceo",
|
||||
"budgetCents": 50000
|
||||
},
|
||||
{
|
||||
"ref": "eng-1",
|
||||
"name": "Engineer",
|
||||
"role": "engineer",
|
||||
"reportsTo": "eng-lead",
|
||||
"budgetCents": 30000
|
||||
},
|
||||
{
|
||||
"ref": "designer",
|
||||
"name": "Designer",
|
||||
"role": "designer",
|
||||
"reportsTo": "ceo",
|
||||
"budgetCents": 20000
|
||||
},
|
||||
{
|
||||
"ref": "ops",
|
||||
"name": "Ops Agent",
|
||||
"role": "devops",
|
||||
"reportsTo": "ceo",
|
||||
"budgetCents": 20000
|
||||
}
|
||||
],
|
||||
|
||||
"goals": [
|
||||
{
|
||||
"ref": "north-star",
|
||||
"title": "Launch MVP",
|
||||
"level": "company"
|
||||
},
|
||||
{
|
||||
"ref": "build-product",
|
||||
"title": "Build the product",
|
||||
"level": "team",
|
||||
"parentRef": "north-star",
|
||||
"ownerRef": "eng-lead"
|
||||
},
|
||||
{
|
||||
"ref": "design-brand",
|
||||
"title": "Establish brand identity",
|
||||
"level": "agent",
|
||||
"parentRef": "north-star",
|
||||
"ownerRef": "designer"
|
||||
}
|
||||
],
|
||||
|
||||
"projects": [
|
||||
{
|
||||
"ref": "mvp",
|
||||
"name": "MVP",
|
||||
"description": "The first shippable version"
|
||||
}
|
||||
],
|
||||
|
||||
"issues": [
|
||||
{
|
||||
"title": "Set up CI/CD pipeline",
|
||||
"status": "todo",
|
||||
"priority": "high",
|
||||
"projectRef": "mvp",
|
||||
"assigneeRef": "ops",
|
||||
"goalRef": "build-product"
|
||||
},
|
||||
{
|
||||
"title": "Design landing page",
|
||||
"status": "todo",
|
||||
"priority": "medium",
|
||||
"projectRef": "mvp",
|
||||
"assigneeRef": "designer",
|
||||
"goalRef": "design-brand"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Templates use `ref` strings (not UUIDs) for internal cross-references. On import, the system maps refs to generated UUIDs.
|
||||
|
||||
### Import Flow
|
||||
|
||||
```
|
||||
1. Parse and validate the template JSON
|
||||
2. Check for ref uniqueness and dangling references
|
||||
3. Insert agents (topological sort by reportsTo)
|
||||
4. Insert goals (topological sort by parentRef)
|
||||
5. Insert projects
|
||||
6. Insert issues (resolve projectRef, assigneeRef, goalRef to real IDs)
|
||||
7. Log activity events for everything created
|
||||
```
|
||||
|
||||
### Export Flow
|
||||
|
||||
You can also export a running company as a template:
|
||||
|
||||
```
|
||||
GET /api/templates/export → downloads the current company as a template JSON
|
||||
```
|
||||
|
||||
This makes companies shareable and clonable.
|
||||
|
||||
---
|
||||
|
||||
## Company Store
|
||||
|
||||
The Company Store is a registry for discovering and installing modules and templates. For v1, it's a curated GitHub repo with a JSON index. Later it could become a hosted service.
|
||||
|
||||
### Index Format
|
||||
|
||||
```json
|
||||
{
|
||||
"modules": [
|
||||
{
|
||||
"id": "observability",
|
||||
"name": "Observability",
|
||||
"description": "Token tracking, cost metrics, and agent performance",
|
||||
"repo": "github:paperclip/mod-observability",
|
||||
"version": "0.1.0",
|
||||
"tags": ["metrics", "monitoring", "tokens"]
|
||||
}
|
||||
],
|
||||
"templates": [
|
||||
{
|
||||
"id": "startup-in-a-box",
|
||||
"name": "Startup in a Box",
|
||||
"description": "5-agent startup team",
|
||||
"url": "https://store.paperclip.dev/templates/startup-in-a-box.json",
|
||||
"tags": ["startup", "team"]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### CLI Commands
|
||||
|
||||
```bash
|
||||
pnpm paperclip store list # browse available modules and templates
|
||||
pnpm paperclip store install <module-id> # install a module
|
||||
pnpm paperclip store import <template-id> # import a company template
|
||||
pnpm paperclip store export # export current company as template
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Module Candidates
|
||||
|
||||
### Tier 1 — Build first (core extensions)
|
||||
|
||||
| Module | What it does | Key hooks |
|
||||
|--------|-------------|-----------|
|
||||
| **Observability** | Token usage tracking, cost metrics, agent performance dashboards, Prometheus export | `agent:heartbeat`, `budget:spend_recorded` |
|
||||
| **Revenue Tracking** | Connect Stripe/crypto wallets, track income, show P&L against agent costs | `budget:spend_recorded` |
|
||||
| **Notifications** | Slack/Discord/email alerts on configurable triggers | All hooks (configurable) |
|
||||
|
||||
### Tier 2 — High value
|
||||
|
||||
| Module | What it does | Key hooks |
|
||||
|--------|-------------|-----------|
|
||||
| **Analytics Dashboard** | Burn rate trends, agent utilization over time, goal velocity charts | `agent:heartbeat`, `issue:status_changed`, `goal:completed` |
|
||||
| **Workflow Automation** | If/then rules: "when issue is done, create follow-up", "when budget at 90%, pause agent" | `issue:status_changed`, `budget:threshold_crossed` |
|
||||
| **Knowledge Base** | Shared document store, vector search, agents read/write organizational knowledge | `agent:heartbeat` (for context injection) |
|
||||
|
||||
### Tier 3 — Nice to have
|
||||
|
||||
| Module | What it does | Key hooks |
|
||||
|--------|-------------|-----------|
|
||||
| **Audit & Compliance** | Immutable audit trail, approval workflows, spend authorization | All write hooks |
|
||||
| **Agent Logs / Replay** | Full execution traces per agent, token-by-token replay | `agent:heartbeat` |
|
||||
| **Multi-tenant** | Separate companies/orgs within one Paperclip instance | `server:started` |
|
||||
|
||||
---
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Core infrastructure
|
||||
|
||||
Add to `@paperclip/server`:
|
||||
|
||||
1. **HookBus** — Event emitter with `register()` and `emit()`, using `Promise.allSettled`
|
||||
2. **Module loader** — Scans `modules/`, validates manifests, calls `register(api)`
|
||||
3. **Module API object** — `registerRoutes()`, `on()`, `registerService()`, logger, core service access
|
||||
4. **Module config** — `paperclip.config.json` with per-module config, env var interpolation
|
||||
5. **Module migration runner** — Extends `db:migrate` to discover and run module migrations
|
||||
6. **Emit hooks from core services** — Add `hookBus.emit()` calls to existing CRUD operations
|
||||
|
||||
Add to `@paperclip/ui`:
|
||||
|
||||
7. **Module page loader** — Reads module manifests, generates lazy routes
|
||||
8. **Dashboard widget slots** — Render module-contributed widgets on the Dashboard page
|
||||
9. **Sidebar extension** — Dynamically add module nav items
|
||||
|
||||
Add new package:
|
||||
|
||||
10. **`@paperclip/module-sdk`** — TypeScript types for `ModuleAPI`, `HookEvent`, `HookHandler`, manifest schema
|
||||
|
||||
### Phase 2: First module (observability)
|
||||
|
||||
11. Build `modules/observability` as the reference implementation
|
||||
12. Token metrics table + migration
|
||||
13. Heartbeat hook handler recording token usage
|
||||
14. Dashboard widget showing burn rate
|
||||
15. API routes for querying metrics
|
||||
|
||||
### Phase 3: Templates
|
||||
|
||||
16. Template import endpoint (`POST /api/templates/import`)
|
||||
17. Template export endpoint (`GET /api/templates/export`)
|
||||
18. First template: "Startup in a Box"
|
||||
|
||||
### Phase 4: Company Store
|
||||
|
||||
19. GitHub-based store index
|
||||
20. CLI commands for browse/install/import
|
||||
21. UI page for browsing the store
|
||||
|
||||
---
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **Modules extend, never patch.** Modules add new routes, tables, and hook handlers. They never modify core tables or override core routes.
|
||||
|
||||
2. **Hooks are post-commit, fire-and-forget.** Module failures never break core operations.
|
||||
|
||||
3. **One-way dependency.** Modules depend on core. Core never depends on modules. Module tables can FK to core tables, not the reverse.
|
||||
|
||||
4. **Declarative manifest, imperative registration.** Static metadata in JSON (validated without running code). Runtime behavior registered via the API.
|
||||
|
||||
5. **Namespace isolation.** Module routes live under `/api/modules/<id>/`. Module tables are prefixed `mod_<id>_`. Module config is scoped to its ID.
|
||||
|
||||
6. **Graceful degradation.** If a module fails to load, log the error and continue. The rest of the system works fine.
|
||||
|
||||
7. **Data survives disable.** Disabling a module stops its code but preserves its data. Re-enabling picks up where it left off.
|
||||
171
doc/spec/agents-runtime.md
Normal file
171
doc/spec/agents-runtime.md
Normal file
@@ -0,0 +1,171 @@
|
||||
# Agent Runtime Guide
|
||||
|
||||
Status: User-facing guide
|
||||
Last updated: 2026-02-17
|
||||
Audience: Operators setting up and running agents in Paperclip
|
||||
|
||||
## 1. What this system does
|
||||
|
||||
Agents in Paperclip do not run continuously.
|
||||
They run in **heartbeats**: short execution windows triggered by a wakeup.
|
||||
|
||||
Each heartbeat:
|
||||
|
||||
1. Starts the configured agent adapter (for example, Claude CLI or Codex CLI)
|
||||
2. Gives it the current prompt/context
|
||||
3. Lets it work until it exits, times out, or is cancelled
|
||||
4. Stores results (status, token usage, errors, logs)
|
||||
5. Updates the UI live
|
||||
|
||||
## 2. When an agent wakes up
|
||||
|
||||
An agent can be woken up in four ways:
|
||||
|
||||
- `timer`: scheduled interval (for example every 5 minutes)
|
||||
- `assignment`: when work is assigned/checked out to that agent
|
||||
- `on_demand`: manual wakeup (button/API)
|
||||
- `automation`: system-triggered wakeup for future automations
|
||||
|
||||
If an agent is already running, new wakeups are merged (coalesced) instead of launching duplicate runs.
|
||||
|
||||
## 3. What to configure per agent
|
||||
|
||||
## 3.1 Adapter choice
|
||||
|
||||
Common choices:
|
||||
|
||||
- `claude_local`: runs your local `claude` CLI
|
||||
- `codex_local`: runs your local `codex` CLI
|
||||
- `process`: generic shell command adapter
|
||||
- `http`: calls an external HTTP endpoint
|
||||
|
||||
For `claude_local` and `codex_local`, Paperclip assumes the CLI is already installed and authenticated on the host machine.
|
||||
|
||||
## 3.2 Runtime behavior
|
||||
|
||||
In agent runtime settings, configure heartbeat policy:
|
||||
|
||||
- `enabled`: allow scheduled heartbeats
|
||||
- `intervalSec`: timer interval (0 = disabled)
|
||||
- `wakeOnAssignment`: wake when assigned work
|
||||
- `wakeOnOnDemand`: allow ping-style on-demand wakeups
|
||||
- `wakeOnAutomation`: allow system automation wakeups
|
||||
|
||||
## 3.3 Working directory and execution limits
|
||||
|
||||
For local adapters, set:
|
||||
|
||||
- `cwd` (working directory)
|
||||
- `timeoutSec` (max runtime per heartbeat)
|
||||
- `graceSec` (time before force-kill after timeout/cancel)
|
||||
- optional env vars and extra CLI args
|
||||
|
||||
## 3.4 Prompt templates
|
||||
|
||||
You can set:
|
||||
|
||||
- `bootstrapPromptTemplate`: used for first run/new session
|
||||
- `promptTemplate`: used for subsequent resumed runs
|
||||
|
||||
Templates support variables like `{{agent.id}}`, `{{agent.name}}`, and run context values.
|
||||
|
||||
## 4. Session resume behavior
|
||||
|
||||
Paperclip stores session IDs for resumable adapters.
|
||||
|
||||
- Next heartbeat reuses the saved session automatically.
|
||||
- This gives continuity across heartbeats.
|
||||
- You can reset a session if context gets stale or confused.
|
||||
|
||||
Use session reset when:
|
||||
|
||||
- you significantly changed prompt strategy
|
||||
- the agent is stuck in a bad loop
|
||||
- you want a clean restart
|
||||
|
||||
## 5. Logs, status, and run history
|
||||
|
||||
For each heartbeat run you get:
|
||||
|
||||
- run status (`queued`, `running`, `succeeded`, `failed`, `timed_out`, `cancelled`)
|
||||
- error text and stderr/stdout excerpts
|
||||
- token usage/cost when available from the adapter
|
||||
- full logs (stored outside core run rows, optimized for large output)
|
||||
|
||||
In local/dev setups, full logs are stored on disk under the configured run-log path.
|
||||
|
||||
## 6. Live updates in the UI
|
||||
|
||||
Paperclip pushes runtime/activity updates to the browser in real time.
|
||||
|
||||
You should see live changes for:
|
||||
|
||||
- agent status
|
||||
- heartbeat run status
|
||||
- task/activity updates caused by agent work
|
||||
- dashboard/cost/activity panels as relevant
|
||||
|
||||
If the connection drops, the UI reconnects automatically.
|
||||
|
||||
## 7. Common operating patterns
|
||||
|
||||
## 7.1 Simple autonomous loop
|
||||
|
||||
1. Enable timer wakeups (for example every 300s)
|
||||
2. Keep assignment wakeups on
|
||||
3. Use a focused prompt template
|
||||
4. Watch run logs and adjust prompt/config over time
|
||||
|
||||
## 7.2 Event-driven loop (less constant polling)
|
||||
|
||||
1. Disable timer or set a long interval
|
||||
2. Keep wake-on-assignment enabled
|
||||
3. Use on-demand wakeups for manual nudges
|
||||
|
||||
## 7.3 Safety-first loop
|
||||
|
||||
1. Short timeout
|
||||
2. Conservative prompt
|
||||
3. Monitor errors + cancel quickly when needed
|
||||
4. Reset sessions when drift appears
|
||||
|
||||
## 8. Troubleshooting
|
||||
|
||||
If runs fail repeatedly:
|
||||
|
||||
1. Check adapter command availability (`claude`/`codex` installed and logged in).
|
||||
2. Verify `cwd` exists and is accessible.
|
||||
3. Inspect run error + stderr excerpt, then full log.
|
||||
4. Confirm timeout is not too low.
|
||||
5. Reset session and retry.
|
||||
6. Pause agent if it is causing repeated bad updates.
|
||||
|
||||
Typical failure causes:
|
||||
|
||||
- CLI not installed/authenticated
|
||||
- bad working directory
|
||||
- malformed adapter args/env
|
||||
- prompt too broad or missing constraints
|
||||
- process timeout
|
||||
|
||||
## 9. Security and risk notes
|
||||
|
||||
Local CLI adapters run unsandboxed on the host machine.
|
||||
|
||||
That means:
|
||||
|
||||
- prompt instructions matter
|
||||
- configured credentials/env vars are sensitive
|
||||
- working directory permissions matter
|
||||
|
||||
Start with least privilege where possible, and avoid exposing secrets in broad reusable prompts unless intentionally required.
|
||||
|
||||
## 10. Minimal setup checklist
|
||||
|
||||
1. Choose adapter (`claude_local` or `codex_local`).
|
||||
2. Set `cwd` to the target workspace.
|
||||
3. Add bootstrap + normal prompt templates.
|
||||
4. Configure heartbeat policy (timer and/or assignment wakeups).
|
||||
5. Trigger a manual wakeup.
|
||||
6. Confirm run succeeds and session/token usage is recorded.
|
||||
7. Watch live updates and iterate prompt/config.
|
||||
Reference in New Issue
Block a user