Architecture Overview

High-Level Architecture

Understanding how Spry processes Markdown into executable workflows through its layered architecture.

╔═════════════════════════════════════════════════════════════╗
║                         SPRY CLI                            ║
║                                                             ║
║     ┌──────────┐      ┌──────────┐      ┌──────────┐        ║
║     │  axiom   │      │    rb    │      │    sp    │        ║
║     │ (graph)  │      │(runbook) │      │(sqlpage) │        ║
║     └────┬─────┘      └────┬─────┘      └────┬─────┘        ║
╚══════════╪══════════════════╪══════════════════╪════════════╝
           │                  │                  │
           └──────────┬───────┴────────┬─────────┘
                      │                │
                      ▼                ▼
╔═════════════════════════════════════════════════════════════╗
║                       lib/axiom                             ║
║                 [Semantic Graph Engine]                     ║
║                                                             ║
║   ┌─────────┐   ┌──────────┐   ┌──────────┐   ┌─────────┐   ║
║   │   io    │──▶│  remark  │──▶│   edge   │──▶│projection│  ║
║   │(loader) │   │(plugins) │   │ (rules)  │   │ (views)  │  ║
║   └─────────┘   └──────────┘   └──────────┘   └────┬────┘   ║
║                                                      │      ║
║                      ┌───────────────────────────────┘      ║
║                      │                                      ║
║                      ▼                                      ║
║              ┌───────────────┐                              ║
║              │  Playbook/    │                              ║
║              │  Task Model   │                              ║
║              └───────┬───────┘                              ║
║                      │                                      ║
║                      ▼                                      ║
║              ┌───────────────┐                              ║
║              │ orchestrate   │                              ║
║              │   (runner)    │                              ║
║              └───────┬───────┘                              ║
╚══════════╪══════════════════╪══════════════════╪════════════╝
                       │
                       │ uses
                       ▼
╔═════════════════════════════════════════════════════════════╗
║                    lib/universal                            ║
║                  [Shared Utilities]                         ║
║                                                             ║
║  ┌────────┐  ┌────────┐  ┌────────┐  ┌────────┐  ┌──────┐   ║
║  │ shell  │  │  task  │  │  code  │  │ render │  │direct│   ║
║  │        │  │  (DAG) │  │ (lang) │  │(template)│ │ -ive │  ║
║  └────────┘  └────────┘  └────────┘  └────────┘  └──────┘   ║
╚═════════════════════════════════════════════════════════════╝

Processing Pipeline

Spry transforms Markdown through six distinct stages, each building upon the previous to create executable workflows.

Stage 1: Input/Loading

Location: lib/axiom/io/

The input stage loads resources from multiple sources including files, URLs, and glob patterns.

// Load resources from files, URLs, or globs
const resources = await markdownASTs({
  sources: ["runbook.md", "*.md", "https://example.com/doc.md"],
  cwd: process.cwd()
});

Key components:

resource.ts - Abstract resource loading
mod.ts - Pipeline orchestration
VFile integration for metadata tracking

Stage 2: Parsing

Location: lib/axiom/remark/

Markdown is parsed into MDAST (Markdown Abstract Syntax Tree) using the unified/remark ecosystem.

// Parser pipeline
unified()
  .use(remarkParse)
  .use(remarkFrontmatter)
  .use(remarkGfm)
  .use(remarkDirective)

The parser produces a tree structure:

interface Root {
  type: "root";
  children: Node[];
  data?: {
    documentFrontmatter?: Record<string, unknown>;
  };
}

interface Code {
  type: "code";
  lang: string;
  meta: string;
  value: string;
  data?: {
    codeFM?: ParsedCodeFrontmatter;
  };
}

Stage 3: AST Enrichment

Location: lib/axiom/remark/

Remark plugins enhance the AST with semantic information:

Plugin	Purpose
`doc-frontmatter.ts`	Parse YAML frontmatter
`code-directive-candidates.ts`	Identify PARTIAL directives
`actionable-code-candidates.ts`	Mark executable cells
`code-contribute.ts`	Handle file includes
`node-decorator.ts`	Add semantic decorations

Note: Plugin Order Matters

Plugins execute sequentially, with each building on previous AST transformations. The order is critical for proper semantic analysis.

Stage 4: Graph Construction

Location: lib/axiom/edge/

Edge rules create semantic relationships between document elements:

// Build graph from AST
const graph = buildGraph(root, typicalRules());

Edge Types:

containedInSection - Section hierarchy
codeDependsOn - Task dependencies
frontmatterClassification - Semantic roles
sectionSemanticId - Section identity
nodesClassification - Node roles

Stage 5: Projection

Location: lib/axiom/projection/

Projections transform the semantic graph into domain-specific models optimized for different use cases.

FlexibleProjection

UI-neutral view of document structure:

const flex = buildFlexibleProjection(graph);

// Returns:
interface FlexibleProjection {
  documents: ProjectedDocument[];
  nodes: ProjectedNode[];
  edges: ProjectedEdge[];
  hierarchies: ProjectedHierarchy[];
}

Use Cases:

Document analysis and querying
Building custom tooling
Exploring document structure
Creating visualizations

PlaybookProjection

Executable task model for runbook operations:

const playbook = buildPlaybookProjection(graph);

// Returns:
interface PlaybookProjection {
  executables: Executable[];
  materializables: Materializable[];
  directives: Directive[];
  tasks: ExecutableTask[];
}

Use Cases:

Running automated workflows
Executing multi-step processes
Building CI/CD pipelines
Creating interactive runbooks

TreeProjection

Hierarchical view of document structure:

const tree = buildTreeProjection(graph);

Use Cases:

Navigation interfaces
Document outlines
Hierarchical analysis

Stage 6: Execution

Location: lib/axiom/orchestrate/

Execute tasks from the playbook using a DAG-based execution engine:

// Build execution plan
const plan = executionPlan(tasks);

// Execute in topological order
await executeDAG(plan, {
  executor: shellExecutor,
  onTaskStart: (task) => console.log(`Starting: ${task.taskId()}`),
  onTaskComplete: (task, result) => console.log(`Done: ${task.taskId()}`),
});

Warning: Dependency Resolution

The execution planner uses Kahn's algorithm for topological sorting. Circular dependencies are detected and reported as errors before execution begins.

Key Modules

Understanding Spry's module organization helps you navigate the codebase and extend functionality.

lib/axiom

The semantic graph engine and core transformation pipeline.

Module	Description
`graph.ts`	Graph construction
`edge/`	Edge rules
`projection/`	View builders
`orchestrate/`	Execution engine
`io/`	Resource loading
`remark/`	AST plugins
`mdast/`	AST utilities
`text-ui/`	CLI interface

lib/axiom/
├── mod.ts              # Public exports
├── graph.ts            # Graph building
├── edge/               # Edge rules
│   ├── mod.ts
│   ├── orchestrate.ts  # Rule pipeline
│   ├── rule/           # Individual rules
│   └── pipeline/       # Rule compositions
├── io/                 # I/O and parsing
│   ├── mod.ts
│   └── resource.ts
├── mdast/              # AST utilities
│   ├── data-bag.ts
│   ├── node-issues.ts
│   └── ...
├── projection/         # Graph projections
│   ├── flexible.ts
│   ├── playbook.ts
│   └── tree.ts
├── remark/             # Remark plugins
│   ├── actionable-code-candidates.ts
│   ├── code-directive-candidates.ts
│   └── ...
├── text-ui/            # Terminal interfaces
└── web-ui/             # Web interface

lib/universal

Shared utilities used across the system.

Module	Description
`task.ts`	DAG execution
`shell.ts`	Process spawning
`code.ts`	Language registry
`directive.ts`	Directive parsing
`posix-pi.ts`	CLI flag parsing
`render.ts`	Template interpolation
`resource.ts`	Resource abstraction

lib/universal/
├── task.ts             # DAG execution
├── shell.ts            # Shell commands
├── resource.ts         # Resource loading
├── code.ts             # Code parsing
├── directive.ts        # Directive parsing
├── event-bus.ts        # Event system
├── watcher.ts          # File watching
└── ...

lib/courier

Data movement protocol implementations.

lib/courier/
├── protocol.ts         # DataMP protocol
├── singer.ts           # Singer adapter
└── airbyte.ts          # Airbyte adapter

lib/playbook

Domain-specific playbook implementations.

Module	Description
`sqlpage/`	SQLPage integration

lib/playbook/
├── README.md           # Architecture docs
└── sqlpage/
    ├── cli.ts          # SQLPage CLI
    ├── content.ts      # Content generation
    ├── interpolate.ts  # Template interpolation
    └── orchestrate.ts  # Orchestration

Data Flow Example

Understanding data flow through the pipeline helps you debug issues and optimize workflows.

Markdown to Tasks

Step 1: Load Markdown File

runbook.md
    ↓ [io/resource.ts]
VFile with content

Step 2: Parse to AST

    ↓ [remark/parser pipeline]
MDAST (raw)
    ↓ [remark/plugins]
MDAST (enriched with codeFM, decorations)

Step 3: Build Semantic Graph

    ↓ [edge/orchestrate.ts]
Graph { root, edges: [...relationships] }

Step 4: Create Projection

    ↓ [projection/playbook.ts]
PlaybookProjection { tasks: [...] }

Step 5: Plan Execution

    ↓ [orchestrate/task.ts]
ExecutionPlan { layers: [[task1], [task2, task3], [task4]] }

Step 6: Execute Tasks

    ↓ [universal/shell.ts]
Process execution with capture
    ↓
Results + captured output

Extension Points

Spry's architecture makes it straightforward to add new capabilities through well-defined extension points.

1. Custom Remark Plugins

Add new AST transformations:

const myPlugin: Plugin<[], Root> = () => {
  return (tree) => {
    visit(tree, "code", (node) => {
      // Transform code nodes
    });
  };
};

2. Custom Edge Rules

Add new relationship types:

const myRule: EdgeRule = {
  rel: "myRelationship",
  apply: function* (root) {
    // Yield edges
    yield { from: nodeA, to: nodeB, rel: "myRelationship" };
  },
};

3. Custom Projections

Create new domain views:

function myProjection(graph: Graph): MyProjection {
  // Transform graph into domain-specific view
  return { ... };
}

4. Custom Executors

Add new execution strategies:

const myExecutor: TaskExecutor = async (task, context) => {
  // Custom execution logic
  return { exitCode: 0, output: "..." };
};

Type System

Key types used throughout Spry's architecture:

// From lib/axiom/
type Graph<Rel, Edge> = {
  root: Root;
  edges: Edge[];
};

type GraphEdge<Rel> = {
  rel: Rel;
  from: Node;
  to: Node;
};

// From lib/universal/
type Task<Baggage> = {
  taskId: () => string;
  taskDeps?: () => string[];
} & Baggage;

type ExecutionPlan<T> = {
  tasks: T[];
  layers: T[][];
  order: T[];
};

How is this guide?

Architecture Overview

On this page