Serverless Second Brain

What it is

The production implementation of the serverless second brain described in the essay of the same name. While the essay defines the architecture — memory, compute, and interface layers with "two doors" for humans and agents — this repository is the code that brings it to life.

Available as source code.

Architecture

The system separates three layers with clear responsibilities:

• Memory: DynamoDB single-table design with GSIs for bidirectional queries + S3 for long-form content
• Compute: 5 specialized Lambda functions (capture, search, graph, connect, flag) orchestrated by Step Functions
• Interface: two doors — API Gateway REST for the human SPA, AgentCore Gateway for MCP agents

Bedrock provides classification (Claude) and embeddings (Titan 1,024 dimensions).

Progress

The project follows four phases defined in the essay. All four phases are complete and deployed to dev:

What works today

Human door (API Gateway REST):

• POST /capture — ingests text, classifies with Bedrock Claude, generates embeddings with Titan, persists to DynamoDB + S3 via Step Functions
• GET /search?q= — hybrid keyword + semantic search with cosine similarity over 1,024-dimension embeddings
• GET /graph — full knowledge graph (nodes + bidirectional edges)
• GET /nodes/{id} — single node with edges and related nodes
• GET /health — health check

Agent door (AgentCore MCP Gateway):

• read_node — reads a node by slug with metadata, edges, and related nodes
• list_nodes — lists nodes with filters by type, status, and tags
• search — hybrid keyword + semantic search
• add_node — creates a seed node with automatic Bedrock classification
• connect_nodes — creates bidirectional edges with audit trail
• flag_stale — flags a node for human review without modifying it

Surfacing (daily digest):

• 5 analyzers: stale seeds, orphan nodes, missing connections, promotion candidates, content gaps
• EventBridge daily cron at 8 AM UTC → Surfacing Lambda → SNS email
• Configurable thresholds via environment variables (STALE_DAYS, MIN_EDGES, SIMILARITY_THRESHOLD)
• Verified in dev: 17 stale seeds, 1 orphan, 40 content gaps in ~2.5s for 170 nodes

Infrastructure:

• CloudFront + S3 for static frontend hosting
• Full migration of ~160 nodes from jonmatum.com MDX to DynamoDB + S3
• Embedding backfill for all existing nodes
• CI/CD with GitHub Actions OIDC (no static credentials)
• Smoke test script with 12 endpoint verifications
• GitHub Actions OIDC role in bootstrap Terraform

Resilience (deep review #3):

• invokeWithRetry() with exponential backoff for Bedrock throttling (1s, 2s, 4s)
• Capture pipeline creates bidirectional edges (consistency with connect_nodes)
• batchGetNodes() eliminates N+1 queries in Graph Lambda
• CORS on all error paths, Content-Type on all responses
• Prod deployment in CI/CD gated on environment approval

Agent door: MCP Gateway

The "agent door" exposes Lambda functions as MCP tools via Bedrock AgentCore. Any MCP-compatible agent can discover and use the tools semantically:

Write safety

Write operations follow strict controls:

• Audit trail: every mutation creates an AUDIT# item in DynamoDB with actor, action, changes, and 90-day TTL
• Seed-only: nodes created by agents start as seed — human review required for promotion
• No deletes: agents cannot delete nodes, only flag them for review with flag_stale
• Actor tracking: every operation records the actor (agent:{session_id} or api)
• Existence validation: connect_nodes verifies both nodes exist before creating edges

// Audit trail on every write operation
const audit: AuditItem = {
  PK: `AUDIT#${now}`,
  SK: `NODE#${slug}`,
  action: "connect",
  actor,
  changes: { source, target, edge_type, weight },
  ttl: Math.floor(Date.now() / 1000) + 90 * 86400,
};
await putAudit(audit);

DynamoDB design

Single-table design with four item types:

Two GSIs enable inverse queries and status filters:

• GSI1: SK (hash) + PK (range) — "what nodes point to serverless?"
• GSI2: GSI2PK (status) — "all seeds not updated in 7 days"

Capture pipeline

Step Functions orchestrates the full pipeline with automatic retry on Bedrock throttling:

Each step is a separate Lambda invocation. Express Workflow (synchronous) keeps the response within API Gateway timeout.

Hybrid search

The Search Lambda combines keyword matching and semantic similarity:

// Hybrid search: keyword + semantic
const keywordResults = await queryByKeywords(query, table);
const queryEmbedding = await generateEmbedding(query);
const allEmbeddings = await scanEmbeddings(table);
 
const semanticResults = allEmbeddings
  .map(item => ({
    slug: item.PK.replace("NODE#", ""),
    score: cosineSimilarity(queryEmbedding, item.embedding),
  }))
  .sort((a, b) => b.score - a.score);
 
// Combine scores with configurable weights
const combined = mergeResults(keywordResults, semanticResults, {
  keywordWeight: 0.3,
  semanticWeight: 0.7,
});

At current scale (~160 nodes, ~700KB of vectors) the in-memory scan is sufficient. The scalability benchmark (issue #12) will evaluate alternatives for 10K+ nodes.

Infrastructure as code

All infrastructure is defined with Terraform using reusable modules:

infra/
  bootstrap/              → State backend (S3 + DynamoDB lock)
  modules/
    dynamodb/             → Single-table design + GSIs
    lambda/               → Compute functions
    api-gateway/          → Human door (REST)
    step-functions/       → Pipeline orchestration
    s3/                   → Content and frontend
    cloudfront/           → CDN + security headers
    iam/                  → Roles and policies
    agentcore-gateway/    → Agent door (MCP)
    sns/                  → Notifications
  environments/
    dev/                  → Dev config (deployed)
    prod/                 → Prod config

CI/CD uses GitHub Actions with OIDC — no static AWS credentials:

• terraform-plan.yml — plan on PRs
• terraform-apply.yml — apply on merge to main
• lambda-deploy.yml — function packaging and deployment

Cost

Scales to zero. No minimum costs beyond S3 storage:

Architecture decision records

The repository includes 12 ADRs (Architecture Decision Records) in docs/decisions/ documenting every technical decision with context, alternatives evaluated, benchmark data, and revisit criteria:

Next steps

• AgentCore Runtime (issue #8) — hosting the reasoning agent in serverless microVMs with access to all Gateway tools
• Observability (issue #14) — CloudWatch dashboards, alarms, X-Ray tracing
• Domain-agnostic config (issue #13) — make the system deployable for any domain with terraform apply
• Authentication (issue #17) — implement ADR-003: Cognito + public/private visibility model

Why it matters

This project translates a reference architecture into deployable code. The goal is for any builder to take the repository, configure their domain (legal, research, education) in terraform.tfvars, and deploy a complete second brain with terraform apply. The essay explains the "why" behind each decision; the code implements the "how."

All four phases already demonstrate the architecture works: capture with automatic classification, hybrid semantic search, a bidirectional knowledge graph, an MCP door for AI agents with write safety controls, and a proactive daily digest that identifies forgotten seeds and missing connections — all serverless, all in Terraform, scaling to zero at ~$0.51/month idle cost.

References

• From Prototype to Production: A Serverless Second Brain — Essay defining the complete architecture.
• GitHub Repository — Project source code.
• AWS Serverless Lens — AWS, 2024. Framework for serverless applications.
• DynamoDB Developer Guide — AWS, 2024. Single-table design guide.
• Bedrock AgentCore — AWS, 2025. MCP Gateway and Runtime for agents.
• AgentCore Gateway Developer Guide — AWS, 2025. MCP gateway documentation for Lambda tools.
• MCP Specification — Anthropic, 2025. Interoperability protocol for agents.
• Step Functions Developer Guide — AWS, 2024. Serverless workflow orchestration.