Software Documentation That Actually Ships With the Product

Documentation should capture decisions, constraints, and operating knowledge without slowing the engineering team down.

March 22, 202517 min readOpenStair Engineering

Illustration of product documentation, engineering notes, and team knowledge flow

Software Documentation That Ships With the Product

Software engineering teams almost universally agree that poor documentation degrades productivity, prolongs incident response, and slows developer onboarding. Yet across the industry, documentation remains one of the most consistently neglected subsystems in the software delivery lifecycle.

Features are designed, code is written, automated test suites execute in continuous integration pipelines, and deployments ship to production. Then, as an afterthought, an engineer is asked to "update the wiki."

That mental model—treating documentation as an administrative postscript rather than an integral part of the product—guarantees documentation failure. Disconnected documentation repositories drift silently out of sync with production reality. Code examples rot as dependencies upgrade. Architecture diagrams describe systems that were refactored months prior. When an incident occurs at 2:00 AM, the on-call engineer discovers that the runbook references environment variables and service endpoints that no longer exist.

Documentation that actually ships—and remains accurate across years of rapid architectural evolution—requires treating technical writing with the same engineering rigor applied to source code. This article establishes the architectural patterns, epistemological frameworks, and automated verification pipelines necessary to build documentation that stays synchronized with production software.

Traditional Wiki Model (Drift & Bitrot):
[Code PR Merged] ──(Temporal Gap)──> [Manual Wiki Edit "Later"] ──> [Stale Documentation]

Docs-as-Code Model (Synchronous Verification):
┌────────────────────────────────────────────────────────────────────────┐
│ Atomic Git Pull Request                                               │
│ ├── Source Code Implementation                                         │
│ ├── Automated Test Suite                                               │
│ └── Co-located Documentation (Diátaxis Structure + Automated CI Gates) │
└────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
       [Automated CI: Syntax + Links + Vale + Schema Drift Check]
                                    │
                                    ▼
       [Production Deployment: Code & Documentation Ship Together]

The Diátaxis Framework: Eliminating Epistemological Friction

The most frequent reason technical documentation becomes unreadable or frustrating is epistemological confusion: attempting to serve conflicting cognitive needs within the same document.

When an engineer is troubleshooting an active database connection timeout during a production outage, they need immediate, prescriptive operational steps. Forcing that engineer to wade through four pages detailing the philosophical origins of distributed consensus creates severe cognitive friction. Conversely, when a software architect evaluates whether to migrate from an event-driven messaging queue to a distributed log broker, a terse list of command-line flags fails to provide the conceptual context necessary to make an informed architectural decision.

To solve this problem systematically, technical documentation must adopt the Diátaxis framework, an architectural taxonomy formalized by Daniele Procida. Diátaxis models technical documentation across two orthogonal axes of human cognition:

Practical Steps (Action / Doing) vs. Theoretical Knowledge (Cognition / Understanding): Does the document guide physical execution, or does it build mental models?
Learning-Oriented (Acquisition) vs. Task/Work-Oriented (Application): Is the reader an apprentice acquiring foundational capability, or a practitioner completing a defined operational goal?

These two axes generate four distinct, non-overlapping quadrants:

                          PRACTICAL STEPS (ACTION)
                                    │
                                    │
             Tutorials              │            How-To Guides
       (Learning-Oriented)          │         (Task/Work-Oriented)
     "Help me get started"          │        "Help me solve a problem"
                                    │
────────────────────────────────────┼────────────────────────────────────
                                    │
            Explanation             │              Reference
       (Learning-Oriented)          │         (Task/Work-Oriented)
       "Help me understand"         │          "Help me find facts"
                                    │
                                    │
                        THEORETICAL KNOWLEDGE (COGNITION)

The Aviation Metaphor

To understand how these four quadrants interact, consider flight training:

Tutorials: A first flying lesson with a licensed flight instructor. The instructor tells the student exactly when to push the throttle, how to pull back on the yoke, and how to level off. The student does not choose the destination or study the thermodynamics of jet turbines; the goal is simply experiencing flight safely and building confidence.
How-To Guides: A flight checklist for navigating through an instrument landing system (ILS) in severe weather. The pilot already knows how to fly; they need a precise, verifiable sequence of operational steps to land safely under specific environmental conditions.
Reference: The flight operating handbook and avionics specifications. It lists stall speeds, maximum takeoff weights, hydraulic pressure limits, and fuel burn rates. It contains zero narrative or storytelling; it provides exact, authoritative facts.
Explanation: A textbook on the aerodynamics of swept-wing aircraft and fluid dynamics. It explains why supersonic shockwaves form and how wing sweep delays wave drag. It contains no checklists or button-pushing guides; its sole purpose is illuminating principles.

The Four Quadrants in Software Systems

1. Tutorials (Learning-Oriented / Practical Action)

Tutorials serve beginners who need to build competence from scratch. A tutorial is an orchestrated journey where the author takes total responsibility for the reader's success.

Pedagogical Contract: The reader follows instructions verbatim and achieves a guaranteed, working result.
Authoring Rule: Eliminate choices, branches, and alternative implementations. Do not explain multiple database drivers or discuss theoretical edge cases. Provide exactly one working path. If the tutorial says "Run command X and you will see output Y," output Y must occur reliably on a clean machine.
Example: "Building Your First Authenticated REST Endpoint with OpenStair Core in 10 Minutes."

2. How-To Guides (Task-Oriented / Practical Action)

How-To guides serve developers who already understand the fundamentals and need to solve a specific real-world problem.

Pedagogical Contract: The reader brings a specific objective and prerequisite knowledge. The guide provides a reproducible recipe.
Authoring Rule: Focus strictly on the goal. State assumptions and prerequisites upfront (e.g., "Prerequisites: An active Kubernetes cluster with kubectl v1.28+ configured"). Do not re-explain what Kubernetes pods are. Provide sequential, numbered steps with validation checks at each stage.
Example: "How to Configure Keyset Cursor-Based Pagination for High-Throughput Event Streams."

3. Reference (Task-Oriented / Theoretical Knowledge)

Reference documentation provides an austere, authoritative catalog of technical facts. It reflects the machine's reality without narrative interpretation.

Pedagogical Contract: The reader knows what they are looking for and needs immediate, unambiguous verification of a signature, property, constraint, or flag.
Authoring Rule: Prioritize completeness, precision, and structural consistency. Every parameter must define its data type, default value, optionality, valid ranges, and error codes. Do not insert multi-step operational tutorials into reference pages; link to How-To guides instead.
Example: "OpenStair Event Streaming API v2: Endpoint and Schema Reference."

4. Explanation / Architecture (Learning-Oriented / Theoretical Knowledge)

Explanation documents illuminate the architectural reasoning, design trade-offs, and historical context of a system.

Pedagogical Contract: The reader wants to understand why the software is built a certain way, how subsystems interact, and what constraints governed major engineering decisions.
Authoring Rule: Discursive, comparative, and reflective. Connect disparate concepts, discuss trade-offs that were evaluated and rejected, explain failure recovery boundaries, and provide high-level topology diagrams. Do not turn an explanation into a copy-paste CLI tutorial.
Example: "Consistency, Concurrency, and Isolation Boundaries in OpenStair's Distributed Transaction Pipeline."

Cognitive Failure Modes of Collapsing Quadrants

When technical writers or engineers collapse these quadrants, documentation quality degrades rapidly:

The Bloated Reference: Embedding narrative walkthroughs and setup guides into an API reference table. Developers searching for an HTTP status code must scroll through three pages of tutorial prose.
The Tangled How-To: Pausing mid-procedure during an incident recovery guide to provide five paragraphs of historical context on database indexing algorithms. The operator's working memory is consumed by historical trivia while the production service is failing.
The Over-Engineered Tutorial: Overwhelming a newcomer by presenting five alternative configuration approaches, deep security implications, and edge-case exceptions before they have even verified hello world. The learner experiences cognitive overload and abandons the guide.
The Action-Starved Explanation: Providing an abstract 10-page thesis on microservices decoupled communication without a single link to an actionable How-To guide or schema specification.

Diátaxis Quadrant Comparison

Docs-as-Code (DaC): Repository Topology and Atomic Pull Requests

For decades, organizations attempted to manage software documentation through external, hosted SaaS wikis (such as Confluence, Notion, or internal enterprise portals). While convenient for non-technical team members, external wikis introduce structural failure modes when applied to production engineering documentation.

The Mechanics of Wiki Bitrot

External wikis decay because their operational lifecycle is fundamentally decoupled from the software delivery pipeline:

Decoupled Write-Paths: Source code is modified inside an integrated development environment (IDE), reviewed on a version control platform (GitHub, GitLab), and merged through pull requests. Updating an external wiki requires switching contexts to a separate browser application with an independent authentication layer and an entirely separate editor. Engineers instinctively defer the wiki update, and deferred documentation is rarely written.
Temporal and Version Dissonance: An external wiki typically maintains a single, global "current" state. In contrast, real-world software exists simultaneously across multiple branches and releases: v1.4.2 running in production, v2.0.0-rc1 validating in staging, and v2.1.0-dev active in local development. When documentation lives in a centralized wiki, readers cannot determine which version of the text corresponds to their deployed application.
Absence of Programmatic Verification: External wikis cannot participate in continuous integration pipelines. They cannot be statically typed, linted for broken links, audited for broken code snippets, or checked for schema drift.

Documentation Storage Topologies

Engineering organizations must select a documentation topology that reflects their release cadence, team boundaries, and deployment infrastructure.

| Criterion | Monorepo Co-location (/docs in Service Repo) | Dedicated Documentation Repository | Disconnected SaaS Wiki (Confluence / Notion) | | :--- | :--- | :--- | :--- | | Version Parity with Code | Absolute: Documentation is committed, branched, and tagged with code | Moderate: Requires cross-repo synchronization or submodule tracking | None: Single mutable state; cannot represent branch variations | | Pull Request Review Workflow | Unified: Code, unit tests, and documentation reviewed in a single PR | Fragmented: Requires dual PR coordination across repositories | Non-existent: Edits happen out-of-band without peer code review | | CI/CD Automated Validation | Native: Full linting, link checking, and contract verification in CI | High: Dedicated doc builds, but misses immediate code AST access | Impossible: Cannot run custom AST linters or programmatic assertions | | Access Boundaries | Restricted to engineers with repository write access | Accessible to technical writers and cross-functional teams | Accessible to entire enterprise (product, sales, support) | | Tooling Overhead | Minimal: Uses existing Git tooling, markdown parsers, and IDEs | Moderate: Requires dedicated build pipelines and release orchestration | Low initial setup; high long-term maintenance debt | | Drift & Bitrot Likelihood | Lowest: Broken docs can block merges directly in CI | Moderate: Drift occurs when cross-repo updates are skipped | Highest: Documentation inevitably rots within months |

The Atomic Pull Request Pattern

The foundational operational practice of Docs-as-Code is the Atomic Pull Request.

The Atomic PR Rule: Any pull request that modifies public interfaces, internal architecture, configuration parameters, operational runbooks, or deployment infrastructure MUST modify the corresponding documentation within the exact same Git commit.

Feature Branch: feat/keyset-pagination
┌─────────────────────────────────────────────────────────────────────────┐
│ Commit 1: feat(api): implement cursor pagination query router           │
│ Commit 2: test(api): add boundary tests for cursor deserialization      │
│ Commit 3: docs(api): update OpenAPI spec & add keyset How-To guide      │
│ Commit 4: docs(adr): add ADR-0014 documenting pagination trade-offs     │
└─────────────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼  Open Pull Request #318
┌─────────────────────────────────────────────────────────────────────────┐
│ CI Pipeline Execution:                                                  │
│ ├── Unit & Integration Test Suites               [PASSED]               │
│ ├── TypeScript Typecheck & Project Build         [PASSED]               │
│ ├── Markdownlint & Structural AST Parsing        [PASSED]               │
│ ├── Vale Prose & Style Validation                [PASSED]               │
│ ├── Relative Link & Heading Anchor Check         [PASSED]               │
│ └── OpenAPI Contract Drift Assertion             [PASSED]               │
└─────────────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼  Peer Review & Merge
┌─────────────────────────────────────────────────────────────────────────┐
│ Git History: Code, Tests, Specs, and Documentation Evolve Atomically    │
└─────────────────────────────────────────────────────────────────────────┘

When code and documentation evolve atomically in the same commit:

Reviewers evaluate the documentation during the standard code review process. A reviewer can challenge an ambiguous parameter explanation before the code merges.
When an engineer bisects an old regression using git bisect, checking out commit 8a2f4c yields the exact documentation that described the system at that precise instant in history.
Pull request templates enforce accountability via mandatory checklist items:
- [x] Public API changes reflected in OpenAPI definitions
- [x] New configuration variables documented in environment reference
- [x] Operational changes added to service runbook

Architectural Decision Records (ADRs): Capturing Why Code Is the Way It Is

Standard Git commit messages explain what lines of code changed. Integration tests prove that the system meets its functional requirements. However, neither artifact reliably answers the most critical long-term question:

Why was the system designed this way, what alternative solutions were evaluated and rejected, and what operational constraints forced this specific trade-off?

When architectural context is lost, future engineers encounter non-obvious code structures, assume the original authors were negligent or uninformed, and execute well-intentioned refactors that unknowingly reintroduce past architectural failures.

To preserve institutional memory, engineering teams utilize Architectural Decision Records (ADRs), a structured format formalized by Michael Nygard.

The Anatomy of an ADR

An ADR is a short, immutable, plain-text document stored directly in version control (typically under docs/adr/ or doc/architecture/decisions/). It adheres to an explicit lifecycle:

Status Field: An explicit state machine:
- Proposed: Under active RFC review; not yet accepted.
- Accepted: Approved and implemented in the system.
- Superseded by ADR-XXXX: A subsequent decision has rendered this architecture obsolete, linking explicitly to the new record.
- Deprecated: The architecture is phased out without an immediate direct replacement.
Context: Explains the technical, business, and operational forces driving the decision. It documents constraints, performance profiles, team capabilities, and the state of the technology ecosystem at the time of writing.
Decision: States clearly and unequivocally what architectural choice was made.
Consequences: An honest evaluation of the decision's impact. Crucially, this section must document negative consequences, trade-offs accepted, and new technical debt incurred, rather than acting as a promotional brochure for the chosen technology.

ADR vs. RFC vs. Living Documentation vs. Git Commits

Production ADR Case Study

Below is an authentic, production-grade ADR demonstrating the Nygard format applied to a distributed systems database pagination challenge.

# ADR-0014: Adopt Keyset Cursor Pagination for Public Event Stream APIs

- **Status**: Accepted
- **Date**: 2025-02-14
- **Deciders**: Architecture Working Group, Core Platform Team
- **Consulted**: Database Operations, Public API Client Working Group

## Context

Our public `/v1/events` endpoint currently utilizes traditional SQL offset pagination (`LIMIT $1 OFFSET $2`). As our event volume has grown, two critical engineering issues have emerged:

1. **Database Performance Degradation**: PostgreSQL must scan and discard $N$ rows before returning the requested page. For queries with large offsets (`OFFSET 100000 LIMIT 50`), database query execution time degrades from 4ms to over 850ms, causing database connection pool saturation and CPU spikes during high-concurrency client crawls.
2. **Result Inconsistency Under Concurrent Writes**: High ingestion volume (averaging 400 new event writes per second) causes "page drift." When new rows are inserted while a client paginates through results, records shift across page boundaries, resulting in duplicate deliveries or skipped events for downstream consumers.

We evaluated three architectural alternatives:
- Option A: Retain `LIMIT / OFFSET` and enforce a hard cutoff at `OFFSET 10000`.
- Option B: Adopt keyset cursor-based pagination utilizing `(created_at, id)` tuples.
- Option C: Route historical event queries to an external Elasticsearch cluster.

Option A artificially restricts legitimate data auditing workflows. Option C introduces significant distributed infrastructure overhead and eventual consistency delays.

## Decision

We will replace offset pagination with **keyset cursor pagination** across all high-throughput event streaming endpoints:

1. Paginated queries will sort deterministically by `(created_at DESC, id DESC)`.
2. Clients will pass an opaque `cursor` parameter containing a base64url-encoded JSON payload: `{"t": 1739520000000, "i": "evt_01HXYZ..."}`.
3. The SQL query planner will execute index seek operations using tuple comparisons:
   `WHERE (created_at, id) < ($1, $2) ORDER BY created_at DESC, id DESC LIMIT $3`
4. A composite B-tree index on `(created_at DESC, id DESC)` will be applied to the `events` table via zero-downtime concurrent indexing.
5. Offset pagination (`page` and `offset` query parameters) will be marked as deprecated on `/v1/events` and scheduled for removal in API v2.

## Consequences

### Positive
- **Predictable Query Latency**: Database seeks execute in $O(\log N)$ b-tree lookup time plus $O(K)$ scan time for the requested page limit $K$, completely eliminating the $O(N)$ row-scanning penalty of high offsets.
- **Idempotent Continuation**: Concurrent record insertions at the head of the table do not shift the cursor boundary, eliminating duplicate and skipped records during long-running client ingestion jobs.

### Negative & Trade-offs Accepted
- **No Arbitrary Page Jumping**: Clients can no longer jump directly to arbitrary page numbers (e.g., "Go to page 450"). Navigation is strictly sequential (`next` and `previous`).
- **Client State Complexity**: API consumers must persist and echo opaque cursor tokens rather than computing simple integer arithmetic for page numbers.
- **Index Storage Overhead**: The composite B-tree index on `(created_at DESC, id DESC)` increases storage requirements by approximately 18% on the primary database partition.

Automated Documentation Verification in CI/CD: Treating Drift as a Build Failure

The central tenet of modern software engineering is that untested code is broken code. The exact same principle applies to technical documentation:

Untested documentation is drifting documentation.

If an engineering team relies on human vigilance to catch broken links, outdated parameter references, malformed markdown, or non-inclusive language, the documentation system will inevitably rot. Production-ready Docs-as-Code implementations establish automated validation gates within the CI/CD pipeline that fail pull requests when documentation integrity is compromised.

Docs-as-Code CI/CD Verification Pipeline:
┌────────────────────────────────────────────────────────────────────────┐
│ Pull Request Triggered                                                 │
└────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│ Gate 1: Markdown AST & Structural Linting (markdownlint-cli2)          │
│ └── Verifies heading hierarchy, unclosed fences, table formatting      │
└────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│ Gate 2: Prose, Style, & Terminology Linting (Vale)                     │
│ └── Enforces brand style guides, active voice, banned jargon           │
└────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│ Gate 3: Relative & Absolute Link Integrity (TypeScript Link Checker)   │
│ └── Verifies relative file paths exist and #heading-anchors match AST  │
└────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│ Gate 4: API Contract Drift Detection (Code-to-Spec Generation)         │
│ └── Compares runtime Zod/Fastify schemas against committed OpenAPI spec│
└────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│ Gate 5: Executable Code Snippet Verification (Sandbox Runner)          │
│ └── Executes TypeScript/Shell blocks in docs to confirm zero bitrot    │
└────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│ All Gates Passed ──> Merge Permitted                                   │
└────────────────────────────────────────────────────────────────────────┘

The Five Automated Verification Gates

Syntax & Structural Integrity: Validates markdown Abstract Syntax Trees (AST). Tools like markdownlint enforce strict heading hierarchies (forbidding jumping from an H2 to an H4), ensure valid table syntax, and detect unclosed backtick fences.
Prose & Style Governance (Vale): Enforces editorial guidelines programmatically. Vale scans markdown prose against established style packages (such as the Google Developer Documentation Style Guide or Microsoft Writing Style Guide). It flags passive voice, warns against deprecated acronyms, and ensures consistent terminology (e.g., enforcing "read-only replica" instead of outdated master/slave terminology).
Link & Anchor Integrity: Validates both relative local paths (../architecture/caching.md) and heading anchor fragments (#cache-invalidation). Broken links produce 404 errors that directly degrade developer trust.
Contract Drift Detection: Eliminates the synchronization gap between application code and API documentation. Rather than maintaining an OpenAPI YAML file by hand, engineers derive API contracts directly from runtime type definitions (such as TypeScript Zod schemas, TypeSpec, or Fastify route schemas). In CI, a build step re-generates the documentation artifact and runs git diff --exit-code. If an engineer changes a route schema in code but forgets to commit the updated documentation artifact, CI immediately fails.
Executable Code Snippets: Code examples embedded in documentation must be compilable. By extracting fenced code blocks and executing them against current library dependencies in an automated test environment, teams eliminate the phenomenon of developers copying and pasting non-functional boilerplate.

Implementation: Production GitHub Actions Workflow

Below is a complete, syntactically valid GitHub Actions workflow (.github/workflows/docs-ci.yml) demonstrating an automated documentation verification pipeline.

name: Documentation CI

on:
  push:
    branches:
      - main
    paths:
      - "docs/**"
      - "content/**"
      - "scripts/**"
      - ".github/workflows/docs-ci.yml"
      - "src/contracts/**"
  pull_request:
    branches:
      - main
    paths:
      - "docs/**"
      - "content/**"
      - "scripts/**"
      - ".github/workflows/docs-ci.yml"
      - "src/contracts/**"

concurrency:
  group: docs-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  markdown-lint:
    name: Markdown AST & Style
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v4

      - name: Setup Node.js Environment
        uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"

      - name: Install Dependencies
        run: npm ci

      - name: Run markdownlint
        run: npx markdownlint-cli2 "docs/**/*.md" "content/**/*.mdx"

  vale-prose-lint:
    name: Vale Prose & Style Guide
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v4

      - name: Run Vale Prose Linter
        uses: errata-ai/vale-action@v2
        with:
          files: '["docs", "content"]'
          reporter: github-pr-check
          fail_on_error: true
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

  link-integrity:
    name: Relative Link & Anchor Verification
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v4

      - name: Setup Node.js Environment
        uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"

      - name: Install Dependencies
        run: npm ci

      - name: Validate Internal Documentation Links
        run: npx tsx scripts/validate-doc-links.ts docs content/blog

  contract-drift-check:
    name: API Contract Drift Detection
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v4

      - name: Setup Node.js Environment
        uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"

      - name: Install Dependencies
        run: npm ci

      - name: Re-generate OpenAPI Specification from Source Types
        run: npm run docs:generate-schemas

      - name: Assert Zero Drift in Committed Contract Files
        run: |
          if ! git diff --exit-code docs/reference/openapi.json; then
            echo "Error: Generated OpenAPI specification does not match committed file."
            echo "Run 'npm run docs:generate-schemas' locally and commit the updated schema."
            exit 1
          fi

Implementation: Production TypeScript Link Validator

Below is a self-contained, robust Node.js/TypeScript validation script (scripts/validate-doc-links.ts) that programmatically parses markdown files, extracts relative links and anchor fragments, and verifies that target files and heading anchors exist on disk.

#!/usr/bin/env node

/**
 * OpenStair Technologies — Documentation Link & Anchor Validator
 * 
 * Recursively scans documentation directories, extracts local markdown links
 * and heading anchors, and verifies that targets exist on disk.
 */

import fs from "node:fs";
import path from "node:path";

interface BrokenLinkDiagnostic {
  sourceFile: string;
  lineNumber: number;
  rawLink: string;
  resolvedTarget: string;
  reason: string;
}

// Convert markdown heading text into a GitHub-compatible anchor slug
function slugifyHeading(heading: string): string {
  return heading
    .toLowerCase()
    .trim()
    .replace(/[^\w\s-]/g, "")
    .replace(/\s+/g, "-");
}

// Extract all heading slugs from a markdown file's content
function extractHeadingAnchors(fileContent: string): Set<string> {
  const anchors = new Set<string>();
  const lines = fileContent.split(/\r?\n/);

  for (const line of lines) {
    const trimmed = line.trim();
    // Match Markdown ATX headings (# Heading)
    const match = trimmed.match(/^#{1,6}\s+(.+)$/);
    if (match) {
      const headingText = match[1].trim();
      anchors.add(slugifyHeading(headingText));
    }
  }

  return anchors;
}

// Recursively collect all markdown/mdx files within target directories
function discoverMarkdownFiles(directories: string[]): string[] {
  const markdownFiles: string[] = [];

  function traverse(currentPath: string) {
    if (!fs.existsSync(currentPath)) return;

    const stat = fs.statSync(currentPath);
    if (stat.isDirectory()) {
      const entries = fs.readdirSync(currentPath);
      for (const entry of entries) {
        traverse(path.join(currentPath, entry));
      }
    } else if (stat.isFile()) {
      const ext = path.extname(currentPath).toLowerCase();
      if (ext === ".md" || ext === ".mdx") {
        markdownFiles.push(path.resolve(currentPath));
      }
    }
  }

  for (const dir of directories) {
    traverse(path.resolve(process.cwd(), dir));
  }

  return markdownFiles;
}

// Main execution function
function validateDocumentationLinks(targetDirectories: string[]): boolean {
  const files = discoverMarkdownFiles(targetDirectories);
  const diagnostics: BrokenLinkDiagnostic[] = [];

  // Pre-load heading anchors for all known markdown files
  const fileAnchorCache = new Map<string, Set<string>>();

  for (const file of files) {
    try {
      const content = fs.readFileSync(file, "utf8");
      fileAnchorCache.set(file, extractHeadingAnchors(content));
    } catch (error) {
      console.error(`Failed to read file for anchor indexing: ${file}`, error);
    }
  }

  // Regex matching inline markdown links: [Text](URL)
  const linkRegex = /\[(?:[^\]\\]|\\.)*\]\(([^)\s]+)(?:\s+"[^"]*")?\)/g;

  for (const file of files) {
    const content = fs.readFileSync(file, "utf8");
    const lines = content.split(/\r?\n/);
    let inCodeBlock = false;

    for (let i = 0; i < lines.length; i++) {
      const line = lines[i];
      const trimmed = line.trim();

      // Track code fence state to avoid checking URLs inside code examples
      if (trimmed.startsWith("```") || trimmed.startsWith("~~~")) {
        inCodeBlock = !inCodeBlock;
        continue;
      }

      if (inCodeBlock) continue;

      let match: RegExpExecArray | null;
      while ((match = linkRegex.exec(line)) !== null) {
        const rawLink = match[1];

        // Skip external URLs, mailto, and protocol-relative links
        if (
          rawLink.startsWith("http://") ||
          rawLink.startsWith("https://") ||
          rawLink.startsWith("mailto:") ||
          rawLink.startsWith("//")
        ) {
          continue;
        }

        const [linkTarget, anchor] = rawLink.split("#");
        let targetFilePath: string;

        if (!linkTarget || linkTarget === "") {
          // Intra-document link: #section-heading
          targetFilePath = file;
        } else if (linkTarget.startsWith("/")) {
          // Root-relative path: resolve against workspace root
          targetFilePath = path.resolve(process.cwd(), `.${linkTarget}`);
        } else {
          // Relative path: resolve against current file directory
          targetFilePath = path.resolve(path.dirname(file), linkTarget);
        }

        // Check if target file exists on disk
        if (!fs.existsSync(targetFilePath)) {
          // Check for implicit extension resolution (.md or .mdx)
          const mdCandidate = `${targetFilePath}.md`;
          const mdxCandidate = `${targetFilePath}.mdx`;

          if (fs.existsSync(mdCandidate)) {
            targetFilePath = mdCandidate;
          } else if (fs.existsSync(mdxCandidate)) {
            targetFilePath = mdxCandidate;
          } else {
            diagnostics.push({
              sourceFile: path.relative(process.cwd(), file),
              lineNumber: i + 1,
              rawLink,
              resolvedTarget: path.relative(process.cwd(), targetFilePath),
              reason: "Target file does not exist on disk",
            });
            continue;
          }
        }

        // Check anchor integrity if an anchor was specified
        if (anchor) {
          let targetAnchors = fileAnchorCache.get(targetFilePath);

          if (!targetAnchors) {
            try {
              const targetContent = fs.readFileSync(targetFilePath, "utf8");
              targetAnchors = extractHeadingAnchors(targetContent);
              fileAnchorCache.set(targetFilePath, targetAnchors);
            } catch {
              targetAnchors = new Set();
            }
          }

          const normalizedAnchor = slugifyHeading(anchor);
          if (!targetAnchors.has(normalizedAnchor)) {
            diagnostics.push({
              sourceFile: path.relative(process.cwd(), file),
              lineNumber: i + 1,
              rawLink,
              resolvedTarget: `${path.relative(process.cwd(), targetFilePath)}#${anchor}`,
              reason: `Heading anchor "#${anchor}" not found in target file`,
            });
          }
        }
      }
    }
  }

  // Report diagnostic results
  if (diagnostics.length > 0) {
    console.error(`\n❌ Found ${diagnostics.length} broken documentation link(s):\n`);
    for (const d of diagnostics) {
      console.error(`  - ${d.sourceFile}:${d.lineNumber}`);
      console.error(`    Link:   "${d.rawLink}"`);
      console.error(`    Target: ${d.resolvedTarget}`);
      console.error(`    Reason: ${d.reason}\n`);
    }
    return false;
  }

  console.log(`\n✅ All internal documentation links and anchors verified successfully across ${files.length} files.`);
  return true;
}

// CLI entry point
const targetDirs = process.argv.slice(2);
if (targetDirs.length === 0) {
  console.error("Usage: npx tsx scripts/validate-doc-links.ts <dir1> [dir2...]");
  process.exit(1);
}

const success = validateDocumentationLinks(targetDirs);
process.exit(success ? 0 : 1);

Actionable Production Documentation Audit Checklist

Before releasing major software features or restructuring documentation repositories, engineering teams should evaluate their documentation assets against this five-pillar audit matrix:

1. Epistemological Alignment (Diátaxis Discipline)

[ ] Quadrant Isolation: Is each document strictly isolated to a single Diátaxis quadrant (Tutorial, How-To, Reference, or Explanation)?
[ ] No Tangled Prose: Are How-To guides stripped of historical and architectural digressions?
[ ] No Instructional Reference: Are API reference tables free of copy-paste multi-step tutorials?
[ ] Guaranteed Tutorials: Has the beginner tutorial been validated on a clean machine environment without implicit global dependencies?

2. Repository Topology & Version Parity

[ ] Git Co-location: Is core architectural and API documentation stored directly in version control alongside the source code it describes?
[ ] Branch Parity: Does checkout of a historical git tag (e.g., git checkout v2.3.0) yield documentation matching that exact software release?
[ ] Atomic PR Discipline: Do pull requests modifying public APIs, configuration parameters, or runbooks update the corresponding documentation files in the same commit?

3. CI/CD Automated Verification Gates

[ ] Syntax & AST Validation: Does CI execute markdownlint to prevent malformed syntax, skipped headings, and unclosed blocks?
[ ] Prose & Style Governance: Is Vale configured with an official style guide to enforce active voice, clear writing, and consistent branding?
[ ] Link & Anchor Integrity: Does an automated script parse and verify all internal relative links and #heading-anchors on every PR?
[ ] Contract Drift Prevention: Are OpenAPI/JSON Schema specifications generated directly from code types and verified against uncommitted diffs in CI?

4. Architectural Decision Governance (ADR Quality)

[ ] Standard Format: Do architectural decisions adhere to the Michael Nygard format (Title, Status, Context, Decision, Consequences)?
[ ] State Machine Hygiene: Are superseded ADRs explicitly linked to their replacements (e.g., Superseded by ADR-0022)?
[ ] Honest Trade-offs: Does the "Consequences" section explicitly document technical debt, operational costs, and negative side-effects?

5. Operational Readiness & Incident Runbooks

[ ] Zero Secret Exposure: Are all private credentials, API keys, and production secrets strictly excluded from documentation and replaced with references to secret managers?
[ ] Actionable Failure Modes: Do troubleshooting documents describe specific error signatures, causes, and step-by-step remediation procedures rather than generic platitudes?
[ ] Clear Ownership: Does every subsystem documentation directory have an explicit code-owner team defined in .github/CODEOWNERS?

Conclusion: Documentation as Compiled Engineering Infrastructure

High-performing engineering organizations do not treat documentation as a speculative, post-implementation luxury. They treat it as compiled engineering infrastructure.

By structuring technical content through the Diátaxis framework, teams eliminate the cognitive friction that makes documents hard to navigate. By co-locating documentation in Git repositories and enforcing atomic pull requests, teams eliminate temporal drift and bitrot. By establishing automated CI/CD validation pipelines for links, prose style, and API contracts, teams ensure that documentation quality is maintained with the same automated rigor applied to production code.

When documentation ships alongside the product, it ceases to be an unread, drifting administrative burden. It becomes what it was always meant to be: an authoritative, tested, and reliable interface connecting the software with the engineers who build, operate, and scale it.