Application Logging: What Should You Actually Log?

During local development, logging is often treated as a casual diagnostic convenience. Engineers sprinkle console.log() or logger.info() statements across functions to verify variable shapes, trace execution paths, or confirm that an API route returned a response. In a local terminal with a single concurrent user, free-form text output is harmless and immediate.

In production distributed systems, logging ceases to be a casual debugging tool. It becomes the primary forensic record of system behavior, system degradation, security intrusions, and operational failure. When an outage occurs across hundreds of microservice replicas processing thousands of concurrent requests per second, unstructured string logging rapidly disintegrates:

• Unstructured strings cannot be queried reliably: Searching through gigabytes of free-form text requires expensive regular expressions and full-text scans that strain log indexing engines.
• Missing distributed context isolates events: An isolated error log in a payment worker provides no insight into which upstream user request or API gateway call initiated the transaction.
• Uncalibrated log volume degrades performance: Emitting verbose logs inside high-frequency execution loops consumes CPU cycles, saturates operating system I/O buffers, and inflates cloud observability bills.
• Unredacted payloads violate privacy compliance: Carelessly logging request bodies or headers leaks authentication tokens, passwords, and Personally Identifiable Information (PII) into log aggregation disks, triggering regulatory non-compliance under GDPR, PCI DSS, and HIPAA.

Building a resilient logging infrastructure requires treating logs not as arbitrary human-readable text, but as structured, machine-parseable event streams. This guide details the engineering principles, standard schemas, distributed tracing patterns, security practices, and pipeline architectures required to operate production application logging at scale.

---

1. Structured Logging vs. String Formatting

The foundational shift in modern logging is the move from unstructured human-oriented strings to machine-parseable structured data, almost universally formatted as JSON Lines (application/x-ndjson or .jsonl).

The Failure Mode of String Formatting

Consider a traditional logging pattern using string interpolation or concatenation:

// Anti-pattern: Unstructured string concatenation
logger.info(`User ${userId} placed order ${orderId} with ${itemCount} items for $${totalAmount}`);

When written to standard output, this produces lines such as:

2026-09-15 08:30:12 [INFO] User usr_94821 placed order ord_10283 with 3 items for $149.50
2026-09-15 08:30:13 [INFO] User usr_31028 placed order ord_10284 with 1 items for $29.00

While easily readable by an engineer scrolling a terminal, this format creates acute operational liabilities in a centralized log aggregation cluster (such as OpenSearch, Elasticsearch, ClickHouse, or Grafana Loki):

1. High Cardinality and Index Fragmentation: Every single log line is a unique string. Full-text search engines must tokenize and index every word, rapidly expanding the inverted index size and memory footprint.
2. Fragile Ingestion Parsers: To extract userId or orderId for dashboarding or alerting, the log collector must run regular expressions (such as Logstash Grok filters). If a developer changes the message phrasing to "User usr_94821 submitted order...", the parser silently fails, breaking downstream alerts and analytical pipelines.
3. Impossibility of Numerical and Structured Filtering: In an unstructured string, finding orders where totalAmount > 100 requires scanning and regex-parsing every log record. In a structured system, context.total_amount > 100 is a direct, indexed numerical range query executed in milliseconds.

The JSON Lines Standard Schema

In structured logging, each log record is emitted as a single, self-contained JSON object terminated by a newline character (\n).

A production log schema must balance consistency across microservices with flexibility for domain-specific context. Below is the standardized field taxonomy aligned with OpenTelemetry semantic conventions:

| Field Name | Type | Requirement | Description & Semantic Standard | | :--- | :--- | :--- | :--- | | timestamp | string | Mandatory | ISO 8601 / RFC 3339 UTC timestamp with millisecond or microsecond precision (e.g., 2026-09-15T08:30:12.184Z). Always use UTC to eliminate multi-region clock ambiguity. | | level | string | Mandatory | Normalized uppercase severity level: FATAL, ERROR, WARN, INFO, DEBUG, or TRACE. | | message | string | Mandatory | A concise, static, human-readable summary of the event (e.g., "Order checkout completed"). Avoid dynamic variables in the message string itself. | | service.name | string | Mandatory | Canonical identifier of the emitting application or service (e.g., "checkout-service"). | | service.version | string | Mandatory | The semantic version or Git commit hash of the deployed artifact (e.g., "2.4.1" or "git-7f8a9b2"). | | deployment.environment | string | Mandatory | Deployment tier (e.g., "production", "staging", "development"). | | trace_id | string | Conditional | 32-character hexadecimal string representing the globally unique distributed trace identifier. | | span_id | string | Conditional | 16-character hexadecimal string representing the active execution span within the trace. | | context | object | Optional | Arbitrary key-value pairs representing domain-specific event context (e.g., order_id, customer_id, duration_ms). | | error | object | Conditional | Standardized error payload including name, message, stack, and optional code or nested cause. |

Here is how the checkout event appears under a structured JSON schema:

{
  "timestamp": "2026-09-15T08:30:12.184Z",
  "level": "INFO",
  "message": "Order checkout completed",
  "service": {
    "name": "checkout-service",
    "version": "2.4.1"
  },
  "deployment": {
    "environment": "production"
  },
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "context": {
    "user_id": "usr_94821",
    "order_id": "ord_10283",
    "item_count": 3,
    "total_amount": 149.50,
    "currency": "USD",
    "duration_ms": 42.6
  }
}

Because the message string remains static ("Order checkout completed"), log aggregators can group millions of records instantly by message template, while the numerical fields (item_count, total_amount, duration_ms) can be filtered, aggregated, and graphed natively.

---

2. Distributed Tracing Correlation: The W3C Trace Context Standard

In a microservice architecture, a single user action (such as clicking "Purchase") triggers a cascade of downstream network calls: the API gateway authenticates the token, the checkout service reserves inventory, the payment gateway charges the credit card, and the notification worker dispatches an email.

If an error occurs in the payment service, an isolated log record that reads "Database connection timed out" is virtually useless unless the on-call engineer can correlate it with the specific checkout request that failed upstream.

The W3C Traceparent Specification

To achieve cross-service correlation, production services adhere to the W3C Trace Context specification (W3C Recommendation 23 November 2021). The core mechanism is the traceparent HTTP header, which propagates execution context across service boundaries:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ── ──────────────────────────────── ──────────────── ──
             │                 │                        │          │
          Version           Trace ID                Parent ID    Flags

The header components are defined as:

1. Version (00): 2 hexadecimal characters representing the current specification version. (Version ff is explicitly invalid).
2. Trace ID (4bf92f3577b34da6a3ce929d0e0e4736): 32 hexadecimal characters (16 bytes) representing a globally unique identifier for the entire distributed transaction. All zero bytes (000...000) is invalid.
3. Parent ID / Span ID (00f067aa0ba902b7): 16 hexadecimal characters (8 bytes) identifying the specific caller span that issued the request.
4. Trace Flags (01): 2 hexadecimal characters (8-bit bitmap). The least significant bit (01) indicates whether the trace was sampled for recording.

Propagating Context In-Process and Over the Wire

Correlating logs with distributed traces requires two distinct propagation steps:

1. Wire Propagation: Inbound HTTP, gRPC, and message queue consumers must extract the traceparent header. When making outbound HTTP calls or publishing message queue events (Kafka record headers, RabbitMQ message properties, AWS SQS attributes), the service must inject the active traceparent.
2. In-Process Propagation: Inside the application runtime, passing trace identifiers explicitly through every internal function argument pollutes domain logic. Modern runtimes provide asynchronous context propagation primitives:
   • Node.js: AsyncLocalStorage from node:async_hooks stores request-scoped context across asynchronous call chains.
   • Java / Kotlin: Mapped Diagnostic Context (MDC) via SLF4J / Logback utilizes ThreadLocal or scoped values.
   • Go: context.Context carries trace metadata explicitly across function boundaries.

Bidirectional Observability Pivoting

When trace_id and span_id are consistently embedded in every structured log line, modern observability platforms (such as Grafana Tempo and Loki, Jaeger, Datadog, or AWS X-Ray) enable seamless bidirectional navigation:

• Trace-to-Logs: While inspecting a slow distributed trace waterfall, clicking on an individual span automatically executes a log query for all logs matching trace_id == $traceId AND span_id == $spanId.
• Log-to-Trace: While investigating an ERROR log in a centralized dashboard, clicking the trace_id link instantly opens the end-to-end trace waterfall, exposing every upstream caller, downstream dependency, and network latency breakdown.

---

3. Log Level Discipline and Operational Semantics

One of the most common operational failure modes is log level inflation: developers mark common client validation errors as ERROR, flooding on-call alerting channels. Over time, engineers become desensitized to alerts, leading to alert fatigue where real outages go unnoticed.

Conversely, some teams log fatal infrastructure disconnections as INFO or DEBUG, leaving systems silently degraded.

Definitive Log Level Semantics

Production applications must enforce strict semantic definitions for each severity level:

| Level | Severity | Operational Semantics | Production Status | Alerting Implication | Concrete Example | | :--- | :--- | :--- | :--- | :--- | :--- | | FATAL | 60 | The application has encountered an unrecoverable failure and must terminate immediately. | Always Enabled | Triggers an immediate, high-priority on-call pager alert. | Database connection pool fails to initialize during service bootstrap; encryption keys missing from vault. | | ERROR | 50 | An operation or user request failed completely and could not be satisfied, or an unhandled exception occurred. | Always Enabled | Aggregated into error-rate metrics; triggers alerts when thresholds are breached. | Payment processor returned an unexpected 500 error; unhandled null pointer exception during order checkout. | | WARN | 40 | An unexpected or degraded condition occurred, but the system either recovered automatically or the overall operation succeeded. | Always Enabled | Monitored on dashboards; should never directly page an on-call engineer at night. | Primary Redis cache timed out, falling back to database read; circuit breaker tripped open; request succeeded after 2 retries. | | INFO | 30 | Major operational lifecycle events and key business milestones. Low volume. | Always Enabled | Stored for audit trails, reporting, and operational health tracking. | HTTP server started listening on port 8080; database migration completed; user registered; batch job finished 10,000 records. | | DEBUG | 20 | Verbose diagnostic information intended for developers during active troubleshooting. High volume. | Disabled in normal production; enabled dynamically per request or service during incidents. | Not alerted; used exclusively for active issue investigation. | Cache hit/miss diagnostics; routing branch taken; parsed payload structure; intermediate state transition. | | TRACE | 10 | Extremely fine-grained execution flow. Massive volume. | Disabled in production; enabled only in local development or staging. | None. | Entering/exiting individual private methods; raw socket byte payloads; TLS handshake details. |

The High-Frequency Loop Trap

A catastrophic anti-pattern is placing INFO or DEBUG log statements inside tight execution loops:

// Anti-pattern: Logging inside high-frequency loops
for (const item of batchOf50000Items) {
  processItem(item);
  logger.info({ item_id: item.id }, "Item processed successfully"); // 50,000 I/O operations!
}

If a background worker executes this loop for 50,000 records:

• The application executes 50,000 synchronous or buffered string serializations.
• Standard output pipe buffers saturate, creating operating system thread contention or freezing the single-threaded event loop.
• The log aggregator must ingest, index, and store 50,000 records for a single batch operation, multiplying log volume by several orders of magnitude.

The Solution: Batch Summary Logging:

// Production pattern: Aggregate metrics and log once per batch
const startTime = performance.now();
let successCount = 0;
const failures: Array<{ id: string; reason: string }> = [];

for (const item of batchOf50000Items) {
  try {
    processItem(item);
    successCount++;
  } catch (err) {
    failures.push({ id: item.id, reason: (err as Error).message });
  }
}

const durationMs = Math.round(performance.now() - startTime);

if (failures.length > 0) {
  logger.warn({
    total_records: batchOf50000Items.length,
    success_count: successCount,
    failed_count: failures.length,
    failures: failures.slice(0, 10), // Sample first 10 to prevent payload bloat
    duration_ms: durationMs,
  }, "Batch processing completed with partial failures");
} else {
  logger.info({
    total_records: batchOf50000Items.length,
    success_count: successCount,
    duration_ms: durationMs,
  }, "Batch processing completed successfully");
}

By logging a single summary record containing aggregate counts and a bounded sample of failures, log volume drops from 50,000 lines to exactly 1 line, while diagnostic utility increases.

---

4. Security, Privacy, and PII Masking at the Boundary

Application logs are among the most frequent vectors for data exposure. When an application logs raw HTTP request headers, query parameters, or deserialized request payloads, sensitive user data and credentials leak into centralized storage.

The Legal and Compliance Imperative

Unredacted logging directly violates major regulatory frameworks:

• GDPR (Article 32): Mandatory security of processing. Storing unencrypted personal data (names, email addresses, IP addresses, location data) in unsegmented log files violates data minimization principles.
• PCI DSS (Requirement 3.4): Strictly forbids storing sensitive authentication data (card validation codes / CVV) after authorization, and requires Primary Account Numbers (PANs / credit card numbers) to be rendered unreadable wherever stored.
• HIPAA: Classifies Electronic Protected Health Information (ePHI) as strictly protected data; logging medical identifiers or diagnoses creates severe compliance liabilities.
• SOC 2: Requires strict access controls and confidentiality safeguards across all audit and diagnostic storage tiers.

Why Downstream Log Scrubbing Is Insufficient

A common architectural mistake is relying on downstream log forwarders (e.g., a Logstash regex filter or an AWS CloudWatch subscription filter) to scrub sensitive data.

This approach fails the principle of defense-in-depth:

1. Unencrypted Local Disk Exposure: Container engines write raw container stdout streams to node disk files (e.g., /var/log/pods/...). If scrubbing occurs downstream, unredacted secrets linger in unencrypted container log files on every worker node.
2. Developer and Sidecar Access: Any developer, automated daemon, or metrics scraper with read access to the Kubernetes node or kubectl logs can read plaintext passwords and tokens directly from the container stdout stream.
3. Parser Failure and Regex Bypasses: Complex nested JSON payloads, URL encoding, or multiline strings often bypass downstream regex filters, causing unredacted data to leak into indexing engines.

The Golden Rule: Redaction must occur in-process at the serialization boundary before any byte is written to standard output, disk, or network sockets.

Multi-Tiered Redaction Architecture

A robust in-process redaction engine applies two complementary techniques:

1. Key-Based Denylisting: Recursively walks all context objects and replaces values for sensitive property keys (case-insensitively matching keys such as password, secret, token, authorization, api_key, cvv, credit_card, ssn).
2. Pattern-Based Regex Masking: Scans all string values (including URLs and error messages) for signatures matching sensitive formats, such as Bearer tokens and credit card numbers.

Incoming Context Object
          │
          ▼
┌────────────────────────────────────────┐
│   Recursive Key Matcher                │
│   (password, token, apiKey, secret...) │
└──────────────────┬─────────────────────┘
                   │ Matches -> "[REDACTED]"
                   ▼
┌────────────────────────────────────────┐
│   String Pattern Sanitizer             │
│   (Bearer tokens, Credit Card PANs)    │
└──────────────────┬─────────────────────┘
                   │ Matches -> "[REDACTED_PAN]"
                   ▼
Safe, Sanitized JSON Output Stream

---

5. Pipeline Architecture: The 12-Factor Event Stream vs. In-Process Shipping

How should application logs travel from the running process to long-term storage and analytical dashboards?

The 12-Factor App Principle XI

The Twelve-Factor App methodology establishes a clear standard for containerized systems: Treat logs as event streams.

"A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout."

In modern containerized deployments (such as Kubernetes, Docker, or AWS ECS), the host infrastructure manages the collection lifecycle:

┌──────────────────────────────────────┐
│        Application Container         │
│  Writes JSON Lines to stdout/stderr  │
└──────────────────┬───────────────────┘
                   │ (OS Pipe)
                   ▼
┌──────────────────────────────────────┐
│       Container Runtime (CRI)        │
│     (containerd / Docker daemon)     │
│   Writes to /var/log/pods/*.log      │
└──────────────────┬───────────────────┘
                   │ (Tails local files)
                   ▼
┌──────────────────────────────────────┐
│       Node-Level Daemon Agent        │
│      (Fluent Bit / Vector DaemonSet) │
│  Buffers on disk, batches, compresses │
└──────────────────┬───────────────────┘
                   │ (HTTPS / gRPC bulk push)
                   ▼
┌──────────────────────────────────────┐
│       Centralized Log Storage        │
│ (ClickHouse / OpenSearch / Loki / S3)│
└──────────────────────────────────────┘

This decoupled architecture provides four critical operational benefits:

1. Kernel-Level Isolation: Writing to stdout relies on lightweight operating system pipes. The application does not manage TLS handshakes, network retry queues, or HTTP connection pools for logging.
2. Host-Level Buffering: If the centralized log storage cluster experiences an outage or network partition, the node-level daemon agent buffers logs on the host node's local disk. The application continues serving customer traffic without interruption.
3. Infrastructure Enrichment: The node-level agent automatically enriches each log line with Kubernetes pod names, namespaces, container IDs, and node labels without requiring the application code to query the Kubernetes API.
4. Zero Impact on Process Lifecycle: When the application process terminates, the container runtime finishes draining the pipe buffer, ensuring final shutdown logs are preserved.

The Fatal Anti-Pattern: In-Process HTTP Log Shipping

Many development teams attempt to simplify infrastructure by installing an "HTTP Log Appender" directly inside the application process (e.g., using an SDK that sends HTTP POST requests directly to Datadog, Loggly, or an Elasticsearch cluster).

In production under load, in-process HTTP log shipping is an architectural disaster:

• Socket and Connection Exhaustion: Every HTTP request sent to the log collector consumes local network sockets, file descriptors, and CPU time, competing directly with revenue-generating user requests.
• Latency Cascades and Thread Starvation: When the log ingestion endpoint experiences network latency or rate-limits traffic with HTTP 429 Too Many Requests, the in-process logger must either block worker threads or queue records in application memory.
• Out-of-Memory (OOM) Death Spirals: In an outage, errors multiply exponentially. If the logging endpoint slows down while error volume surges, the in-process in-memory buffer balloons, triggering intense garbage collection thrashing and eventual kernel SIGKILL (OOM kill). The application crashes precisely because of its logging library during the exact moment it needs to record errors.
• Disappearance of Crash Logs: When an application process crashes due to a fatal segmentation fault, unhandled exception, or memory exhaustion, all in-memory buffers are instantly destroyed. The fatal error that caused the crash is never sent over HTTP, leaving engineers with zero diagnostic clues.

---

6. Runtime Performance, Concurrency, and Backpressure

Even when writing to standard output, logging is not free. In high-throughput backend services handling tens of thousands of requests per second, logging mechanics can create significant latency jitter and CPU overhead.

The Mechanics of Standard Output Buffering

In POSIX operating systems, standard output (stdout) is connected via a pipe or file descriptor.

• Pipe Buffer Limits: On Linux, an operating system pipe has a default buffer capacity (historically 64 KB).
• The Blocking Boundary: When an application writes to stdout, the write call completes almost instantaneously as long as the kernel pipe buffer has available capacity. However, if the downstream process (such as Docker or containerd) cannot read from the pipe as fast as the application writes, the pipe buffer fills completely.
• Event Loop Starvation: Once the pipe buffer is full, subsequent write() system calls block. In single-threaded event loop runtimes (such as Node.js or Python asyncio), a blocked stdout write freezes the entire event loop, halting all network I/O and HTTP request handling.
• Thread Lock Contention: In multi-threaded runtimes (such as Java, Go, or Rust), standard output is typically protected by a process-wide mutex. If 64 worker threads simultaneously attempt to write log lines during a traffic spike, they contend on the single stdout mutex, serializing parallel execution and degrading throughput.

Backpressure Strategy: Dropping vs. Blocking

When an application produces logs faster than the operating system or logging infrastructure can consume them, the logging system faces an unavoidable trade-off: block the application or drop logs.

Log Buffer Approaching Capacity
                 │
        ┌────────┴────────┐
        ▼                 ▼
[Strict Audit Systems]  [High-Throughput Web Services]
        │                         │
Block Execution           Drop Non-Critical Logs
(Preserves 100% logs,     (Preserves API latency & uptime,
sacrifices user latency)  increments dropped_records metric)

1. Blocking Mode: The application slows down or stalls to ensure that every single log line is persisted. This strategy is mandatory for financial ledgers, audit compliance events, and security access logs.
2. Dropping Mode: In high-throughput public-facing APIs, customer availability is paramount. When log buffers exceed safe capacity (e.g., 80% full), the logging library discards TRACE, DEBUG, and INFO records while preserving WARN and ERROR records.

Whenever logs are dropped, the system must increment an atomic counter metric (e.g., logging.dropped_records_total) exposed to your metrics scraping agent (such as Prometheus). This alerts engineers to buffer saturation without destroying application availability.

---

7. Complete Implementation: Production-Grade TypeScript Structured Logger

Below is a complete, self-contained, type-safe TypeScript implementation of a production structured logger. It includes:

1. W3C Traceparent Header Parsing: Extracts and validates trace_id and span_id according to the W3C Trace Context recommendation.
2. In-Process Recursive PII Redaction: Sanitizes sensitive object keys and regex patterns (Bearer tokens and credit card numbers) before serialization.
3. Robust Error Serialization: Serializes standard errors, stack traces, error codes, and nested ES2022 cause properties.
4. Structured JSON Lines Output: Writes newline-delimited JSON directly to process.stdout.write.
5. Contextual Child Loggers: Enables request-scoped contextual enrichment without mutating base logger configuration.

import process from "node:process";

export type LogLevel = "fatal" | "error" | "warn" | "info" | "debug" | "trace";

const LOG_LEVEL_SEVERITY: Record<LogLevel, number> = {
  fatal: 60,
  error: 50,
  warn: 40,
  info: 30,
  debug: 20,
  trace: 10,
};

export interface LogContext {
  [key: string]: unknown;
}

export interface TraceContext {
  trace_id: string;
  span_id: string;
  trace_flags: string;
}

export interface LoggerOptions {
  serviceName: string;
  serviceVersion: string;
  environment: string;
  minLevel?: LogLevel;
  customSensitiveKeys?: string[];
}

export interface SerializedError {
  name: string;
  message: string;
  stack?: string;
  code?: string | number;
  cause?: unknown;
}

// ---------------------------------------------------------------------------
// 1. W3C Trace Context Parser
// ---------------------------------------------------------------------------
const W3C_TRACEPARENT_REGEX = /^([0-9a-f]{2})-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$/i;

export function parseTraceParent(headerValue?: string | null): TraceContext | null {
  if (!headerValue || typeof headerValue !== "string") {
    return null;
  }
  const match = headerValue.trim().match(W3C_TRACEPARENT_REGEX);
  if (!match) {
    return null;
  }
  const [, version, traceId, spanId, flags] = match;

  // W3C rules: version ff is invalid; all-zero IDs are invalid
  if (version === "ff" || traceId === "00000000000000000000000000000000" || spanId === "0000000000000000") {
    return null;
  }

  return {
    trace_id: traceId.toLowerCase(),
    span_id: spanId.toLowerCase(),
    trace_flags: flags.toLowerCase(),
  };
}

// ---------------------------------------------------------------------------
// 2. Recursive PII and Secret Redactor
// ---------------------------------------------------------------------------
const DEFAULT_SENSITIVE_KEYS = new Set([
  "password",
  "passwd",
  "secret",
  "token",
  "authorization",
  "api_key",
  "apikey",
  "access_token",
  "refresh_token",
  "private_key",
  "credit_card",
  "card_number",
  "cvv",
  "ssn",
]);

const BEARER_TOKEN_PATTERN = /Bearer\s+[A-Za-z0-9._~+/-]+=*/gi;
const CREDIT_CARD_PATTERN = /\b(?:\d[ -]*?){13,16}\b/g;

export function redactSensitiveData(data: unknown, sensitiveKeys: Set<string>, depth = 0): unknown {
  if (depth > 8) {
    return "[MAX_DEPTH_EXCEEDED]";
  }
  if (data === null || data === undefined) {
    return data;
  }
  if (typeof data === "string") {
    return data
      .replace(BEARER_TOKEN_PATTERN, "Bearer [REDACTED]")
      .replace(CREDIT_CARD_PATTERN, "[REDACTED_PAN]");
  }
  if (typeof data !== "object") {
    return data;
  }
  if (Array.isArray(data)) {
    return data.map((item) => redactSensitiveData(item, sensitiveKeys, depth + 1));
  }

  const result: Record<string, unknown> = {};
  for (const [key, value] of Object.entries(data as Record<string, unknown>)) {
    const normalizedKey = key.toLowerCase().replace(/[-_]/g, "");
    if (sensitiveKeys.has(normalizedKey) || sensitiveKeys.has(key.toLowerCase())) {
      result[key] = "[REDACTED]";
    } else {
      result[key] = redactSensitiveData(value, sensitiveKeys, depth + 1);
    }
  }
  return result;
}

// ---------------------------------------------------------------------------
// 3. Error Serializer
// ---------------------------------------------------------------------------
export function serializeError(err: unknown, depth = 0): SerializedError | unknown {
  if (!(err instanceof Error) || depth > 4) {
    return err;
  }

  const serialized: SerializedError = {
    name: err.name,
    message: err.message,
    stack: err.stack,
  };

  if ("code" in err && (typeof err.code === "string" || typeof err.code === "number")) {
    serialized.code = err.code;
  }

  if ("cause" in err && err.cause !== undefined) {
    serialized.cause = serializeError(err.cause, depth + 1);
  }

  return serialized;
}

// ---------------------------------------------------------------------------
// 4. Structured Logger Class
// ---------------------------------------------------------------------------
export class StructuredLogger {
  private readonly minSeverity: number;
  private readonly baseContext: LogContext;
  private readonly sensitiveKeys: Set<string>;

  constructor(
    private readonly options: LoggerOptions,
    parentContext: LogContext = {}
  ) {
    this.minSeverity = LOG_LEVEL_SEVERITY[options.minLevel || "info"];
    this.sensitiveKeys = new Set(DEFAULT_SENSITIVE_KEYS);
    if (options.customSensitiveKeys) {
      for (const key of options.customSensitiveKeys) {
        this.sensitiveKeys.add(key.toLowerCase().replace(/[-_]/g, ""));
      }
    }

    this.baseContext = {
      service: {
        name: options.serviceName,
        version: options.serviceVersion,
      },
      deployment: {
        environment: options.environment,
      },
      ...parentContext,
    };
  }

  /**
   * Creates a contextual child logger with inherited properties.
   */
  public child(childContext: LogContext): StructuredLogger {
    return new StructuredLogger(this.options, {
      ...this.baseContext,
      ...childContext,
    });
  }

  private write(level: LogLevel, message: string, context?: LogContext, err?: unknown): void {
    const severity = LOG_LEVEL_SEVERITY[level];
    if (severity < this.minSeverity) {
      return;
    }

    const record: Record<string, unknown> = {
      timestamp: new Date().toISOString(),
      level: level.toUpperCase(),
      message,
      ...this.baseContext,
    };

    if (context) {
      const sanitized = redactSensitiveData(context, this.sensitiveKeys) as Record<string, unknown>;
      Object.assign(record, sanitized);
    }

    if (err) {
      record.error = serializeError(err);
    }

    // Write single-line JSON directly to stdout
    process.stdout.write(JSON.stringify(record) + "\n");
  }

  public fatal(message: string, context?: LogContext, err?: unknown): void {
    this.write("fatal", message, context, err);
  }

  public error(message: string, context?: LogContext, err?: unknown): void {
    this.write("error", message, context, err);
  }

  public warn(message: string, context?: LogContext, err?: unknown): void {
    this.write("warn", message, context, err);
  }

  public info(message: string, context?: LogContext): void {
    this.write("info", message, context);
  }

  public debug(message: string, context?: LogContext): void {
    this.write("debug", message, context);
  }

  public trace(message: string, context?: LogContext): void {
    this.write("trace", message, context);
  }
}

End-to-End Application Usage Example

Here is how the structured logger integrates into an HTTP request lifecycle with trace extraction, child logger scoping, PII sanitization, and error handling:

// 1. Initialize the global root logger
const rootLogger = new StructuredLogger({
  serviceName: "checkout-service",
  serviceVersion: "2.4.1",
  environment: "production",
  minLevel: "info",
});

// 2. Incoming HTTP Request Handler
interface IncomingHttpRequest {
  headers: Record<string, string | undefined>;
  method: string;
  path: string;
  body: Record<string, unknown>;
}

async function handleCheckoutRequest(req: IncomingHttpRequest): Promise<{ status: number; body: unknown }> {
  const startTime = performance.now();

  // Extract W3C Traceparent from inbound request headers
  const trace = parseTraceParent(req.headers["traceparent"]);

  // Create an enriched, request-scoped child logger
  const reqLogger = rootLogger.child({
    trace_id: trace?.trace_id,
    span_id: trace?.span_id,
    http: {
      method: req.method,
      route: req.path,
    },
  });

  reqLogger.info("Checkout transaction initiated", {
    user_id: req.body.userId,
    item_count: Array.isArray(req.body.items) ? req.body.items.length : 0,
    // Sensitive keys are automatically redacted in the output stream
    authorization: req.headers["authorization"],
    credit_card: req.body.cardNumber,
  });

  try {
    // Business logic simulation
    if (!req.body.items) {
      throw new Error("Invalid checkout payload: items array is missing");
    }

    const durationMs = Math.round(performance.now() - startTime);

    reqLogger.info("Checkout transaction completed successfully", {
      order_id: "ord_88192",
      duration_ms: durationMs,
    });

    return { status: 200, body: { success: true, orderId: "ord_88192" } };
  } catch (err) {
    const durationMs = Math.round(performance.now() - startTime);

    reqLogger.error("Checkout transaction failed", {
      order_id: "ord_88192",
      duration_ms: durationMs,
    }, err);

    return { status: 500, body: { error: "Internal server error" } };
  }
}

When executed, the logger emits the following single-line JSON to standard output, demonstrating automatic trace correlation, redaction of sensitive credentials, and structured error serialization:

{"timestamp":"2026-09-15T08:35:40.102Z","level":"INFO","message":"Checkout transaction initiated","service":{"name":"checkout-service","version":"2.4.1"},"deployment":{"environment":"production"},"trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7","http":{"method":"POST","route":"/api/v1/checkout"},"user_id":"usr_4401","item_count":2,"authorization":"[REDACTED]","credit_card":"[REDACTED]"}

---

8. Common Application Logging Anti-Patterns

Even experienced engineering organizations encounter predictable pitfalls when managing production logs. Below is a taxonomy of frequent anti-patterns, their operational risks, and their architectural remedies:

| Anti-Pattern | Operational Risk | Production Remedy | | :--- | :--- | :--- | | Unstructured String Formatting | Prevents structured filtering; causes index bloat and fragile Grok parsing failures. | Emit single-line JSON Lines (application/x-ndjson) with standardized static messages. | | Logging Inside Tight Loops | Causes I/O saturation, pipe buffer deadlocks, and astronomical cloud storage costs. | Aggregate metrics in-memory and log a single summary record after loop completion. | | In-Process HTTP Log Appenders | Memory leaks during outages, socket exhaustion, and loss of crash logs on process termination. | Write to standard output (stdout); delegate routing, buffering, and delivery to host daemon agents (Fluent Bit / Vector). | | Logging Raw Secrets and PII | Regulatory non-compliance (GDPR, PCI DSS), credential exposure during security breaches. | Enforce in-process redaction at the serialization boundary using key denylists and regex patterns. | | Swallowing Errors with Vague Logs | Obscures root cause (logger.error("Failed") without passing the actual Error object). | Always serialize the error name, message, stack trace, and nested cause properties into structured fields. | | Inconsistent Field Names Across Teams | Team A uses userId, Team B uses user_id, and Team C uses uid, breaking unified dashboards. | Establish an organization-wide logging schema enforced via shared libraries or linting. | | Stack Traces at WARN or INFO | Triggers false alarms in log monitoring tools that grep for "at [function] (file.js)". | Reserve stack traces strictly for ERROR and FATAL levels where unexpected failure occurred. |

---

9. Production Logging Readiness Checklist

Before promoting an application service to production, verify its logging implementation against this engineering checklist:

Schema & Formatting

• [ ] Every log line is emitted as valid, single-line JSON terminated by a newline (\n).
• [ ] Timestamps are formatted in ISO 8601 / RFC 3339 UTC with millisecond precision.
• [ ] Log messages are concise, static strings; dynamic variables reside in structured context fields.
• [ ] Core resource attributes (service.name, service.version, deployment.environment) are present on all records.

Distributed Tracing Correlation

• [ ] Inbound W3C traceparent headers are parsed and validated.
• [ ] trace_id and span_id are injected into all request-scoped log records.
• [ ] Tracing headers are propagated across outbound HTTP requests and message broker events.
• [ ] Asynchronous execution context is preserved across callbacks and promises (AsyncLocalStorage, MDC).

Log Level Discipline

• [ ] FATAL is reserved strictly for unrecoverable startup crashes and process halts.
• [ ] Expected user input validation failures (e.g., HTTP 400, 422) are logged at INFO or WARN, not ERROR.
• [ ] WARN is used for recovered failures (retries, cache fallbacks, circuit breakers) and does not page on-call engineers.
• [ ] No log statements exist inside high-iteration loops without batch summarization.
• [ ] DEBUG and TRACE are disabled by default in high-traffic production environments.

Security & Privacy

• [ ] In-process redaction executes at the serialization boundary before I/O.
• [ ] Sensitive keys (password, authorization, token, secret, cvv, ssn) are scrubbed automatically.
• [ ] Regex sanitizers mask inline Bearer tokens and credit card PANs in URLs and text fields.
• [ ] Standard output streams do not leak raw authorization headers or sensitive database connection strings.

Pipeline Architecture & Runtime Performance

• [ ] The application writes directly to stdout/stderr without managing log files or network delivery.
• [ ] A dedicated host daemon (e.g., Fluent Bit, Vector) tails logs, buffers on local disk, and ships to central storage.
• [ ] Logging does not block the application's main thread or event loop under load.
• [ ] If backpressure occurs, the system drops non-critical logs and increments a logging.dropped_records_total metric.

---

Conclusion

Application logging is not an arbitrary string dumping mechanism; it is the primary sensory organ of distributed systems architecture. When designed with deliberate engineering discipline, logs provide instantaneous forensic clarity, allowing engineering teams to diagnose subtle race conditions, correlate cascading service failures, and resolve complex outages in minutes rather than hours.

Achieving this level of operational maturity requires treating logs as first-class software contracts. Standardize on JSON Lines, bind every event to W3C distributed trace contexts, enforce uncompromising privacy redaction at the in-process serialization boundary, and rely on the 12-factor architecture to decouple application execution from ingestion pipelines.

By shifting from reactive, unstructured text output to disciplined, machine-readable event streams, teams build systems that are observable by design, secure by default, and resilient under production pressure.