Skip to content

runbook.md

Latest commit

 

History

History
219 lines (147 loc) · 4.47 KB

runbook.md

File metadata and controls

219 lines (147 loc) · 4.47 KB
Raw

Log Monitoring Service – Runbook

1. System Overview

The Log Monitoring Service processes application.log, evaluates error thresholds, and triggers alerts when predefined conditions are met. The system is designed to be defensive, test-driven, and safe under failure conditions.

This document describes how the system behaves, what it expects, how it fails, and how it should be monitored in a production environment.

2. Input Format Expectations

Log File

  • File Name: application.log
  • Format: Structured JSON log entries (one entry per line)
  • Encoding: UTF-8

Required Fields per Log Entry

Each log line must contain:

  • timestamp (ISO-8601 string)
  • service (string)
  • level (INFO, WARNING, ERROR, CRITICAL)
  • message (string)

Accepted Log Levels

  • INFO
  • WARNING
  • ERROR
  • CRITICAL

Malformed Entries

If a log line:

  • Is not valid JSON
  • Is missing required fields
  • Contains unknown log level

Then:

  • The entry is skipped
  • The event is logged as malformed
  • The system continues processing

Malformed entries must not crash the system.

3. Threshold Definitions

Alert Trigger Rules

Current threshold rules:

  • Trigger alert when 3 or more ERROR entries occur within 2 minutes for the same service
  • Trigger alert immediately for any CRITICAL entry
  • Threshold evaluation is performed per service

Post-Alert Behavior

  • Alert is triggered once per threshold breach
  • Repeated alerts require threshold to reset (cooldown behavior if implemented)
  • System continues processing after alert

Default Behavior

If threshold configuration is invalid or missing:

  • Log configuration error
  • Fail safely (exit or fallback to default thresholds)

4. Processing Flow

High-Level Flow

  1. Read application.log
  2. Parse entries
  3. Validate required fields
  4. Filter relevant log levels
  5. Evaluate threshold conditions
  6. Trigger alert if needed
  7. Log summary
  8. Exit safely

5. Logging Expectations

The system must log:

  • Start of processing
  • End of processing
  • Number of lines processed
  • Number of malformed lines skipped
  • Threshold evaluation result
  • Alert triggered (or not)
  • Alert failure events
  • Configuration errors
  • File read errors

Logging must:

  • Avoid exposing sensitive data
  • Be consistent and structured
  • Support troubleshooting

6. Failure Behavior

Malformed Input

  • Logged
  • Skipped
  • Processing continues

Log File Read Error

  • Error logged
  • System exits cleanly

Alert Handler Failure

  • Failure logged
  • Processing continues (unless explicitly configured otherwise)

Threshold Misconfiguration

  • Error logged
  • Safe fallback OR controlled shutdown

No failure should result in silent success.

7. Alerting Dependency Assumptions

Alerting is treated as an external dependency.

  • Alert handler is isolated and mockable
  • Alert mechanism may fail
  • Alerting failure must not crash core processing
  • Alert success or failure must be logged

Future integrations (email, SMS, webhook) are out of scope unless explicitly implemented.

8. Operational Concerns

What to Monitor

Operators should monitor:

  • Frequency of ERROR and CRITICAL events
  • Number of malformed log entries
  • Alert trigger frequency
  • Alert failure frequency
  • File read failures
  • Repeated threshold breaches

Warning Signals

Potential instability indicators:

  • High malformed entry rate
  • Repeated alert failures
  • Rapid repeated alerts (alert fatigue)
  • Threshold misconfiguration errors

If Alerting Fails Repeatedly

  • Investigate alerting dependency
  • Confirm network/service availability
  • Validate configuration
  • Review error logs

9. Out of Scope

The following are not currently implemented:

  • Real email/SMS integration
  • Distributed tracing
  • Log rotation handling
  • Multi-file processing
  • Advanced rate limiting
  • Persistent alert state storage

10. Known Constraints

  • Single-file processing only
  • No persistent state between runs
  • No distributed coordination
  • No external metrics reporting
  • No real-time streaming ingestion

11. Version Alignment

This runbook reflects:

  • Current implementation behavior
  • Current threshold rules
  • Current failure handling logic
  • Current logging structure

If code behavior changes, this document must be updated immediately.

Engineering Reminder

The runbook must always match reality.

If the runbook and implementation disagree, production will expose it.

This document is part of the system—not an afterthought.