<< Back to article

System Monitoring

System monitoring is the practice of observing the health, performance, and availability of infrastructure and applications. Effective monitoring enables rapid detection of issues, capacity planning, and informed decision-making.

The Three Pillars of Observability

Metrics

Metrics are numerical measurements collected over time.

  • Counter: Monotonically increasing value (requests served, errors).
  • Gauge: Point-in-time value (CPU usage, memory, queue depth).
  • Histogram: Distribution of observations (request latency).
  • Summary: Similar to histogram but calculated client-side.

Common metrics:

  • CPU: Usage, load average, steal time.
  • Memory: Used, free, cached, swap.
  • Disk: I/O ops, throughput, latency, space usage.
  • Network: Bandwidth, packets per second, errors, drops.
  • Application: Request rate, error rate, latency (RED method).

Logs

Logs are timestamped records of events. They provide context for metrics and traces.

  • Structured logging: JSON or key-value pairs for machine parsing.
  • Log levels: DEBUG, INFO, WARN, ERROR, FATAL.
  • Log aggregation: Centralised storage (ELK, Loki, Splunk).
  • Log rotation: Prevent disk exhaustion.

See Article - logging/journal for logging details.

Traces

Distributed tracing tracks requests across service boundaries. Each trace consists of spans representing operations.

  • OpenTelemetry: Vendor-neutral observability framework.
  • Jaeger, Zipkin: Distributed tracing systems.
  • Span attributes: Service name, operation, duration, status.

Monitoring Tools

System Monitoring

  • top: Basic process viewer.
  • htop: Enhanced interactive process viewer.
  • iostat: CPU and I/O statistics.
  • vmstat: Virtual memory and system statistics.
  • netstat / ss: Network connections.
  • sar: Historical performance data.

Infrastructure Monitoring

  • Prometheus: Pull-based metrics collection with time-series database.
  • Grafana: Visualisation dashboards.
  • Zabbix: Enterprise monitoring with alerting.
  • Nagios: Classic monitoring system.
  • Icinga: Nagios fork with modern features.

Log Management

  • ELK Stack: Elasticsearch, Logstash, Kibana.
  • Loki: Lightweight log aggregation (Grafana Labs).
  • Fluentd: Unified logging layer.
  • rsyslog / syslog-ng: Traditional syslog daemons.

APM (Application Performance Monitoring)

  • Datadog APM: Commercial full-stack monitoring.
  • New Relic: Application and infrastructure monitoring.
  • Dynatrace: AI-powered full-stack monitoring.

Alerting

Alerts notify operators of issues requiring attention.

  • Threshold-based: Alert when a metric exceeds a limit (CPU > 90%).
  • Anomaly detection: Alert when behaviour deviates from baseline.
  • SLO-based: Alert when error budget is exhausted.
  • On-call rotation: Escalate alerts to the right person (PagerDuty, Opsgenie).

SLOs, SLIs, and SLAs

  • SLI (Service Level Indicator): Quantitative measure of service level (latency p99, error rate).
  • SLO (Service Level Objective): Target value for an SLI (p99 latency < 200ms).
  • SLA (Service Level Agreement): Contractual commitment to an SLO, with consequences for breach.