Back to projects

Scheduled Monitoring & Alerting System

Condition-based monitoring of DB and logs, with automatic email and SMS alerts

Runs scheduled checks against two data sources, a database and logs, using configurable monitoring rules, and automatically notifies the right people when a condition is met. Each rule has its own data source, time window, threshold and recipients, a suppression window stops the same person from being notified repeatedly on the same channel, and a distributed lock prevents the same rule from running twice.

Role
Requirements Analysis, Architecture Design, Development, Testing
Tech
JavaSpring BootPostgreSQLOpenSearchDistributed LockSchedulerEmailSMS
01.

The Problem

When a system misbehaves, relying on people to query the database or dig through logs means problems are noticed late.

This system turns "what to check, where to check, how many records within what time, and who to notify" into configurable rules. A scheduler runs them automatically and notifies the right people as soon as a condition is met.

Another problem is over-alerting: while a condition keeps holding, notifying on every scheduler run means the same people keep receiving repeat messages. A suppression mechanism keeps the same person from being notified again on the same channel within the suppression window.

02.

How It Works

  1. 01

    Scheduler fires

    Each rule is triggered on its own schedule.

  2. 02

    Acquire lock

    Take the rule's distributed lock so only one worker handles that rule at a time.

  3. 03

    Load rule

    Read the rule: data source, query, time window, threshold and recipients.

  4. 04

    Query the source

    DB rules query PostgreSQL; log rules query OpenSearch.

  5. 05

    Evaluate

    Count matches within the time window and compare against the threshold.

  6. 06

    Check suppression

    For each recipient and each channel, check whether they were already notified within the suppression window, and skip them if so.

  7. 07

    Notify

    When the condition is met, send email or SMS to recipients that are not suppressed, and record the notification time.

  8. 08

    Release lock

    Release the lock once processing is done.

03.

Flexible Monitoring Rules

Every rule is an independent configuration. Different rules can use different data sources, time windows, thresholds and recipients.

Data source
DB or log
Query
Which records or log entries to count
Time window
For example the last 3 days or 7 days
Threshold
For example more than 5 matches
Recipients
Each person's own name, email and phone number
Channels
Email, SMS
Suppression on/off
Whether suppression is enabled for the rule
Suppression window
How long to stay silent after notifying the same person on the same channel

Example of Changing a Rule

  • TodayNotify when more than 5 records appear within 3 days.
  • TomorrowChange it to notify only when more than 5 records appear within 7 days.

Example rule configuration

{
  "ruleId": "timeout-errors-3d",
  "source": "LOG",
  "query": "level:ERROR AND message:timeout",
  "window": { "unit": "DAY", "value": 3 },
  "threshold": { "operator": ">", "count": 5 },
  "recipients": [
    { "name": "Alice", "email": "alice@example.com", "phone": "0900-000-001" },
    { "name": "Bob",   "email": "bob@example.com",   "phone": "0900-000-002" }
  ],
  "channels": ["EMAIL", "SMS"],
  "suppression": { "enabled": true, "unit": "MINUTE", "value": 60 }
}

Illustrative configuration only. Names, emails and phone numbers are made up.

04.

Suppressing Repeat Notifications

While a condition keeps holding, every scheduler run sees it as an anomaly. Notifying on every run would keep bothering the same people, so each rule has a configurable suppression window.

  • >Each rule can turn suppression on or off; when it is off, no suppression check is made.
  • >When on, suppression is decided per recipient and per channel.
  • >Once someone has been notified on a channel within the window, they are not notified again until the window has passed.
  • >A person's email and SMS are tracked separately and do not affect each other.
05.

Multithreading & Distributed Lock

The scheduler can fire on several threads and several nodes at once. If the same rule ran concurrently, one anomaly would be checked and notified more than once.

Worker A: gets the lock, runs the rule
Worker B: no lock, does not run
  • >A distributed lock is taken per rule, so only one worker can process that rule at a time.
  • >A worker that does not get the lock does not run the rule, so no duplicate notifications are sent.
  • >Different rules do not block each other and can still run in parallel.