Threshold Tuning for Alerts #

Threshold tuning for alerts is the discipline of turning noisy maritime telemetry into a small, trustworthy stream of actionable signals by treating every alerting boundary as a versioned, observable, hot-reloadable parameter rather than a hardcoded constant. Within the Container Tracking & AIS Event Synchronization domain, this is the control plane that guards pipeline health: it decides when a berth-approach deviation, a stalled gate move, or a synchronization drift becomes an operator-facing alert. In high-throughput port environments, alert fatigue is a direct operational liability — static boundaries applied to dynamic vessel kinematics and tidal constraints generate false positives that drown out genuine anomalies, delaying crane dispatch, misallocating yard slots, and triggering unnecessary port state inspections. Effective tuning fuses two independent data sources before it fires, applies hysteresis to reject transient noise, and records every evaluation in an immutable audit trail for shipping operations teams, port authorities, and the Python automation engineers who own the on-call rotation.

Ingestion Boundary & Protocol Handling #

Thresholds can only be as reliable as the telemetry they evaluate, so the tuning layer sits downstream of a hardened ingestion boundary that validates every metric before it reaches the comparison logic. Two feeds dominate. Positional metrics arrive through the AIS Data Stream Integration layer as fragmented NMEA 0183 sentences carrying ITU-R M.1371 payloads — position reports (types 1/2/3), Class B reports (type 18), and static/voyage data (type 5). Landside operational metrics — gate throughput, crane cycle times, dwell counters — arrive through adaptive Terminal API Polling Strategies as REST/JSON or UN/EDIFACT CODECO/COARRI messages. Neither can be trusted in raw form: AIS is subject to satellite occlusion, terrestrial receiver saturation, and VHF multipath, while terminal APIs redeliver, batch, and occasionally emit stale snapshots.

The ingestion boundary enforces three guarantees on every metric before an evaluation runs. It validates structure (required fields present, types correct), it validates semantics (coordinates within range, SOG within 0–102.2 knots, MMSI in the assigned MID space), and it stamps provenance (source system, ingestion time, and a monotonic sequence number) so that every alert decision is fully reconstructable. Metrics that fail these checks never reach the threshold engine — they route to a quarantine topic and the last-known-good value holds, preventing a single corrupt broadcast from tripping a spurious alert. Because thresholds are compared against a smoothed signal rather than instantaneous readings, the boundary also applies temporal smoothing over sliding windows (30s, 2m, 15m) and deduplicates broadcasts keyed on (mmsi, timestamp) with a bounded LRU window before the value is handed to the state machine.

Python Data Structure Mapping #

Mapping ITU-R M.1371 and terminal payloads to Python requires explicit schema validation at the ingestion boundary, not defensive try/except scattered through the evaluation code. Production pipelines use pydantic models with strict type coercion so that an out-of-range value is rejected structurally rather than silently producing a NaN comparison later. An AIS positional report maps directly to a typed PositionReport; the threshold configuration itself is a separate versioned model so that boundaries are diffable, auditable, and hot-reloadable from a centralized store (etcd, Consul, or a cloud-native config service) with feature-flag gating for canary rollouts.

from datetime import datetime
from decimal import Decimal
from pydantic import BaseModel, Field, field_validator


class PositionReport(BaseModel):
    """A validated ITU-R M.1371 type 1/2/3 position report."""

    mmsi: int = Field(..., ge=100_000_000, le=999_999_999)  # 9-digit MMSI
    lat: Decimal = Field(..., ge=Decimal("-90"), le=Decimal("90"))
    lon: Decimal = Field(..., ge=Decimal("-180"), le=Decimal("180"))
    sog: float = Field(..., ge=0.0, le=102.2)   # knots
    cog: float = Field(..., ge=0.0, le=359.9)   # degrees
    ts_utc: datetime
    position_accuracy: bool = False             # AIS high-accuracy flag

    @field_validator("ts_utc")
    @classmethod
    def must_be_utc(cls, v: datetime) -> datetime:
        if v.tzinfo is None:
            raise ValueError("timestamp must be timezone-aware UTC")
        return v


class ThresholdBand(BaseModel):
    """A versioned, hysteresis-aware boundary for one metric."""

    metric: str
    upper_bound: float
    lower_bound: float          # re-arm level; lower_bound < upper_bound
    ttl_seconds: int = 60       # sustain time before PRE_ALERT -> ALERT
    version: str                # e.g. "sog-drift@2026-07-01"
    confidence_floor: float = Field(0.75, ge=0.0, le=1.0)

The lower_bound field is not the mirror of upper_bound; it is the re-arm level that implements hysteresis. A metric must climb past upper_bound to arm an alert and fall back below lower_bound to clear it, so a signal oscillating around a single line never flaps between states. This directly filters GPS multipath drift and transient VHF interference, which are the dominant sources of false positives in yard telemetry; applying the same two-level band to zone entry and exit is the technique behind Suppressing flapping geofence alerts with hysteresis. Unit normalisation happens here too: speeds are coerced to knots, timestamps to UTC-aware datetime, and coordinates to WGS84 decimal degrees, matching the equipment identity conventions established by the Container Hierarchy Data Models specification so that an alert can always be tied back to a specific box.

Evaluation itself is state-aware rather than a bare comparison. Instead of if speed > threshold: alert, the engine runs a deterministic finite-state machine whose transitions are the ones drawn in the diagram above:

import structlog
from dataclasses import dataclass

log = structlog.get_logger()


@dataclass
class AlertState:
    state: str = "NORMAL"       # NORMAL | PRE_ALERT | ALERT
    armed_at: datetime | None = None


def evaluate(
    value: float, band: ThresholdBand, st: AlertState, now: datetime
) -> AlertState:
    if st.state == "NORMAL" and value > band.upper_bound:
        log.info("threshold.arm", metric=band.metric, value=value,
                 bound=band.upper_bound, version=band.version)
        return AlertState(state="PRE_ALERT", armed_at=now)
    if st.state == "PRE_ALERT":
        if value < band.lower_bound:                      # recovered within hysteresis
            return AlertState(state="NORMAL")
        sustained = (now - st.armed_at).total_seconds()
        if sustained >= band.ttl_seconds:
            log.warning("threshold.fire", metric=band.metric, value=value,
                        sustained_s=sustained, version=band.version)
            return AlertState(state="ALERT", armed_at=st.armed_at)
    if st.state == "ALERT" and value < band.lower_bound:
        return AlertState(state="NORMAL")
    return st

Validation, Quarantine & Compliance Auditing #

Threshold evaluation is a regulated activity because the alerts it produces feed customs risk assessment, port-facility security, and SOLAS-driven load decisions. Validation is therefore multi-tier. The structural tier is the pydantic boundary above. The semantic tier confirms that a metric is physically plausible in context — an SOG of 40 knots for a container vessel alongside a berth is structurally valid but semantically impossible, so it is quarantined rather than allowed to trip an alert. The regulatory tier tags each evaluation with the compliance context it touches: a threshold breach on a Verified Gross Mass figure is checked against SOLAS Chapter VI before it can gate a loading decision, and access-zone deviations are mapped onto the API scopes defined by the Maritime Security Boundary Setup work so a feed authorized only for berth-approach telemetry can never raise a gate-release alert.

Quarantine and dead-letter routing are deliberately distinct. Semantically implausible but parseable metrics go to a quarantine topic where the last-known-good value continues to hold the threshold state, keeping operations flowing while an exception is triaged. Structurally broken input that cannot be parsed at all goes to a dead-letter queue (DLQ) for out-of-band remediation. This distinction is what lets the alerting plane degrade without either crashing or silently guessing.

Every evaluation — fired, suppressed, or quarantined — emits one structured audit record. Regulatory frameworks (IMO SOLAS V/19, customs manifest requirements, environmental emission reporting) demand immutable, tamper-evident trails, so the record is written to an append-only datastore or write-ahead log, and its retention aligns with port-authority mandates (typically 3–7 years). Structured logging uses structlog with a JSON renderer and a correlation ID injected at ingestion, so a single manifest can be traced from raw broadcast to alert decision. This mirrors the endorsement-chain audit discipline of the Bill of Lading Schema Mapping layer.

import hashlib
import structlog

log = structlog.get_logger()


def audit_evaluation(
    asset_id: str, value: float, band: ThresholdBand,
    state: str, source_confidence: float, correlation_id: str
) -> None:
    """Emit one immutable, hash-chained audit line per evaluation."""
    payload = f"{asset_id}|{value}|{band.version}|{state}|{source_confidence}"
    audit_hash = hashlib.sha256(payload.encode()).hexdigest()
    log.info(
        "threshold.audit",
        asset_id=asset_id,
        metric=band.metric,
        value=value,
        applied_bound=band.upper_bound,
        hysteresis_state=state,
        source_confidence=source_confidence,
        threshold_version=band.version,
        correlation_id=correlation_id,
        audit_hash=audit_hash,        # references prior entry for a tamper-evident chain
    )

Downstream Integration #

A fired alert is only useful if it lands in the right operational context. The tuning layer does not actuate operations directly; it emits idempotent alert events that downstream consumers subscribe to. Synchronization-drift alerts — where AIS and terminal feeds disagree about a container’s state for longer than the configured TTL — are consumed by the Container Status Mapping Rules engine, which decides whether to hold a milestone at PENDING_VERIFICATION rather than advance it on a single untrusted source. Berth-approach and dwell-time alerts feed the Port Call Workflow Design state machine so that pilotage, tug assignment, and customs pre-clearance stay aligned with what the vessels and boxes are actually doing. Setting those dwell boundaries so they fire before a box crosses into chargeable time — without paging on ordinary yard delay — is the subject of Tuning dwell-time thresholds for demurrage alerts.

A hard rule keeps false positives out of the operational bus: an alert only fires when two independent data sources converge on a threshold breach. A speed anomaly seen only in AIS, with no corroborating change in crane cycle time or gate throughput, is scored at reduced confidence and held below the firing threshold. When the alert concerns document-derived metrics rather than live telemetry, the metrics inherit the normalisation discipline of the Document Ingestion & EDI Parsing Workflows domain before they are evaluated. For spatial alerting specifically — geofenced yard zones, crane operational envelopes, tidal-adjusted berth boxes — the boundaries are tuned in Tuning geofence thresholds for yard tracking, which adjusts spatial radii by vessel draft, tidal variation, and equipment kinematics.

Fallback Chains & Uptime Guarantees #

When primary telemetry degrades or the configuration store becomes unreachable, the alerting plane must degrade gracefully without triggering cascading failures. The design principle is that evaluation always returns a typed, confidence-scored answer rather than crashing or guessing. A tiered fallback chain carries that load, and each tier applies progressively wider thresholds to preserve safety margins while suppressing false triggers:

1. Primary AIS stream — full-resolution telemetry, tightest bands, source confidence ≈ 0.95.
2. Secondary terrestrial VHF feed — lower update rate, widened bands, confidence ≈ 0.85.
3. Last-known-good state with an exponential-decay timer — the held value ages out, confidence decaying toward the confidence_floor; alerts require a larger margin to fire.
4. Manual override queue — requires explicit operator acknowledgement; no automatic firing.

Circuit breakers wrap every external dependency (AIS aggregator, terminal API, config store), and transient fetch failures use exponential backoff before the breaker opens and the chain steps down a tier. Configuration is never hardcoded: if the config store is unreachable, the engine holds the last successfully loaded ThresholdBand set and logs a degraded-mode warning rather than reverting to compiled defaults.

from tenacity import retry, stop_after_attempt, wait_exponential
import structlog

log = structlog.get_logger()

FALLBACK_CONFIDENCE = {"primary": 0.95, "vhf": 0.85, "cached": 0.75}


@retry(stop=stop_after_attempt(3),
       wait=wait_exponential(multiplier=2, min=4, max=30))
def fetch_metric(source: str) -> float:
    ...  # raises on transient failure; tenacity backs off, then the breaker opens


def resolve_with_fallback(sources: list[str]) -> tuple[float, float]:
    """Return (value, confidence) from the first healthy tier."""
    for source in sources:                    # ["primary", "vhf", "cached"]
        try:
            value = fetch_metric(source)
            return value, FALLBACK_CONFIDENCE[source]
        except Exception as exc:              # tier exhausted -> step down
            log.warning("fallback.step_down", source=source, error=str(exc))
    log.error("fallback.manual_override_required")
    raise RuntimeError("all telemetry tiers exhausted")

For memory-constrained deployments handling storm-induced telemetry surges, sliding windows use bounded collections.deque ring buffers, historical baselines are held in memory-mapped arrays, and stream processing is generator-based to avoid heap fragmentation. Profile with tracemalloc, cap queue depths to prevent OOM, and enforce strict object lifecycle management so a burst of buffered frames after a reconnection cannot exhaust the heap.

Step-by-step Implementation Guide #

The following steps compose the models and functions above into a runnable evaluation loop. Each step is independently testable against synthetic telemetry.

1. Load versioned threshold bands. Read ThresholdBand records from the centralized config store, validate them with pydantic, and cache the set in memory; on load failure, hold the previous set and log a degraded-mode warning.
2. Validate the incoming metric. Parse the raw AIS/terminal payload into a PositionReport (or the relevant typed model) at the ingestion boundary; route structural failures to the DLQ and semantically implausible values to quarantine.
3. Smooth over the sliding window. Append the validated value to the bounded deque for its (metric, asset) key and compute the windowed value the threshold compares against, discarding entries older than the window TTL.
4. Resolve the source through the fallback chain. Call resolve_with_fallback to obtain (value, confidence); if confidence is below band.confidence_floor, widen the effective band or suppress firing.
5. Run the state machine. Call evaluate with the smoothed value, the band, and the current AlertState; hysteresis and the TTL govern the NORMAL → PRE_ALERT → ALERT transitions.
6. Require dual-source convergence. Before emitting an ALERT, confirm a second independent source corroborates the breach; a single-source breach stays in PRE_ALERT.
7. Emit an audited, idempotent alert event. Call audit_evaluation to write the hash-chained record and publish the alert event with a deterministic idempotency key to downstream consumers.
8. Feed the recalibration loop. Compare fired alerts against ground-truth yard events to compute precision/recall, and stage any boundary change behind a feature flag with a signed configuration diff.

Troubleshooting Common Failures #

Related topics #

• Suppressing flapping geofence alerts with hysteresis — two-level entry/exit bands that stop a box on a zone edge from flapping.
• Tuning dwell-time thresholds for demurrage alerts — setting dwell boundaries that fire before chargeable time without paging on ordinary delay.
• Tuning geofence thresholds for yard tracking — adaptive spatial boundaries by draft, tide, and equipment kinematics.
• AIS Data Stream Integration — parsing and normalising raw NMEA/AIS feeds into the metrics thresholds evaluate.
• Terminal API Polling Strategies — rate-limit-aware polling that supplies landside throughput and dwell metrics.
• Container Status Mapping Rules — the state engine that consumes synchronization-drift alerts.
• Port Call Workflow Design — the downstream workflow that acts on berth and dwell alerts.

↑ Back to Container Tracking & AIS Event Synchronization.