Terminal API Polling Strategies #

Terminal API polling strategies define how a landside ingestion service pulls container status, gate transactions, and equipment assignments from Terminal Operating Systems (TOS) at a cadence that stays fresh without exhausting carrier rate limits — the pull-side counterpart to the push-based AIS Data Stream Integration feed inside the parent Container Tracking & AIS Event Synchronization domain. For shipping operations teams, port authorities, and Python automation engineers, reliable polling is a control plane, not a background job: naive fixed-interval requests either starve the state machine of berthing events or trip a 429 storm that blacklists the client for hours. This page covers how TOS payloads arrive, how they are typed and validated, how their output is fed downstream, and how the loop degrades gracefully when an endpoint goes dark.

Ingestion Boundary & Protocol Handling #

TOS vendors — Navis N4, Tideworks Mainsail, Kaleris, and the port community systems that wrap them — expose container events over RESTful JSON endpoints, occasionally alongside a legacy CODECO/COARRI EDI drop. The ingestion boundary is the point where those requests are shaped, throttled, and authenticated before any payload touches business logic. Two disciplines govern it: an adaptive cadence and a rate-limit-aware transport.

Fixed-interval cron jobs fail under operational variance because a berth that is idle for six hours needs the same client that must catch a 40-move-per-hour discharge in near real time. Polling intervals must therefore be driven by the vessel’s operational phase, aligned to the same milestones the Port Call Workflow Design state machine tracks:

The transport itself must survive high concurrency. Production loops use httpx.AsyncClient with a persistent connection pool so the TCP and TLS handshakes are amortised across cycles rather than paid per request — critical when a single facility integration fans out to dozens of endpoints. Each cycle carries a terminal-specific sequence_id cursor and an ISO 8601 event_timestamp, and requests are made conditional (If-None-Match / If-Modified-Since) so an unchanged resource returns 304 Not Modified and costs no payload transfer against the rate budget — the full pattern, including how a returned ETag is stored and replayed, is worked through in Adaptive polling with ETag conditional requests. Cursor-based pagination with monotonic sequence tracking guarantees that a mid-cycle failure resumes from the last acknowledged token instead of re-fetching the full dataset. Idempotent request signatures — a deterministic hash of (endpoint, cursor, phase) — let the client dedupe retries so a redelivered page never double-counts a gate move.

Python Data Structure Mapping #

Raw TOS JSON rarely aligns with internal data models. Every field must be coerced into a typed, validated structure at the boundary — loose dictionaries propagate silent drift into the state store. The canonical model binds ISO 6346 container coding, UN/LOCODE terminal identifiers, and TOS status codes into a frozen dataclass whose __post_init__ guards reject malformed identifiers before they enter the queue. Equipment identity resolves against the nested relationships defined by the Container Hierarchy Data Models specification, so a bare box number is always tied back to its size-type and any grouping.

from dataclasses import dataclass
from enum import Enum
from datetime import datetime
import re

class ContainerStatus(Enum):
    YARD = "YARD"
    GATE_IN = "GATE_IN"
    GATE_OUT = "GATE_OUT"
    VESSEL_ONBOARD = "VESSEL_ONBOARD"
    CUSTOMS_HOLD = "CUSTOMS_HOLD"

@dataclass(frozen=True)
class TerminalEvent:
    sequence_id: int
    iso_6346: str
    un_locode: str
    status: ContainerStatus
    event_ts: datetime
    raw_payload_hash: str

    def __post_init__(self) -> None:
        # ISO 6346: 4 letters (owner + category) + 6 serial digits + 1 check digit
        if not re.match(r"^[A-Z]{4}\d{7}$", self.iso_6346):
            raise ValueError(f"Invalid ISO 6346 format: {self.iso_6346}")
        # UN/LOCODE: 2-letter ISO country code + 3-char location code
        # (letters A-Z and digits 2-9; 0 and 1 are excluded to avoid O/I confusion)
        if not re.match(r"^[A-Z]{2}[A-Z2-9]{3}$", self.un_locode):
            raise ValueError(f"Invalid UN/LOCODE format: {self.un_locode}")

Field coercion rules matter as much as the shapes. Timestamps arrive in local port time as often as UTC and must be normalised to timezone-aware UTC datetime objects before comparison; weights carried on the same payload are normalised to kilograms; and proprietary TOS status strings are held as-is here and translated separately, because the mapping is version-controlled and hot-reloadable. Keeping the transport model (TerminalEvent) distinct from the resolved lifecycle state means a vendor renaming ONBOARD to LOADED is a one-line adapter change, not a schema migration.

Validation, Quarantine & Compliance Auditing #

Validation runs as a three-tier gate at the ingestion boundary, and anything that fails a tier is quarantined rather than dropped:

1. Structural — required fields present, types correct, JSON well-formed. A truncated page or an unexpected null fails here.
2. Semantic — ISO 6346 modulo-11 check digit valid, UN/LOCODE resolvable against the reference registry, event_ts within a plausible window and not regressing behind the last committed state for that container.
3. Regulatory — Verified Gross Mass present and within SOLAS Chapter VI limits where the event gates a load, and the source credential scoped to the facility zone that produced the event.

Rejected payloads route to a quarantine topic with the full raw snapshot and the failing tier attached, which keeps a single malformed record from poisoning the batch while preserving it for replay once the upstream defect is fixed. This is deliberately distinct from the dead-letter queue described below: quarantine holds structurally or semantically invalid input awaiting a rules or data fix, whereas the DLQ holds well-formed events that could not be resolved even in degraded mode.

Maritime operations require deterministic audit trails for SOLAS, ISPS, and port-authority review, so every poll cycle emits structured JSON logs through structlog — never bare print() — carrying a correlation ID that threads a single logical cycle across retries. Each cycle records poll_start_ts, poll_end_ts, latency_ms, sequence_gap_detected, fallback_activated, and payload_bytes_received, plus the request/response hashes that let an auditor reconstruct exactly what was fetched. Logs ship to a centralized, immutable store; port authorities routinely audit terminal data pipelines during incident investigations, and a missing cycle is itself a finding. The same append-only discipline applied to commercial documents in the Bill of Lading Schema Mapping layer governs the polling audit trail — each entry references the idempotency key of the event that produced it.

Downstream Integration #

Polling exists to feed the correlation core, not to be an endpoint. Normalised TerminalEvent records are published to a unified event bus before any downstream system sees them, so that state drift can never propagate directly into yard planning or billing. From the bus, three consumers matter:

• State resolution. Events flow into the Container Status Mapping Rules engine, which fuses the terminal move with a plausible AIS state before committing a transition — a VESSEL_ONBOARD event is only trusted when the vessel is moored at the expected berth.
• Port-call orchestration. The resolved states emit idempotent events consumed by the Port Call Workflow Design state machine so berth allocation and customs pre-clearance stay aligned with physical container movement.
• Alerting. Cycle telemetry feeds the Threshold Tuning for Alerts layer, which decides whether a latency spike or sequence gap is transient noise or a systemic outage.

Where a facility still delivers container events as EDI rather than REST, those messages pass through the normalisation discipline documented in the Document Ingestion & EDI Parsing Workflows domain before joining the same bus, so the polling client and the EDI path converge on one typed contract. The exact REST call construction — headers, pagination cursors, and conditional-request semantics — is covered in depth in Polling terminal operating systems via REST APIs.

Fallback Chains & Uptime Guarantees #

Terminal APIs experience scheduled maintenance windows, transient network partitions, and aggressive rate-limiting. A production loop classifies every response deterministically before deciding to retry:

• Recoverable: 429 Too Many Requests, 502 Bad Gateway, 503 Service Unavailable
• Non-recoverable: 400 Bad Request, 401 Unauthorized, 404 Not Found

Recoverable failures retry with exponential backoff and jitter (base=2s, max=60s, jitter=0.1) to avoid a thundering herd when a terminal comes back online. A circuit breaker wraps each endpoint and opens after consecutive failures, halting polling for that endpoint until a lightweight health check passes — this stops the client from burning its rate budget against a wall. When the primary endpoint is unavailable, operations route through a tiered fallback chain rather than blocking:

1. Primary — live TOS REST endpoint at the phase-appropriate cadence (confidence ≥ 0.95).
2. Secondary — a Redis-backed cursor cache serving last-known-good state (confidence 0.75) so dashboards and provisional planning keep working.
3. Tertiary — AIS position and ETA correlation from the AIS Data Stream Integration feed (confidence 0.50), which informs yard planning but never actuates a gate release. Reconstructing the missed cycles once the endpoint recovers — reconciling the AIS-derived state against the resumed cursor — is the technique covered in Backfilling terminal API gaps with AIS events.
4. Quaternary — a manual reconciliation queue plus a port-authority alert.

Every fallback transition is logged, auditable, and reversible, and degraded polling is never allowed to silently overwrite a verified yard state. Events that cannot be resolved even at the lowest tier — an unknown LOCODE, a malformed page that survived quarantine replay — are raised as an UnresolvableEvent and routed to a dead-letter queue for out-of-band remediation.

Step-by-step Implementation Guide #

Each step below is runnable in isolation and uses type annotations plus structlog for structured logging.

Step 1 — Configure the async client and structured logger #

Build one long-lived httpx.AsyncClient with keep-alive and explicit timeouts, and bind a correlation ID so every log line in a cycle is joinable.

import httpx
import structlog

structlog.configure(processors=[structlog.processors.JSONRenderer()])
log = structlog.get_logger()

def build_client() -> httpx.AsyncClient:
    limits = httpx.Limits(max_keepalive_connections=20, max_connections=100)
    timeout = httpx.Timeout(connect=5.0, read=15.0, write=5.0, pool=5.0)
    return httpx.AsyncClient(limits=limits, timeout=timeout)

Step 2 — Resolve the adaptive cadence #

Choose the interval from the vessel’s operational phase rather than a fixed constant.

def poll_interval_seconds(phase: str) -> int:
    tiers: dict[str, int] = {"alongside": 20, "yard": 240, "offpeak": 600}
    return tiers.get(phase, 600)

Step 3 — Fetch one page with a conditional cursor request #

Send the last ETag so an unchanged resource returns 304 and spends no payload budget.

async def fetch_page(
    client: httpx.AsyncClient, url: str, cursor: str, etag: str | None
) -> httpx.Response:
    headers = {"If-None-Match": etag} if etag else {}
    resp = await client.get(url, params={"cursor": cursor}, headers=headers)
    log.info("poll.fetch", url=url, cursor=cursor, status=resp.status_code)
    return resp

Step 4 — Validate and normalise into typed events #

Reject malformed payloads at the boundary; construct TerminalEvent objects that self-validate in __post_init__.

from datetime import datetime, timezone
import hashlib, json

def to_event(record: dict) -> TerminalEvent:
    payload_hash = hashlib.sha256(
        json.dumps(record, sort_keys=True).encode()
    ).hexdigest()
    return TerminalEvent(
        sequence_id=int(record["seq"]),
        iso_6346=record["container"].upper(),
        un_locode=record["locode"].upper(),
        status=ContainerStatus(record["status"]),
        event_ts=datetime.fromisoformat(record["ts"]).astimezone(timezone.utc),
        raw_payload_hash=payload_hash,
    )

Step 5 — Wrap the cycle in retry, backoff, and a circuit breaker #

Classify the failure, retry only recoverable codes, and open the breaker on repeated failure.

from tenacity import retry, stop_after_attempt, wait_exponential_jitter

RECOVERABLE = {429, 502, 503}

@retry(stop=stop_after_attempt(5),
       wait=wait_exponential_jitter(initial=2, max=60, jitter=0.1))
async def poll_once(client: httpx.AsyncClient, url: str, cursor: str) -> httpx.Response:
    resp = await fetch_page(client, url, cursor, etag=None)
    if resp.status_code in RECOVERABLE:
        log.warning("poll.recoverable", status=resp.status_code, url=url)
        resp.raise_for_status()
    return resp

Step 6 — Persist the cursor and publish to the event bus #

Advance the cursor atomically after acknowledgement, and prune tokens older than 24 hours to bound state storage.

async def commit(redis, endpoint: str, event: TerminalEvent, bus) -> None:
    await redis.set(f"cursor:{endpoint}", event.sequence_id)
    await bus.publish("terminal.events", event)
    log.info("poll.commit", endpoint=endpoint, seq=event.sequence_id,
             fallback_activated=False)

Troubleshooting Common Failures #

For long-running async workers, address memory pressure directly: stream-parse large payloads with ijson or orjson instead of a synchronous json.loads() in the event loop, yield events from generators rather than accumulating lists, reuse the httpx.AsyncClient instance, and evict acknowledged sequence tokens from Redis after 24 hours. Monitor RSS and GC pause times with tracemalloc and psutil; when memory exceeds 80%, apply graceful backpressure by extending poll intervals and deferring non-critical reconciliation.

Frequently Asked Questions #

Related topics #

• Adaptive polling with ETag conditional requests — storing and replaying the ETag so unchanged resources return 304 and spend no rate budget.
• Backfilling terminal API gaps with AIS events — reconstructing missed poll cycles from the AIS feed once an endpoint recovers.
• Polling terminal operating systems via REST APIs — header, pagination, and conditional-request construction for TOS REST calls.
• AIS Data Stream Integration — the push-based vessel feed that backs the tertiary fallback tier.
• Container Status Mapping Rules — deterministic state resolution the polled events feed into.
• Threshold Tuning for Alerts — turning cycle telemetry into transient-vs-systemic alerting.
• Port Call Workflow Design — the orchestration state machine that consumes resolved container states.

↑ Back to Container Tracking & AIS Event Synchronization.