IFCSUM EDI Message Parsing #

IFCSUM EDI Message Parsing is the deterministic pipeline that turns a UN/EDIFACT International Forwarding and Consolidation Summary interchange — the aggregated manifest that carriers and consolidators exchange ahead of a vessel call — into a typed, validated, audit-traceable record that terminal and customs systems can act on without re-reading raw segments. Unlike the OCR-and-NLP path behind PDF Bill of Lading Extraction, an IFCSUM (specifically the 311 message variant) arrives as strictly structured segments, so parsing is deterministic rather than probabilistic — but that determinism comes with a hard obligation to handle carrier dialects, conditional qualifiers, and exact segment sequencing. As a discipline inside Document Ingestion & EDI Parsing Workflows, a production IFCSUM parser must guarantee sub-second throughput, zero silent field loss, and a full audit trail to satisfy port authority SLAs and customs pre-arrival mandates.

Ingestion Boundary & Protocol Handling #

Production ingestion must decouple transport receipt from business logic so a slow or malformed feed never blocks the downstream terminal operating system (TOS). IFCSUM interchanges arrive over AS2, SFTP, or a REST gateway and land on a high-throughput staging queue before any segment is parsed. The boundary applies the same syntax-tolerant, semantics-strict posture used across the parent Document Ingestion & EDI Parsing Workflows: accept anything parseable at the wire, then apply uncompromising checks inward.

The boundary executes a strict, fail-fast sequence:

1. Envelope validation. Parse the UNB interchange header and UNZ trailer first. Extract sender and receiver IDs, the syntax version, and the interchange control reference, and cross-check them against an authorized trading-partner registry. Never read delimiters from a default — the UNA service string advice, when present, declares the component (:), data (+), decimal, and release (?) characters for this interchange. The defensive parsing this demands, including how carriers silently rotate separators between feeds, is covered in Handling UNA service strings and delimiter drift.
2. Acknowledgment generation. Emit a CONTRL message synchronously on successful envelope validation to satisfy the EDI handshake; a functional APERAK follows later if business-level rejection occurs.
3. Message isolation. Dispatch validated interchanges to the Async Batch Processing Pipelines layer for segment-level work. The parser isolates individual UNH/UNT message blocks, verifies the type identifier (IFCSUM:311), and routes each payload to the appropriate mapper by directory version (for example D95B versus D01B).

Transport metadata, routing identifiers, and control references are stripped and written to an immutable audit store before the core business segments enter the normalization engine. Staging this way means malformed envelopes fail fast without ever consuming segment-parsing compute.

Python Data Structure Mapping #

Mapping UN/EDIFACT to Python demands explicit type safety, strict index tracking, and carrier-dialect normalization. The parser runs as a generator-based state machine, tracking loop boundaries — NAD party loops, CNI consignment loops, GID goods loops — while resolving composite separators (:), data separators (+), and the release character (?) without corrupting embedded free text. Implicit dictionaries are unacceptable in production; every field carries a declared type, a source alias, and a validation verdict.

from __future__ import annotations

from datetime import datetime, timezone
from decimal import Decimal
from enum import Enum
from typing import Optional

import structlog
from pydantic import BaseModel, ConfigDict, Field, field_validator

log = structlog.get_logger("ifcsum.parser")


class PartyQualifier(str, Enum):
    # UN/EDIFACT NAD party function qualifiers (DE 3035)
    SHIPPER = "CZ"     # Consignor / shipper
    CONSIGNEE = "CN"   # Consignee
    CARRIER = "CA"     # Carrier
    NOTIFY = "NI"      # Notify party


class TransportMode(str, Enum):
    VESSEL = "1"
    RAIL = "2"
    ROAD = "3"


class IFCSUMHeader(BaseModel):
    model_config = ConfigDict(populate_by_name=True)
    message_ref: str = Field(alias="BGM_C106_1004")
    doc_type: str = Field(alias="BGM_C002_1001")
    issue_date: Optional[datetime] = None


class TransportDetails(BaseModel):
    model_config = ConfigDict(populate_by_name=True)
    mode: TransportMode = Field(alias="TDT_8067")
    vessel_imo: int = Field(alias="TDT_C222_8213", ge=1_000_000, le=9_999_999)
    voyage_number: Optional[str] = Field(default=None, alias="TDT_C222_8212")


class Consignment(BaseModel):
    model_config = ConfigDict(populate_by_name=True)
    consignment_ref: str = Field(alias="CNI_C503_1004")
    goods_desc: Optional[str] = None
    gross_weight_kg: Optional[Decimal] = Field(default=None, alias="MEA_C174_6314")
    weight_unit: Optional[str] = Field(default=None, alias="MEA_C174_6411")
    imdg_class: Optional[str] = Field(default=None, alias="DGS_C205_7124")

    @field_validator("gross_weight_kg")
    @classmethod
    def positive_weight(cls, v: Optional[Decimal]) -> Optional[Decimal]:
        if v is not None and v <= 0:
            raise ValueError("gross weight must be positive")
        return v


class IFCSUMPayload(BaseModel):
    header: IFCSUMHeader
    transport: TransportDetails
    consignments: list[Consignment] = Field(default_factory=list)
    raw_segment_indices: dict[str, int] = Field(default_factory=dict)

Field resolution follows a strict precedence chain: primary carrier field → dialect alias → computed default → explicit None. Standard coercions include DTM values in CCYYMMDDHHMM (format qualifier 203) into timezone-aware UTC datetimes, MEA+WT+...:KGM weights into Decimal kilograms held at full precision until the presentation boundary, and LOC codes into verified UN/LOCODE strings. Segment positions are preserved in raw_segment_indices so a validation failure can be localized to the exact byte range in the original interchange. The segment-by-segment parsing detail — escape-sequence resolution, positional indexing, and memory-safe streaming — is covered in Parsing IFCSUM 311 messages with Python. Where the same cargo also appears as individual bills, the consolidated record reconciles against the Bill of Lading Schema Mapping layer so consignment weights and party IDs agree across both feeds.

Validation, Quarantine & Compliance Auditing #

IFCSUM data must align with UN/EDIFACT directory rules and local customs pre-arrival requirements. Validation runs as three deterministic tiers, and the tier that fails decides where the record goes.

1. Structural validation. Enforce mandatory segment presence, correct loop nesting, and code-list membership (UN/LOCODE, INCOTERMS 2020, ISO 6346 container identifiers). A malformed envelope, a missing UNT, or an unreadable release sequence is a format defect: the parser halts, returns a structured CONTRL/APERAK rejection, and routes the interchange to a dead-letter queue (DLQ).
2. Semantic validation. Cross-check weight and volume consistency, validate hazardous goods (IMDG class codes in DGS segments), verify seal formats, and confirm the IMO number’s mod-11 check digit. These are recoverable business exceptions — they route to a QUARANTINE topic, preserving throughput while ops teams triage.
3. Regulatory validation. Align extracted data with IMO FAL Convention pre-arrival manifest requirements and WCO Data Model fields. Missing customs-required attributes — HS codes in GID, VGM below tare weight — are flagged for manual review, never silently forwarded.

The DLQ-versus-quarantine split is load-bearing: unrecoverable corruption a human cannot fix without engineering stays in the DLQ, while a stale registry or a missing optional field lands in quarantine and reconciles automatically once an authoritative source recovers. Every validation event is emitted as a structured JSON log carrying control_ref, segment_index, error_code, field_path, original_value, and compliance_status, so root-cause analysis never requires parsing a raw EDI dump. That immutable trail is what satisfies customs audits and port state control inspections, and it is written under the zero-trust controls defined in Maritime Security Boundary Setup.

Downstream Integration #

A parsed IFCSUM record is a fan-out point, not an endpoint. The consolidated manifest feeds the TOS berth and yard plan, customs pre-arrival declarations, and stowage planning, so the schema tags regulatory attributes and operational flags explicitly rather than forcing consumers to re-infer meaning from free text.

Because an IFCSUM aggregates many consignments, each referencing one or more containers with distinct seals and cargo, resolving those references into the Container Hierarchy Data Models Vessel → Bay → Tier → Row → Container topology lets equipment tracking, reefer monitoring, and dangerous-goods segregation run deterministically off a shared structure. As the vessel’s estimated arrival firms up, the manifest emits idempotent milestones — keyed on a deterministic (message_ref, consignment_ref, event_hash) tuple — consumed by the Port Call Workflow Design state machine, so an AS2 retransmission or an API retry collapses to a no-op instead of double-triggering a berth plan. Physical movement of the consolidated boxes is then correlated against the live position feed from AIS Data Stream Integration, closing the loop between the declared manifest and the observed vessel. Records that clear every tier are the same shape a general-purpose validator would accept, which is why the Schema Validation Frameworks layer can gate them without carrier-specific branching.

Fallback Chains & Uptime Guarantees #

Carrier dialect drift is inevitable, so uptime depends on graceful degradation rather than brittle rejection. When the parser meets a non-standard qualifier or a deprecated segment repetition, it walks a deterministic fallback chain:

1. Strict mode. Validate against the canonical UN/EDIFACT schema for the declared directory version. This clears the large majority of compliant feeds.
2. Tolerant mode. Activated when a structural error traces to a known carrier extension. Apply dialect-specific mapping overrides loaded from versioned YAML configs, tag the record data_quality="NORMALISED", and log every override as a telemetry event so schema maintainers see drift before it spreads.
3. Raw preservation. If both tiers fail, serialize the payload to the DLQ with full context — transport headers, partial parse state, exception traceback — and keep terminal operations moving.

Resilience is reinforced with circuit breakers on external registry lookups (open after five consecutive 5xx/timeout errors and fall back to cached code lists), exponential backoff with jitter on transient transport failures, and SHA-256 checksums that bind every parsed payload to its raw EDI counterpart for audit integrity. Monitoring tracks parse success rate, fallback activation frequency, and DLQ backlog depth so schema updates land before a carrier change stalls a vessel-clearance cycle.

Step-by-step Implementation Guide #

The reference pipeline below turns a raw IFCSUM 311 interchange into a validated, routed record. Each step is runnable in isolation and uses type annotations with structlog for structured JSON logging.

Step 1 — Validate the envelope and read the service string #

import structlog

log = structlog.get_logger()


def read_service_string(raw: str) -> dict[str, str]:
    # UNA<comp><data><decimal><release><space><segment>; default per ISO 9735
    if raw.startswith("UNA"):
        una = raw[3:9]
        d = {"component": una[0], "data": una[1], "release": una[3], "segment": una[5]}
    else:
        d = {"component": ":", "data": "+", "release": "?", "segment": "'"}
    log.info("service_string", **d)
    return d

Step 2 — Isolate the IFCSUM message and confirm its type #

def isolate_messages(raw: str, seg: str) -> list[str]:
    blocks, depth = [], []
    for segment in raw.split(seg):
        tag = segment[:3]
        if tag == "UNH":
            depth = [segment]
        elif tag == "UNT":
            depth.append(segment)
            blocks.append(seg.join(depth))
        elif depth:
            depth.append(segment)
    return blocks


def assert_ifcsum(unh_line: str) -> None:
    if "IFCSUM:311" not in unh_line:
        raise ValueError(f"unexpected message type: {unh_line!r}")

Step 3 — Tokenise segments and resolve the release character #

def tokenise(message: str, d: dict[str, str]) -> list[list[str]]:
    out: list[list[str]] = []
    for seg in message.split(d["segment"]):
        if not seg.strip():
            continue
        # honour the release char so "?+" is a literal, not a separator
        parts, buf, i = [], "", 0
        while i < len(seg):
            ch = seg[i]
            if ch == d["release"] and i + 1 < len(seg):
                buf += seg[i + 1]
                i += 2
                continue
            if ch == d["data"]:
                parts.append(buf); buf = ""
            else:
                buf += ch
            i += 1
        parts.append(buf)
        out.append(parts)
    return out

Step 4 — Map indexed segments into the typed payload #

def build_payload(tokens: list[list[str]]) -> IFCSUMPayload:
    idx: dict[str, list[list[str]]] = {}
    for seg in tokens:
        idx.setdefault(seg[0], []).append(seg)

    header = IFCSUMHeader(
        BGM_C002_1001=idx["BGM"][0][1].split(":")[0],
        BGM_C106_1004=idx["BGM"][0][2],
    )
    transport = TransportDetails(
        TDT_8067=idx["TDT"][0][1],
        TDT_C222_8213=int(idx["TDT"][0][8].split(":")[0]),
    )
    payload = IFCSUMPayload(header=header, transport=transport)
    log.info("payload_structurally_valid", message_ref=header.message_ref)
    return payload

Step 5 — Apply semantic checks, then route #

def route(payload: IFCSUMPayload, *, locode_ok: bool, vgm_ok: bool) -> str:
    if not locode_ok:
        log.warning("quarantine", ref=payload.header.message_ref, reason="LOCODE_UNKNOWN")
        return "QUARANTINE"
    if not vgm_ok:
        log.warning("quarantine", ref=payload.header.message_ref, reason="VGM_MISSING")
        return "QUARANTINE"
    log.info("payload_accepted", ref=payload.header.message_ref)
    return "ACCEPTED"

A pydantic.ValidationError raised in Step 4 is a structural defect bound for the DLQ; a False verdict in Step 5 is a recoverable business exception bound for quarantine.

Troubleshooting Common Failures #

Frequently Asked Questions #

Related #

• Handling UNA service strings and delimiter drift — defensively reading per-interchange delimiters when carriers rotate separators between feeds
• Parsing IFCSUM 311 messages with Python — the segment-level parsing detail beneath this pipeline
• PDF Bill of Lading Extraction — the OCR/NLP counterpart for unstructured documents
• Async Batch Processing Pipelines — the queue layer that isolates and dispatches interchanges
• Schema Validation Frameworks — the general-purpose gate for typed records
• Bill of Lading Schema Mapping — the per-shipment layer IFCSUM output reconciles against

Up: Document Ingestion & EDI Parsing Workflows — the parent domain governing ingestion, parsing, and validation.