Aonex
Enterprise Multi-Marketplace Catalog Intelligence & Autonomous Data Reconciliation Platform
01. The Multi-Channel Commerce Data Fragmentation Problem
In high-volume e-commerce, product data is notoriously dirty, inconsistent, and distributed across fragmented silos. Brands and multi-channel retailers manage thousands of SKU variations across supplier spreadsheets (CSV/XLSX), direct-to-consumer storefronts (Shopify), and global sales marketplaces (Amazon, eBay). Each channel enforces conflicting taxonomies, attribute schemas, image standards, and validation constraints.
Traditional Product Information Management (PIM) software relies on fragile manual entry or rudimentary regex scrapers that fail when web page layouts change. When data arrives with missing barcodes (GTIN/EAN), corrupted variant matrices, or mismatched specifications, the result is search invisibility, listing rejections, and abandoned purchases.
With Aonex, we engineered an autonomous, high-throughput catalog intelligence platform. The system ingests raw product observations across any channel, applies multi-tier extraction heuristics, executes deterministic multi-source reconciliation, and leverages multi-modal Vision LLMs to classify categories, extract structured attributes, and export synchronized crosswalk feeds.
“Aonex treats product cataloging as a continuous distributed pipeline: raw observations are immutably recorded, identities are scored probabilistically, and canonical attributes are arbitrated deterministically with hallucination-grounded AI enrichment.”
02. Distributed Monorepo Topology with Turborepo, Bun & Hono
To maintain strict type safety and maximize code sharing across the ingestion workers, enrichment microservices, and HTTP API layer, we structured the backend as a high-performance Turborepo monorepo running on the Bun runtime with Hono.
The monorepo separates concerns into specialized domain packages:
// High-level Monorepo Workspace Dependency Topology
apps/
├── api/ // Hono HTTP server (sub-millisecond routing, JWT auth)
└── worker/ // BullMQ distributed job processor pool
packages/
├── catalog/ // Canonical catalog service (write, reconcile, score)
├── ingestion/ // Structured extraction (JSON-LD, microdata, DOM)
├── ingestion-browser/ // Headless Playwright fallback for SPA rendering
├── multi-source-reconciler/ // Identity matching & winning-value arbitration
├── taxonomy-enrichment/ // Vision LLM inference, grounding & isotonic calibration
├── connector-gateway/ // Marketplace facade (Shopify GraphQL, eBay APIs)
└── db/ // Drizzle ORM schema + 55 relational migrations
Database persistence is built on PostgreSQL 16 managed via Drizzle ORM. The data layer utilizes partitioned observation tables, composite GIN indexing on normalized JSONB attribute vectors, and materialized database views to maintain P95 query latencies under 45ms across millions of records.
03. The 4-Tier Ingestion Spine & Headless Browser Escalation
External storefronts and supplier URLs exhibit radically different rendering patterns—ranging from server-rendered HTML with structured schema to client-hydrated React/Vue SPAs with aggressive anti-bot protections. To maximize extraction speed while guaranteeing 99%+ extraction completeness, we built a 4-tier escalating ingestion spine:
- Tier 1 · Structured Schema Extraction (< 80ms): High-speed parsing of embedded
JSON-LD,schema.org Microdata, and OpenGraph tags with zero full-DOM rendering overhead. - Tier 2 · DOM Heuristics & Regex Parsers: Semantic tree traversals extracting tabular specification keys, price-currency normalization, and automated
selector-healthtelemetry detecting layout changes before extractions break. - Tier 3 · Headless Browser Escalation (Playwright): For complex JavaScript-hydrated single-page applications or sites requiring interactive state rendering, the ingestion engine escalates to a pooled Playwright Chromium container to render the full DOM and capture high-resolution screenshots for Vision LLM inspection.
- Tier 4 · Multi-Source GTIN / Barcode Backfill: Automated barcode resolution querying global GTIN/EAN/UPC registry databases with mathematical check-digit validation.
04. Observation Ledger & Deterministic Multi-Source Reconciliation
When a product is observed from multiple sources (e.g. a merchant's Shopify store, an eBay marketplace listing, and a supplier CSV feed), field values frequently conflict. For instance, titles may vary in verbosity, pricing may include different currency conversions, and dimensions may use conflicting units.
Aonex solves this through an Append-Only Observation Ledger. Raw extractions are never overwritten; instead, every incoming data point is logged as an immutable observation linked to its source provenance, reliability prior, and timestamp.
// Conceptual Architecture: Deterministic Multi-Source Reconciliation
interface ProductReconciler {
identityScore: (obsA: ProductObservation, obsB: ProductObservation) => number;
resolveWinningValue: <T>(field: string, observations: FieldObservation<T>[]) => {
winningValue: T;
confidence: number; // Calibrated score [0.0 - 1.0]
provenance: SourceTier; // Selected source authority
};
}
The reconciliation engine evaluates composite identity weights (GTIN matching, MPN sanitization, and brand/title Levenshtein similarity). Once identity is proven, a deterministic arbitration algorithm selects the winning canonical field value based on source authority tiers and recency decay curves.
05. Multi-Modal Vision LLM Enrichment, Grounding & Isotonic Calibration
Raw catalog titles and descriptions often lack structured, marketplace-mandated attributes (e.g. neck style, wattage, material composition, closure type). Aonex integrates a dual-stage intelligence engine to enrich catalogs autonomously:
Deterministic Taxonomy Classification
Maps incoming products into a hierarchical taxonomy tree containing over 1,200 standardized e-commerce category leaves, ensuring immediate schema compliance with Google Shopping, Amazon, and eBay standards.
Multi-Modal Vision LLM Inference
High-throughput Vision LLM models inspect both normalized text descriptions and rendered product imagery to detect subtle visual attributes (colors, patterns, connector types, packaging counts) that are absent from raw text feeds.
Hallucination Grounding & Isotonic Confidence Calibration
To eliminate AI hallucinations, all model outputs are validated against strict leaf-level attribute schemas. We implemented an isotonic regression calibration layer that converts raw model probabilities into reliable confidence scores, flagging low-confidence values for automated human-in-the-loop review.
06. High-Throughput Worker Queues & Distributed Job Architecture
Processing 50,000+ product updates per hour requires a resilient asynchronous execution engine. In apps/worker, we deployed a multi-concurrency BullMQ cluster powered by Redis 7.
The worker architecture isolates workloads into specialized priority queues:
- Real-Time Ingestion Queue: Handles merchant webhooks and single-URL crawls with sub-second prioritization.
- Batch Enrichment Queue: Batches product payloads for token-efficient Vision LLM inference, respecting API rate limits with exponential backoff and jitter.
- Transactional Outbox Poller: Guarantees exactly-once event delivery between Postgres database writes and external message queues, preventing orphaned tasks during server restarts.
- Continuous Watchdog & Drift Detectors: Runs automated hourly and daily health checks measuring Population Stability Index (PSI) and attribute null-rates to catch upstream schema drift.
07. Multi-Marketplace Connector Gateway & Automated Schema Crosswalks
Once product catalogs are normalized and enriched, Aonex synchronizes the canonical catalog with downstream sales channels via the connector-gateway:
Shopify Integration: Utilizes the Shopify GraphQL Admin API with Bulk Operation mutations to sync thousands of variant metafields, price updates, and inventory levels in single batch payloads.
eBay Inventory & Trading Integration: Maps canonical taxonomy attributes directly into eBay item specifics, category IDs, and listing policies with automated aspect validation.
Automated Schema Crosswalks: A declarative translation layer transforms canonical entities into channel-specific JSON feeds (Amazon SP-API, Google Merchant Center, custom feeds), ensuring zero manual formatting overhead for retailers.
08. Real-Time Command Centre, Time-Series Price Analytics & Frontend Engineering
The frontend is built with Next.js 15 App Router and React 19. Designed for enterprise merchandising teams, the application combines high-density data tables with responsive visual analytics:
- Command Centre: Real-time operations dashboard tracking ingestion throughput, active worker concurrency, queue backlog depths, and channel health indicators.
- Interactive Taxonomy & Catalog Explorer: Hierarchical category browser with instant variant inspection, winning value overrides, and confidence badges.
- Time-Series Price Indexing: Built with
Recharts, rendering dynamic competitor pricing trends, market distribution curves, and margin volatility indexes. - WebGL Aesthetic & Motion: Integrated subtle WebGL background canvas visualizations (
Three.js / OGL) and fluid micro-interactions withFramer Motion 12.
Technical Specifications
Visual Gallery
01 / 06
Fig 01 · Command Centre & Real-Time System Telemetry01
Fig 02 · Hierarchical Category Taxonomy & Attribute Schema02
Fig 03 · 4-Tier Ingestion Console & Extraction Anomaly Lab03
Fig 04 · Multi-Modal Vision LLM Enrichment & Grounding Studio04
Fig 05 · Time-Series Price Index & Competitor Trend Analysis05
Fig 06 · Multi-Marketplace Channel Optimization & Crosswalk Engine06