Zum Inhalt springen
SAP, DATEV and Dynamics experts
Message QueuesEvent-DrivenDead-Letter Handling

The invisible layer that holds your systems together

In complex system landscapes with ERP, shop, PIM, CRM and accounting, you need a central instance that orchestrates data flows, transforms formats and reliably handles errors. Our middleware solutions take on exactly this role: asynchronous, resilient and scalable.

When two systems communicate, a direct API connection often suffices. But as soon as three, four or more systems are involved, dedicated middleware becomes indispensable. Without central orchestration, point-to-point connections emerge that grow exponentially complex with each new integration. Our middleware solutions serve as central hubs: they receive data from source systems, transform it into target formats and reliably distribute it to all connected systems, even when individual recipients are temporarily unavailable.

Why middleware is indispensable in e-commerce integrations

The typical system landscape of a mid-market online retailer comprises at least four systems: the shop as sales channel, the ERP system for inventory management, accounting software for financial reporting, and often a PIM system for product data. Without middleware, each system would need to communicate directly with every other, resulting in six bilateral connections for four systems. With six systems, that becomes fifteen connections. Each connection must be developed, tested, maintained and adapted for updates.

Middleware reduces this complexity to a hub-and-spoke architecture: each system communicates only with the middleware, which serves as central mediator. This means: with four systems, only four connections instead of six. With six systems, only six instead of fifteen. And when a new system is added, only a single new connection to the middleware is needed, rather than connecting it to all existing systems.

Core functions of our middleware solutions

Message Routing

Messages are rule-based routed to the correct target systems. Content-based routing enables distribution based on message content: product updates go to the shop, invoice data to DATEV, stock changes to warehouse management.

Data Transformation

Data formats are automatically converted between source and target systems. XML to JSON, different date formats, unit conversion, currency conversion and field-level mapping rules are centrally managed in the middleware.

Error Handling and Retry

Circuit breaker pattern for controlled interruption during system outages. Retry strategies with exponentially growing wait times. Dead-letter queues for messages that could not be delivered after defined attempts.

Persistent Message Queues

Messages are persistently stored before forwarding to target systems. Even during a complete system outage, no data is lost. After recovery, all pending messages are automatically delivered.

Monitoring and Dashboards

Real-time monitoring of all message flows: throughput, latency, error rate and queue depth. Configurable alerts notify on anomalies. Historical metrics enable capacity planning and trend analysis.

Horizontal Scaling

The middleware scales horizontally with growing data volume. Additional worker processes handle more messages in parallel. Clustering ensures availability even when individual nodes fail.

Architecture patterns for different integration scenarios

Depending on system landscape complexity and latency and data volume requirements, we employ different architecture patterns. The following three patterns cover the most common scenarios in our e-commerce integration projects.

Hub-and-Spoke

Classic integration pattern: all systems communicate via the central middleware. Ideal for landscapes with three to six systems where the middleware centrally manages routing, transformation and error handling.

Event-Driven Architecture

Systems publish events on data changes, processed by interested consumers. Publish-subscribe pattern for maximum decoupling. Ideal for highly dynamic scenarios with many parallel data flows.

Microservices Integration

For complex transformation logic, we use specialized microservices: one for SAP communication, one for pricing calculation, one for DATEV preparation. Orchestration via a central message queue.

Dead-letter handling: What happens when things go wrong

In distributed systems, errors are not the exception but the rule. Network interruptions, timeout scenarios, malformed data and temporary system outages occur regularly in daily operations. The question is not whether errors occur, but how the system handles them.

Our middleware implements a multi-level error handling concept: transient errors (network timeouts, brief unavailability) are handled by automatic retries with exponentially growing wait times. When a target system is unreachable for longer, the circuit breaker activates and prevents the queue from being flooded with unsuccessful delivery attempts. Messages that could not be delivered after a configurable number of attempts move to the dead-letter queue.

The dead-letter queue is not a black hole. Every message in the DLQ contains the complete context: original message, error descriptions of all delivery attempts, timestamps and target information. Via a management interface, administrators can analyze failed messages, fix the root cause and reprocess them. Automated reports inform the operations team about DLQ status and escalate at critical thresholds.

Technology stack of our middleware solutions

RabbitMQ

Proven message broker with support for persistent messages, flexible routing topologies (Direct, Topic, Fanout), clustering and management API. Our preferred choice for most e-commerce integrations.

Redis Streams

For scenarios with extremely high throughput and low latency, we use Redis Streams as a lightweight alternative. Consumer groups enable parallel processing with at-least-once delivery.

Node.js Workers

Asynchronous event processing in Node.js for maximum throughput in I/O-intensive operations. Type-safe implementation with TypeScript, comprehensive test coverage and containerized deployment.

Middleware in the context of the overall architecture

The middleware is the connecting piece between all other integration components. It consumes the REST and GraphQL APIs of connected systems, orchestrates data flows and delivers normalized data to all target systems. In our projects, the middleware works closely with the ERP connectors for SAP and Dynamics 365, the DATEV interface and the shop adapter.

A key advantage of middleware architecture is the replaceability of individual components. When you change your ERP system, only the ERP connector needs adaptation. The middleware, shop connectivity and DATEV interface remain unchanged. Likewise, the shop can be changed without rebuilding the ERP integration. This modular architecture protects your investment and gives you long-term flexibility. Contact us for advice on your middleware architecture.

Observability: Complete insight into every data flow

In a middleware architecture, observability is not optional but a necessity. When data flows through multiple processing steps, it must be traceable at all times where a message is, how long processing took and whether errors occurred. We implement a three-level observability concept: metrics show throughput, latency and error rate at queue and endpoint level. Logs document every processing step with context (message content, source and target system, transformation steps). And traces enable end-to-end tracking of a single message through all processing stages.

Metrics are visualized in real-time dashboards that give both the technical operations team and business departments insight. The operations team sees queue depth, processing time and error rate. The business department sees the number of synchronized products, orders and invoices per period. Configurable alerts notify on anomalies: when queue depth exceeds a threshold, error rate increases or processing time exceeds a critical value, the operations team is automatically informed.

Migrating existing integrations to middleware architecture

Many companies already have point-to-point integrations between their systems in use. These organically grown connections often work but are difficult to maintain, extend and diagnose when problems occur. Migration to a middleware architecture can proceed step by step: we start with the most critical data flow, implement it via middleware and deactivate the direct connection. Step by step, all data flows are migrated to the central architecture without interrupting ongoing operations.

Parallel operation during migration ensures no data is lost at any point. The old and new integrations run simultaneously, results are compared, and only when the new integration delivers identical results is the old one deactivated. This conservative approach requires somewhat more effort but eliminates the risk of data loss or operational interruption during migration.

Fault Tolerance and Resilience

A middleware is only as good as its error handling. Network interruptions, API timeouts, data inconsistencies and load spikes are not exceptions but expected operational states. Our middleware architectures implement automatic retry mechanisms with configurable backoff strategies, dead-letter queues for persistent errors, circuit breaker patterns to protect overloaded systems and compensation logic for partially processed transactions. Every error case is captured in structured logs that enable root cause analysis and traceability. Monitoring dashboards show the state of all integration flows in real time.

Scaling and Performance

B2B integrations frequently process large data volumes: thousands of product master records with prices and stock levels, hundreds of orders per hour during peak times, complex price configurations with customer-specific terms. Our middleware is designed for these volumes: message-queue-based processing enables horizontal scaling, batch processing optimizes handling of large datasets and caching strategies reduce redundant API calls. The architecture is dimensioned to remain stable even during load peaks such as seasonal order surges or ERP mass imports.

Maintenance and Evolution

A middleware connects systems that evolve independently. ERP updates, shop updates, new API versions and changed business processes require regular adjustments to integration logic. Our maintenance packages include proactive monitoring, adaptation to API changes in both systems and quarterly reviews of data quality and performance. Through automated contract tests, we detect breaking changes early and can react before the new version goes live.

Documentation and Operations Handbook

Every middleware implementation is fully documented: architecture diagrams, data flow definitions, mapping matrices, error handling logic and operations handbook with troubleshooting guide. This documentation enables your internal team to understand the middleware and independently perform basic operational tasks. For knowledge transfer, we offer onboarding sessions where we explain architecture decisions and walk through typical operational scenarios. Our experience shows that documentation quality ultimately determines the maintainability and evolvability of an integration long-term.

Consulting and Architecture Planning

Every middleware project begins with thorough requirements analysis: which systems need to be connected? What data flows are required and at what frequency? What error scenarios must be covered? In structured workshops, we develop the complete integration specification and decide on the optimal architecture: synchronous REST calls for time-critical data, asynchronous message queues for batch processing or event-based architectures for real-time synchronization. The architecture decision has long-term consequences for scalability, maintainability and operating costs. Our experience from dozens of middleware projects helps make the right decision for your specific use case.

Security and Compliance

Middleware processes business-critical data and is therefore a sensitive element of the system architecture. Our implementations consistently rely on encryption in transit and at rest, granular access controls and complete audit logs for every data movement. For companies with special compliance requirements, we adapt the security architecture to the respective regulatory specifications. GDPR-compliant data processing and traceable audit trails are our standard and ensure your integration stands on secure ground not only technically but also legally.

GDPR-compliant data processing, traceable audit trails and encrypted data transmission are our standard. For companies with special regulatory requirements, we individually adapt the security architecture. Our middleware solutions are designed to fully meet both the technical and legal requirements for processing sensitive business data.

If you need a robust, maintainable and secure connection between your shop and your ERP system, talk to us about your requirements. In the free initial consultation, we analyze your system landscape and provide an initial assessment of the optimal integration architecture.

Our middleware expertise spans various integration scenarios: from simple point-to-point connections between shop and ERP to complex hub-and-spoke architectures that centrally orchestrate multiple systems. In every case, reliability, maintainability and transparency take priority. Every transaction is traceable, every error is diagnosable, and every change is documented. These principles are the foundation for an integration that works not only at go-live but remains maintainable and extensible years later.

Frequently asked questions about middleware