Core Concepts: Events, Producers, and Consumers
In event-driven architecture, three roles define the system. Producers are components that emit events when something notable happens: a user signs up, a document is uploaded, an AI job completes. Consumers are components that subscribe to events and react to them: send a welcome email, index the document in a vector database, notify the user that their AI report is ready. The message broker is the infrastructure that sits between producers and consumers, receiving events, storing them temporarily, and delivering them to the appropriate consumers. Common message brokers include Apache Kafka for high-throughput enterprise systems, RabbitMQ for general-purpose messaging, and cloud-native options like AWS SQS, Google Pub/Sub, and Azure Service Bus. The critical property of event-driven architecture is asynchrony: the producer emits an event and moves on immediately without waiting for consumers to finish processing. This means that long-running AI tasks, like processing a document with an LLM or running a multi-step agentic workflow, can be triggered by a user action and processed in the background while the user's request returns immediately.
Event-Driven Architecture for AI Workflows
Event-driven architecture is a natural fit for AI workflow automation because AI processing is often asynchronous by nature. When a user uploads a document and asks an AI to summarise it, the summarisation might take 3-10 seconds for a long document with an LLM call. You do not want the user's browser waiting for that HTTP response. Instead, the upload triggers an event, a background worker picks up the event, calls the LLM, and sends the result via a websocket or push notification when it is ready. This pattern scales to complex multi-step agentic workflows. A user submits a research request. An event is emitted. A planning agent picks it up, creates a list of sub-tasks, and emits events for each sub-task. Specialist agents consume those events, perform their tasks, and emit result events. An aggregation agent consumes all the results and emits a final completion event. Each step is decoupled, each can be scaled independently, and failures in one step do not cascade to others.
When Event-Driven Architecture Is Worth the Complexity
Event-driven architecture adds real complexity: you need to manage a message broker, handle event ordering and deduplication, design for eventual consistency, and debug asynchronous failures that are harder to trace than synchronous ones. This complexity is worth it in specific situations. If your AI product has long-running background jobs that should not block user-facing requests, you need at minimum a job queue, which is a simplified form of event-driven architecture. If you have multiple independent components that need to react to the same event (user signup triggers: send welcome email, provision trial account, notify sales team, create analytics profile), event-driven architecture lets each consumer react independently without the signup handler needing to know about all of them. If you need to process high volumes of events reliably, message queues with at-least-once delivery guarantees are more reliable than synchronous HTTP calls that can fail and are not retried. For most AI MVPs, a simple job queue (using a tool like BullMQ with Redis, or a managed queue service) is sufficient, and full event-driven architecture with a message broker like Kafka is premature.
Event Sourcing: A Related but Distinct Pattern
Event sourcing is often confused with event-driven architecture but is a distinct pattern. Event sourcing stores the state of a system as a sequence of immutable events rather than as the current state. Instead of updating a database record to reflect the current value, you append an event that describes what changed. The current state is derived by replaying the event history. Event sourcing is powerful for audit-heavy applications (financial systems, healthcare records, compliance-critical AI decisions) because it gives you a complete, immutable history of every change. For a regulated AI product in the UK, where the FCA or NHS Digital may require an audit trail of AI-assisted decisions, event sourcing can satisfy compliance requirements that a standard mutable database does not. However, it also adds significant implementation complexity and should only be adopted if the audit requirements genuinely require it, not as a general-purpose architecture.
Practical EDA Patterns for AI MVPs
For most AI MVPs, the relevant event-driven patterns are lighter-weight than full EDA with a dedicated message broker. The job queue pattern covers 80% of the use cases: long-running AI tasks are submitted to a queue, workers process them asynchronously, and results are delivered via a push mechanism (websocket, server-sent events, or a polling endpoint). This can be implemented with BullMQ and Redis, which adds minimal operational overhead and is deployable on any Node.js platform. The webhook pattern covers a second major use case: notifying external systems when events occur in your AI product. When an AI job completes, send an HTTP POST to a customer's webhook endpoint with the result. This is simpler to implement than a full message broker and sufficient for most integration use cases. Only when you have genuinely high-throughput requirements or a complex fan-out pattern with many independent consumers does a dedicated message broker become justified.
Event-Driven Architecture and UK Compliance
Event-driven architecture has specific GDPR implications. Events often contain personal data (user IDs, email addresses, document content). If events are stored in a message broker, that broker is processing personal data and requires a Data Processing Agreement under UK GDPR. Events stored in a Kafka topic or message queue have a retention period that must be managed in compliance with your data retention policy. If a user exercises their right to erasure, personal data in event streams is harder to delete than personal data in a relational database, because events are typically immutable. Design your event payloads with data minimisation in mind: include only the identifiers needed to process the event, not full personal data, and look up the full data from the primary database at processing time. This approach means that deleting a user's data from the primary database satisfies erasure without requiring mutation of the event store.