The Power of Message Queues and Event-Driven Architecture

Consider that you have to ask a colleague some question. You have two options,

1. Call them

   • You must wait on the line till they pick up. When they are busy or away, then you are left holding the phone, with no option to do anything.

2. Send a text/email

   • You send a message and you are back at work. They write and respond when they are not busy.

In software architecture, the first one is called Synchronous (direct HTTP calls) and the second Asynchronous (the use of Message Queues).

In the case of small apps, calling is okay. However, when dealing with such large systems as Uber or Netflix, calling is a disaster. To address this, developers are relying on Message Queues and Event-Driven Architecture.

1. The Issue - The Barefoot Chain

Service A In a classic "Synchronous" system, the Service B is called right away by Service A.

• Example - A user orders a pizza. The Payment Service is instructed by the Order Service to charge the card.
• The Danger - In the event of the Payment Service being unavailable or slow, the Order Service will freeze. The user will be shown a spinning wheel and finally an error. The entire chain is ruined since a single link broke.

2. The Solution - Message Queues (The Buffer)

A Message Queue (such as RabbitMQ, Apache Kafka or AWS SQS) is an intermediary - a virtual mailbox.

Service A does not call Service B and instead refuses to call and invoices him/her (Charge User $20) and places a message in the Queue and agrees with the user that the order is accepted. Service B (the consumer) links to the Queue and when ready, the message is processed.

Why is this better?

• Traffic Spikes - When 10,000 orders are received simultaneously, the Queue stores them in a safe manner. Service B serves them non crashingly. This is called "Load Leveling."
• Resilience - When Service B malfunctions, the messages merely remain in the Queue. Service B is restarted and it continues where it was interrupted. No data is lost.

3. Event-Driven Architecture (EDA) - Something Happened

Having queues, you can change all your attitude to Event-Driven Architecture. A conventional system has the Order Service giving orders to the res, Payment Service, charge this. Inventory Service, stock to be deducted. Email Service, send receipt." It is dictatorial and generates close dependencies.

In EDA, the Order Service simply screams about a situation - Order Placed! It does not matter who is listening.

• It is heard by the Payment Service and the card is billed.
• It is heard by the Inventory Service and a deduction of stock made.
• It is heard by the Analytics Service and changes the dashboard.

This makes the system Decoupled. It is possible to create a new service (say a Rewards Service) and be able to listen to the events of Order Placed without ever having to do anything with the original Order Service code.

Conclusion

Scale Decoupling

The key to creating a system that makes it in the real world is Messages Queues and Event-Driven Architecture. They enable other components of your application to be independent, gracefully fail and scale indefinitely. It is the difference between a house of cards (the slightest knock overthrows the whole structure) and a well-run relay race.