Introducing CQRS

The command stack

In a CQRS scenario, the command stack is concerned only about the performance of tasks that modify the state of the application. As usual, the application layer receives requests from the presentation and orchestrates their execution. So what’s going to be different in a CQRS scenario?

As shown in Figure 10-1, CQRS is about having distinct domain layers where the business logic—and objects required to have it implemented—is simpler to write because of the separation of concerns. This is already a benefit, but CQRS doesn’t stop here. Additionally, it lays the groundwork for some more relevant design changes.

In the rest of the chapter, we drill down into concepts that slowly emerged as people increasingly viewed query and command stacks separately. These concepts are still evolving and lead toward the event-sourcing supporting architecture we’ll discuss thoroughly in the next couple of chapters.

Getting back to presentation

A command is an action performed against the back end, such as registering a new user, processing the content of a shopping cart, or updating the profile of a customer. From a CQRS perspective, a task is monodirectional and generates a work flow that proceeds from the presentation down to the domain layer and likely ends up modifying some storage.

Tasks are triggered in two ways. One is when the user explicitly starts the task by acting on some UI elements. The other is when some autonomous services interact asynchronously with the system. As an example, you can think of how a shipping company interacts with its partners. The company might have an HTTP service that partners invoke to place requests.

The command placed updates the state of the system, but the caller might still need to receive some feedback.

Tasks triggered interactively

Imagine a web application like the I-Buy-Stuff online store we presented in Chapter 9. When the user clicks to buy the content of the shopping cart, she triggers a business process that creates an order, places a request for delivery, and processes payment—in a nutshell, it modifies the state of multiple systems, some interactively and some programmatically.

Yet the user who originally triggered the task is there expecting some form of feedback. That’s no big deal when commands and queries are in the same context—the task modifies the state and reads it back. But what about when commands and queries are in separated contexts? In this case, you have two tasks: one triggered interactively and one triggered programmatically. Here’s some code from an ASP.NET MVC application:

[HttpPost]
[ActionName(“AddTo”)]
public ActionResult AddToShoppingCart(int productId, int quantity=1)
{
    // Perform the requested task using posted input data
    var cart = RetrieveCurrentShoppingCart();
    cart = _service.AddProductToShoppingCart(cart, productId, quantity);
    SaveCurrentShoppingCart(cart);

    // Query task triggered programmatically
    return RedirectToAction(“AddTo”);
}

[HttpGet]
[ActionName(“AddTo”)]
public ActionResult DisplayShoppingCart()
{
    var cart = RetrieveCurrentShoppingCart();
    return View(“shoppingcart”, cart);
}

The method AddToShoppingCart is a command triggered interactively, as evidenced by the HttpPost attribute. It reads the current state of the shopping cart, adds the new item, and saves it back. The command Add-to-Shopping-Cart ends here, but there’s a user who still needs some feedback.

In this specific case—an ASP.NET application—you need a second command triggered programmatically that refreshes the user interface by placing a query or, like in this case, performing an action within the realm of the application layer. This is the effect of RedirectToAction, which places another HTTP request—a GET this time—that invokes the DisplayShoppingCart method.

What if you have a client-side web application—for example, a Single-Page application? In this case, you use some JavaScript to trigger the call to a Web API or SignalR endpoint. The task completes, but this time there’s no strict need for the web back end to execute a second task programmatically to get back to the presentation. The nature of the client makes it possible to display feedback in the form of an acknowledgment message:

$(“#buttonBuy”).click(function() {
   // Retrieve input data to pass
   ...
   $.post(url, { p1: ..., p2: ... })
    .done(function(response) {
        // Use the response from the task endpoint to refresh the UI
        ...
    });
});

A similar mechanism applies when the client is a desktop or mobile application, whether it’s Microsoft Windows, Windows Store, Windows Phone, Android, iOS, or something else.

Tasks triggered programmatically

A system that exposes a public HTTP API is subject to receive calls from the outside. The call merely consists of the invocation of a method through an HTTP request, and the response is just an HTTP response. Here, the fundamentals of HTTP rule over all. Here’s a sample Web API template for tasks to be invoked programmatically:

public class ExternalRequestController : ApiController
{
    public HttpResponseMessage PostDeliveryRequest(DeliveryRequest delivery)
    {
       // Do something here to process the delivery request coming from a partner company
       ...

       // Build a response for the caller:
       // Return HTTP 201 to indicate the successful creation of a new item
       var response = Request.CreateResponse<String>(HttpStatusCode.Created, “OK”);

       // Add the location of new item for their reference
       var trackingId = ...;
       var path = “/delivery/processed/” + delivery.PartnerCode + “/” + trackingId;
       response.Headers.Location = new Uri(Request.RequestUri, path);
       return response;
   }
}

In this example, you have a Web API controller that receives delivery requests from a partner company. The request is processed and generates a tracking ID that must be communicated back to indicate the success of the operation.

There are various ways you can do this, and it mostly depends on your personal perspective regarding Web APIs. If you’re a REST person, you would probably go with the code shown earlier. If you’re more inclined toward remote procedure calls (RPCs), you can just return the tracking ID as a plain string in a generic HTTP 200 response.

Formalizing commands and events

All software systems receive input from some front-end data source. A data source can be any number of things, like a sensor connected to a hardware device that pumps real-time data, a feed asynchronously provided by a remote service, or—the most common scenario—a presentation layer equipped with a comfortable user interface. The input data travels from the front end to the application layer, where the processing phase of the input data is orchestrated.

Abstractly speaking, any front-end request for input processing is seen as a message sent to the application layer—the recipient. A message is a data transfer object that contains the plain data required for any further processing. In such an architecture, it is assumed that messages are fully understood by the recipient. Such a definition of a message leaves room for a number of concrete implementations. In most cases, you might want to start with a Message base class that acts as a data container:

public class Message
{
    // Optionally, a few common properties here.
    // The class, however, can even be a plain marker with no properties.
    ...
}

The front end can deliver the message to the application layer in a number of ways. Commonly, the delivery is a plain method invocation—for example, an application service invoked from within an ASP.NET MVC controller method. In more sophisticated scenarios, such as where scalability is the top priority, you might want to have a service bus in your infrastructure that also supports brokered messaging. In this way, you ensure delivery of the message to the intended recipient under any conditions, including when the recipient is not online.

Events vs. commands

There are two types of messages: commands and events. In both cases, messages consist of a packet of data. Some subtle differences exist, however, between events and commands.

A command is an imperative message that sounds like an explicit request made to the system to have some tasks performed. Here are some other characteristics of a command:

• A command is directed at one handler.
• A command can be rejected by the system.
• A command can fail while being executed by some handler.
• The net effect of the command can be different depending on the current state of the system.
• A command doesn’t generally trespass the boundaries of a given bounded context.
• The suggested naming convention for commands says that they should be imperative and specify what needs to be done.

An event is a message that serves as a notification for something that has already happened. It has the following characteristics:

• An event can’t be rejected or canceled by the system.
• An event can have any number of handlers interested in processing it.
• The processing of an event can, in turn, generate other events for other handlers to process.
• An event can have subscribers located outside the bounded context from which it originated.

Writing an event class

In terms of source code, commands and events are both classes derived from Message. Dealing with commands and events through different classes makes the design of the system more logical and simpler overall. Here’s a sample command class:

public class CheckoutCommand : Message
{
    public string CartId { get; private set; }
    public string CustomerId { get; private set; }

    public CheckoutCommand(string cartId, string customerId)
    {
        CartId = cartId;
        CustomerId = customerId;
    }
}

Conversely, here’s the layout of an event class.

public class DomainEvent : Message
{
   // Common properties
   ...
}

public class OrderCreatedEvent : DomainEvent
{
    public string OrderId { get; private set; }
    public string TrackingId { get; private set; }
    public string TransactionId { get; private set; }

    public OrderCreatedEvent(string orderId, string trackingId, string transactionId)
    {
        OrderId = orderId;
        TrackingId = trackingId;
        TransactionId = transactionId;
    }
}

As you can see, the structure of event and command classes is nearly the same except for the naming convention. A fundamental guideline for designing domain events is that they should be as specific as possible and clearly reveal intent.

As an example, consider the form through which a customer updates the default credit card he uses in transactions. Should you fire a rather generic CustomerUpdated event? Or is a more specific CreditCardUpdated event preferable? Both options lead to a working solution, but which option works for you should be evident and stand on its own. It depends on the ubiquitous language and the level of granularity you have in place. We believe that a finer granularity here is a significant asset.

We generally recommend that the intent of the event be made clear in the name and all ambiguity be removed. If some ambiguity around a single event surfaces, you’ll probably find that it’s safer to have two distinct events.

Which properties should you have in the base class DomainEvent?

We’d say that at a minimum you want to have a timestamp property that tracks the exact time at which the event was fired. Moreover, you might want to have a property containing the name (or ID) of the user who caused the firing of the event. Another piece of data you might want to have is a version number that handlers can use to determine whether they can or cannot handle the event. The point here is that the definition of an event might change over time. A version number can help in this regard.

The implementation of the version number is completely up to the team. It can be a Version object as well as a string or a number. It can be bound to the application build number, or it can even be referred to the version of the event class:

public class DomainEvent : Message
{
    public DateTime TimeStamp { get; private set; }
    public DomainEvent()
    {
        TimeStamp = DateTime.Now;
    }
}

An event class should be considered immutable for the simple reason that it represents something that has already happened. Immutable here means that there should be no way to alter the value of properties. The combination of private setters, no write methods, and a plain constructor will do the trick.

Handling commands and events

Commands are managed by a processor that usually is referred to as a command bus. Events are managed by an event bus component. It is not unusual, however, that commands and events are handled by the same bus. Figure 10-3 presents the overall event-based architecture of a CQRS solution. This architecture is more standard and is an alternative to the one we mentioned earlier that was based on the TS pattern.

Any interaction that takes place in the user interface generates some requests to the system. In an ASP.NET MVC scenario, these requests take the form of controller actions and methods in the application layer. In the application layer, a command is created and pushed to some machinery for actual processing.

The bus component

The command bus holds a list of known business processes that can be triggered by commands. Active instances of such processes can be further advanced by commands. Processing a command can sometimes generate an event within the domain; the generated event is published to the same command bus or to a parallel event bus, if any. Processes that handle commands and related events are usually referred to as sagas.

The command bus is a single class that receives messages (requests of executing commands and notifications of events) and finds a way to process them. The bus doesn’t actually do the work itself; instead, it selects a registered handler that can take care of the command or event. Here’s a possible template for a bus class that handles commands and events. We use the interface IHandles as a placeholder for actions. The interface has a single void method:

public interface IHandles
{
   void Handle(T message);
}

The bus uses the interface to handle both commands and events:

public class Bus
{
    private static readonly Dictionary<Type, Type> SagaStarters =
             new Dictionary<Type, Type>();
    private static readonly Dictionary<string, object> SagaInstances =
             new Dictionary<string, object>();

    public static void RegisterSaga<TStartMessage, TSaga>()
    {
        SagaStarters.Add(typeof(TStartMessage), typeof(TSaga));
    }

    public static void Send<T>(T message) where T : Message
    {
        // Publish the event
        if (message is IDomainEvent)
        {
            // Invoke all registered sagas and give each
            // a chance to handle the event.
            foreach(var saga in SagaInstances)
            {
               var handler = (IHandles<T>) saga;
               if (handler != null)
                   handler.Handle(message);
            }
        }

        // Check if the message can start one of the registered sagas
        if (SagaStarters.ContainsKey(typeof (T)))
        {
            // Start the saga creating a new instance of the type
            var typeOfSaga = SagaStarters[typeof (T)];
            var instance = (IHandles<T>) Activator.CreateInstance(typeOfSaga);
            instance.Handle(message);

            // At this point the saga has been given an ID;
            // let's persist the instance to a (memory) dictionary for later use.
            var saga = (SagaBase) instance;
            SagaInstances.Add(saga.Data.Id, instance);
            return;
        }

        // The message doesn't start any saga.
        // Check if the message can be delivered to an existing saga instead
        if (SagaInstances.ContainsKey(message.Id))
        {
            var saga = (IHandles<T>) SagaInstances[message.Id];
            saga.Handle(message);

            // Saves saga back or remove if completed
            if (saga.IsComplete())
                SagaInstances.Remove(message.Id);
            else
                SagaInstances[message.Id] = saga;
        }
    }
}

The bus has two internal dictionaries: one to map start messages and saga types, and one to track live instances of sagas. In the latter dictionary, you can typically have multiple instances of the same saga type that are bound to different IDs.

The saga component

In general, a saga component looks like a collection of logically related methods and event handlers. Each saga is a component that declares the following information:

• A command or event that starts the process associated with the saga
• Commands the saga can handle and events the saga is interested in

Whenever the bus receives a command (or an event) that can start a saga, it creates a new saga object. The constructor of the saga generates a unique ID, which is necessary to handle concurrent instances of the same saga. The ID can be a GUID as well as a hash value from the starter command or anything else, like the session ID. Once the saga is created, it executes the command or runs the code that handles the notified event. Executing the command mostly means writing data or executing calculations.

At some point in the lifetime of the saga instance, it might be necessary to send another command to trigger another process or, more likely, fire an event that advances another process. The saga does that by pushing commands and events back to the bus. It might also happen that a saga stops at some point and waits for events to be notified. The concatenation of commands and events keeps the saga live until a completion point is reached. In this regard, you can also think of a saga as a workflow with starting and ending points.

Events raised by sagas pass through the bus, and the bus passes them as messages to whomever subscribed to that event. Raising an event from within the saga requires code like that shown here:

// Raise the PaymentCompleted event
var theEvent = new PaymentCompletedEvent( /* add transaction ID */ );
theEvent.SagaId = message.Cart.Id;
Bus.Send(theEvent);

From the application layer, you invoke the bus as shown in the following example. This example simulates a scenario in which an ASP.NET MVC application is called back from the payment page on a bank service gateway:

public ActionResult CompleteProcessOrder(String transactionId)
{
    // Retrieve shopping cart from session state
    var cart = RetrieveCurrentShoppingCart();

    // Prepare and queue a Process-Order command
    var command = new ProcessOrderCommand(transactionId, cart.CartId);
    Bus.Send(command);

    // Refresh view: in doing so, results of previous command might be captured, if ready.
    return RedirectToAction(“Done”);
}

As you might have noticed, the command bus doesn’t return a response. This is not coincidental. The refresh of the user interface—wherever necessary—is left to a subsequent read command that queries data from any place—storage, cache, or whatever—where output is expected to be.

What we mostly want from a command bus is to decouple the application layer from the domain services. Doing so opens up new opportunities, such as handling domain service calls asynchronously. Other scenarios that the command bus can simplify are adding cross-cutting filters along the way, such as transactions, logging, and general injection points for optional logic. In any of these cases, all you need to do is change the command-bus class—no changes are required to the application layer and domain services.

The combined effect of commands and events

When you write systems based on very dynamic business rules, you might want to seriously consider leaving the door open to making extensions and changes to certain workflows without patching the system.

Think, for example, of an e-commerce website that launches a marketing campaign so that any user performing certain actions through the site will accrue points on some sort of a customer-loyalty program. Because of the huge success of the campaign, the company decides to extend the incentives to more actions on the site and even double the points if a given action (say, buying suggested products) is repeated over time. How would you effectively handle that?

You can certainly fix and extend the application layer, which turns out to be the nerve center that governs the workflow. Anyway, when the business logic is expressed through the composition of small, independent and even autonomous commands and events, everything is much easier to handle. This is a classic scenario for collaborative systems. Let’s say you use the flowchart in Figure 10-4 to implement the process of an order being submitted.

Having the entire business logic of the workflow in a single method might make maintenance and testing problematic. However, even in the simple case of having split blocks in the workflow in simple commands, it might be difficult to do things properly. Let’s consider the update of the customer’s status in the fidelity program after an order is processed.

Instead of having the Register-Order command invoke the Update-Fidelity-Card command, isn’t it more flexible if the Register-Order command just fires an event saying that a new order has been created? With this approach, a handler for that event can kick off and check whether the customer is eligible for Gold status and fire another event for other handlers to jump in and update databases. This would isolate each action, keep each form of validation small and in reusable pieces, and keep the amount of logic in each command to the bare minimum.

Sagas and transactions

Looking back at Chapter 8 where we introduced the Domain Model architecture, we can say that a saga is the evolution of the application service concept.

In an application service you orchestrate the tasks that serve a user request. The application service takes care of setting up transactions--possibly distributed transactions--and like a state-machine it coordinates the next step until the end of a workflow is reached. Abstractly speaking, a saga does the same except that it removes the need for a single component to know about the entire workflow and uses events to break an otherwise monolithic workflow into the combined effect of smaller command handlers raising events to coordinate with others.

This makes the entire solution a lot more flexible and reduces the surface of code subject to changes when business changes.

In addition, a saga reduces the need of distributed transactions when long-running processes are involved that span across multiple bounded contexts. The structure of a saga--a collection of rather independent command and event handlers--lends to easily define compensating behavior for each command executed. Thus if at some point a saga command fails, it can execute the compensating behavior for what it knows has happened and raise an event at the saga level to undo or compensate the work completed.

Downsides of having a command bus

There are contrasting feelings about the command bus. A common criticism is that it just adds an extra layer and contributes to making the code less easy to read. You need to go through some more classes to figure out what happens.

This aspect can be mitigated by using proper naming conventions so that the name of the handler matches the name of the command class being queued to the bus. In this way, you know exactly where to look, and looking into the implementation of methods takes only a few more clicks than with application services, as discussed in past chapters.

With classic application services, you see the call to the method right from controllers and can use a tool like ReSharper to jump directly to the source code. With a command bus, you see the command and know by convention the name of the class in which you need to look.

Readymade storage

Most real-world systems write data and read it back later. Before CQRS appeared on the horizon, reads and writes took place within the same stack and often within the same transactional context. CQRS promotes the neat separation of queries and commands and pushes the use of different stacks. As you saw in this chapter, though, the internal architecture of the two stacks can be significantly different. You typically have a domain layer in the command stack and a far simpler data-access layer in the query stack.

What about databases, then? Should you use different databases too: one to save the state of the system and one to query for data?

Optimizing storage for queries

Many real-world systems use a single database for reading and writing purposes—and this happens regardless of CQRS architectures. You can have distinct command and query models and still share the same database. It’s all about the scenarios you deal with.

Our thought is that using CQRS—at least in its lightweight form of using different models—is mostly beneficial for every system because it splits complexity, enables even different teams to work in parallel, and can suggest using something even simpler than a Domain Model for each stack. So CQRS is also about simplification.

However, for most practical purposes, using a single relational database is still the best option. When CQRS is used for scalability in a highly collaborative system, however, you might want to think about a couple of other aspects. (CQRS with a single database is sometimes referred to as hybrid-CQRS.)

In the query stack, what you query for has nearly the same schema as the view model. Most likely, these view models come from tables that are largely denormalized and that have very few links to other tables. In other words, the tables you really need are specialized for the queries you need to run.

A question that naturally arises is this: do you actually need a relational database for queries? We say you don’t strictly need a relational database, but it’s probably the best option you have, even though NoSQL solutions might be handier at times. More importantly—we’d say—you don’t strictly need the same database where the state of the system is persisted.

Creating a database cache

If you’re using one database to store the state of the system and another database to store data in a format that is quick and effective to query for the purposes of presentation, who’s in charge of keeping the two databases in sync?

This seems to be the final step of nearly any command whose action affects the query database. To avoid giving too many responsibilities to each command, the most obvious approach is that any command whose action affects the query database fires an event at the end. The event handler will then take care of updating the query database. The query database doesn’t fully represent the business behind the application; it only contains data—possibly denormalized data—in a format that matches the expectations of the user interface. For example, if you’re going to display pending orders, you might want to have a query database just for the information you intend to display for orders. In doing so, you don’t store the customer ID or product ID, just the real name of the customer, a full description of the product, and a calculated total of the order that includes taxes and shipping costs.

Stale data and eventual consistency

When each stack in CQRS owns its database, the command database captures the state of the system and applies the “official” data model of the business domain. The query database operates as a cache for data readymade for the presentation’s needs. Keeping the two databases in sync incurs a cost. Sometimes this cost can be postponed; sometimes not.

If you update the query database at the end of a command, just before the end of the saga or at any point of the workflow when it makes sense, you automatically keep the command and query databases in sync. Your presentation layer will constantly consume fresh data.

Another approach that is sometimes used in the concrete implementations consists of delaying the synchronization between the command and query databases. The command database is regularly updated during the execution of the command so that the state of the application is consistent. Those changes, though, are not replicated immediately to the query side. This typically happens for performance reasons and to try to keep scalability at the maximum.

When the query and command databases are not in sync, the presentation layer might show stale data and the consistency of the entire system is partial. This is called eventual consistency—at some point, the databases are consistent, but consistency is not guaranteed all the time.

Is working with stale data a problem? It depends.

First and foremost, there should be a reason for having stale data; often, the reason is to speed up the write action to gain greater scalability. If scalability is not a concern, there’s probably no reason for stale data and eventual consistency. Beyond that, we believe that very few applications can’t afford displaying stale data for a short amount of time (with “short amount of time” being defined by the context). In many write-intensive systems, writes are sometimes performed in the back end and only simulated on the presentation to give the illusion of full synchronicity. The canonical example is when you post on a social network and the following two things happen:

• The back end of the system is updated through a command.
• At the end of the command, the DOM of the page is updated via JavaScript with the changes.

In other words, what appears as the real status of the system is simply the effect of a client-side command. At some point, however, in a few seconds (or minutes or days) something happens that restores full consistency so that when the page is displayed from scratch reading from the server, it presents aligned data.

Eventual consistency is commonly achieved through scheduled jobs that run periodically or via queues that operate asynchronously. In this case, the saga ends by placing an event to a separate bus, possibly on a persistent queue. The queue will then serve events one at a time, causing the handler to fire and update the query database.