Design data implementation

Skill 2.5: Design for Cosmos DB storage

Azure Table Storage is a good NoSQL service, but it may not be the best option for your needs. If you need more advance features like secondary indexes for your data, global distribution, or latency guarantee, you should use Cosmos DB. This also allows you to move from your on-premises NoSQL storage, like MongoDB or Casandra, to a SaaS model..

This is a NoSQL storage service that offers more advanced capabilities than Azure Table Store, but it is still compatible with it. This way, you can move from your current Azure Table Store solution to Cosmos DB without changing your code.

Azure Cosmos DB

There are situations where you need to provide a solution for data storage that can be distributed across several regions in the world, while ensuring a good level of throughput and elasticity. If you need to globally distribute your solution, it makes sense to think that latency is also a requirement for your application.

Azure Cosmos DB provides you with a service that is a multi-model database, globally distributed, with guaranteed throughput, latency, availability and consistency; it also enables you to scale independently each part of the distribution.

One of the main advantages of the global distribution that Cosmos DB offers is that you don’t need to make changes to your application when you want to change or add additional regions to the replication. When you use the Cosmos DB multi-homing API, you can configure your application for using logical endpoints, which are region-agnostic endpoints, for accessing your Cosmos DB account. These logical endpoints allow your application to access the storage transparently in the case of a failover of the region. If you need more granular control from the application to redirect read and writes to specific regions, you can use physical endpoints.

Cosmos DB allows you to configure an unlimited number of replicas, outside of geo-fencing restrictions like Germany or China. You can add or remove new replicas from your configuration dynamically. You can also configure each region for read, write, or read/write operations. You can only have a single write region, but you can have as many read regions as you want. Read queries are always routed to the nearest region to the request. This way Cosmos DB can ensure that the latency for read requests is always minimal. When you add a new region to the database account, Azure ensures that the region will be available within 30 minutes anywhere in the world, as long as your database size is up to 100 TB.

While working with Highly Available (HA) services, automatic failover is a key feature of the any HA system. Cosmos DB provides two different levels of automatic failover for the region that is configured for write operations:

• Regional If a regional outage happens, Cosmos DB automatically moves the requests to another region. During this transition there is a potential data loss during the regional outage.

• Internal There are internal failover mechanism for protecting you from failures at the database, collection, or partition level. These automatic failovers are transparent for you, and you don’t have any control over them.

If you need to test the availability features of Cosmos DB with your application, you can manually start a failover operation. When you make a manual failover operation, Azure guarantees that there will be zero data loss. When dealing with failover, Cosmos DB allows you to configure failover priorities. You can use these priorities for instructing Cosmos DB in which order an automatic failover should happen. The following procedure shows how to add additional regions to your database replication and configure failover priorities:

1. Sign into the management portal (http://portal.azure.com).

2. Select Azure Cosmos DB on the navigation panel on the left side of the window.

3. Select the name of your database, which opens the Overview blade for your database.

4. Under the Settings section, select Replicate Data Globally (see Figure 2-27).

   

   FIGURE 2-27 Configured replicas across the globe

5. In the Replicate Data Globally blade click on Add New Region. The button switches into a dropdown control. You can also add new regions by clicking on the hexagon icon in the world map.

6. Select a region in the dropdown and click OK. You need to repeat this operation with every region you want to add to the database replication.

7. Once you are done adding regions, click the Save button in the upper-left corner of the Replicate Data Globally blade.

8. For enabling and configuring the failover priority, click on the Automatic Failover icon. This is the fourth icon in the upper menu.

9. Select Enable Automatic Failover to On. You can change the order by dragging each region to the new position in the list.

Cosmos DB also ensures, by SLA, that Latency is P99 for reads and synchronously indexed writes. This means that 99% of the operations will have a latency, measured in milliseconds (ms), below 10ms. Azure can achieve this by ensuring that database operations are performed locally in the nearest region to the user. To achieve this level of low latency, Cosmos DB needs to perform asynchronous replication. This also means that Cosmos DB cannot employ strong consistency, because this consistency model imposes big restrictions that lead to greater latency values. Cosmos DB uses well-defined, relaxed consistency models.

Cosmos DB uses different data models like document, graph, key-value, table or column-family. You can work with these data models through different APIs, which expose different features, depending on your needs. We will review these APIs in the next sections.

MongoDB API

You can connect to your Azure Cosmos DB using the MongoDB API. MongoDB is a NoSQL storage system that uses a document data model. Similar to JSON objects, a MongoDB document is composed of pairs of fields-value, where a value can be other documents, arrays, or arrays of documents.

When using the MongoDB API, you can reuse your already existing libraries, code, and tools for accessing your Cosmos DB databases. MongoDB API is compatible with MongoDB 3.4 (version 5) wire protocol as well as MongoDB aggregation pipeline. By using MongoDB API you can take advantage of the features of Cosmos DB:

• Scalability You can scale up and down storage and throughput elastically. You can configure the resources assigned for each collection on each database.

• Multi-region replication As with any other Cosmos DB database, the MongoDB database is replicated across all regions that you configure. Cosmos DB provides transparent failover capabilities through the use of multi-homing APIs.

• No server management As with any other managed service, you don’t need to worry about MongoDB servers’ configuration and management details.

• Tunable consistency levels You can change the consistency level at the account level. Since MongoDB 3.4 has such a session consistency level, if you configure this level and access Cosmos DB using the MongoDB API, this will be treated as eventual consistency.

• Automatic indexing When you create a MongoDB database, Cosmos DB will automatically create an index for all properties of each document in the database. There is no special need for a schema or secondary index for this automatic action.

• Enterprise grade Out of the box, Cosmos DB provides you with several local replicas of your database. These replicas are part of the way that Azure uses for ensuring the 99.99% level of availability. You don’t need to make any special configuration for this.

The following procedure shows how to create your first Cosmos DB database. The procedure is equivalent to all APIs, so you just need to select the API that you want to use for accessing the database. Once you select the API for accessing the database, you cannot change it:

1. Sign into the management portal (http://portal.azure.com).

2. Click on Create A Resource on the upper-left corner of the window. On the Azure Marketplace, select Databases and then select Azure Cosmos DB.

3. On the Azure Cosmos DB New Account blade, enter an account ID. This will be the name of your account. You can create several databases inside your account.

4. On the API property, choose MongoDB in the dropdown (see Figure 2-28). Remember that you cannot change this value later.

   

   FIGURE 2-28 Selecting Cosmos DB API

5. Depending on your needs, select an existing resource group or create a new one.

6. Select the location more appropriate for you. This will be the write region for your account.

7. If you want to enable geo-redundancy, click on the checkbox below the location dropdown. If you don’t enable geo-redundancy during account creation, you can enable that later. If you decide to enable the geo-redundancy at this point, you need to add additional replicas later.

8. Click on Create.

SQL API

Another way to access to your Cosmos DB databases is using SQL API. This way, you can access your NoSQL storage by using your SQL skills. Added to the features that we already mentioned in previous sections, you can program your own stored procedures, triggers and user defined functions (UDFs) using standard JavaScript.

When you work with SQL API, you use a document model. This means that the information is organized in databases, collections, and documents. There are other types of items like triggers, stored procedures, or user defined functions that are also stored as JSON documents inside a database. We have two different types of resources inside a Cosmos DB account:

• System-defined resources These resources have a well-known schema since they are part of the system itself. System-defined resources are databases, users, permissions, and collections.

• User-defined resources These are the resources defined by the user with an arbitrary schema. These resources are documents, stored procedures, triggers, UDFs and attachments.

Figure 2-29 shows the relationship between different resources types.

FIGURE 2-29 Resources relationship in Cosmos DB

You can access your databases, collections, and documents by using the existing REST API making requests using HTTP/HTTPS. Microsoft also provides SDKs for languages .NET, Node.js, Java, JavaScript, and Python.

Graph API

When you deploy your Cosmos DB for use with the Graph API, you are telling Azure that you want to use a Graph data model. Graph API appears as a Gremlin (graph) option in the API dropdown during the deployment of a new Cosmos DB account.

A graph data model is useful when your entities and the relationship between them are equally important and you need to define properties for both types of elements. Azure Cosmos DB implements the property graph model. In this model each entity is known as a vertex and represents discrete objects like a car, a person, or a place. Vertices have relationships between them called edges. Both vertices and edges have properties. Using NoSQL engines is usually a good option for implementing graphs, thanks to the schema-free structure.

You typically use a graph for solving problems related with social networking, content management, geospatial, or recommendations. Graph algorithms are usually faster and perform better than traditional SQL and NoSQL databases, usually by orders of magnitude, thanks to the algorithms associated with the graph mathematics, like depth-first search or breadth-first search.

Graph API is compatible with Apache TinkerPop graph traversal language, Gremlin, or any other TinkerPop-compatible graph system.

Azure Tables API

Cosmos DB allows you to connect to your database using the same API calls that you use for your Azure Table Storage. This allows you to move from Table Storage to Cosmos DB without changing a line of code. Making this change, you will benefit from the improvements made for Cosmos DB:

• Latency With Cosmos DB you are guaranteed that latency is less than 10ms for reads operations and less than 15ms for write operations at 99th percentile.

• Throughput Using Table Store you have a throughput limit of 20,000 operations per second. With Cosmos DB you have no such limit, supporting more than 10 million of operations per second.

• Global distribution You can configure only a single region for read/write operations with an optional read-only second region for high availability when you use Table Storage.

• Indexing Azure Table Store provides a single primary index based on PartitionKey and RowKey properties. You cannot configure additional indexes.

• Query Table Store use indexes for queries only when you include the primary key on your query. For any other query without primary key, it uses scans, which have great impact on performance.

• Consistency You cannot choose the consistency model with Table Store. Within the primary site a strong consistency model is used along with a consistency model inside the secondary site.

• Pricing It’s optimized for storage when using Table Storage. Cosmos DB pricing is optimized for throughput.

Design for cost, performance, data consistency, availability, and business continuity

When planning for deploying Azure Cosmos DB, one of the aspects that you need to consider is the amount of resources that you need for your solution and the associated costs for those resources. When dealing with resources and costs, you need to consider an important concept, Request Unit (RU). This unit is the measure for assigning resources per partition and for billing. You can consider a partition or physical partition as a server. When you need to assign resources to your Cosmos DB account, you make it by adding RU per seconds. Remember, billing is performed hourly. Each RU has assigned a fixed amount of resources (Memory, Core and IOPS). This unit or currency simplifies the model for provisioning throughput to the application, since you don’t need to differentiate between read and write capacity units. As a rule of thumb, you should consider that a write operation needs five times the number of RUs needed for a read operation of the same size. This means that if you need one RU for reading a document of 1KB size, you will need five RUs for writing a document of 1KB size.

Cosmos DB offers a low latency guarantee for read and write operations. Azure can provide this feature thanks to consistency models used on data replication. Depending on your needs, you can configure five well-defined different consistency models for your Cosmos DB account:

• Strong Guarantees that the read operation returns the most recent version of an item. Any write operation will be available for reading only when has been committed by the majority quorum of replicas. The client never sees partially committed data. If you configure your account with this consistency model, you cannot associate more than one region with your account. The cost associated with read operations is higher than sessions or eventual consistency models.

• Bounded-staleness You configure a staleness value based on the number of versions K or the time interval t. This level of consistency guarantees that reads may lag writes by a maximum K number of versions or t time-interval. This consistency level is ideal when you want to keep low latency guarantee, but have a strong consistency. You can associate any number of regions with your account when you use this consistency model. Costs associated with read operations are equivalent to a strong consistency model.

• Session The consistency model is scoped to a client session. This consistency model is ideal for scenarios where a user or device typically reads its own writes. You can associate any number of regions with your account when using this consistency level. Costs associated with read operations are lower than strong or bounded-staleness, but higher than eventual consistency.

• Consistent Prefix The replication within the group will eventually converge only if there are no further write operations. This model of consistency guarantees that reads are always ordered. This means that if you wrote A, B, C data, when you read it, you can receive A or A, B or A, B, C, but never A, C or B, A, C. You can associate any number of regions with your account when you use this consistency model.

• Eventual The replication within the group will eventually converge only if there is no further write operations. There is no guarantee of the order when you perform read operations. You can associate any number of regions with your account when you use this consistency model. This consistency level has the lower cost when performing read operations.

Cosmos DB also provides business continuity features, like high availability (HA) and backup.

High availability is provided at several levels. You can configure high availability at a regional level by enabling and adding regions to geo-replications. Azure replicates your write operations across all regions that you configure in your account. But Azure also provides HA locally to each region. When you create an Azure Cosmos DB account, Azure creates a Cosmos DB container. This container automatically is divided in three different partitions. Each partition is made highly available by creating a replica set. When you configure geo-replication and add additional regions, a new layer of partition is added to the Cosmos DB container.

Azure performs backups of your Cosmos DB account automatically every four hours and keeps the last two backups. To ensure that the backup process does not impact on the latency of your account, Cosmos DB uses a separate Azure Blob Storage account. Taking a backup doesn’t consume any provisioned RU from your account. These automatic backups are also resilient against regional disaster by replicating the backup data to another region using geo-redundant storage (GRS). Although only two last backups are available for recovery, if you accidentally delete a database or collection, your backup is maintained up to 30 days. If you need to have a longer retention time, you can use Azure Cosmos DB Data Migration Tool and schedule additional backups. You can only perform a restore by opening a support ticket.

We have reviewed the advanced features that Microsoft offers you when using Cosmos DB and how this unique NoSQL service can help you with your storage needs. This service is especially useful for applications that need to replicate worldwide. You can also take advantage of the multi API provided by Cosmos DB to transparently migrate your application from other services like MongoDB, Casandra, or Azure Table Storage.