NoSQL data stores and their use in .NET

Practice

Most .NET developers very rarely stop to think about where and how their application's data is stored. SQL Server's excellent integration with .NET makes it a universal and convenient storage that we use for any task without a second thought.

Modern DBMSs are a wonderful example of the fusion of excellent engineering solutions and decades of experience in building reliable, optimized data stores. However, more and more developers are choosing NoSQL data stores, which have lately been considered fashionable and progressive.

Recently I decided to explore the capabilities of NoSQL and how convenient it is to work with them in .NET, and in this article I'll share my experience and impressions with you.

Looking back a little, I'd call 2009 the year of ORM systems. Engines like Linq to SQL, Entity Framework, and NHibernate became mandatory tools in every .NET developer's toolkit.

This year you increasingly hear enthusiastic reviews of MongoDb, CouchDB, Cassandra, and other stores, and «select fun, profit from real_world where relational=false;» has become the new motto of web developers working in dynamic languages. NoSQL stores, or document-oriented storages, have firmly established themselves in the market and are gaining ever more popularity.

Moreover, such giants as Amazon (Dynamo), Google (BigTable), Twitter (Cassandra), and many others are setting the example.

So what's wrong with relational DBMSs?

Many projects have highly dynamic data. System entities often change their structure, similar entities may differ slightly in their data set, and some existing entities may over time «acquire» new properties. Because of the strict data structure in the database, any changes to the entity schema must be reflected in the table structure, constantly complicating the model (for example, in the case of inheritance) and requiring migration scripts to be prepared. These changes also require modifying SQL queries or object mappings, and on top of that, data access speed can suffer significantly (remember the queries Entity Framework generates for Table-per-Type inheritance). In the end, we have to maintain two heterogeneous models — relational in the database and object-oriented in the code — and often suffer performance losses as a result.

Also, many projects don't require strict support for ACID principles. It's far more important to have fast data processing with minimal response time, as well as fast and simple horizontal scalability. Furthermore, sometimes the volume of data being processed is so large that the only feasible way to work with it is to process it in parallel on a cluster.

The concept of NoSQL stores

Let me try to list what I consider the most characteristic features:

Schema-less data. Instead of strictly structured tables, we get collections of documents, and in some systems, simply key-value sets. The structure of each document can be individual — in effect, it describes itself. Supporting documents with different structures falls on the shoulders of the application logic.

Data Sharding. Very simple horizontal scalability. Add a server to the cluster and get a more performant store. Most NoSQL systems are cross-platform, which lets you build a distributed store out of practically any set of servers. Some stores are limited to a single server for reads and writes, with the rest of the servers being read-only (MongoDB), while others (CouchDB) allow reads and writes on all shards.

Fast index lookup. This is a top priority for any NoSQL system. Using B-trees allows for very fast lookups at a low cost for data modification.

MapReduce. A mechanism for parallel processing of large volumes of data on clusters. In the Map step, one of the computers (the master node) splits the task into parts, distributing them among the other computers (worker nodes). In the Reduce step, the master node collects the intermediate results and assembles them into the final result. This significantly improves the efficiency of distributed data processing. For example, Google's implementation of MapReduce can sort a petabyte of data in just a few hours.

MongoDB

For my tests I settled on MongoDB. It's one of the most popular document-oriented systems running on Windows, with a wealth of good reviews and bright prospects for development.

So, we download MongoDB for Windows — there's nothing to install, everything is run from the console (hello again, world of cross-platform tools).

We create the folder C:\data\db (this can be reconfigured).

We launch mongod.exe from the console — this is the server. At http://localhost:28017 you can enjoy the austere activity monitor.

Let's launch mongo.exe and try running a couple of queries. It's a realm of democracy here. You don't need to create anything — the database and collections will appear as soon as you write something into them.

For example:

use testdb
db.Movie.save({ Name: "Fight Club" Year: 1999 })
db.Movie.save({ Name: "Avatar", Year: 2010, Director: "James Cameron" })

We just created two documents in the Movie collection. The structure is similar to JSON, and it's called BSON (Binary JSON). Notice that this also created the database at C:\data\db

Let's try selecting the data:

db.Movie.find()
{ "_id" : ObjectId("4bc37626a20c0000000072fd"), "Name" : "Fight Club", "Year" : 1999 }
{ "_id" : ObjectId("4bc37633a20c0000000072fe"), "Name" : "Avatar", "Year" : 2010, "Director" : "James Cameron" }

As you can see, the documents got their identifiers automatically.

Now let's select a specific record:

db.Movie.find({Name:"Avatar"})
{ "_id" : ObjectId("4bc37633a20c0000000072fe"), "Name" : "Avatar", "Year" : 2010, "Director" : "James Cameron" }

Using MongoDB in .NET

Support for C# and .NET is community-driven. The most fully-featured driver is mongodb-csharp. It's still far behind the equivalents for other languages, but it's the best we've got.

With it, connecting to the database looks like this:

var mongo = new Mongo();
mongo.Connect();
var db = mongo.getDB("testdb");

Adding a new document:

var movies = db.GetCollection("Movie");
var movie = new Document();
movie["title"] = "Star Wars";
…
movies.Insert(movie);

Searching for documents:

var criteria = new Document();
criteria ["title"] = "Star Wars";
var result = movies.FindOne(criteria);

So, we already have a spartan feature set. It's also worth noting that the current functionality includes "Basic Linq Support," which, frankly, doesn't really save this driver as a product, given that the official wiki consists of three sentences and six lines of code.

It's also worth taking a look at the NoRM project. The project is still very young — the first commit was made on January 30, 2010. Even so, it already offers more convenient data access.

Let's declare the Movie class:

public class Movie
{
public Movie()
{
this.Id = ObjectId.NewObjectId();
}
public ObjectId Id { get; set; }
public string Name { get; set; }
public int Year { get; set; }
public string Director { get; set; }
}

ObjectId – a class representing a unique MongoDB identifier, from the Norm library.

After that, you can add data as follows:

var provider = new MongoQueryProvider(dbName);
provider. DB.GetCollection().Insert(new Movie {Name = "Star Wars", Director = "George Lucas"});

You can select data using Linq:

var movie = new MongoQueryProvider(provider).Where(m => m.Name == "Star Wars").FirstOrDefault();

In this case, interacting with the store takes on a palatable form that's familiar to .NET developers.

The programming model imposes certain restrictions on the data format in the database. The collection name must match the class name, and the class itself must contain all the properties that exist in the collection's documents. If a document contains only some of the properties, the rest simply won't be populated.

You can get around these restrictions using Flyweight objects:

var flys = provider. DB.GetCollection("collectionOfUnknowns");
foreach(var fly in flys)
{
//check whether the value exists
var value = fly["apropertyname"] ?? "property not set";
Console.Writeline(value);
}

Conclusions

NoSQL stores are becoming more and more popular thanks to their flexible data storage and scalability. Their main niche is projects written in dynamic programming languages, as well as projects with very large volumes of data. In statically typed languages using NoSQL is less convenient, and C# is perhaps the least suitable of the modern languages for it, given how sparse the driver APIs are. However, with the arrival of the DLR and the dynamic capabilities of C# 4.0, the situation could change significantly for the better. On top of that, some interesting projects have recently started to appear that make working with it more and more convenient.

On the other hand, it's worth waiting for Windows Azure to become the present rather than the future. This platform offers various ways of storing data, both SQL and non-SQL. It would be interesting to see in practice whether NoSQL stores can hold their own against the competition on the .NET platform.