Introduction Orleans, especially when building applications requiring rich querying capabilities

Microsoft Orleans is a remarkable framework for building distributed systems. It abstracts much of the complexity associated with distributed computing, making it easier for developers to build scalable and fault-tolerant applications.

Orleans operates on the concept of virtual actors, known as grains. These grains are the primary units of computation and state. While this model offers many advantages, it poses challenges when it comes to querying.

In a typical Orleans setup Orleans, especially when building applications requiring rich querying capabilities:

• You can easily fetch a grain(s) if you know their ID(s).

• You cannot perform complex queries, especially those that rely on indexes or other database-specific features.

The latter point can be a bit of a headache with Orleans, especially when building applications requiring rich querying capabilities.

While it's straightforward to query grains when you know their IDs, querying based on other criteria, such as indexes, is not natively supported. This limitation can be a significant hindrance in some cases.

Orleans.Providers.MongoDBRepository Orleans, especially when building applications requiring rich querying capabilities

This (I hope) useful library aims to provide a seamless hybrid approach to bridge the gap between Orleans and the traditional use of MongoDB when complex querying capabilities are needed, thus allowing developers to harness the full power of both Orleans and MongoDB.

In this article, we'll explore the challenges posed by Orleans's querying limitations, how the Orleans.Providers.MongoDB.Repository library addresses these challenges and how to leverage this library in your projects.

If you are not familiar with the Repository Pattern, here's a short description:

A Repository acts as a bridge between the application's core logic and the database. It behaves like an in-memory collection of domain objects. Clients set up queries and send them to the Repository to get results. You can add or remove objects from the Repository just like a regular collection. The Repository handles the background tasks of mapping these operations to the database. Essentially, it offers an object-oriented perspective of the database, ensuring a clear boundary between the application's main logic and the database interactions.

How It Works?

The Orleans.Providers.MongoDBRepository library is designed to address the querying limitations of Orleans when used with MongoDB as a storage provider.

It provides automatic MongoDB repository class creation using Roslyn incremental source generators. The library is intended to be used with Microsoft Orleans and its MongoDB storage provider (Orleans.Providers.MongoDB).

Key Features

Typed Grain State MongoDB Documents

Since Orleans.Providers.MongoDB stores grain states in the same document layout for all grain state types; I've been able to leverage this to provide the generic MongoDbGrainDocument<TGrainState> class which uses the actual grain state types; thus, all operations on Mongo grain collections in the repository are correctly typed by default. More on this in the 'Document Model' section.

Reusing MongoDB Client

Orleans.Providers.MongoDB.Repository reuses the MongoDB client created when setting up a MongoDB storage provider with Orleans.Providers.MongoDB library.

One MongoDB client per application is recommended to avoid connection pool exhaustion. If you need to connect to multiple clusters/nodes, then implement IMongoClientFactory interface provided by Orleans.Providers.MongoDB library and implement client handling as you need.

Here you can read about it a bit and also read official MongoDB documentation about the clients and connection.

Automatic Repository Creation

The library will generate the necessary repository partial class with all the required functionality by adding attributes to your counterpart partial classes.

This means developers can focus on implementing repository methods without worrying about the boilerplate code for database connections, getting the right collection, and other boilerplate tasks.

Access to MongoDB Database and Collections

The library exposes (among other things) the IMongoCollection to the repository, allowing developers to modify the underlying collection settings. This is particularly useful when you need to add additional indexes allowing for complex queries, leveraging MongoDB's powerful querying engine.

Installation

Open a Nuget Browser and search for Orleans.Providers.MongoDBRepository package and install it. By installing this, two more 'child' packages will also be installed:

• Orleans.Providers.MongoDBRepository.Abstractions

• Orleans.Providers.MongoDBRepository.Generator

If you prefer manual installation via console, just hit:

dotnet add package Orleans.Providers.MongoDBRepository

Child Packages

Orleans.Providers.MongoDBRepository.Abstractions

This package consists of interfaces and similar used by the library. It's a convenient way not to depend on the main library where you don't need it.

Orleans.Providers.MongoDBRepository.Generator

This package is the source generator; you will not use it directly as it does its thing automatically. It uses an incremental source generator to produce grain repository classes. For any (partial) class marked with the MongoDbGrainRepository attribute, this generator will produce a (partial) class with the same name in which all needed grain repository code is implemented.

How It Works

To create a MongoDB grain repository, create a partial class and annotate it with MongoDbGrainRepository attribute. The attribute takes one argument, which is the type of the grain state. Each grain repository handles a single grain type (its state) and has direct access to the underlying MongoDB collection, where the state data for those grains are stored.

Grains and States

First, define your grains and their states as you normally do in Orleans. For example, if you have a grain class UserGrain and its state is defined something like this:

User Grain

using Orleans.Runtime;

// Interface for the UserGrain is a standard IGrainWithStringKey
// so there is nothing special to present about it.
public sealed class UserGrain : IUserGrain
{
	// MEMBERS 
	private readonly IPersistentState<UserModel> _state;

	public UserGrain(
		[PersistentState(stateName: nameof(UserModel), storageName: "UserStorage")]
		IPersistentState<UserModel> state
	)
	{
		_state = state;
	}

  ... grain methods

}

User Model (Grain State)

Note: Interface and implementation of the model are here together just for brevity

public interface IUserModel
{
	/// <summary>
	///   Username; must be unique.
	/// </summary>
	string EmailAddress { get; set; }

	/// <summary>
	///   Email address; must be unique.
	/// </summary>
	string Username { get; set; }
}

[GenerateSerializer]
public record struct UserModel : IUserModel
{
	[Id(0)] public string Username { get; set; }
	[Id(1)] public string EmailAddress { get; set; }
}

The User Repository

You can create a new grain repository like this:

using MongoDB.Driver;
using Orleans.Providers.MongoDB.Repository;

[MongoDbGrainRepository(typeof(UserModel))] // Note that we are using the same UserModel record as the grain
public partial class UserRepository : IUserRepository
{
	public async Task<List<IUserModel>?> GetAll(int page, int pageSize)
	{
        // Some method to query the grains directly with the MongoDb driver
		var userDocuments = await Collection.Find(_ => true)
			.Skip((page - 1) * pageSize)
			.Limit(pageSize)
			.ToListAsync();

		var users = userDocuments.Select(ud => ud.Doc as IUserModel).ToList();
		return users;
	}

    ...

}

// Interface for the UserRepository
// Nothing special, just a regular interface
public interface IUserRepository
{
	/// <summary>
	///   Gets users with paging
	/// </summary>
	/// <param name="page">Page number (zero based).</param>
	/// <param name="pageSize">Number of items per page.</param>
	Task<List<IUserModel>?> GetAll(int page, int pageSize);

    ...
}

By adding the attribute and specifying the type, the library will automatically generate the other UserRepository partial class with all the necessary grain repository functionality. This includes connecting to the database, fetching the right collection, and more.

Register Repository

Orleans.Providers.MongoDB.Repository provides several extension methods on IServiceCollection and OrleansISiloBuilder therefore setting up repositories is straightforward.

Here is an example of how repositories can be set up using extension methods on Orleans ISiloBuilder:

IHostBuilder hostBuilder = Host.CreateDefaultBuilder(args);
hostBuilder.ConfigureServices((hostContext, services) =>
{
    // Get some options from IConfiguration object
	MongoDbConnectionOptions? mongoConfigOptions = 
		hostContext.Configuration.GetSection("StorageProviders:ConnectionStrings:MongoDb").Get<MongoDbConnectionOptions>();	
	
	MongoDbGrainRepositoryOptions? profileConfigOptions = 
		hostContext.Configuration.GetSection("StorageProviders:ProfileRepository").Get<MongoDbGrainRepositoryOptions>();
	
	ArgumentNullException.ThrowIfNull(mongoConfigOptions, nameof(mongoConfigOptions));
	ArgumentNullException.ThrowIfNull(usersRepositoryOptions, nameof(usersRepositoryOptions));
	ArgumentNullException.ThrowIfNull(profileConfigOptions, nameof(profileConfigOptions));

	// Configure Orleans and MongoDB
	services
		.Configure<ClusterOptions>(options =>
		{
			options.ClusterId = "ResearchOrleansCluster";
			options.ServiceId = "UserRepositoryService";
		})
		.AddOrleans(siloBuilder =>
		{
			siloBuilder
				.ConfigureServices(siloServices =>
				{
					// Serialize as BSON instead of JSON (part of 
					siloServices.AddSingleton<IGrainStateSerializer, BsonGrainStateSerializer>();
				})
				.UseMongoDBClient(mongoConfigOptions.ConnectionString)
				.UseMongoDBClustering(options =>
				{
					options.DatabaseName = "OrleansClusteringTable";
					options.Strategy = MongoDBMembershipStrategy.SingleDocument;
				})

				// UserGrain State Storage
				.AddMongoDBGrainStorage("UserGrainStorage", options =>
				{
					options.DatabaseName = "DSI_UsersDb";
					options.CollectionPrefix = string.Empty; // No prefix ("Grains") on collection names
				})

				// Add the repository services
				// -----------------------------------------------------------------------

				// UsersRepository with manual options definition
				.AddMongoDbRepository<IUserRepository, UserRepository>(options =>
				{
					options.DatabaseName = "DSI_UsersDb";
					options.CollectionName = "UserModel";
				})

				// ProfileRepository with MongoDbGrainRepositoryOptions options instance
				.AddMongoDbRepository<IProfileRepository, ProfileRepository>(profileConfigOptions);
		})

		// Add other services
        ....
})

This example is the most basic if you ever did something with Orleans and its MongoDb storage provider. It sets up Orleans in the simplest way using MongoDb as storage.

Note:As you probably noticed, I set string.Empty on options.CollectionPrefix when configuring the MongoDB storage provider. MongoDB storage provider automatically adds the "Grains" prefix to the collections, and I don't like unpredictable manipulations; hence I'm removing it with this line.

The additional lines at the bottom add two repositories, one for users and the other for profiles:

• The UserRepository is added by the manual setup of repository options

• The ProfileRepository is added just by passing options object retrieved from IConfiguration got from the hostContext.

This two-way approach is a flexible way to add and configure your MongoDB repositories in most cases.

MongoDbGrainRepositoryOptions Class

MongoDB repositories use the options class named MongoDbGrainRepositoryOptions , which is a part of the library and provides an easy way to use strongly typed options objects and validate them when using IConfiguration and IOptions pattern.

The Advanced Usage

The examples so far showed how to set up the repository. In this stage, you have a repository and can query it using exposed IMongoCollection object. Unfortunately, only queries on grain ID would be performant since they are indexed. You'll probably hit the wall for all queries that target other fields of the grain state and find yourself scanning the whole collection.

So, what can we do about it? Let's first see how the stored grain state looks:

The Document Model

Orleans.Providers.MongoDB storage provider persists each grain state in a document structure like this one (this is the persisted UserModel state object we discussed above):

_id: "user/01H7RBNBW8209NV85XRWA78QEG",
_etag: "c6b962c8-b5b5-4aa4-bc55-c82aabaa94f8",
_doc: {
  Username: "winne.pooh",
  EmailAddress: "winne.pooh@robbins.com"
}

Where:

• _id is a standard Orleans grain ID. In this case, I used grain with string keys, and my keys are ULIDs.

• _etag is a standard Orleans mechanism to track changes in the grain state (short description)

• _doc is an object of actual serialized state (in our case UserModel as described before)

Note:Orleans.Providers.MongoDB automatically creates an index on _id field for fast grain ID queries.

Strongly Typed States

Leveraging this 'stable' structure, the library provides a MongoDbGrainDocument<TGrainState> class which provides an identical structure as the MongoDB documents and a typed state object (got from the attribute set on the repository class).

With this setup, all queries in the repository will have proper type-checking, autocompletion, and all other stuff, making queries easy to write.

Modifying the Collection Settings (adding additional indexes)

As mentioned, we can now query over other state fields, but it would not be performant since those aren't indexed; hence we would scan the whole collection when searching for something.

Orleans.Providers.MongoDB.Repository provides a simple way to customize the collection upon repository initialization just by implementing IInitializableGrainRepository interface (which is the part of the library).

In that stage, one can check for and add indexes where needed.

IInitializableGrainRepository Interface

Using this interface, you will need to implement Task OnInitialized() method in your MongoDB repository class.

This method will be called when the repository has been initialized. This is the right time to modify the collection settings if needed. In our case, I want to add two new indexes; one on the EmailAddress and the other on the Username fields of the persisted grain state.

Here's the implementation of this method with code that adds two new indexes on the UserModel collection:

 public Task OnInitialized()
	{
		if(!MongoDbUtility.IndexExists(Collection, "_emailAddress_"))
		{
			// Create an index on the EmailAddress field
			CreateIndexOptions indexOptions = new CreateIndexOptions { Name = "_emailAddress_", Unique = true };
			CreateIndexModel<MongoDbGrainDocument<UserModel>> emailIndexModel =
				new CreateIndexModel<MongoDbGrainDocument<UserModel>>(
					Builders<MongoDbGrainDocument<UserModel>>.IndexKeys.Ascending("_doc.EmailAddress"), indexOptions);
			Collection.Indexes.CreateOne(emailIndexModel);
		}

		if(!MongoDbUtility.IndexExists(Collection, "_username_"))
		{
			// Create an index on the Username field
			CreateIndexOptions usernameIndexOptions = new CreateIndexOptions { Name = "_username_", Unique = true };

			CreateIndexModel<MongoDbGrainDocument<UserModel>> usernameIndexModel =
				new CreateIndexModel<MongoDbGrainDocument<UserModel>>(
					Builders<MongoDbGrainDocument<UserModel>>.IndexKeys.Ascending("_doc.Username"), usernameIndexOptions);
			Collection.Indexes.CreateOne(usernameIndexModel);
		}

		return Task.CompletedTask;
	}

After this, if you check your database and UserModel collection, you'll notice that two new indexes have been added for the correct members of the document. Now we can query our collection by username and email pretty fast.

The Example Project

The library contains a console app project that creates two types of Orleans grains and corresponding repositories. The app also provides the UI for filling the database with mock data and fetching it using repositories.

Conclusion

I hope this library will be useful for developers working with Orleans and MongoDB. It addresses the weaker side of Orleans, allowing for rich querying capabilities while maintaining the simplicity and scalability that Orleans offers.

The library allows developers to focus on what truly matters: building robust and scalable applications by automating much of the boilerplate code associated with setting up database repositories.

Whether you're a Orleans veteran or just getting started, I hope that Orleans.Providers.MongoDBRepository will find a place as a valuable tool to have in your toolkit.

Have a great day!