12 December 2013 CI Team

 

image.png

Those of you who use RavenDB might have heard about ESENT already. In the inside RavenDB uses the “Extensible Storage Engine” which is included into Windows since XP. Read here why Ayende is looking for an alternative for a while – although ESENT is basically reliable but that’s just a side note.

ESENT is used by Microsoft in many different areas – like the Active-Directory or the Exchange Mailbox information’s. So basically it is used whenever a huge amount of information’s need some structure. I’m going to quote this Blogpost because it describes the features of ESENT pretty well.

Features

Significant technical features of ESENT include:

- ACID transactions with savepoints, lazy commits, and robust crash recovery.
- Snapshot isolation.
- Record-level locking (multi-versioning provides non-blocking reads).
- Highly concurrent database access.
- Flexible meta-data (tens of thousands of columns, tables, and indexes are possible).
- Indexing support for integer, floating point, ASCII, Unicode, and binary columns.
- Sophisticated index types, including conditional, tuple, and multi-valued.
- Columns that can be up to 2GB with a maximum database size of 16TB.

Note: The ESENT database file cannot be shared between multiple processes simultaneously. ESENT works best for applications with simple, predefined queries; if you have an application with complex, ad-hoc queries, a storage solution that provides a query layer will work better for you.

 

First steps with ESENT – ManagedESENT

Of course it is possible to talk in C++ to ESENT but it just doesn’t look right. A better alternative is:

 

image

The project consists of two different parts:

- A .NET wrapper to ESENT.dll

- PersistentDictionary which takes the .NET wrapper and offers a nice API

Additionally you’ll get some additional documentation.

 

.NET wrapper

Who plans to use ESENT in a more “pure” way could use the .NET wrapper. But you better know the insides of ESENT before you do so. Here is the code of a low-level sample-application.

 

namespace EsentSample { using System; using System.Text; using Microsoft.Isam.Esent.Interop; public class EsentSample { /// <summary> /// Main routine. Called when the program starts. /// </summary> /// <param name="args"> /// The arguments to the program. /// </param> public static void Main(string[] args) { JET_INSTANCE instance; JET_SESID sesid; JET_DBID dbid; JET_TABLEID tableid; JET_COLUMNDEF columndef = new JET_COLUMNDEF(); JET_COLUMNID columnid; // Initialize ESENT. Setting JET_param.CircularLog to 1 means ESENT will automatically // delete unneeded logfiles. JetInit will inspect the logfiles to see if the last // shutdown was clean. If it wasn't (e.g. the application crashed) recovery will be // run automatically bringing the database to a consistent state. Api.JetCreateInstance(out instance, "instance"); Api.JetSetSystemParameter(instance, JET_SESID.Nil, JET_param.CircularLog, 1, null); Api.JetInit(ref instance); Api.JetBeginSession(instance, out sesid, null, null); // Create the database. To open an existing database use the JetAttachDatabase and // JetOpenDatabase APIs. Api.JetCreateDatabase(sesid, "edbtest.db", null, out dbid, CreateDatabaseGrbit.OverwriteExisting); // Create the table. Meta-data operations are transacted and can be performed concurrently. // For example, one session can add a column to a table while another session is reading // or updating records in the same table. // This table has no indexes defined, so it will use the default sequential index. Indexes // can be defined with the JetCreateIndex API. Api.JetBeginTransaction(sesid); Api.JetCreateTable(sesid, dbid, "table", 0, 100, out tableid); columndef.coltyp = JET_coltyp.LongText; columndef.cp = JET_CP.ASCII; Api.JetAddColumn(sesid, tableid, "column1", columndef, null, 0, out columnid); Api.JetCommitTransaction(sesid, CommitTransactionGrbit.LazyFlush); // Insert a record. This table only has one column but a table can have slightly over 64,000 // columns defined. Unless a column is declared as fixed or variable it won't take any space // in the record unless set. An individual record can have several hundred columns set at one // time, the exact number depends on the database page size and the contents of the columns. Api.JetBeginTransaction(sesid); Api.JetPrepareUpdate(sesid, tableid, JET_prep.Insert); string message = "Hello world"; Api.SetColumn(sesid, tableid, columnid, message, Encoding.ASCII); Api.JetUpdate(sesid, tableid); Api.JetCommitTransaction(sesid, CommitTransactionGrbit.None); // Use JetRollback() to abort the transaction // Retrieve a column from the record. Here we move to the first record with JetMove. By using // JetMoveNext it is possible to iterate through all records in a table. Use JetMakeKey and // JetSeek to move to a particular record. Api.JetMove(sesid, tableid, JET_Move.First, MoveGrbit.None); string buffer = Api.RetrieveColumnAsString(sesid, tableid, columnid, Encoding.ASCII); Console.WriteLine("{0}", buffer); // Terminate ESENT. This performs a clean shutdown. Api.JetCloseTable(sesid, tableid); Api.JetEndSession(sesid, EndSessionGrbit.None); Api.JetTerm(instance); } } }

All it does is to save one value – that’s it. Uhhh… I know but as I said before it is Low-Level ;-)

A better inside is available in the Stock-Sample. In this example a complex data base is created by using the ManagedEsent API and it saves stock information’s. But there is still an easier way:

 

Persistent Dictionary

This alternative behaves mainly like a dictionary – but saved as an Esent-database. There are some more details in the documentation.

 

public static void Main(string[] args) { var dictionary = new PersistentDictionary<string, string>("Names"); Console.WriteLine("What is your first name?"); string firstName = Console.ReadLine(); if (dictionary.ContainsKey(firstName)) { Console.WriteLine("Welcome back {0} {1}", firstName, dictionary[firstName]); } else { Console.WriteLine( "I don't know you, {0}. What is your last name?", firstName); dictionary[firstName] = Console.ReadLine(); } }

The code is quite simple and in the background a folder named “Names” is created. That’s where the Esent database is situated:

image

Unfortunately I don’t know if it’s possible to readout the information with a tool (without code).

 

About the Performance

It takes a while to enter 1.000.000 entries into a database with using a PersistentDictionary and it takes even longer with using GUIDs instead of Integer. Maybe it takes some time to get expressive numbers. The team published some performance information here and here.

 

LINQ support!

There is a LINQ support for the PersistentDictionary since Version 1.6.

 

Windows Store Apps

Since Esent is a part of Windows and some sections might use the API it is possible to use Esent with Windows Store Apps. It should work since version 1.8 but it seems like the PersistentDictionary is not supported actually. Read more about it here.

 

More links about ManagedEsent

- Performance of Persistent Dictionary (CodePlex Discussion)

- Ayendes Fork

- Blog by a Microsoft employee about Esent and the ManagedEsent API

 

Esent Serialization

Beside the “PersistentDictionary” there is another project using the ManagedEsent:

 

image

 

Basically the project helps to integrate data structures easily in Esent. Have a look on it in this demo.

 

Result

Esent has a nasty API but is still used even in Windows 8. With the help of tools like ManagedEsent API, PersistentDictionary and Esent Serialization the first steps are quite easy. If they are the tools of choice depends mainly on the purpose – since there are numerous alternatives. But the fact that RavenDB is partly build on Esent is a proof that it is at least worth a shot.

P.S.: Since the code in this blogpost is mainly from official sources I won’t upload anything on GitHub. The necessary ManagedEsent.dll is available on NuGet.

 

Question: Anyone some experience with Esent to share?