Securing NetMQ

Inspired by the coming ZMTP v3.0 and CurveZMQ I started to develop security  layer for NetMQ as well.

As CurveZMQ the library is currently working over NetMQ sockets, in the future with ZMTP v3.0 it will probably be part of the library.

NetMQ Secure Channel is based on TLS 1.2 and DTLS 1.2, after spending a lot of time with both RFC (TLS & DTLS) I’m happy to say that the library covered most of the the features.

Before diving into some code I need to say that the library is not yet ready for production use.

Client

using (var socket = context.CreateDealerSocket())
{
  socket.Connect("tcp://127.0.0.1:5556");
 
  SecureChannel secureChannel = new SecureChannel(ConnectionEnd.Client);
 
  // we are not using signed certificate so we need to validate 
  // the certificate of the server, by default the secure channel 
  // is checking that the source of the 
  // certitiface is a root certificate authority
  secureChannel.SetVerifyCertificate(c =>  true);
 
  IList outgoingMessages = new List();
 
  // call the process message with null as the incoming message 
  // because the client is initiating the connection
  secureChannel.ProcessMessage(null, outgoingMessages);
 
  // the process message method fill the outgoing messages list with 
  // messages to send over the socket
  foreach (NetMQMessage outgoingMessage in outgoingMessages)
  {
    socket.SendMessage(outgoingMessage);
  }
  outgoingMessages.Clear();
 
  // waiting for a message from the server
  NetMQMessage incomingMessage= socket.ReceiveMessage();
 
  // calling ProcessMessage until ProcessMessage return true 
  // and the SecureChannel is ready to encrypt and decrypt messages
  while (!secureChannel.ProcessMessage(incomingMessage, outgoingMessages))
  {
    foreach (NetMQMessage outgoingMessage in outgoingMessages)
    {
      socket.SendMessage(outgoingMessage);
    }
    outgoingMessages.Clear();
 
    incomingMessage = socket.ReceiveMessage();  
  }
 
  foreach (NetMQMessage outgoingMessage in outgoingMessages)
  {
    socket.SendMessage(outgoingMessage);
  }
  outgoingMessages.Clear();
 
  // you can now use the secure channel to encrypt messages
  NetMQMessage plainMessage = new NetMQMessage();
  plainMessage.Append("Hello");
 
  // encrypting the message and sending it over the socket
  socket.SendMessage(secureChannel.EncryptApplicationMessage(plainMessage));
}

Server

// we are using dealer here, but we can use router as well, we just have to manager
// SecureChannel for each identity
using (var socket = context.CreateDealerSocket())
{
  socket.Bind("tcp://*:5556");
 
  SecureChannel secureChannel = new SecureChannel(ConnectionEnd.Server);
 
  // we need to set X509Certificate with a private key for the server
  X509Certificate2 certificate = new X509Certificate2("NetMQ.Testing.pfx", "1");
  secureChannel.Certificate = certificate;
 
  IList outgoingMessages = new List();
 
  // waiting for message from client
  NetMQMessage incomingMessage = socket.ReceiveMessage();
 
  // calling ProcessMessage until ProcessMessage return true 
  // and the SecureChannel is ready to encrypt and decrypt messages
  while (!secureChannel.ProcessMessage(incomingMessage, outgoingMessages))
  {
    foreach (NetMQMessage outgoingMessage in outgoingMessages)
    {
      socket.SendMessage(outgoingMessage);
    }
    outgoingMessages.Clear();
 
    incomingMessage = socket.ReceiveMessage();
  }
  foreach (NetMQMessage outgoingMessage in outgoingMessages)
  {
    socket.SendMessage(outgoingMessage);
  }
  outgoingMessages.Clear();
 
  // this message is now encrypted
  NetMQMessage cipherMessage = socket.ReceiveMessage();
 
  // decrypting the message
  NetMQMessage plainMessage = secureChannel.DecryptApplicationMessage(cipherMessage);
  Console.WriteLine(plainMessage.First.ConvertToString());
}

Code

Both client and server should call ProcessMessage until the method return true.

First call to client ProcessMessage should be with null as the incomingMessage because client is the one initiating the connection.

Calling ProcessMessage fill the outgoingMessages list with messages that need to be send to the other peer. Make sure to send those.

Summary

The code is not yet merge into NetMQ, you can find the source code at my github.

Learning from past mistakes the SecureChannel protocol is versioned, so any breaking changes will happen in new a version.
Server will be able to support multiple versions of clients.

Some notes about the library:

  • Only RSA, AES (128/256) and SHA (1/256) combination exist.
  • Renegotiation is not supported inside the library, however it’s very simple to implement over the library.
  • Alert layer from TLS is missing completely from the layer, might be implement in future versions.
  • Only block ciphers are supported, compression is not supported.
  • Client cannot authenticate with a certificate.
  • You can send multipart messages.
  • The messages doesn’t have to be ordered (like DTLS)
  • Make sure to catch and handle NetMQSecurityException.

Array Length in C#

As you know in C# you can query the length of the array with Length property, but it’s not so obvious, in c++ for example you allocate the array to the size you want but the length is unknown after the allocation.

My question is, how in C# the language can tell you the length of the array, or to ask this question in another way, where does C# save the array length, my first guess was that the first item in an array is the array length, I decide to go and search for it. First I tried to search in the first item:

  static void Main(string[] args)
  {
    int[] array = new int[6];
 
    array[0] = 0;
    array[1] = 1;
    array[2] = 2;
    array[3] = 3;
    array[4] = 4;
 
    var handle = GCHandle.Alloc(array, GCHandleType.Pinned);
 
    IntPtr address = handle.AddrOfPinnedObject();
 
    int value = Marshal.ReadInt32(address);
    Console.WriteLine(value);
 
    address += 4;
    value = Marshal.ReadInt32(address);
    Console.WriteLine(value);
 
    address += 4;
    value = Marshal.ReadInt32(address);
    Console.WriteLine(value);
 
    address += 4;
    value = Marshal.ReadInt32(address);
    Console.WriteLine(value);
 
    address += 4;
    value = Marshal.ReadInt32(address);
    Console.WriteLine(value);
 
    handle.Free();
 
    Console.ReadLine();
  }

And the output:

3

So you can see my guess was wrong, the first item is actually the first item.

Then I tried the 7 item (last + 1), which make no sense because it’s has to be the first otherwise the language won’t know where to search for it.

Finally I tried to search before the first item which actually prove right:

  static void Main(string[] args)
  {
    int[] array = new int[6];
 
    array[0] = 0;
    array[1] = 1;
    array[2] = 2;
    array[3] = 3;
    array[4] = 4;
 
    var handle = GCHandle.Alloc(array, GCHandleType.Pinned);
 
    IntPtr address = handle.AddrOfPinnedObject();
 
    int value = Marshal.ReadInt32(address-4);
    Console.WriteLine(value);
 
    value = Marshal.ReadInt32(address);
    Console.WriteLine(value);
 
    address += 4;
    value = Marshal.ReadInt32(address);
    Console.WriteLine(value);
 
    address += 4;
    value = Marshal.ReadInt32(address);
    Console.WriteLine(value);
 
    address += 4;
    value = Marshal.ReadInt32(address);
    Console.WriteLine(value);
 
    address += 4;
    value = Marshal.ReadInt32(address);
    Console.WriteLine(value);
 
    handle.Free();
 
    Console.ReadLine();
  }

And the output was:
4

To summarize C# save the length of the array one item before the first item of the array (or 4 bytes before the beginning of the array).
Which actually make the size of the array num_of_items * size_of_item + 4 (which is size of integer holding the array size).

Structures in C# & .Net

You are probably not sure why I’m writing post about structures in the Advanced C# category. I hope that by the end of the post you will be sure, because structures have some interesting attributes in .net that worth understanding, especially when writing high performance and low level code in C#.

Structures are value type

What does that mean? Let’s dive directly into an example:

struct Number
{
  public int Value;
}

static void Main(string[] args)
{
  Number n;
  n.Value = 5;

  Number n2 = n;
  Console.WriteLine("Value of n2 after copyied from n: {0}", n2.Value);

  n2.Value = 10;
  Console.WriteLine("Value of n2 after a change: {0}", n2.Value);
  Console.WriteLine("Value of n after the change to n2: {0}", n.Value);
}

Before showing you the result, let’s talk about some interesting points in this code, I didn’t initialize the n variable and didn’t get null reference exception, why is that? because when you create a structure variable it’s created on the stack, it’s not a pointer to memory on the heap. Now let’s take a look at the result:

2

Strange? doing the same with classes would show 10 for both n and n2, but with structures n keep the original value. Why? because it’s value type, it’s not like a class that when you are doing assignment you just copy the reference, you copy the entire structure. n and n2 are two different location on memory, when the assignment happen it just copied all the memory from n to n2, when we changed n2 the n didn’t change.

Deep Copy

So we know that every time we use assignment with structure we are actually coping memory, this copy thing can be a good for case when you want an object to have deep copy capability, instead of manual copy all the object and fields or using serialization solutions.

The rule for using structure for deep copy is that the structure will only hold primitives and other structures (which hold primitives), string will be copied as well but arrays won't be deep copeied.

Some example can be the memento pattern, which is a pattern when an object can go back to previous state, you can hold all the state of the object (which is a class) in a structure and when you start editing copy the structure to another field inside the class, when you want to rollback just copy the structure back to the main field.

Serialization

I think is the fastest way to serialize an object in .net, what I’m I talking about? because structure is a block of memory, you can copy this block of memory into a byte array, no serialization and no other stuff, just copy into a byte array and your are ready to save to a file, send with socket or write to a stream. How you do this? following is a short example:

int size = Marshal.SizeOf(n);
byte[] bytes = new byte[size];
 
var handle = GCHandle.Alloc(n, GCHandleType.Pinned);
IntPtr ptr = handle.AddrOfPinnedObject();
 
Marshal.Copy(ptr, bytes, 0, size);
 
handle.Free();

What happened here? We got the memory location of the structure with line 4 & 5 (this trick is usually used with interop) then copied the memory to the byte array and released the pointer and handle.

NetMQ Lesson #1 – Basics

So you probably already know that NetMQ is port of ZeroMQ (also spelled ØMQ, 0MQ or ZMQ) to .Net.

If you are not familiar with ZeroMQ or NetMQ please read my previous post about introducing NetMQ.

Transport

In NetMQ you can choose from different transport, in this post I will focus on TCP.

  • TCP – communication over the network.
  • INPROC – in process communication, or inter-thread communication.
  • PGM – reliable multicast implemented with Microsoft implementation of PGM.
  • IPC – inter process communication, on NetMQ it’s exactly the same as TCP.

Patterns

As with transports NetMQ comes with different communication patterns, in this post I will cover request-response pattern.

  • Request/Response – client send request and server answer with response.
  • Pub/Sub – client subscribe for messages and server distribute message to all subscribed clients.
  • Dealer/Router – in simple words its client-server communication, dealer is the client and the router is the server, more about this in the future.
  • Push/Pull – one peer push messages and one or more workers peek those messages, good to build a pipeline.
  • Pair – one to one connection, for use with INPROC transport.
  • Dealer/Dealer, Router/Router – more advanced patterns for some cases.

Multi-threading

When you are entering the world of high performance systems (like trading platforms) you soon find out that everything you learned about multi-threading is wrong.

When you are writing high performance systems you don’t use locks, not reader writer lock, not mutex, not monitor and not any other type of lock. If you are writing very good code you even not using .net concurrent collection (except blocking collection, which is one pattern you can use) or interlocked. Now there is very easy way to do it, just don’t share data between threads, but this is another topic worth posting about.

The reason I’m telling you this is that NetMQ (and ZeroMQ) is coming from that world, you are not sharing the socket (which I explain later what is) between threads, the socket is belonging to the thread that create it. If you want to pass data to another thread you can use NetMQ with INPROC as transport (or blocking collection or ring buffer or any other inter-thread communication library). The topic of multi-threading is covered very good in the ZeroMQ guide.

Socket like API

So the basic of NetMQ is the socket object, no matter how many clients you have connected you have one socket, and not matter to how may servers you are connected to you have one socket. When you want to send a message you just call the Send method and when you want to receive a message you just call the Receive message.

Code

So let’s start with simple request response client server example (you need to add a reference to NetMQ, you can find the library on nuget):

  class Program
  {
    static void Main(string[] args)
    {
      using (NetMQContext context = NetMQContext.Create())
      {
        Task serverTask = Task.Factory.StartNew(() => Server(context));
        Task clientTask = Task.Factory.StartNew(() => Client(context));
        Task.WaitAll(serverTask, clientTask);
      }
    }

    static void Server(NetMQContext context)
    {
      using (NetMQSocket serverSocket = context.CreateResponseSocket())
      {
        serverSocket.Bind("tcp://*:5555");

        while (true)
        {
          string message = serverSocket.ReceiveString();

          Console.WriteLine("Receive message {0}", message);

          serverSocket.Send("World");

          if (message == "exit")
          {
            break;
          }
        }
      }
    }

    static void Client(NetMQContext context)
    {
      using (NetMQSocket clientSocket = context.CreateRequestSocket())
      {
        clientSocket.Connect("tcp://127.0.0.1:5555");

        while (true)
        {
          Console.WriteLine("Please enter your message:");
          string message = Console.ReadLine();
          clientSocket.Send(message);

          string answer = clientSocket.ReceiveString();

          Console.WriteLine("Answer from server: {0}", answer);

          if (message == "exit")
          {
            break;
          }
        }
      }
    }
  }

Remember what I told you about not sharing anything between threads? So you are actually allowed to share the NetMQContext between threads. It’s the only NetMQ object that you allowed to share between threads.

So as you can see using NetMQ is pretty easy. One important thing to note, string is not the only thing you can send, NetMQ is actually about delivering binary data, the Send method that receives string is just another overload of the receive method, more usually you will pass byte array and do the object serialization your self or with another library.

You can create different application for the client and for the server and launch multiple clients, in the request response pattern you don’t have to specify which client you sending the response to, the socket just know. In more advanced scenario (dealer/router pattern) you can specify to which client you are sending the message.

Introducing NetMQ - .net port of ZeroMQ

Update: this post was written more than a year ago and since then the NetMQ has become the default ZeroMQ choice for .Net developers. Also the project has a growing community with more than 4,000 downloads on nuget and it is a production ready library.

NetMQ is C# port of ZeroMQ.

ZeroMQ

ZeroMQ  is a high-performance asynchronous messaging library aimed at use in scalable distributed or concurrent applications. It provides a message queue, but unlike message-oriented middleware, a ZeroMQ system can run without a dedicated message broker. The library is designed to have a familiar socket-style API.

If you are not familiar with ZeroMQ, (also spelled ØMQ, 0MQ or ZMQ) you should be, becuse besides being a very fast messaging library it’s also changes the way you think about programming. If you really want to learn about ZeroMQ, make sure to read Pieter Hintjens book “Code Connected”. The ebook version is free and you can get it here or you can also read it on the ZeroMQ website.

This book will change the way you think about distributed enterprise systems and about programming all together. It did for me and I read it 3 6 times already, front to back and I will probably read it again.

How it all started

So the NetMQ project started as a weekend project (I have plenty of those, but most of them stay on my computer).

The original reason was that the library was missing some features that I really needed in order to push ZeroMQ at my company. But not being familiar with the library and C++ not being my native programming language I decided to port it to C#.

Lucky for me, somebody already did part of the job by porting ZeroMQ to Java. With java being more close to C# then C++, I took the java project as my base project. And 30 hours later and after a lot of coffee and no sleep ZeroMQ was ported to C#. It took me around 4 more month to do the extra 20% and make it ready for use.

Why NetMQ?

So if you want to use ZeroMQ or already are using ZeroMQ and you are developing with .Net, I would suggest that your try out NetMQ. You might be asking yourself, “why should I use NetMQ and not ZeroMQ with the C# binding?” I have multiple answers for this question:

  • Running unmanaged code inside a managed application can do some nasty stuff like memory leak and weird no access errors.
  • It is easier to debug with native c# code and you can download the code and just debug your system.
  • You can contribute. If you are using .Net, then C#  is probably better than your C++, and NetMQ gives you the opportunity to give something back to the community and get your name on the contributors list.
  • Update: As of 2014 C# binding (CLRZMQ) is no longer maintained and NetMQ is the default choice for ZeroMQ and .Net.
  • Update: As of 2014 NetMQ is a stable project with a growing community and is in production use by multiple companies.

NetMQ is being maintained by a community. You can visit our website at github, and download the nuget package.