Transactions - both single node and distributed - are hardwired in Windows - since Win 95

Transactions or "Atomic Transactions" to be precise, are very well known to anyone who has worked with databases. With the recent advent of NoSQL databases and the CAP theorem being used/abused by anyone and everyone, words like "consistency" and "transactional model" have become run-of-the-mill jargon. But what is actually interesting is that the concept of transactions or transactional model goes beyond our typical RDBMS. Things get even more challenging when we try to achieve transactions in a distributed system. Because transactions inherently lock the resource(s)/data they are operating on until the transaction completes, those resources can become inaccessible altogether very easily in a distributed setup if one of the node fails or if there is some problem with the network or any such thing, there by increasing the complexity of implementing distributed transactions by many folds compared to transactions on a single node.

Today I was trying to figure out if there is a way to "simulate" (albeit it will be very crude) some sort of transactions in my application which uses MongoDB (which doesn't support transactions by design - to avoid the locking mentioned above, although ironically there is a global write lock..!!). Searching on the internet lead me to this blog of a Raven DB developer. The author there mentions that RavenDB supports both sharding and transactions, which means it has implemented distributed transaction support. At first read I was pretty impressed (this was the first time I had heard about RavenDB). Before I could ask the author about the implementation details I saw a comment in which the author had mentioned that they use DTC (which again was a new thing). Turns out DTC, Distributed Transaction Controller, is a service that is baked right in the Windows OS itself, that too dating back to the Windows 95 days (wow.. now I am impressed with Windows..!). Here is the MSDN article describing the service.

The MSDN article clearly explains the basics of distributed transactions and how it is modeled. What is worth noting is that, by abstracting out the code for carrying out distributed transactions as a service, multiple resource managers (like different databases, queue servers, file servers/managers, etc..) can all interact together in a single transaction. For example, lets say that you have web application where in a client request results in a job being picked up from a queue for processing and simultaneously you update the status of the job in a DB and also create a new file associated with the start of the job. Very evidently all the three resource managers and the web application itself can be (very likely will be) on different nodes. With something like DTC you can easily create a new transaction, send across a commit message and you will get a success notification only if all three actions were successful or else none of the actions go through. Of course, this is possible only if all the three resource managers involved here adhere to the Microsoft's DTC specification and provide the necessary interface to work with it.

The previous example might make DTC appear like this Jason Bourne kind of super dude who can take care of all the heavy lifting and also do it very efficiently. But remember even Bourne gets shot at and also loses his girl. So DTC is not fully immune to problems either. Here is one blog post titled "My beef with MSDTC and two phase commits". It is definitely worth reading. Note that my impression about DTC is purely based on reading the documentation. I have not written a single line of code using DTC.

Microsoft's Virtual Wifi adapter ( or virtual wifi card) -- cool technology

I wasn't aware of the very interesting research on Virtual Wifi Adapters that Microsoft guys have been carrying out. Apparently they have been doing it for quite some time now. What this research group is trying to do is basically allow us to have an unlimited number of "virtual" wireless cards on our computers, each connecting to a different wireless connection, and all of it using just a single physical card. That is some awesome stuff.. !

A couple of days ago I opened up the Dell Support Center tool on my laptop and it popped up a message saying a device on my system is in the disabled state. I was pretty startled to see that, as pretty much every device on my laptop is used by me daily. On clicking the message it told me that the disabled device is "Microsoft Viritual Wifi Miniport". That did not make any sense to me. I had absolutely no clue about this device.

Searching the internet led me to this Microsoft page (along with several others, of course) which gave me a fair idea of what this device might be, but nothing concrete. It was this fine article on istartedsomething.com that clearly explained what this is all about. In the same article the author tells us that Microsoft has been carrying out research in this regard for a few years now. But nothing was given to end users until Windows 7 baked in this wifi card virtualization natively. And not just that, all WiFi card providers are expected to add support for this virtualization in their drivers if they want their drivers to be properly signed digitally and recognized by Windows during installation. I say that is "Wicked cool".. :)

About the technology itself, it can be described as a way to make "software copies" of your Wireless card and use those copies to "connect to multiple networks simultaneously".  Although research prototypes can apparently create any number of virtual devices over the single actual hardware device, Windows 7 limits it to just one copy/virtual device.

This whole research is doubly fascinating.

First because the applications of this research work are very interesting. One such application is explained in the article mentioned above. It talks about being able to connect to an existing wireless access point with your laptop and at the same time making your laptop a wireless access point in itself. It means, if someone is far from the actual access point and your laptop happens to be closer, he/she can connect to your laptop instead and your laptop will forward their connections to the actual wireless access point. Of course, this can only happen when the two laptops involved are in the same security/trust group. I wouldn't go on and connect via some random stranger's laptop. It is like letting that person look at all the data coming in and going out of your computer over the internet (or network in general). Despite such caveats, this is very much a practical use case. May be you wouldn't use it to be a hop in the network (or more like a virtual signal booster), but you may use it to make P2P/direct connections with another laptop close by for sharing files instead of doing over the wireless LAN. Or, if access to wireless network is possible only after you authenticate via certificate (like in a corporate setup) and the certificates can be put only one of your laptops (the official company laptop), the connection sharing will indeed come in handy.

Secondly, and more importantly, the complexities associated with this are lot and that makes it all the more exciting. If we delve a little deeper into what this virtual adapter is and how it works, we will see that it is actually a piece of software sitting between the actual device driver and the rest of the network stack (i.e. all above the MAC layer in the OSI model). This little piece of software is supposed to appear as one or more "devices" to the OS and hence it invariably has to have its own device driver. That is the "Virtual WiFi Filter" driver or VWifi driver. This VWifi driver tells the OS that there are multiple wireless cards and the OS then allows the user to connect to different available wireless connections via these virtual cards. But note that all this time, there is only one physical card and hence at any given point in time that one physical card can be connected to (or can communicate with) only one wireless network. It is the job of the virtual adapter software to cycle over all the virtual wireless cards and service the network requests made through them using the one physical card, in a time shared manner all the while keeping it transparent to the user. Although it sounds very similar to the kernel's process scheduling which makes use of the single processor in a time shared manner, this is actually somewhat different because of the way wireless networks work.

Note that different wireless networks behave differently. They might be operating at different frequencies, they might be having different authentication/encryption schemes, the bandwidth might be different and probably many other factors that I can't think of right now. So every time the actual wireless card switches to a different connection, there can be a shift in all or some of the above mentioned attributes. The card might have to step up or down its operating frequency, change to a different encryption scheme and all of this on the fly. Now that is a lot of work to do and in fact all of this switching can just drag the network performance to the ground. This makes the design and implementation of the virtualization software pretty challenging. This and many other challenges/caveats are discussed in this Microsoft Research's FAQ page.

I have been very excited about this research work ever since I read it and have been meaning to try out writing some network code using the virtual adapter. Sadly I have zero experience in network programming in Windows and currently don't have enough time to read up all of that. I hope such a thing will come up in Linux some time soon, if it isn't already there.