A testimonial to one of the best mentors I have had

I have previously written about my internship at IBM in this post and this post. The first post makes it pretty clear that the internship was a very productive (very likely most productive) part of my engineering student life. The second post briefly talks about Gautham Pai as being my guru. Both are very much true and I am thankful to Gautham for having provided me the opportunity to be on the Eclifox team. Although I have expressed my gratitude to Gautham a few times, I never really wrote it down anywhere, neither in my blog nor on any social n/w site. A few months ago Gautham started his own company Jnaapti, which is a technical skill development company. Basically he is doing what he is very good at, i.e. bringing the best out of anyone willing to learn and succeed. As part of the company operations he conducts training sessions for various corporate clients and also mentors engineering students helping them understand their subjects better using some useful project as a means of teaching. He is experimenting with various educational methodologies and different ways to teach/mentor students remotely. I am very confident that his efforts are going to change the landscape of computer science engineering education vastly. Having been mentored and guided by Gautham at various points, I thought now would be a good time to pen down a testimonial and finally put that gratitude in words. So here it goes :

Every software engineer who has been in the industry, even for a small amount of time, surely knows the gap between academic teaching and the industry requirements and the initial uphill task of coping up when a fresh engineering graduate joins any company. It would not have been different for me, if not for Gautham's guidance as my senior at college and my mentor at IBM during my internship. With Gautham's mentoring, the internship was probably one of the most productive spans in my 4 years of engineering studies and also the one packed with maximum learning. Additionally it opened up a number of opportunities for me which I previously did not even know existed - like my participation in Google Summer code and later being with the Mozilla community for quite some time and many others.

Traits like general intelligence, theoretical understanding of the subjects and the ability to solve problems are undoubtedly necessary, but not sufficient. An engineer should be able to think not just about the next line of the code that he is going to write but also think about product that he is building or helping build. He should also know that any technology is just a tool to get the work done and not something that limits you. That way you just pick any new tool that you come across and find useful or necessary for the job. This also means you keep up with the latest happenings in the tech world via blogs, articles, mailing lists etc. Above all the zeal to do more, to come up with new ideas, to start executing those ideas and the persistence to see them through, in the course carefully managing a team as a leader, are what will make an engineer truly successful.

I, of course, did not realize or understand all this during my internship. These were not handed out to me in bulleted list of To-Dos. Rather it was all nicely baked into the project that I (with a few friends) carried out and I was set on the right path without any additional effort. More than that, all of this was demonstrated to us in practice by Gautham himself and some of it just rubbed off on me, making me a much better problem solver, much better product developer, much better ENGINEER, than I would have been otherwise. Now when I look back at my internship days and my days as an engineer after that, I clearly see the impact and how much it has helped. That's Gautham and his way of mentoring. Thank you Gautham for letting me be part of the Eclifox team and for your guidance till date and the future too. :). (For the readers : Eclifox was what we built during our internship - http://buzypi.in/2007/10/11/eclifox-bringing-eclipse-to-the-browser/  and I am very proud of it.)

 In case you are wondering where did all of this finally land me, here is my linkedIn profile. :)
Keep up the great work Gautham. Wish you all the success and happiness.

Transactions - both single node and distributed - are hardwired in Windows - since Win 95

Transactions or "Atomic Transactions" to be precise, are very well known to anyone who has worked with databases. With the recent advent of NoSQL databases and the CAP theorem being used/abused by anyone and everyone, words like "consistency" and "transactional model" have become run-of-the-mill jargon. But what is actually interesting is that the concept of transactions or transactional model goes beyond our typical RDBMS. Things get even more challenging when we try to achieve transactions in a distributed system. Because transactions inherently lock the resource(s)/data they are operating on until the transaction completes, those resources can become inaccessible altogether very easily in a distributed setup if one of the node fails or if there is some problem with the network or any such thing, there by increasing the complexity of implementing distributed transactions by many folds compared to transactions on a single node.

Today I was trying to figure out if there is a way to "simulate" (albeit it will be very crude) some sort of transactions in my application which uses MongoDB (which doesn't support transactions by design - to avoid the locking mentioned above, although ironically there is a global write lock..!!). Searching on the internet lead me to this blog of a Raven DB developer. The author there mentions that RavenDB supports both sharding and transactions, which means it has implemented distributed transaction support. At first read I was pretty impressed (this was the first time I had heard about RavenDB). Before I could ask the author about the implementation details I saw a comment in which the author had mentioned that they use DTC (which again was a new thing). Turns out DTC, Distributed Transaction Controller, is a service that is baked right in the Windows OS itself, that too dating back to the Windows 95 days (wow.. now I am impressed with Windows..!). Here is the MSDN article describing the service.

The MSDN article clearly explains the basics of distributed transactions and how it is modeled. What is worth noting is that, by abstracting out the code for carrying out distributed transactions as a service, multiple resource managers (like different databases, queue servers, file servers/managers, etc..) can all interact together in a single transaction. For example, lets say that you have web application where in a client request results in a job being picked up from a queue for processing and simultaneously you update the status of the job in a DB and also create a new file associated with the start of the job. Very evidently all the three resource managers and the web application itself can be (very likely will be) on different nodes. With something like DTC you can easily create a new transaction, send across a commit message and you will get a success notification only if all three actions were successful or else none of the actions go through. Of course, this is possible only if all the three resource managers involved here adhere to the Microsoft's DTC specification and provide the necessary interface to work with it.

The previous example might make DTC appear like this Jason Bourne kind of super dude who can take care of all the heavy lifting and also do it very efficiently. But remember even Bourne gets shot at and also loses his girl. So DTC is not fully immune to problems either. Here is one blog post titled "My beef with MSDTC and two phase commits". It is definitely worth reading. Note that my impression about DTC is purely based on reading the documentation. I have not written a single line of code using DTC.

Microsoft's Virtual Wifi adapter ( or virtual wifi card) -- cool technology

I wasn't aware of the very interesting research on Virtual Wifi Adapters that Microsoft guys have been carrying out. Apparently they have been doing it for quite some time now. What this research group is trying to do is basically allow us to have an unlimited number of "virtual" wireless cards on our computers, each connecting to a different wireless connection, and all of it using just a single physical card. That is some awesome stuff.. !

A couple of days ago I opened up the Dell Support Center tool on my laptop and it popped up a message saying a device on my system is in the disabled state. I was pretty startled to see that, as pretty much every device on my laptop is used by me daily. On clicking the message it told me that the disabled device is "Microsoft Viritual Wifi Miniport". That did not make any sense to me. I had absolutely no clue about this device.

Searching the internet led me to this Microsoft page (along with several others, of course) which gave me a fair idea of what this device might be, but nothing concrete. It was this fine article on istartedsomething.com that clearly explained what this is all about. In the same article the author tells us that Microsoft has been carrying out research in this regard for a few years now. But nothing was given to end users until Windows 7 baked in this wifi card virtualization natively. And not just that, all WiFi card providers are expected to add support for this virtualization in their drivers if they want their drivers to be properly signed digitally and recognized by Windows during installation. I say that is "Wicked cool".. :)

About the technology itself, it can be described as a way to make "software copies" of your Wireless card and use those copies to "connect to multiple networks simultaneously".  Although research prototypes can apparently create any number of virtual devices over the single actual hardware device, Windows 7 limits it to just one copy/virtual device.

This whole research is doubly fascinating.

First because the applications of this research work are very interesting. One such application is explained in the article mentioned above. It talks about being able to connect to an existing wireless access point with your laptop and at the same time making your laptop a wireless access point in itself. It means, if someone is far from the actual access point and your laptop happens to be closer, he/she can connect to your laptop instead and your laptop will forward their connections to the actual wireless access point. Of course, this can only happen when the two laptops involved are in the same security/trust group. I wouldn't go on and connect via some random stranger's laptop. It is like letting that person look at all the data coming in and going out of your computer over the internet (or network in general). Despite such caveats, this is very much a practical use case. May be you wouldn't use it to be a hop in the network (or more like a virtual signal booster), but you may use it to make P2P/direct connections with another laptop close by for sharing files instead of doing over the wireless LAN. Or, if access to wireless network is possible only after you authenticate via certificate (like in a corporate setup) and the certificates can be put only one of your laptops (the official company laptop), the connection sharing will indeed come in handy.

Secondly, and more importantly, the complexities associated with this are lot and that makes it all the more exciting. If we delve a little deeper into what this virtual adapter is and how it works, we will see that it is actually a piece of software sitting between the actual device driver and the rest of the network stack (i.e. all above the MAC layer in the OSI model). This little piece of software is supposed to appear as one or more "devices" to the OS and hence it invariably has to have its own device driver. That is the "Virtual WiFi Filter" driver or VWifi driver. This VWifi driver tells the OS that there are multiple wireless cards and the OS then allows the user to connect to different available wireless connections via these virtual cards. But note that all this time, there is only one physical card and hence at any given point in time that one physical card can be connected to (or can communicate with) only one wireless network. It is the job of the virtual adapter software to cycle over all the virtual wireless cards and service the network requests made through them using the one physical card, in a time shared manner all the while keeping it transparent to the user. Although it sounds very similar to the kernel's process scheduling which makes use of the single processor in a time shared manner, this is actually somewhat different because of the way wireless networks work.

Note that different wireless networks behave differently. They might be operating at different frequencies, they might be having different authentication/encryption schemes, the bandwidth might be different and probably many other factors that I can't think of right now. So every time the actual wireless card switches to a different connection, there can be a shift in all or some of the above mentioned attributes. The card might have to step up or down its operating frequency, change to a different encryption scheme and all of this on the fly. Now that is a lot of work to do and in fact all of this switching can just drag the network performance to the ground. This makes the design and implementation of the virtualization software pretty challenging. This and many other challenges/caveats are discussed in this Microsoft Research's FAQ page.

I have been very excited about this research work ever since I read it and have been meaning to try out writing some network code using the virtual adapter. Sadly I have zero experience in network programming in Windows and currently don't have enough time to read up all of that. I hope such a thing will come up in Linux some time soon, if it isn't already there.