Invalid credentails : OmniAuth + oAuth2 + Rails 4 encrypted cookie store + simultaneous requests

OmniAuth is a very well know gem in the Ruby/Rails world. Almost every Rails application out there is probably using it to authenticate with one of the various mechanisms it supports. OmniAuth is just awesome.!

I have been using OmniAuth for about 3 years now in various Rails projects and it has worked very well, although I have had to monkey patch it once or twice to allow me to exploit some of the Facebook features (like authenticated referrals). But all in all, it just works as advertised and you will have an authentication system in pretty much no time at all.

Yesterday, however, I was having quite a bit of hard time getting OmniAuth to do a simple "Login with Facebook" oAuth2 authorization. It was something that had worked seamlessly on innumerable previous occasions. But yesterday it just kept failing repeatedly, succeeding only once in a while. And it always failed with the same obscure error "Invalid Credentials" during the callback phase. (OmniAuth operates in three phases : Setup, Request and Callback. Apart from OmniAuth wiki on github, this is a good place to read about it : http://www.slideshare.net/mbleigh/omniauth-from-the-ground-up). The fact that the error message was not so helpful made this whole process a lot more frustrating. After some hunting on the inter-webs I found that the culprit could be a bad "state" parameter.

Wait, what is this "state" parameter?

Background : oAuth-2 CSRF protection

oAuth2 specifies the uses of a non-guessable secure random string to be used as a "state" parameter to prevent CSRF attacks on oAuth. More details here : http://tools.ietf.org/html/rfc6749#section-10.12. This came out almost a year back and many oAuth providers implement it already, including Facebook. OmniAuth also implemented it last year. Although not written with the best grammar, this article will tell you why this "state" parameter is needed and what happens without it.

To sum it up, oAuth client (our web application) creates a random string, stores it in an accessible place and also sends it to the oAuth provider (Ex : Facebook) as "state" during the request phase. Facebook will keep it, authenticate the end user, ask for permissions and when granted sends a callback to our web application by redirecting the user back to our website "with the state" as a query parameter. The client (our web application) compares the "state" sent by the provider and the one it had stored previously and proceeds only if they match. If they don't then there is no proof that the callback that our web application received is actually from the provider. It could be from some other attacker trying to trick our web application into thinking (s)he is someone else.

In case of OmniAuth oAuth2, this state parameter is stored as a property in the session with the key 'omniauth.state' during the request phase. The result of the request phase is a redirect to the provider's URL. The new session with the "state" stored in it will be set on the client's browser when it receives this redirect (302) response for the request /auth/:provider (This is the default OmniAuth route to initiate the request phase). After the provider (Facebook) authenticates the user and user authorizes our application, the provider makes a callback to our application by redirecting the user back to our web application at the callback URL /auth/:provider/callback along with the "state" as a query parameter. When this callback URL is requested by the browser, the previously stored session cookie containing the 'omniauth.state' property is also sent to our web application.

OmniAuth checks both of these and proceeds only if they match. If they don't match it raises the above mentioned "Invalid Credentials" error. (Yeah, I know, not really a helpful error message..!).

Ok, that is good to hear, but why will there be a mismatch?

A mismatch is possible only if the session cookie stored on the user's browser is changed such that the 'omniauth.state' property is removed from it or altered after the request phase has set it. This can happen if a second request to our web application was initiated while the request phase of oAuth was running and it completed after the request phase completed but before the callback phase started. Sounds complex? The diagram below illustrates it.

The diagram makes it clear as to when and how the 'omniauth.state' gets removed from the session leading to the error. However, apart from the timeline requirements (i.e. when requests start and end), there is another essential criteria for this error to occur :

The response of the "other simultaneous request" must set a new session cookie, overriding the existing one. If it does not explicitly specify a session cookie in the response headers, the client's browser will retain the existing cookie and 'omniauth.state' will be preserved in the session.

Now, from what I have observed, Rails (or one of Rack middlewares) has this nifty feature of not serializing the session and setting the session cookie in the response headers, if the session has not changed in the course of processing a request. So, in our case, if the intermediate simultaneous request does not make any changes to the session, Rails will not explicitly set the session cookie, there by preventing the loss of 'omniauth.state' property in the session.

Ok, then why will the session cookie change and lose the 'omniauth.state' property?

One obvious thing is that the "other simultaneous request" might change the session - either add or remove or edit any of the properties. There however is another player involved.

This is where the "Encrypted Cookie Store" of Rails 4 comes into picture. Prior to Rails 4, Rails did not encrypt its session cookie. It merely signed it and verified the signature when it had to de-serialize the session from the request cookie. Read how Rails 3 handles cookies for a detailed breakdown. Rails 4 goes one step ahead and encrypts the session data with AES-256 (along with the old signing mechanism. More details on that coming up in a new post). The implementation used is AES-256-CBC from OpenSSL. I am not a Cryptography expert, but the way I understand it, the property of AES is that it results in a different cipher text every time you run the encryption for the same message plain text. Or it could also be because the Rails encryption scheme initializes the encryptor with a random initialization vector every time it encrypts a session (Implementation here). Either ways, the session cookie contents are always new for every request even when the actual session object or session contents remain unchanged. As a result Rails will always set the session cookie in the response header for every request and result in the browser updating that cookie in its cookie store.

In our case, this results in the session being clobbered at the end of the "other simultaneous request" and we end up losing the 'omniauth.state' property and oAuth fails.

Umm.. ok, but when and how does this happen in real world, if at all it can?!

All these requirements/constraints described above, especially the timing constraints makes one wonder if this can really happen in the real world. Well, for starters it happened to me (hence this blog post..!!). I also tried to think of scenarios other than mine where this would happen. Here are a couple that I could think of :

Scenario - 1) FB Login is in a popup window and the "Simultaneous request" is a XHR - Ex : an analytics or tracking request.

Here is the flow :

1. User clicks on a "Login with FB" button on your website.
2. You popup the FB Login page in a new popup window. Request phase is initiated. But there is a small window of time before the redirect response for '/auth/facebook' is received and 'omniauth.state' is set.
3. During that small window of time, in the main window, you send an XHR to your web app to, may be, track the click on the "Login with FB" button. You might do this to just track usage or for some A/B testing or to build a funnel, etc. This request sends the session without the 'omniauth.state'.
4. While the XHR is in progress, the redirect from the request phase is complete and the session with 'omniauth.state' is set. The user now sees FB Login page loading and proceeds to login once it is loaded.
5. While the user is logging in to FB and approving our app, the XHR has completed and has come back with a session without 'omniauth.state'. This is stored by the browser now.
6. Once user logs in and approves your app, the callback state starts. But the session sent to your web app is now missing the 'omniauth.state'. 
7. oAuth fails.

How big a deal is this scenario?

If you are indeed making a XHR in the background, then this scenario needs to be taken care of. Since the "other simultaneous" request is automatically triggered every time, it is very likely that session will get clobbered.

How to solve this?

You can either first send the XHR and then in the response handler of that XHR, you can open the FB Login page in the popup. Also have a timeout just to make sure you don't wait for too long (or forever) before you receive a response for that XHR.

Alternatively, if you can push your tracking events in a queue stored in a cookie, you can do that and then open the FB Login page. Once the FB Login completes, you can pull that event out of the queue and send it. As a backup have a code that runs on every new page load to look for pending events from the queue in the cookie and send those events.

With HTML5 in place, its probably better to use the localstorage for the queue than the cookie. But again that needs user's permission. Your call.

Scenario - 2) FB Login is in the same window/tab but User has the website opened in two tabs.

Here is the flow :

1. User has your website opened in a browser tab - Tab-1
2. User opens a link on your website in a second tab - Tab-2 (Ctrl + Click or 'Open in a new tab' menu item). This request sends the session without 'omniauth.state'.
3. While that Tab-2 is loading, user clicks on "Login with FB" in Tab-1 initiating the request phase.
4.  If the request loading in Tab-2 is a little time consuming the redirect of the request phase of oAuth in Tab-1 completes before request in Tab-2, setting the session with 'omniauth.state'. After that FB Login page is shown and user proceeds to login and authorize.
5. While the user is logging in, the request in Tab-2 completes, but with a session that is missing 'omniauth.state'.
6. After logging in to FB, the callback phase is initiated with a redirect to your web app, but with a session that doesn't have 'omniauth.state'. 
7. oAuth fails. 

How big a deal is this scenario?

Not a big deal actually. In your web app, in the oAuth failure handle, you can just redirect the user back to /auth/facebook, redoing the whole process again and guess what - this time it will succeed and that too without the user having to do anything because the user is already logged in to FB and has also authorized your app. But just to be on the safer side, you would want to be careful about this loop going infinite (i.e. You start FB auth, it fails and the failure handler restarts the FB auth). Setting a cookie (different from the session cookie) with the attempt count should be good. If the attempt count crosses a certain limit, send the user back to homepage or show up an error page or show a lolcats video, c'mon be creative.

Ok, those are two scenarios that I could think of. I am not sure if there are more.

Can OmniAuth change something to solve this?

I believe so. If OmniAuth uses a different signed and/or encrypted cookie to store the state value instead of the session cookie none of this session clobbering would result in loss of the state value. OmniAuth is a Rack based app and relies on the Session middleware. I am not entirely sure, but it can probably use the Cookie middleware instead. Just set its own '_oa_state' cookie and use that during callback for verification.

Will you send a pull request making this change?

I am not sure. I first will hit the OmniAuth mailing list and find out what the wise folks there have to say about this. If it makes sense and nobody in the awesome Ruby community provides an instant patch, I will try and send a patch myself.

THE END
 
Ok, so that was the awesome ride through oAuth workings inside the OmniAuth gem. In the course I got to know quite a bit of Rails and also Ruby internals. Looking forward to writing posts about those too. Okay, okay.. fine. I will try and keep those posts short and not make them this long..!!

Till then, happy oAuthing. :-/ !

P.S : Security experts, excuse me if I have used "authentication" and "authorization" in the wrong places. I guess I have used it interchangeably as web applications typically do both with oAuth2.

Creating Wildcard self-signed certificates with openssl with subjectAltName (SAN - Subject Alternate Name)

For the past few hours I have been trying to create a self-signed certificate for all the sub-domains for my staging setup using wildcard subdomain.

There are a lot of guides and tutorials on the internet out there which explain the process of creating a self-signed certificate using openssl with a good amount details. Further there are also certain guides to create a self-signed cert for wildcard domain. It's fairly easy. You just specify that your Common Name (CN) a.k.a FQDN is *.yourdomain.com while creating the certificate signing request (CSR).

This will take care of all of your sub-domains under yourdomain.com (like www.yourdomain.com or mail.yourdomain.com), however your top level bare domain (yourdomain.com) itself is not covered under this certificate. When you use a certificate generated by specifying *.yourdomain.com the browsers will throw up an error when you hit your server with the top level domain name https://yourdomain.com/.

To address this X.509 certificate standard allows for a type of extension named subjectAltName (http://en.wikipedia.org/wiki/SubjectAltName). Using this you can specify that there are a few other domain names for which this certificate is valid.

This requires specifying the use of this extension while generating the certificate request AND while signing the certificate. To do this you will have to add a few things to your openssl configuration file (typically /etc/ssl/openssl.cnf on a Ubuntu like machine). Alternatively you can copy the config file to another location, add these extension stuff there and then specify the new config file for all your openssl commands. The commands shown below assume the default config file at /etc/ssl/openssl.cnf was updated with extension details.

Here is one blog post which details the updates needed for the openssl config file. http://grevi.ch/blog/ssl-certificate-request-with-subject-alternative-names-san. It also has commands for generating the private key, converting the key to a format which does not ask for password (a.k.a unencrypted key), generating the certificate request (CSR) and finally signing the certificate. The steps mentioned there until the generation of certificate request are correct. It is the last step of signing the certificate that is missing one small piece of information because of which the extension mentioned above (subjectAltName) doesn't get added to the final certificate, despite they being present in the certificate request.

After a lot of searching on the internet, copy pasting the commands exactly, trying my luck at IRC (irc.freenode.net#openssl) the answer finally appeared to me in the man pages (duh..! RTFM dude..!!).  The man page for x509 (man x509) command of openssl has this little entry :

-extfile filename
           file containing certificate extensions to use. If not specified then no extensions are added to the certificate.

So turns out that just specifying the extensions in the openssl config file is not sufficient, but you must also specify the same file as the file containing the extensions to be included in the command line using the above -extfile option.

With that added, you will get a self-signed certificate for your wildcard subdomain which is also valid for your top level bare domain.

Changes made to /etc/ssl/openssl.cnf

Uncomment the req_extensions = v3_req

req_extensions = v3_req # The extensions to add to a certificate request

Add subjectAltName to v3_req section

[ v3_req ]

# Extensions to add to a certificate request

basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

Finally add the alternate names for which you want this certificate to be valid. It would be your toplevel bare domain

[alt_names]
DNS.1 = yourdomain.com

This last section [alt_names] will not be present in the file. You can add it right after the [ v3_req ] section.

Once config update is done, create your certificate.
Here are the commands that I used to create such a certificate :

Create a private key :

openssl genrsa -des3 -out ssl/staging/yourdomain.com.key 2048

This will ask you for a password. Key in a simple one and remember it.

Convert the private key to an unencrypted format

openssl rsa -in ssl/staging/yourdomain.com.key -out ssl/staging/yourdomain.com.key.rsa

This will ask you for a password. Key in the same thing that you used in the previous step.

Create the certificate signing request

openssl req -new -key ssl/staging/yourdomain.com.key.rsa -out ssl/staging/yourdomain.com.csr

This will ask you for a bunch of fields. Enter *.yourdomain.com when it asks for Common Name (or FQDN). The fields after that can be left blank by just hitting return (Enter) key.

Sign the certificate with extensions

openssl x509 -req -extensions v3_req -days 365 -in ssl/staging/yourdomain.com.csr -signkey ssl/staging/yourdomain.com.key.rsa -out ssl/staging/yourdomain.com.crt -extfile /etc/ssl/openssl.cnf

Note that here we specify the openssl config file as the file file containing extensions as that is where we have defined it. Probably we can put the extensions in a separate file too, but I haven't tried that. (Homework?! :P)

Thats it. Now you have a self-signed wildcard subdomain certificate which is valid for your top level domain too.

Despite this the first time you hit your SSL enabled website with the above generated certificate your browser will show the standard "Invalid Certificate - Untrusted" page. It is because your certificate is self-signed. You can view the errors in the "Technical Details" section.

Rails cookie handling -- serialization and format

A typical Rails cookie has this format : cookie-value--signature (the two dashes are literal). The "cookie-value" part is a url encoded, base64 encoded string of the binary dump (via Marshal.dump) of whatever was set in the session. The signature part is a HMAC-SHA1 digest, created using the cookie-value as the data and a secret key. This secret key is typically defined in [app-root]/config/initializers/secret_token.rb.

Let us try and reverse engineer a session cookie for a local app that I am running. I am using Devise for authentication, which in turn uses Warden. I use the Firecookie extension to Firebug to keep track of cookies. It is pretty handy.

Here is the session cookie set by Rails:

# Cookie as seen in Firebug
BAh7B0kiGXdhcmRlbi51c2VyLnVzZXIua2V5BjoGRVRbCEkiCVVzZXIGOwBGWwZvOhNCU09OOjpPYmplY3RJZAY6CkBkYXRhWxFpVGkvaQGsaQGwaRBpAdFpCGk9aQHtaQBpAGkGSSIiJDJhJDEwJEZseHh3c293Q29LcHhneWMxODR2b08GOwBUSSIPc2Vzc2lvbl9pZAY7AEYiJTUwNDdkOTMwNDNkNGEzOTA4YTkwN2U2MDY5OGRmOTdm--51f90f7176326f61636b89ee9a1fce2a4972d24f

As mentioned at the beginning it has two parts separated by two dashes (--).

The cookie value in this case is :

# The cookie-value part
BAh7B0kiGXdhcmRlbi51c2VyLnVzZXIua2V5BjoGRVRbCEkiCVVzZXIGOwBGWwZvOhNCU09OOjpPYmplY3RJZAY6CkBkYXRhWxFpVGkvaQGsaQGwaRBpAdFpCGk9aQHtaQBpAGkGSSIiJDJhJDEwJEZseHh3c293Q29LcHhneWMxODR2b08GOwBUSSIPc2Vzc2lvbl9pZAY7AEYiJTUwNDdkOTMwNDNkNGEzOTA4YTkwN2U2MDY5OGRmOTdm

The signature is :

51f90f7176326f61636b89ee9a1fce2a4972d24f

Whenever Rails gets a cookie it verifies that the cookie is not tampered with, by verifying that the HMAC-SHA1 signature of the cookie-value sent matches the signature sent. We can also do the verification ourselves here. Fire up irb and try the following :

$ irb

irb(main):003:0> cookie_str = "BAh7B0kiGXdhcmRlbi51c2VyLnVzZXIua2V5BjoGRVRbCEkiCVVzZXIGOwBGWwZvOhNCU09OOjpPYmplY3RJZAY6CkBkYXRhWxFpVGkvaQGsaQGwaRBpAdFpCGk9aQHtaQBpAGkGSSIiJDJhJDEwJEZseHh3c293Q29LcHhneWMxODR2b08GOwBUSSIPc2Vzc2lvbl9pZAY7AEYiJTUwNDdkOTMwNDNkNGEzOTA4YTkwN2U2MDY5OGRmOTdm"
=> "BAh7B0kiGXdhcmRlbi51c2VyLnVzZXIua2V5BjoGRVRbCEkiCVVzZXIGOwBGWwZvOhNCU09OOjpPYmplY3RJZAY6CkBkYXRhWxFpVGkvaQGsaQGwaRBpAdFpCGk9aQHtaQBpAGkGSSIiJDJhJDEwJEZseHh3c293Q29LcHhneWMxODR2b08GOwBUSSIPc2Vzc2lvbl9pZAY7AEYiJTUwNDdkOTMwNDNkNGEzOTA4YTkwN2U2MDY5OGRmOTdm"


# This cookie_secret comes from [app-root]/config/initializers/secret_token.rb. Obviously you need to keep this secret for your production apps.
irb(main):005:0> cookie_secret = '392cacbaac74af104375eb91324e254ba232424130e69022690aa98c1d0dfade159260588677e2859204298181385a83b923e58c4ef24bb3a40bdad9a41431b4'
=> "392cacbaac74af104375eb91324e254ba232424130e69022690aa98c1d0dfade159260588677e2859204298181385a83b923e58c4ef24bb3a40bdad9a41431b4"

irb(main):006:0> OpenSSL::HMAC.hexdigest(OpenSSL::Digest::SHA1.new, cookie_secret, cookie_str)
=> "51f90f7176326f61636b89ee9a1fce2a4972d24f"

As can be seen the HMAC-SHA1 hexdigest generated with the cookie-value matches the signature part of the cookie. Hence the cookie is not tampered with.

Now that the cookie authenticity is validated, let us see what information it holds.

Let us retrace the steps taken by Rails to generate this cookie value to get the value stored in the cookie. The steps taken by Rails are :

1. session_dump = Marshal.dump(session)
2. b64_encoded_session = Base64.encode64(session_dump)
3. final_cookie_value = url_encode(b64_encoded_session)

The reverse process would be :

1. url_decoded_cookie = CGI::unescape(cookie_value)
2. b64_decoded_session = Base64.decode64(url_decoded_cookie)
3. session = Marshal.load(b64_decoded_session)

And with a beautiful language like Ruby all these 3 steps can be done in one single line of code. Here it is :
(Btw, I need to require 'mongo' because one of the values contained here is of type BSON::ObjectId which is defined in the mongo gem. Without this Marshal.load will error out)

irb(main):001:0> require 'mongo'
=> true
irb(main):002:0> require 'cgi'
=> true
irb(main):003:0> cookie_str = "BAh7B0kiGXdhcmRlbi51c2VyLnVzZXIua2V5BjoGRVRbCEkiCVVzZXIGOwBGWwZvOhNCU09OOjpPYmplY3RJZAY6CkBkYXRhWxFpVGkvaQGsaQGwaRBpAdFpCGk9aQHtaQBpAGkGSSIiJDJhJDEwJEZseHh3c293Q29LcHhneWMxODR2b08GOwBUSSIPc2Vzc2lvbl9pZAY7AEYiJTUwNDdkOTMwNDNkNGEzOTA4YTkwN2U2MDY5OGRmOTdm"
=> "BAh7B0kiGXdhcmRlbi51c2VyLnVzZXIua2V5BjoGRVRbCEkiCVVzZXIGOwBGWwZvOhNCU09OOjpPYmplY3RJZAY6CkBkYXRhWxFpVGkvaQGsaQGwaRBpAdFpCGk9aQHtaQBpAGkGSSIiJDJhJDEwJEZseHh3c293Q29LcHhneWMxODR2b08GOwBUSSIPc2Vzc2lvbl9pZAY7AEYiJTUwNDdkOTMwNDNkNGEzOTA4YTkwN2U2MDY5OGRmOTdm"

# Reverse engineering the cookie to get the session object
irb(main):004:0> session = Marshal.load(Base64.decode64(CGI.unescape(cookie_str)))
=> {"warden.user.user.key"=>["User", [BSON::ObjectId('4f2aacb00bd10338ed000001')], "$2a$10$FlxxwsowCoKpxgyc184voO"], "session_id"=>"5047d93043d4a3908a907e60698df97f"}

This is the session data that the session cookie was holding. This data is subsequently used by Warden and Devise to fetch the user from the DB and do the authentication.

And that is how Rails handles cookies (at least how Rails 3.0.11 does. I am not sure if things have changed in later versions)

Analysis of the Duqu Trojan worm by Kaspersky Labs

This summary is not available. Please click here to view the post.

Microsoft's Virtual Wifi adapter ( or virtual wifi card) -- cool technology

I wasn't aware of the very interesting research on Virtual Wifi Adapters that Microsoft guys have been carrying out. Apparently they have been doing it for quite some time now. What this research group is trying to do is basically allow us to have an unlimited number of "virtual" wireless cards on our computers, each connecting to a different wireless connection, and all of it using just a single physical card. That is some awesome stuff.. !

A couple of days ago I opened up the Dell Support Center tool on my laptop and it popped up a message saying a device on my system is in the disabled state. I was pretty startled to see that, as pretty much every device on my laptop is used by me daily. On clicking the message it told me that the disabled device is "Microsoft Viritual Wifi Miniport". That did not make any sense to me. I had absolutely no clue about this device.

Searching the internet led me to this Microsoft page (along with several others, of course) which gave me a fair idea of what this device might be, but nothing concrete. It was this fine article on istartedsomething.com that clearly explained what this is all about. In the same article the author tells us that Microsoft has been carrying out research in this regard for a few years now. But nothing was given to end users until Windows 7 baked in this wifi card virtualization natively. And not just that, all WiFi card providers are expected to add support for this virtualization in their drivers if they want their drivers to be properly signed digitally and recognized by Windows during installation. I say that is "Wicked cool".. :)

About the technology itself, it can be described as a way to make "software copies" of your Wireless card and use those copies to "connect to multiple networks simultaneously".  Although research prototypes can apparently create any number of virtual devices over the single actual hardware device, Windows 7 limits it to just one copy/virtual device.

This whole research is doubly fascinating.

First because the applications of this research work are very interesting. One such application is explained in the article mentioned above. It talks about being able to connect to an existing wireless access point with your laptop and at the same time making your laptop a wireless access point in itself. It means, if someone is far from the actual access point and your laptop happens to be closer, he/she can connect to your laptop instead and your laptop will forward their connections to the actual wireless access point. Of course, this can only happen when the two laptops involved are in the same security/trust group. I wouldn't go on and connect via some random stranger's laptop. It is like letting that person look at all the data coming in and going out of your computer over the internet (or network in general). Despite such caveats, this is very much a practical use case. May be you wouldn't use it to be a hop in the network (or more like a virtual signal booster), but you may use it to make P2P/direct connections with another laptop close by for sharing files instead of doing over the wireless LAN. Or, if access to wireless network is possible only after you authenticate via certificate (like in a corporate setup) and the certificates can be put only one of your laptops (the official company laptop), the connection sharing will indeed come in handy.

Secondly, and more importantly, the complexities associated with this are lot and that makes it all the more exciting. If we delve a little deeper into what this virtual adapter is and how it works, we will see that it is actually a piece of software sitting between the actual device driver and the rest of the network stack (i.e. all above the MAC layer in the OSI model). This little piece of software is supposed to appear as one or more "devices" to the OS and hence it invariably has to have its own device driver. That is the "Virtual WiFi Filter" driver or VWifi driver. This VWifi driver tells the OS that there are multiple wireless cards and the OS then allows the user to connect to different available wireless connections via these virtual cards. But note that all this time, there is only one physical card and hence at any given point in time that one physical card can be connected to (or can communicate with) only one wireless network. It is the job of the virtual adapter software to cycle over all the virtual wireless cards and service the network requests made through them using the one physical card, in a time shared manner all the while keeping it transparent to the user. Although it sounds very similar to the kernel's process scheduling which makes use of the single processor in a time shared manner, this is actually somewhat different because of the way wireless networks work.

Note that different wireless networks behave differently. They might be operating at different frequencies, they might be having different authentication/encryption schemes, the bandwidth might be different and probably many other factors that I can't think of right now. So every time the actual wireless card switches to a different connection, there can be a shift in all or some of the above mentioned attributes. The card might have to step up or down its operating frequency, change to a different encryption scheme and all of this on the fly. Now that is a lot of work to do and in fact all of this switching can just drag the network performance to the ground. This makes the design and implementation of the virtualization software pretty challenging. This and many other challenges/caveats are discussed in this Microsoft Research's FAQ page.

I have been very excited about this research work ever since I read it and have been meaning to try out writing some network code using the virtual adapter. Sadly I have zero experience in network programming in Windows and currently don't have enough time to read up all of that. I hope such a thing will come up in Linux some time soon, if it isn't already there.

OpenSSL base64 filter BIO needs an EOL and memory BIO needs to know about EOF

I recently started working with the OpenSSL library to do some https stuff (sort of obviously). OpenSSL apart from having an implementation for the SSL encryption part, it also nifty algorithms for certificate handling and more importantly an abstract I/O layer implementation called BIO which probably stands for Basic I/O or Buffered I/O or something else. I do not know, I could not find it. Nevertheless, the items of interest here are the BIO_f_base64() -- The base64 encode/decode filter BIO and the BIO_s_mem() -- The memory BIO, which can hold data in a memory buffer.

The BIO man page (or its online version present here: http://www.openssl.org/docs/crypto/bio.html) give a nice introduction. For now just consider BIOs as black boxes from which you can read or write data. If the BIO is a filter BIO then the data will be processed whenever you read or write to it.

The name, BIO_f_base64, says it all about the functionality of this BIO. If you read from this BIO, then whatever data is being read is first base64 decoded and given to you. OTOH, if you write something to this BIO it will be base64 encoded and then written to the destination. These BIOs can be arranged in the form of chains to do a series of processing on the data that you are reading or writing, all by just a single call to read() or write(). Its all abstracted. Saves a lot of time.

I was trying to decode some base64 encoded data which I had in a buffer, a char [] to be precise. So if you read up about the BIOs it becomes obvious that you first have to create a memory BIO, which will hold the actual encoded data. Write the encoded data to the memory BIO. Then you chain that memory BIO with a base64 BIO and read from that chain. Any data that you read from the chain will actually come from the memory BIO, but before it reaches you it passes through the base64 BIO. So essentially you are reading from the base64 BIO. As mentioned in the earlier paragraph, when you read from a base64 BIO it decodes the data and gives it you. So the base64 encoded data present in the memory BIO is decoded and presented to you. That's it. base64 decoding is done in one simple read() call !

But there is small catch here. For some reason, which I have yet partially understood, base64 requires that the data it is handling be terminated with a new-line character always. If the data does have any newline character, meaning all your data is present in a single line then you have to explicitly tell that to the BIO by setting the appropriate flag. Here is what the man page says:

The flag BIO_FLAGS_BASE64_NO_NL can be set with BIO_set_flags() to encode the data all on one line or expect the data to be all on one line.

That's about the base64's EOL. Now the other BIO involved here,the memory BIO, is also an interesting guy. When the data it has gets over, it doesn't say "Hey, its over, stop it!". Instead it says "Dude, you got to wait for some more data to arrive. Hang on and keep trying". !!! This is very much suitable, probably when you using the BIO like a PIPE, where you keep pumping data from one end by acquiring it from somewhere and some other guy consumes that data. But in a situation like mine where the data is all fixed I simply want it to tell that the data is all over and I need to stop it. To do this again I will have to explicitly set an appropriate flag and here is what the man page says:

BIO_set_mem_eof_return() sets the behaviour of memory BIO b when it is empty. If the v is zero then an empty memory BIO will
return EOF (that is it will return zero and BIO_should_retry(b) will be false. If v is non zero then it will return v when it
is empty and it will set the read retry flag (that is BIO_read_retry(b) is true). To avoid ambiguity with a normal positive
return value v should be set to a negative value, typically -1.

And this same thing is explained very well here: http://www.openssl.org/support/faq.html#PROG15.

I thank Dr. Stephen N Herson of the OpenSSL project for helping me out in understanding this. Here is the mailing list posting that taught me this thing : http://groups.google.com/group/mailing.openssl.users/browse_thread/thread/f0fc310c1bc6ec65#

Happy BIOing. :-)