AJAX for beginners - what it is and how it evolved

Today's websites or web pages are very complex structures involving several components. It can be daunting for a beginner to understand it all at once. To ease the process let's break things down and see how the various pieces fit together.

At the core of web pages is the HTML. It is the language that browsers understand and it is what web developers use to showcase rich content. Without HTML all our web pages would have been simple text files. Now I assume that you are aware of HTML and how it is used in web pages. Another critical piece that works closely with HTML is CSS, but we can skip that one for now.

Plain HTML web pages are pretty dull. They are not really interactive. What they mainly have apart from text are :

1. Some formatting information which tells the browser how to show the contents.
2. Some media content like images or videos.
3. A bunch of hyperlinks which lead to other similar web pages.
4. In slightly evolved pages there would be forms which allowed a user to send data back to the servers.

It is the third one which helped form this "web" of pages which we call the Internet. How do these links work? Pretty simple actually. Each link typically points to a web page on a server. Whenever a user clicked on a link, the browser would fetch that page, remove the existing page and show the newly downloaded one. Some times the link could point back to the same page in which case clicking on it would just "refresh" the existing page. We all know about the refresh/reload button in the browsers don't we?

There are a few things to note here.

1. These basic web pages were dull in the sense that they were not interactive. They did not respond to user actions except for clicks on links, in which case the existing web page goes away and a fresh page is loaded.
2. When a link was clicked the browser would decide to send a request to the server. The web page itself did not have much control on when the request should be made.
3. When the browser was making a request in response to a link click, there was pretty much nothing you could do with the web page until the request completed. Every request to the server blocked the entire page and if there were multiple requests that had to be made it had to be sequential, one after the other. A fancy technical word to describe this would be "synchronous".
4. An obvious observation is that the data that came from server in response to request from a browser was HTML, as in the data and information on how to display that data were sent back from the server all the time.

While the fourth point appears as not such a big problem, the first three are definitely limitations.

To work around the first problem browser developers came up with JavaScript. Browser folks told web developers that they can make their web pages interactive and responsive to user actions by writing code to handle those interactions in this new language called JavaScript and including it in the web pages. Web developers could now write different pieces of code to handle different user actions. The browser provided a few functions (aka APIs) in JavaScript which the web developers could use to make changes to the web page. This API is called DOM (Document Object Model). Thus web pages became interactive and responsive.

The second and third limitations still existed however. The only way to request data from server was through links or in case of evolved pages through forms. Also the web page was one single unit. Whenever something had to change the whole page had to be reloaded. Imagine something like that with your gmail page. How would it be if the page refreshed every time you received a chat message or worse every time one of your contacts logged in or out of chat? That would be disastrous. The fate would be similar of something like a sports website in which the small corner showing the score needs to be updated a lot more frequently than the rest.

To solve this some people thought of iframes as a solution. Iframe is a HTML tag inside which you can put another HTML page. Now with this arrangement the inner page could be reloaded without affecting the outer page. So the sports website could now put the score part in an iframe and refresh it without affecting the rest of the page. Similarly gmail could break up its page into multiple iframes (It actually does this in reality, but it also does a lot more than this).

With iframes the "synchronous" problem was somewhat solved. Multiple requests could now be made in parallel from a single web page. However the method of making the request largely remained the same, using links or forms. It was either not seamless or almost impossible to make requests in any other manner, like say whenever user typed something (like Google search). Now let's see what all capabilities we have on top of the basic web pages.

1. Web pages can respond to user actions. Some code can be executed on various user actions made on different parts of the page.
2. This code can modify the web page using DOM.
3. Browsers are now capable of making multiple requests in parallel without blocking the web page.

The browser folks now combined these abilities thereby allowing web developers to make requests to servers in response to user actions using JavaScript. These requests could be made in parallel without blocking the page i.e. Asynchronously. Now when the response came back the same JavaScript code could add the HTML received in the response to the existing page using DOM.

The developers went a step ahead and said "why receive HTML every time? In most cases the formatting or presentation information is already there in the web page. What is needed is only new data or content (For example, in the sports website example, all that is needed is the updated score. Information on how to show that score is already present in the web page). So instead of fetching HTML every time let's just fetch data to make things much faster and efficient". And back then the most popular way of representing data in a structured manner was XML. So developers started receiving data in XML format from the server and using that to update the web page. XML was so popular that browsers added support for automatically parsing XML data received in response to these requests.

This approach of making requests and updating parts of web pages became very popular and was much widely used. Since this was predominantly used to make requests over HTTP (The protocol of the web) to fetch XML data this technology was called "XMLHttpRequest" (Internet explorer called it something different, it probably still does).

What all does this XHR offer?

1. Making requests "Asynchronously"
2. Making requests from "JavaScript"
3. Making requests for "XML"

Put together it forms "AJAX - Asynchronous JavaScript and XML".

Notes :

1. Several things have been simplified here as this is aimed at people new to web development and Ajax in particular. This should nevertheless give a good starting point.
2. References to XML should be read as " something other than HTML". Nowadays XML has been largely replaced by JSON because of several advantages.
3. XMLHttpRequest does not necessarily mean making asynchronous request. It supports making synchronous requests also. But I don't know if there is a good reason to do it. Making a sync xhr will block the entire page until that request is complete. Really don't do that.

MongoDB concurrency - Global write lock and yielding locks

There has been lot of hue and cry about MongoDB's global write lock. Quite a few people have said (in blog posts, mailing lists etc) that this design ties down MongoDB to a great extent in terms of performance. I too was surprised (actually shocked) when I first read that the whole DB is locked whenever a write happens - i.e a create or update. I can't even read a different document during this time. It did not make any sense to me initially. Previous to this revelation I was very pleased to see MongoDB not having transactions and always thought about that feature as a tool which avoided locking the DB when running expensive transactions. However this global lock sent me wondering whether MongoDB is worth using at all.. !! I was under the assumption that the art of "record level locking" had been mastered by the database developers. This made MongoDB look like a tool of stone age.

Well I was wrong. Turns out that "Record level locking" is not that easy (and the reasons for that warrant a different post altogether) and from what I understand MongoDB has no plans of implementing such a thing in the near future. However this doesn't mean the DB will be tied up for long durations (long on some scale) for every write operation. The reason is that MongoDB is designed and implemented in ways different than other databases and there are mechanisms in place to avoid delays to a large extent. Here are a couple of things to keep in mind :

MongoDB uses memory mapped files to access it's DB files. So a considerable chunk of your data resides in the RAM and hence results in fast access - fast read all the time and very fast write without journaling and pretty fast write with journaling. This means that for several regular operations MongoDB will not hit the disk before sending the response at all - including write operations. So the global lock that is applied exists only for the duration of time needed to update the record in the RAM. This is orders of magnitude faster than writing to the disk. So the DB is locked for a very tiny amount of time. This global lock is after all not as bad as it sounds at first.

But then the entire database cannot be in RAM. Only a part of it (often referred to as working set) is in RAM. When a record not present in RAM is requested/updated MongoDB hits the disk. Oh no, wait.. so does that mean the DB is locked while Mongo tries read/write that (slow) disk? Definitely not. This is where the "yield" feature comes in. Since 2.0 MongoDB will yield the lock if it is hitting the disk. This means that once Mongo realizes it is going for the disk, it sort of temporarily releases the lock until the data from disk is loaded and available in RAM.

Although I still prefer record level locking in MongoDB, these two above mentioned features are sufficient to reinstate my respect and love for MongoDB. :)

Where does node.js figure in the typical web application stack?

I have been reading up about node.js for a little while now. Now having Javascript as one of my favorite languages I obviously love node.js. It's simply awesome. But it was not until very recently that I started to wonder where would this node.js fit in the typical three-tier web architecture, fondly know as "the stack". The three elements of the stack being Web server, app server and DB (DB is definitely out of question). Within the app server we typically have a language interpreter and a framework (like rails).

I initially thought of node.js as a JS interpreter (similar to the ruby interpreter) and can be used with FCGI. But then again node.js in turn uses the V8 JS engine. So it's not a language interpreter per say. Then I saw several node.js examples showing how easily web based applications can be created. So I started comparing it with other frameworks like Rails or Erlyweb. But no, its not that either. Sure there is a simple HTTP module in node.js but it's in no way anywhere close to these frameworks. So is it a web server then? Definitely not that either, considering the rich feature list of existing web servers like Apache or Nginx. So what is this node.js then?

From what I have understood till now, node.js is just a JS library (not like jQuery or Prototype which are meant to run in the browser context). node.js is more like a ECMA-Script library. If we treat JS as a generic programming language, I believe, we will see quite a few shortcomings, the most significant one being the lack of system i/o facility. I guess ECMAScript was designed to be run in a host environment and hence features like console i/o or file i/o or network i/o were not added. This makes it very hard for JS to be used outside the host environment. This is exactly what node.js provides.

node.js extends the ECMA script and provides these missing aspects which enable JS to be used on the server side for network programming. node.js provides file i/o, socket i/o, process handling, a mechanism for creating modules and specifying dependencies, several network oriented modules and so on. (The complete list is here).

So node.js is really a library, which adds capabilities, although these features are somewhat at a more basic level than the ones provided other libraries. For instance, look at the libxml library in C. The C language compilers come with their own standard library which provides mechanisms to do file i/o. But there is no out of the box provision to deal with XML files/documents. This capability is provided by the libxml library. So the libxml library allows programmers to do something more with C than what is provided by default. There are innumerable number of such libraries which add various types of capabilities.

In somewhat analogous way, node.js is also a library which adds a lot of new capabilities to the Javascript language, although, as stated earlier, these capabilities are much more basic in nature and in most cases are present in other languages as part of their standard offering.

• So node.js is not a new language and hence do not compare it with other languages like Ruby, Python, etc. Javascript is the language here.

• node.js is not a new web application framework. So do not compare it with Rails, Django, Sinatra etc. albeit, note that node.js was apparently developed as a means to write high performance client server programs. Consequently smart folks out there started working on web application frameworks based on node and there are a couple. I know about "Express" which AFAIK, is based on Sinatra and is gaining popularity. Now that is an item comparable to Rails and friends. Questions like will node.js replace rails are, technically speaking, absurd.

• node.js is not a server. Absolutely not. There are node.js based servers, just like Nginx and Apache are C based servers.

What is Cloud? -- Simple terms please

Cloud has been making a lot of noise and almost every tech (or tech related) person knows about it or at least heard of it. Now for those who have just heard about it but do not know what it means here is a quick definition from Dave Neilsen, the founder of Cloud-Camp. He says, "For something to be called cloud, it should have these properties :

• Hosted by someone else
• On-demand. Do not have to wait or call somebody to get it.
  
• Metered somehow. So you know exactly how much you are using and how much you are paying.
  
• Scalable, both ways - up and down as and when you require."
  

He goes on to say that Cloud could mean different things for different people. Here area few examples stating what cloud is for a particular person :

For an IT guy -- Infrastructure as Service
For a Web Developer -- Platform. Just dump your code and don't worry what runs it.
For a Business guy -- SaaS (Software as a Service)

That was pretty neat. Helps me answer the standard question "What the hell is this cloud thing?" in a sane manner. Earlier I could never figure out what a proper answer should be for this question, because there was so much to tell.

Here is my attempt to elaborate on above mentioned examples.

So cloud is basically having the infrastructure to do what you do hosted by someone else and having it totally scalable. For example, in the above list, for a web developer cloud is a platform where he can dump his code and expect it to run as he has designed it. He does not worry about the machines, the network connectivity, the bandwidth. He just pays for those in a metered manner. He scales his platform whenever he wants. He can increase his bandwidth quota, move to a better machine, increase the number of machines and all of this without calling the customer care or the sales guy. He will do it by logging into the cloud services website or he would have a script do this for him automatically, i.e if he is geek enough.

Similarly for a business man, it is software as a service. E-mail service would probably be a good example. The business man does not know what software runs the email system, he does not worry about what version of email server is running, what os it is running on, what DB it is using to store the emails, what protocols it is making use of. If the email contents are not that sensitive he would not even worry about the physical location of the servers storing these emails. He just buys the email software as a service and uses it. All that he probably worries about is how many email accounts are available to him/his company and how reliable/usable they are. At any point he can increase or decrease the number of accounts, once again without making a call.

That's cloud .

Note : I got this definition from one of the IBM developerWorks podcasts which is available here.

Oh, and remember, all this time every reference to Cloud meant "Cloud Computing", not just plain "cloud"

Getting the size of an already loaded page (from cache) in a Firefox extension.

Today this question came up in the IRC (moznet, #extdev). One of the add-on developers wanted to get the size of the page, either bytes or number of characters. The most obvious thing that came to my mind was progress listeners for definitive answers or the content length from the channel for not so critical scenario. But then he said he wants it for an already loaded page. And he further said that the information is already there somewhere as it is shown by the Page Info dialog (Right Click on a web page and select View Page Info). He was indeed right. Somebody in the code is already going through the trouble of calculating the data size and we can just re-use that. And I immediately started the quest to find that out.

As usual to figure out any browser component I opened up DOM Inspector. That tool is improving, which was against my earlier perception (Sorry Shawn Wilsher), though the highlighting part is still screwed up. Nevertheless, locating that particular label "Size" and the textbox in front of it containing the value was not difficult at all. I got the "id" of the textbox containing the size value. (Its "sizetext" :) ).

Next it was MXR (http://mxr.moziila.org/) in action. I did a text search for the id and got a bunch of results, one of which was pageInfo.js with this entry : line 489 -- setItemValue("sizetext", sizeText); . It is here. The very line made it apparent that it is the place where the value is being set and hence it is the place from where I can get to know how the value is being calculated.

Once I saw the code it was very clear and straight forward and pretty simple also. We have the URL. From the URL we get the cache entry for that URL. (Every cache entry has a key and that key is the URL - so neat). We try to get the cache entry from the HTTP Session first and if that fails we try FTP Session. The cache entry has the size as an attribute on itself, so its just getting that attribute value. DONE.

I am not sure how this will behave if we have disabled every type of cache. AFAIK, there will still be some in-memory cache as long as the page is still loaded. Probably good enough.

That was the end of a small but interesting quest. :-)

Making firefox use a little lesser memory

A lot people tell me that firefox uses a lot of memory and it grows like anything when used for a long duration. From the discussions developers and some blog posts/forums it appears the prior to FF3 it was memory leaks which formed a major part of this. Apart from that one of the important features "bfcache - Back/Forward Cache" which, arguably, makes back and forward navigation very fast, uses a lot of memory. In on the pages I read apparently it can be up to 4MB per page. This cache essentially keeps the parsed HTML of the mark-up in the memory along with the Javascript state. Sometimes, or more like most of the times, this can very taxing. You might be the kind of person who does not really uses the Back-Forward navigation and in case you do it you are OK with a slight delay (which may not percievable with a good conenction). If indeed you are that kind then disabling this "bfcache" will probably make your firefox eat lesser memory.

The setting/preference that controls this behavior is : browser.sessionhistory.max_total_viewers

By default it is -1, which is no limit. Setting this to 0 (zero) will disable this cache completely. Essentially you are telling Firefox to not to store the state of any document in the memory. If you want to have this feature with a saner limit you can set it to an integer representing the number of pages you want to be saved in memory.

Like any other pref not exposed through the Options dialog this has to be edited by visiting the page about:config. Open this in a tab and key in (or copy and paste) the preference string. Double click it to edit.

May be you can now lose lesser hair as your other applications will run smoothly. :P

Hari Om.

Undo "Always display images" switch in Gmail

Gmail has this nifty and very useful feature to avoid mostly annoying images being displayed in our incoming emails. Whenever we get an email containing an embedded image, it is not displayed by default. Instead there appear two links at the top of the email saying:

"Display images below"
"Always display images from <this sender's email>"

The first one is a one time thing and the second one is a setting for that particular email id.

I once clicked on the second link accidentally and had unwanted images cluttering my email view. I looked for this thing to be turned off in the settings page, but found nothing. I had a very hard time undoing this accidental setting. Finally I discovered the solution also accidentally. The switch to undo this setting is "hidden" under the "Show Details" link for the email header.

The right end of the very first line of the email, which contains the sender's name/email id", has this "Show Details" link. This shows the email header with several details. At the end is the link saying something like this:

"Do not display images from this sender".

Just use this switch to go back to the good old times of emails without images. :)

My first Mozilla Education Status call

Mozilla is the only open source community which I have understood a little and also to which I have contributed a little. I always wanted to make my college a sort of Mozilla hub with several contributors and many feature developments happening out of my college. It actually started with an idea of a "Complete Open Source Hub", but it got reduced to just "Mozilla Hub" (either because of my laziness or because of lack of resources). Anyways I did not take any proactive steps towards that wish of mine, until recently when our college got the "Autonomous" status. That is when I realized that bringing open source software development in the course mainstream has become a lot easier as the power to form the syllabus and conduct the tests and examination rests with my college itself and not the University.

Just when I thought of doing something tangible, Mozilla came up with their "Mozilla in Education" program. This is a brilliant idea to drive open source into the student community and also give the students opportunities to work on real world applications. I was impressed by this program the very moment I read about it. I decided to present this idea to my HOD at the college and get him to start working offering Mozilla education to students in my college. I started reading more about it and today I also attended my first Mozilla Education status meeting conference call, which happens every week. I got a wealth of information.

This week we had Pascal F presenting to us about the "Design Challenge" program organized by the Mozilla Labs. He talked to us about the way in which they organized this program. It was all very inspiring. They had nearly 30-35 people from different parts of the world (literally). In this contest the students initially submitted mock-ups of their ideas, nearly 40 of them. Amongst those, 30 fully completed mock-ups were chosen for the second level, in which the students got mentoring by some the well known names in the Mozilla community. The mentoring consists of 10 Webinar sessions conducted using WebEx, over a period of 3 weeks. This is going on currently and will end at the end of this month. These mentoring sessions aim at converting those mock-ups into working prototypes. At the end, the best prototypes are given honors.

During the presentation Pascal mentioned some interesting things. Of the students from various countries, it was the students from so called "2nd World Countries" (like Romania, India, Argentina, etc.. ) who showed a lot more interest than their US counterparts. There was tremendous enthusiasm in them where as the students from US expected -- in his own words -- "being entertained" and "spoon-fed". Though this is an alarming thing when viewed from a global perspective, I was personally very happy that Indian students, my fellow country-men, have shown such dedication. Pascal also mentioned that they were so interested that they were up in the middle of the nights for the webinars. All in all, I am even more motivated to bring open source in general and mozilla in particular to my college.

I decided to take this program as an example and present it to my HOD this Saturday and try and make him accept this and similar programs as official ones and the projects done in such programs be considered for the completion of the course goals.

Thank you Pascal, thank you Mozilla. Lets hope to have a Mozilla India Center, at least an unofficial one at my college SJCE. :-)

Browsers are undergoing continuous innovation.

It was sometime since I blogged about anything. I have had several things on mind and many of them are presents as drafts. But this one really caught my attention and I felt I should put in my thoughts about this.

Until a couple of years, people rarely looked beyond Internet Explorer for their browsing experience, though they kept cursing it a lot. After that came the Mozilla Firefox web browser with its pack of addons allowing users to actually customize for their needs. They could actually make the browser do what they wanted it to do and not just set some options. The browser wars had started again. At least I started reading about browsers and started following up on things happening with these browsers. Opera and Safari and a few other, actually used, browsers were not really "news" as such.
This was dying off. Firefox had created a sizable chunk of user base and was pretty stable with it. Except for few traditional enhancements like memory optimizations, bug fixes, etc.. nothing big was happening. It was a lot silent.

Then came the next wave with Google announcing the release of its browser with a nice, easy to understand "comic" book. It came with a whole new paradigm for building browsers with the "process-per-tab" concept. It was really innovative. Though discussions about this happened in other browser communities also, Google Chrome was the first one to implement it. Also it boasted of its super-fast V8 JavaScript Engine and also the browser UI, which gave more screen real estate to content than chrome. This was a real big thing and made the browser wave go much higher than what it had ever been. With the Google branding a vast majority of internet users rushed to have a sneak peak at Google Chrome or may be try it or even keep it as their regular browser. There was a lot of noise about this and people indeed listened. Mozilla and IE people were not silent and did responded very well. Mozilla came up with its ultra-fast "Tracemonkey" JS engine, implementing the trace trees and there by making it much faster than V8. IE8 also has the "process-per-tab" and "private browsing" features first presented by Chrome. But like any other sound this too dampened a little after it was created and browsers were back in the silent phase doing traditional improvements and bug fixes. At least that's how I percieved the situation.

[Edit 06-Mar-09 : Shawn Wilsher suggest that Tracemonkey had appeared before Chrome made the pubic appearance. He certainly knows these things better than me and I believe he is right. But still I wanted to keep my original post as it is and instead I put this separate edit note. :) ]

But I was clearly, totally wrong. People have realized that internet is the place to be in the future and thats where a large part of our life will be. With browser being the main and central interface for people to use that internet, it makes a real good sense to make this browser as robust and reliable as possible. New things keep coming on the internet, both and bad, the latter being more often, and the browser has to keep up with all of it. What we thought yesterday as being a good design apporach might just look senseless tomorrow. There just seems no end to it and researchers appear all ready for it. I am coming across so many innovations happening in the browser domain. This article : Researchers Say Gazelle Browser Offers Better Security -- Campus Technology -- gives us an idea about how much effort the scientists are putting in making our internet lives better. The article is about Gazelle, but it also mentions about another experimental browser OP.

This browser Gazelle, uses the Trident rendering engine (used by IE) but builds upon a OS based process architecture where websites form the processes and they communicate by passing messages like IPC (Inter-process communication).

I am not yet sure how this will all work out. Probably the websites need to built with some intelligence so that it can do the so called IPC when it actually has to. But the evoltion in browsers is for sure. They will not be same as they are now. Several things are going on along the UI front. When these things mature and come togther you can have the scenes of your favourite sci-fi movie scenes right in front of you everyday.

Lets just hope these come soon enough for people like me to enjoy, not when I am all grey hair and toothless.

Hari Om.

Image for a statusbar panel in a firefox extension

I was working on a firefox extension which, like many other extensions, was adding a small button to the status bar. Doing this is very clearly explained here in : this MDC tutorial

I followed the instructions properly but to my surprise I did not see the image on the status bar, but just a square box in the defualt chrome color. I tried forums/web search and the IRC too, but no help. Then I read the statusbarpanel reference page - statusbarpanel - MDC -- completely and it appears that for the image to show up the statusbarpanel has to be of a specific class called : "statusbarpanel-iconic". I just added that and it worked. :-)

Here is my code

<statusbar id="status-bar">
     <statusbarpanel id="graph_status_bar_panel"
              image="chrome://myext/skin/chart_status_bar.png"
              class="statusbarpanel-iconic" onclick="toggleGraphPanel(event)"
              tooltiptext="View Graph Panel" />
</statusbar>

Pictorial representation of blog

I came across this post by Shawn (sdwilsh) and instantly wanted to know what I am concentrating on in my blogs. And here is what my blog has to say:

Here is the link where you can get one for your blog : http://www.wordle.net/

Happy picturing. :-)

Documentation appears by itself -- from the code.

Beautifully Documented Code at Toolness

I am slightly into web and web related development and as part of that I come across various JavaScript (JS) libraries which  claim to make my life a lot easier and many of them actually do that. But several times there is a big hurdle that I have to cross before actually things become easy. Its understanding how to use that library. This sometimes turns out to be a pain, often bigger than the one if I had done everything by myself without any external library.

This is probably all going to change. The lack of documentation might just vanish and all the necessary documentation might start appearing from the code itself. And the steps for building this, collating things and publishing on website are taken care of. You just need to annotate your code with the right tags and the JavaScript linked at the top (Written by Atul) will do the documentation generation and will also display it alongside code.

This is particularly useful for libraries providing various APIs as the users can see a function in the raw code and understand its meaning and purpose and the arguments it expects and its return value. All in once place at one time.

This is really cool. Go check out the tool and give the world a better library to use :-)

Happy (auto)Documenting

The ISP Cat and Mouse game and CDNs finally benefiting out of it.

Politics and policies are everywhere. They say policies are made to govern us and I say many of those are there for inertia - resistance to change. The big and powerful want to be so, always and do not want others to get there. This is well known and very much a cliche. But what has this got to do with ISPs in specific? Here we go:

ISPs are the people who sort of own the "Internet network" physically. It is they who actually connect the various computers by physical cables. And yes, that is why we pay them. For getting us connected to the rest of the world. Now there is no single ISP who has his cables connected to all the computers in the world. In fact there is no one who can even boast about a majority stake in the market. So obviously when the data travels through the internet, it goes through the infrastructure laid and maintained by different ISPs. A simple example will illustrate this :

Let's say a user is connected to the internet via Bharti Airtel connection and he is trying to access, say Indian Railways website, which is hosted on a machine connected to internet, for the sake of illustration, via BSNL connection. So the path of the request from client to the server would involve, both Airtel network path and BSNL network path. The client sends the request to the Airtel server. The Airtel network will route the data in its own network to the extent possible. At one point it needs to request BSNL network to take up the data and then give it to the destination server machine. Now at this cross-over point, Airtel is requesting some service from BSNL. Essentially Airtel is making use of the BSNL's network infrastructure to carry its data. Now there is no point in BSNL giving this for free. So obviously it charges Airtel some amout of money. Airtel does not mind paying it as it mostly gets translated to user charges. This is not really the issue. Problem would arise when BSNL will refuse to take the request and Airtel will have find some other alternate path, which generally ends up to be very very long. Consider this:

Clinet -> Last Airtel machine (router) -- m number of hops
Last Airtel machine -> Destination machine in BSNL network (Direct path) -- 4 hops.
Last Airtel machine -> Destination machine in BSNL network (Indirect path via some other ISP or via some other cross-over point) -- 20 hops

So totally the data has to do m+4 hops if BSNL takes up the request from last Airtel machine. At a time when BSNL is experiencing some heavy traffic in the region where the Airtel-BSNL crossover is happening, it would not be willing to accept more data, that too from a different ISP. So they follow two techniques here:

1. Simply drop the data packets, which will result in bad experience for the end user.
2. As routing happens based on least number of hops, the first BSNL server at the crossover point, will tell the last Airtel machine that the number of hops to the destination machine is actually 25 hops even though it is totally wrong. As a result the last Airtel machine will instead choose the indirect path with 20 hops. This will obviously slows down the internet and again result in bad experience for the end user.

Now you see how policies and profits affect technology. This is, as stated by an electronics professor at my college SJCE, TECHNO-POLITICS.

The solution for this would be to make the data available in every ISPs own network. And that is precisely what the CDN - Content Delivery Network - companies do. These companies have a huge number or servers placed in various parts of the worlds. In most cases they are placed in the data centers of these ISPs. It is symbiosis. With CDNs placing their servers in ISP's data centers, the ISP has a lot of data in its own network, even though the original website (or content owner) might be using a different ISP. This avoids a lot of requests to different ISPs and there by reduce costs significantly. In return the CDN companies get a very sweet deal for the rack space for their machines

offline cache discussion with campd

[ 2:20 am] <brahmana> hi all,
[ 2:20 am] <brahmana> Looks like my earlier question about offline cache got lost here...
[ 2:21 am] <brahmana> I read the HTML 5 spec and understood that the cache will be versioned and hence multiple versions of the same cached element will be present on the client's disk. Is that so?
[ 2:22 am] <campd> yeah
[ 2:22 am] <campd> as of 3.1
[ 2:22 am] <campd> in 3.0, there's only one version
[ 2:23 am] <brahmana> ok..
[ 2:23 am] <campd> brahmana: though they won't stick around long
[ 2:23 am] <brahmana> ok..
[ 2:23 am] <campd> brahmana: when you visit a page, it'll download a new version. Once any page using he old version is navigated away from, it is cleaned up
[ 2:25 am] <brahmana> campd, So everytime the user goes online and visits a website which has offline cache, the cache is refreshed provided no page is using the old cache.
[ 2:26 am] <campd> brahmana: sorta
[ 2:26 am] <campd> brahmana: every time they visit an offline cached website
[ 2:26 am] <campd> brahmana: it will check for a new version of the cache manifest
[ 2:26 am] <brahmana> ok..
[ 2:27 am] <campd> brahmana: if there's a new manifest, a new version of the cache will be created and fetched
[ 2:27 am] <brahmana> campd, ok.. answers my question fully..
[ 2:27 am] <campd> cool
[ 2:27 am] <brahmana> campd, However I have another question..
[ 2:27 am] <campd> ok
[ 2:28 am] <brahmana> Now is this cache application specific? As in if a image with the same src is referenced by two websites, the image will be cached separately for each webapp?
[ 2:28 am] <campd> yes.
[ 2:29 am] <brahmana> ok..
[ 2:30 am] <brahmana> campd, Will this offline cache in anyway affect the regular browsing when the user is online?
[ 2:30 am] <campd> if they're online, browsing an offline app, it will be loaded from the cache first
[ 2:30 am] <campd> it won't affect online browsing of non-offline pages
[ 2:31 am] <brahmana> ok..
[ 2:31 am] <campd> so if http://www.foo.com/offline.html is an offline app that references http://www.bar.com/another.html
[ 2:31 am] <campd> going to http://www.bar.com/another.html will NOT load it from the offline cache
[ 2:31 am] <campd> but going to http://www.foo.com/offline.html WILL be loaded from the offline cache
[ 2:32 am] <brahmana> okay..
[ 2:33 am] <brahmana> campd, Regarding the local storage, Can it be looked at as an extended form of what currently is cookie?
[ 2:34 am] <campd> kinda, yeah
[ 2:34 am] <brahmana> Is there any limit on the amount of data that each web-app gets on this local storage?
[ 2:35 am] <campd> yep
[ 2:35 am] <brahmana> Because the spec says that the web-app can use this to store user created _documents_
[ 2:35 am] <campd> 5 megs for the etld+1
[ 2:35 am] <campd> if the domain has the offline-app permission it gets more, but I forget the exact number
[ 2:35 am] <mconnor> campd: is that right? I thought I remembered some wacky combination thing
[ 2:36 am] <brahmana> oh.. ok.. thats pretty big space..
[ 2:36 am] <campd> (which I assume is the wacky combination mconnor's referring to ;))
[ 2:36 am] <mconnor> no
[ 2:36 am] <mconnor> it was something like "foo.bar.com can have 3 MB, and bar.com can have 2 MB" or something
[ 2:36 am] <mconnor> in whatever combination
[ 2:37 am] <mconnor> maybe that was the spec that got deprecated?
[ 2:37 am] <campd> I think right now it's just "5 for the whole etld"
[ 2:37 am] <campd> err, etld+1

Discussion with biesi, bz and gavin about channel and tabId in #developers

brahmana wonders if biesi was able to examine the UML diagrams

[ 8:24 pm] <biesi> brahmana, sorry not yet
[ 8:24 pm] <brahmana> thought so..
[ 8:24 pm] <brahmana> biesi, anyways another quick question.
[ 8:25 pm] <biesi> brahmana, yes?
[ 8:26 pm] <brahmana> biesi, Can you please have a look at this one: http://wiki.mozilla.org/images/a/a0/Brahmana_URI_Loading_DocShell_Code.jpg -- Which would be the ideal point in the sequence to associate the tabId to the channel created.
[ 8:26 pm] <brahmana> ?

[ 8:26 pm] <brahmana> biesi, I want to know where I will have access to both of them.
[ 8:27 pm] <timeless> brahmana: pretty

[ 8:27 pm] <biesi> brahmana, um

[ 8:28 pm] <biesi> brahmana, nothing in that diagram has access to both :-)
[ 8:28 pm] <brahmana> timeless, thank you.. more here: http://wiki.mozilla.org/User:Brahmana/Netwerk_Docs (just in case)

[ 8:29 pm] <brahmana> biesi, ok.. so how much prior to webBrowser will I have to go to get a tabID ?
[ 8:29 pm] <biesi> brahmana, tabbrowser.xml

[ 8:29 pm] <biesi> its loadURI function or something like that

[ 8:29 pm] <biesi> brahmana, of course some things call loadURI on the web navigation directly...

[ 8:29 pm] <brahmana> biesi, yeah.. thats how I created that sequence diagram..


[ 8:30 pm] <brahmana> biesi, Can't I get hold of the tabID in C++ ?
[ 8:30 pm] <biesi> C++ has no concept of "tab"
[ 8:31 pm] <brahmana> But the <browser> present in each tab corresponds to one nsIWebBrowser, isn't it?
[ 8:31 pm] <brahmana> <browser> == the xul browser element

[ 8:35 pm] <biesi> brahmana, no
[ 8:35 pm] <biesi> there is no nsIWebBrowser in firefox
[ 8:36 pm] <biesi> that's only used for embedding
[ 8:36 pm] <timeless> unfortunately we don't use the same apis everywhere :(
[ 8:37 pm] <brahmana> oh man..
[ 8:37 pm] <biesi> brahmana, there is one docshell per tab
[ 8:37 pm] <brahmana> then the starting of my sequence diagram is wrong..
[ 8:37 pm] <biesi> if that helps you
[ 8:37 pm] <brahmana> yeah.. I am aware of that..
[ 8:38 pm] <brahmana> and I thought it was nsWebBrowser that held a reference to a docShell and made calls on the docShell
[ 8:38 pm] <biesi> ah, no
[ 8:38 pm] <brahmana> But as it appears firefox does not use nsWebBrowser itself...
[ 8:38 pm] <biesi> the browser holds the docshell directly, I believe
[ 8:39 pm] <brahmana> you mean the xul browser ?
[ 8:39 pm] <biesi> yeah
[ 8:39 pm] <brahmana> What C++ object does that map to?
[ 8:39 pm] <brahmana> something under widgets?
[ 8:39 pm] <biesi> the xul browser?
[ 8:39 pm] <brahmana> yeah
[ 8:40 pm] <biesi> um
[ 8:40 pm] <biesi> some xul magic
[ 8:40 pm] <biesi> nsXULElement.cpp perhaps
[ 8:40 pm] <brahmana> oh.. let me see
[ 8:40 pm] <biesi> via the boxObject maybe?
[ 8:41 pm] <biesi> but note that the <browser> is mostly an XBL thingy
[ 8:41 pm] <brahmana> oh man.. this is getting heavily complex..
[ 8:41 pm] <timeless> it really is

[ 8:43 pm] <brahmana> Now the JS call: browser.loadURI() will be a call on the corresponding nsXULElement object, which actually holds a reference to the docShell. Is that right?
[ 8:45 pm] * brahmana requests to put aside the XPCOM stuff that happens in the above sequence..
[ 8:45 pm] <brahmana> sorry, the XPConnect stuff..
[ 8:46 pm] <gavin|> yes
[ 8:46 pm] <gavin|> though the nsXULElement isn't really involved
[ 8:46 pm] <gavin|> apart from being associated with the JS object that implements the XBL methods
[ 8:48 pm] <brahmana> gavin|, Can you please elaborate a little on your last statement..
[ 8:48 pm] <brahmana> ?
[ 8:49 pm] <brahmana> Or is there a doc that I can read up to orient myself a little before asking lots of questions here?
[ 8:49 pm] <gavin|> nsXULElement itself doesn't have anything to do with the XBL implemented methods
[ 8:50 pm] <gavin|> it's just a "container"
[ 8:51 pm] <gavin|> it's not really useful to say that you're interacting with a nsXULElement, because you're really interacting with an XBL bound node
[ 8:51 pm] <gavin|> and the XBL <browser> methods implemented in JS are what matters
[ 8:51 pm] <gavin|> not the nsXULElement class methods
[ 8:52 pm] <brahmana> oh.. ok. so the browser.loadURI() is (most probably) implemented in the JS itself. This JS object holds a reference to the docShell directly and thats how the calls are routed -- makes sense?
[ 8:53 pm] <gavin|> that's about right
[ 8:55 pm] <brahmana> now this XBL/JS implementation is present in tabbrowser.xml?
[ 8:56 pm] <gavin|> and browser.xml, yeah
[ 8:56 pm] <gavin|> the tabbrowser contains <browser>s

[ 8:57 pm] <brahmana> This is the one I should be looking at: http://mxr.mozilla.org/mozilla-central/source/toolkit/content/widgets/browser.xml , right?

[ 8:58 pm] <gavin|> yes


[ 9:04 pm] <brahmana> gavin|, ok.. i figured the exit point to docShell: http://mxr.mozilla.org/mozilla-central/source/toolkit/content/widgets/browser.xml#186 --
[ 9:04 pm] <brahmana> Now I want to get the tabId in which this browser object is, how would I achieve it?

[ 9:04 pm] <gavin|> brahmana: not sure what you mean by "tabId"
[ 9:05 pm] <brahmana> tabIndex, the index of the tab in the tabContainer
[ 9:05 pm] <gavin|> you have a reference to a <browser>, and want to find which tab it's in?
[ 9:06 pm] <brahmana> yes..
[ 9:06 pm] <brahmana> well actually I am inside the browser's definition itself..
[ 9:06 pm] <gavin|> I guess you you need to loop through tabs and compare against their .linkedBrowser
[ 9:06 pm] <gavin|> don't think there's a utility method to do that
[ 9:07 pm] <brahmana> ok.. let me see how I can accomplish that..
[ 9:08 pm] <brahmana> I am thinking this... for(i=0; i < this.parentNode.browsers.length; ++i) if(this.parentNode.browsers[i] == this) return i
[ 9:09 pm] <brahmana> that must work, isn't it?
[ 9:09 pm] <gavin|> probably
[ 9:09 pm] <gavin|> assuming this.parentNode is the tabbrowser
[ 9:09 pm] <brahmana> yeah.. I verified that..
[ 9:09 pm] <gavin|> though the "browsers" getter builds an array by looping through taqbs
[ 9:09 pm] <gavin|> so it would best be avoided
[ 9:09 pm] <gavin|> to avoid having to loop twice
[ 9:10 pm] <brahmana> oh.. instead we directly loop through the tabs..
[ 9:10 pm] <gavin|> yeah
[ 9:10 pm] <gavin|> http://mxr.mozilla.org/seamonkey/source/browser/base/content/tabbrowser.xml#1693

[ 9:11 pm] <brahmana> And if i change the XBL now, is there anything that I need to do during the build?
[ 9:11 pm] <gavin|> you just need to rebuild browser/
[ 9:11 pm] <gavin|> (or toolkit/ if you're touching browser.xml
[ 9:12 pm] <gavin|> why are you changing them, though?
[ 9:12 pm] <brahmana> ok.. thats great.. i can even go for a full build.. :-)

[ 9:12 pm] <brahmana> I want to associate every channel with the tab it is working for..


[ 9:14 pm] <gavin|> brahmana: is therea bug # for this?
[ 9:15 pm] <brahmana> gavin|, oh no.. there isn't.. I saw similar thing in Firebug, Cookie Manager and wanted to explore on that..
[ 9:15 pm] <brahmana> if by any means this is a desirable thing we can have a bug...

[ 9:24 pm] <brahmana> Is the code under xpfe/ still used?
[ 9:25 pm] <gavin|> some of it is
[ 9:25 pm] <brahmana> the window mediator?
[ 9:25 pm] <gavin|> yes

[ 9:26 pm] <brahmana> Along with the tabIndex I would also require some sort of window Id i guess, as tabIndices are not unique across browser windows, isn't it?
[ 9:26 pm] <gavin|> right
[ 9:27 pm] <bz> tabindices need not be unique within a single window either
[ 9:28 pm] <gavin|> er, why wouldn't they be?
[ 9:28 pm] <brahmana> oh.. well if we do not open and close stuff, they should be right?
[ 9:28 pm] <bz> because they're under the control of the page author?
[ 9:28 pm] <bz> And nothing prevents an HTML author from sticking tabindex="2" on every single node in the document
[ 9:28 pm] <gavin|> we're talking browser tabs
[ 9:28 pm] <bz> oh
[ 9:29 pm] <bz> nevermind, then


[ 9:31 pm] <brahmana> gavin|, And about the bug for the stuff I am asking, I assume this isn't really a desired feature, is it?

[ 9:31 pm] <gavin|> brahmana: I still don't really know what the feature is

[ 9:35 pm] <brahmana> gavin|, To observe requests for one tab in observerservice ... there is requirement for coupling tab index with the http channel....

[ 9:35 pm] <brahmana> And that is what I am trying to accomplish, associate the tabIndex with the channel..

[ 9:37 pm] <bz> brahmana: er.... you know the docshell involved in both places, right?


[ 9:39 pm] <brahmana> bz, yeah.. I was talking to gavin and others about the way browser interacts with docShell. But I did not fully understand your question.
[ 9:39 pm] <brahmana> browser as in the xul browser element.

[ 9:40 pm] <bz> brahmana: docshell is he guts of a browser
[ 9:40 pm] <bz> brahmana: the part that actually holds the web page, etc

[ 9:41 pm] <brahmana> bz, yeah.. that was evident from this one: http://www.mozilla.org/projects/embedding/docshell.html and also from the length of nsDocShell.cpp file..

[ 9:43 pm] <bz> ok
[ 9:43 pm] <bz> so you can get from a tab to a docshell
[ 9:43 pm] <bz> you can get from the channel to a docshell (usually)
[ 9:43 pm] <bz> then compare the two
[ 9:43 pm] <bz> for images you're out of luck
[ 9:45 pm] <biesi> brahmana, there can be a <browser> that's not part of a <tabbrowser>
[ 9:46 pm] <brahmana> bz, Whats special about the images?
[ 9:46 pm] <biesi> what's NOT special about images
[ 9:46 pm] <bz> brahmana: they don't so much follow necko rules
[ 9:46 pm] <brahmana> biesi, Are you referring to a situation with single tab?

[ 9:47 pm] <brahmana> bz, biesi yeah I had got the same statement when discussing about the request end notifications..
[ 9:47 pm] <biesi> brahmana, no, I'm referring to extensions or mailnews or whatever that doesn't support tabs
[ 9:47 pm] <brahmana> biesi, oh well.. that is probably not a problem. I don't think we would go beyond firefox.
[ 9:48 pm] <biesi> ok
[ 9:48 pm] <biesi> I have no idea what you're trying to do
[ 9:48 pm] * bz points to "extensions"
[ 9:48 pm] <bz> anyway
[ 9:48 pm] <biesi> I'm just saying, if you want to change browser.xml in mozilla.org
[ 9:48 pm] <biesi> 's repository, you can't assume that there's a tabbrowser
[ 9:49 pm] <brahmana> point fully accepted.. :-)
[ 9:50 pm] <brahmana> bz, I am not sure about getting to a docShell from a channel. Is there is straight forward way?
[ 9:51 pm] <brahmana> s/there is/there a

[ 9:54 pm] <brahmana> moreover the docShell will have reference only to one channel and I assume that is channel corresponding to the base document, i.e the main document request. Is that so?
[ 9:55 pm] <bz> brahmana: the docshell is the channel's loadgroup's notification callbacks
[ 9:55 pm] <bz> brahmana: generally
[ 9:55 pm] <bz> brahmana: we're talking about a channel having a reference to the docshell, not the other way around

[ 9:57 pm] <biesi> the load group has the channels/requests for all the loads
[ 9:57 pm] <biesi> except for iframes of course


[10:00 pm] <brahmana> biesi, bz.. a little out of the current discussion.. When the http-on-modify request is fired, will the connection be already set up? Is there a possibility to change the request URL in that event's listener?
[10:01 pm] <bz> "change" in what sense?
[10:01 pm] <biesi> the connection is not set up yet
[10:01 pm] <biesi> but you can't change the URL
[10:01 pm] * bz really wishes URIs were immutable and that channels' URI were readonly so people wouldn't ask questions like this
[10:01 pm] <biesi> the channel's URI IS readonly
[10:01 pm] <brahmana> ok.. :-)
[10:01 pm] <bz> ah, good

[10:01 pm] <bz> well, that URIs were immutable, then
[10:02 pm] <bz> I thought we were gonna do that for http sometime
[10:02 pm] <bz> flip the bit
[10:02 pm] * brahmana decides not to ask such questions to be compliant with bz's wishes..
[10:02 pm] <bz> (and see what breaks)
[10:02 pm] <biesi> oh we aren't? that sucks :/
[10:02 pm] <bz> well
[10:02 pm] <bz> we're not _yet_
[10:02 pm] <bz> I also wish URIs were immutable by default instead of the nsIMutable mess..
[10:02 pm] <bz> but then again, I could use a pony too


[10:02 pm] <bz> Or better yet, a kayak
[10:02 pm] <bz> it is?
[10:03 pm] * bz looks in his "to check in" folder
[10:03 pm] <biesi> there's so much stuff that I'd like to change if I could...

A web-wizard to create XPCOM components in JS

JavaScript XPCOM Component Wizard

As I might have mentioned in my earlier posts, creating XPCOM components can sometimes be hell. (So wait for js-ctypes to get on-board). Mozilla people know about this pain and hence have published sample code snippets to help you in figuring out what all is necessary and what all is optional. Even the process of creating your component from these snippets can be cumbersome sometimes. So one mozillian, ted, went a step ahead and created a web based wizard to create a component in JS. You just enter a few parameters and a skeletal component is ready for use. It simply saves a lot of time.
Just visit the link at the top and try your hands at it.

Happy Componenting. ;-)

--Brahmana.

Optimized resource utilization by Firefox

I have now started working very closely to firefox and in the course I have discovered a few things. Firefox does some real good job in optimizing using resources, mainly the network resources. I specifically mean the connections established to remote servers and the DNS resolution. Of course the second one is done by caching where as the first one is done by connection sharing (at the network layer).

For example, There are two tabs opened in mozilla. In the first one some page is loading, say http://sribrahmana.blogspot.com and when it is loading the DNS resolution is done and a TCP/IP connection is already established and over that application layer connections are made. Now if you try load to load the same url in the second tab also then a lot of redundant work is avoided. The DNS resolution result which is already available is used instead of resolving the name again. This is possible because of the DNS cache. (There are, of course, ways to disable this cache, in which case redundancy will be there). And also since there is already a network layer connection established for the first tab, and there is every possibility that it is still alive, the same connection is used for the second tab also. This results in lesser network resource utilization and also in one less memory consumption as the memory taken by the new sockets is avoided.

This is some cool thing, but not what I precisely want. I infact do not want it to share connections. I do not yet know how to tell FF to do that. I will put it here once I figure out how.

And Just as an FYI, here is how you can override cache: MDC doc

Firefox extensions can behave like stand alone applications.

XUL Solutions: Creating an uninstall script for an extension
XUL Solutions: Creating a post-install script for an extension

This is one interesting thing that I found today. One of the most common things done when an application is installed is to verify that the installation has been proper and things are where they are. Also the very basic requirement for any "clean" application is to leave no traces of its presence when it is uninstalled. This is easily possible in case of stand alone applications which are either straight forward executables or else have standard installers and uninstallers. They can set registry values, hook up with Program files (or package managers in Linux) and finally do the cleaning when time comes. But things are not as simple with Firefox extensions, because the platform, i.e Firefox, is not as extensive as a full fledged operating system. So I was always under the impression that these extensions are installed and run in the controlled environment constantly monitored by FF. But as it appears, I was wrong. Firefox indeed provides install and uninstall events which you can hook on to and do the necessary cleaning up.

The technical details are in that blog posts. Read it up and have some clean extensions. :-)

Regards,
Brahmana.

Web protocol handler in FF3

This is one awesome thing to happen. Haven't you crunched your face in disappointment and frustration whenever you clicked a "mailto" link on a webpage and unnecessarily have your desktop mail application (mostly unused and not configured one) popping up, hogging quite a bit of resources? The worst case scenario would be, the above described things happening even when you are logged into one of your web based mail services and thats where you wanted to send an email from.

Now all that go away. You Gmail might just give you a new mail compose page when you click on a mailto link. And this is not just for the "mailto" link. There can be web based protocol handlers to any sort of protocol. So all in all FF3 will rock like anything along with making web applications rock harder. So folks be ready to dance for the beat, for the beat will be pretty high paced. :-)

More details here: mfinkle's blog.

JS-CTYPES Status

Mark Finkles Weblog » JS-CTYPES Status

This was what mfinkle recently talked about js-ctypes. As he said because of the hectic FF3 schedule the devs at mozilla were pretty busy and hence nothing could be done. I being a lazy bot, also did not do anything. In the blog post above, Mark talks about the struct support (which I told that I would be doing). Now I exhibited my initial enthusiasm by writing up a doc, which was not really useful (its here ). After that nothing much happened except for a small bug fix.

Now that mfinkle has rekindled the interest I will also try and contribute. The struct support is still pending and I will start getting XPCOM code exchanging struct information with libffi. After that we can look at the other interface of JS exchanging structs with XPCOM to finally complete the loop. In the mean time I am also looking at embedding mozilla. Will put up a post about that very soon.


Again, Get ready to use deadly binaries in your JS code. JS-ctypes is soon going to be here.. ;-)