offline cache discussion with campd

[ 2:20 am] <brahmana> hi all,
[ 2:20 am] <brahmana> Looks like my earlier question about offline cache got lost here...
[ 2:21 am] <brahmana> I read the HTML 5 spec and understood that the cache will be versioned and hence multiple versions of the same cached element will be present on the client's disk. Is that so?
[ 2:22 am] <campd> yeah
[ 2:22 am] <campd> as of 3.1
[ 2:22 am] <campd> in 3.0, there's only one version
[ 2:23 am] <brahmana> ok..
[ 2:23 am] <campd> brahmana: though they won't stick around long
[ 2:23 am] <brahmana> ok..
[ 2:23 am] <campd> brahmana: when you visit a page, it'll download a new version. Once any page using he old version is navigated away from, it is cleaned up
[ 2:25 am] <brahmana> campd, So everytime the user goes online and visits a website which has offline cache, the cache is refreshed provided no page is using the old cache.
[ 2:26 am] <campd> brahmana: sorta
[ 2:26 am] <campd> brahmana: every time they visit an offline cached website
[ 2:26 am] <campd> brahmana: it will check for a new version of the cache manifest
[ 2:26 am] <brahmana> ok..
[ 2:27 am] <campd> brahmana: if there's a new manifest, a new version of the cache will be created and fetched
[ 2:27 am] <brahmana> campd, ok.. answers my question fully..
[ 2:27 am] <campd> cool
[ 2:27 am] <brahmana> campd, However I have another question..
[ 2:27 am] <campd> ok
[ 2:28 am] <brahmana> Now is this cache application specific? As in if a image with the same src is referenced by two websites, the image will be cached separately for each webapp?
[ 2:28 am] <campd> yes.
[ 2:29 am] <brahmana> ok..
[ 2:30 am] <brahmana> campd, Will this offline cache in anyway affect the regular browsing when the user is online?
[ 2:30 am] <campd> if they're online, browsing an offline app, it will be loaded from the cache first
[ 2:30 am] <campd> it won't affect online browsing of non-offline pages
[ 2:31 am] <brahmana> ok..
[ 2:31 am] <campd> so if http://www.foo.com/offline.html is an offline app that references http://www.bar.com/another.html
[ 2:31 am] <campd> going to http://www.bar.com/another.html will NOT load it from the offline cache
[ 2:31 am] <campd> but going to http://www.foo.com/offline.html WILL be loaded from the offline cache
[ 2:32 am] <brahmana> okay..
[ 2:33 am] <brahmana> campd, Regarding the local storage, Can it be looked at as an extended form of what currently is cookie?
[ 2:34 am] <campd> kinda, yeah
[ 2:34 am] <brahmana> Is there any limit on the amount of data that each web-app gets on this local storage?
[ 2:35 am] <campd> yep
[ 2:35 am] <brahmana> Because the spec says that the web-app can use this to store user created _documents_
[ 2:35 am] <campd> 5 megs for the etld+1
[ 2:35 am] <campd> if the domain has the offline-app permission it gets more, but I forget the exact number
[ 2:35 am] <mconnor> campd: is that right? I thought I remembered some wacky combination thing
[ 2:36 am] <brahmana> oh.. ok.. thats pretty big space..
[ 2:36 am] <campd> (which I assume is the wacky combination mconnor's referring to ;))
[ 2:36 am] <mconnor> no
[ 2:36 am] <mconnor> it was something like "foo.bar.com can have 3 MB, and bar.com can have 2 MB" or something
[ 2:36 am] <mconnor> in whatever combination
[ 2:37 am] <mconnor> maybe that was the spec that got deprecated?
[ 2:37 am] <campd> I think right now it's just "5 for the whole etld"
[ 2:37 am] <campd> err, etld+1

Building and embedding spiderMonkey

If you already do not know, SpiderMonkey is the JavaScript engine of the Mozilla web browser. The interesting part about this JS engine is that you can use it in your application and execute JS from your application. Well I can talk a lot about the uses and the advantages of such an embeddable JS engine but thats for another post. Here I will tell you how to build SpiderMonkey and how to embed it in your application and finally how to get your application running with the JS engine.

Actually the first two steps are very well explained in these MDC pages:

http://developer.mozilla.org/En/Building_only_SpiderMonkey
http://developer.mozilla.org/en/JavaScript_C_Engine_Embedder%27s_Guide

• The first one tells you how to get SpiderMonkey and build it. It involves downloading the source and running "make -f Makefile.ref" and thats the end of the build story. You will have the engine binary, both in the form of a reusable library - dynamic and static and also an ready to use JS-execution shell. The reusable library is named libjs.so(dynamic) or libjs.a(static). The interactive JS-Shell will be an executable named js. More about using the Shell here.

• The second one tells you how to write a simple application using this reusable JS engine library. It also explains you the basic types used in the engine and the general essential terminologies. There is a boilerplate program also which you can use to test your first embedding attempt.

With all this great documentation, I was stuck at the point where I have to compile my application with the JS library. After the build I do not get any directory named "sdk" or something similar. It just gives me a build directory with a static library, a dynamic library and an executable. There are also a bunch of object files (.o files) but they are not really of much use. Only two .h files are copied to the object directory and that does not involve the main "jsapi.h" file.

As a result you will end up with a hell lot of compilation errors if you just try to build your application. So there are a couple of steps, probably very evident ones, that you need to do before you build your app.

1. Put all the .h files, header files, from the source (src) directory in the include path when building your app. The best way for this would be to create a spidermonkey folder under the includes directory or your app and providing that directory as the include path to the compiler at build time.
2. Copy the libjs.so to the lib directory of your application and pass it as a linker option (-ljs).
3. The various JS script types map to some system types, probably, based on the OS being used and hence they are present under some #ifdef. If you do not define any OS then you will not get defintions for several types and you end up with compilation errors. To avoid this manually define the OS before you include the first file from spiderMonkey (which is typically jsapi.h). Defining is in the usual way: #define XP_UNIX // -- For linux systems.

With this you should be able to build your application and run it. So you now have a program of yours for which input can be JavaScript and it will compile and run that JS and give you the output.

Happy embedding. :-)

Amazing pitfall, you will laugh

This is one of those funny videos for which I really laughed... Watch this and let me know if you did not laugh...

Specially, the lungi is the added attraction.. :D

How libffi actually works?

I came across this libffi when I thought of working on js-ctypes. Though my contribution remained very close to nil, I got to know about this fantastic new thing libffi. But I did not understand how it actually worked when I first read. I just got a high overview and assumed that the library abstracts out several things that I need not worry about when making calls across different programming languages. I just followed the usage instructions and started looking at the js-ctypes code. That was sufficient to understand the js-ctypes code. But today when I was once again going through the README, I actually came across one line that made things a little more clearer. The first paragraph in "What is libffi?" tells it all. Its the calling conventions that this library exploits. I had written a post about calling conventions some time back. As mentioned in the readme file, its the calling convention whcih is the guiding light for the compiler. So when generating the binary code, thecompiler will assume will that the arguments that are being passed to that function will be available in some place and also it knows about a place where the return value will be kept, so that the code from the calling function will know where to look for it.

Now we know that the ultimate binary code that we get after compilation is the machine code. So it is basically a set of machine supported instructions. These instructions are certainly not as sophisticated as those available in the C - like the functions. So what do these functions transalte to? Nothing but a jump of IP(instruction pointer) to an address where the binary code of that function is. Inside this code, the arguments passed are accessed by referencing some memory location. During compilation the compiler will not know the precise memory locations (obviously). But the compiler has to put in some address there when generating the code. How does it decides on the memory location? This is where the calling conventions come into picture. The compiler follows these standard steps. Since its the compiler who is generating the code for both calling a function, where arguments are sent, and the code of the called function, where those args are received, the compiler will be knowing where it has put the arguments and hence generate code to use the data in those locations. From this point of view, the main(or is it the only one) condition for the binary code of a function to work properly is to have its arguments in the memory locations it thinks they are in and and the end place back the return value in the right location. And from what I have understood till now, it is this point that libffi exploits.

When we are forwarding calls from an interpreted language, like JS, to some binary code generated from a compiled language, like C, there are two things to be done:

0. As discussed earlier, whatever arguments are passed from JS are to be placed in the right locations and later take the return value and give it back to JS.
1. Type conversions -- The types used by JS are not understood by C. So someone should rise up to the occasion and map these JS types to their C counterparts and vice-versa.

The first of these is taken care by libffi. But the second one is very context specific and totally depends on the interpreted language. It does not make sense to just have a catalog of type convertors for every interpreted language on this earth. So these two steps were separated and spread out as two interacting layers between the binary code and the interpreted language code. libffi now takes care of the calling convention part and making sure the binary code runs and gives back the results. And the job of type conversion is the job of another layer which is specific to the interpreted language which wants to call into the binary code. And hence we have these different type converting layers for different interpreted languages: ctypes for python, js-ctypes for JS (and probably some more exist). Well with this renewed and clearer understanding I hope to actually contribute to js-ctypes.

Happy calling (with conventions) ;-)

Something good even duirng these rough patches in market

Value Research: The Complete Guide to Mutual Funds

This website, valuereasearchonline, comes up with some analysis that is new to me almost every time. This is particularly stuck me because, the tool mentioned in this article and the message that it drives has got something to do beyond the markets.

If we can have a method to take advantage of the current volatility, irrespective of whether the markets go down or up, then we should remind ourselves of the good old statement: "Where there is a will there is a way".

Just read this up and you will feel better after the big downfall of yesterday and today. :-)

Kaminsky DNS Vulnerability

An Illustrated Guide to the Kaminsky DNS Vulnerability

With twitter, I have almost stopped blogging. But this was something that needed more than 140 chars. So here is another blog post.
This thing has been making rounds in the web security world for quite some time now. This in fact appeared in the Times of India newspaper recently. The article there mentioned about the sudden patching that the nameservers were going through. This security hole is indeed that serious. With this exploit an attacker can get hold of a complete domain and become an authoritative nameserver for that domain. So any request to resolve name for the hijacked domain can be directed to the attacker's machine.

Now imagine this happening to the most popular online banking website. Yes, things can be very creepy on the internet.

To clearly understand how this attack can happen and how it can be prevented read on the the illustrative guide linked at the beginning of the post. Doesn't matter even if you don't know how DNS works. The illustrative guide explains several things satisfactorily.

Happy Resolving ;-)