Incrementally building Mozilla/Firefox

Mozilla code base is really huge and has variety of files which are built in a variety of ways. It was sort of always confusing for me to figure out where all I should run make after I change any of the files. I generally asked on the IRC and someone just told me where to run make.

Today it was the same thing. But I also thought I would as well learn the logic to decide for myself the next time. Here is the chat transcript of NeilAway answering these questions.

The MDC page (https://developer.mozilla.org/en/Incremental_Build) has almost the same content for the native code. Neil here explains it for all the types of files involved.

Also to add to the following things running : "make check" from the objdir will run the automated tests.

• For xul/js/css/xbl it usually suffices to find the jar.mn (it may be in an ancestor folder) and make realchrome in the corresponding objdir
• For idl you're often looking at a full rebuild, depending on how widely it's used
• For .cpp and .h you obviously have to make in the folder itself, and then look for a corresponding build folder
• Except for uriloader where you use docshell/build and content, dom, editor and view use layout/build
• If you're building libxul or static then this is all wrong
• You don't look for a build folder, I think for libxul you build in toolkit/library and for static you build in browser/app

Changing the start up directory of command prompt -- The safe and simple way

By default, the command prompt window (cmd.exe) starts in a particular directory which depends on quite a few factors. Specifically, the env varibales %HOMEDIR% and %HOMEPATH% matter the most or these are the ones which ultimately decide the location. There are some registry values also, but they are not in the game by default. This leads to the command window to start up in something like C:\Documents and Settings\<username>\ which is not particularly useful as this location is rarely used for anything useful. It can get worse. In a corporate environment it might so happen, more often than not, that your home directory is set to a shared network drive using a group policy and the command prompt starts in that remove drive location.. ! The pain can be aggravated if you are on VPN or something similar.

I personally feel that command prompt, which is mainly programmers tool, should not start in %HOMEDIR% as there are rarely (mostly never) any programming related files are kept. Anyways, there are many ways to solve this problem.

The first of them and the most dangerous is fiddling with the registry. It is mentioned here : http://windowsxp.mvps.org/autoruncmd.htm
The problem with this is that it will screw up make based build environments which spawn multiple child shells (aka command prompts), because the command prompt will start in this changed default location instead of the location where the make had to run.

The next is fairly simple and also elegant. You can create a shortcut to the main exe (C:\Windows\system32\cmd.exe) and in the properties of the shortcut you can provide the directory to start in. This good for mouse users. However for those (most programmers) who start command prompt from the run dialog (by typing cmd) this will fail.

The third solution is to deal with the previous solutions shortcoming. You can just create a batch fail in any of the locations where the run dialog looks into. I prefer C:\Windows\systme32\. Put the following line in the batch file: C:\Windows\system32\cmd.exe /K "cd <path-to-dir>" . Just replace the <path-to-dir> with the path where you want the command to start. I generally put it as C:\.
This will start a command window and execute cd <path-to-folder> and will stay for further inputs. I have named this batch file as sh.bat (obviously to get a linux feel :P ). So now when I press the Windows + R key and type sh, I get the command prompt started in C:\.
Done.
And yes, this is totally safe and will not affect any other application using cmd.exe. :)

OpenSSL base64 filter BIO needs an EOL and memory BIO needs to know about EOF

I recently started working with the OpenSSL library to do some https stuff (sort of obviously). OpenSSL apart from having an implementation for the SSL encryption part, it also nifty algorithms for certificate handling and more importantly an abstract I/O layer implementation called BIO which probably stands for Basic I/O or Buffered I/O or something else. I do not know, I could not find it. Nevertheless, the items of interest here are the BIO_f_base64() -- The base64 encode/decode filter BIO and the BIO_s_mem() -- The memory BIO, which can hold data in a memory buffer.

The BIO man page (or its online version present here: http://www.openssl.org/docs/crypto/bio.html) give a nice introduction. For now just consider BIOs as black boxes from which you can read or write data. If the BIO is a filter BIO then the data will be processed whenever you read or write to it.

The name, BIO_f_base64, says it all about the functionality of this BIO. If you read from this BIO, then whatever data is being read is first base64 decoded and given to you. OTOH, if you write something to this BIO it will be base64 encoded and then written to the destination. These BIOs can be arranged in the form of chains to do a series of processing on the data that you are reading or writing, all by just a single call to read() or write(). Its all abstracted. Saves a lot of time.

I was trying to decode some base64 encoded data which I had in a buffer, a char [] to be precise. So if you read up about the BIOs it becomes obvious that you first have to create a memory BIO, which will hold the actual encoded data. Write the encoded data to the memory BIO. Then you chain that memory BIO with a base64 BIO and read from that chain. Any data that you read from the chain will actually come from the memory BIO, but before it reaches you it passes through the base64 BIO. So essentially you are reading from the base64 BIO. As mentioned in the earlier paragraph, when you read from a base64 BIO it decodes the data and gives it you. So the base64 encoded data present in the memory BIO is decoded and presented to you. That's it. base64 decoding is done in one simple read() call !

But there is small catch here. For some reason, which I have yet partially understood, base64 requires that the data it is handling be terminated with a new-line character always. If the data does have any newline character, meaning all your data is present in a single line then you have to explicitly tell that to the BIO by setting the appropriate flag. Here is what the man page says:

The flag BIO_FLAGS_BASE64_NO_NL can be set with BIO_set_flags() to encode the data all on one line or expect the data to be all on one line.

That's about the base64's EOL. Now the other BIO involved here,the memory BIO, is also an interesting guy. When the data it has gets over, it doesn't say "Hey, its over, stop it!". Instead it says "Dude, you got to wait for some more data to arrive. Hang on and keep trying". !!! This is very much suitable, probably when you using the BIO like a PIPE, where you keep pumping data from one end by acquiring it from somewhere and some other guy consumes that data. But in a situation like mine where the data is all fixed I simply want it to tell that the data is all over and I need to stop it. To do this again I will have to explicitly set an appropriate flag and here is what the man page says:

BIO_set_mem_eof_return() sets the behaviour of memory BIO b when it is empty. If the v is zero then an empty memory BIO will
return EOF (that is it will return zero and BIO_should_retry(b) will be false. If v is non zero then it will return v when it
is empty and it will set the read retry flag (that is BIO_read_retry(b) is true). To avoid ambiguity with a normal positive
return value v should be set to a negative value, typically -1.

And this same thing is explained very well here: http://www.openssl.org/support/faq.html#PROG15.

I thank Dr. Stephen N Herson of the OpenSSL project for helping me out in understanding this. Here is the mailing list posting that taught me this thing : http://groups.google.com/group/mailing.openssl.users/browse_thread/thread/f0fc310c1bc6ec65#

Happy BIOing. :-)