Here are some of the superlatives I found:

“Increasingly enormous data sets”
“Today’s humongous data sets”
“ultra-large-scale data”
“extremely large data sets”
“enormous data sets”

I was thinking of coming up with a new superlative for my abstract. How does “Insanely huge” or “Frickin huge” or “shitload of data” sound?

Here is how you can reproduce it:

Open two terminals, lets call them Terminal A and Terminal B. In Terminal A, do the following:

mkdir /tmp/dummydir
cd /tmp/dummydir
touch file.txt


Now in Terminal B, do the following:

cd /tmp/dummydir
ls

If your eyesight is good, you should see a file named file.txt. Now in Terminal A, do the following:

cd ..
rm -rf dummydir
mkdir dummydir
cd dummydir
touch file.txt
ls

You should see file.txt. Now in Terminal B, do the following:

ls

You will not see any file, although you would expect to see file.txt because you are inside /tmp/dummydir.
One explanation of this behavior is that the inodes for both directories are different, and the current working directory of terminal B is still pointing to the old inode. This can be confirmed by doing the following in both terminals:

stat .


I am not sure if it is a well known behavior, but it’s interesting nonetheless.

alias fname='find . -name'

It lets me search files recursively in a directory starting from the current working directory. Some other aliases I use are:

alias gits='git status'

and

alias gitb='git branch'

I would like to know what aliases you are using for your everyday use, feel free to leave a comment.

Filesystem

A filesystem basically lets you read, write and update data on a storage device. It mostly provides the abstraction of files and directories. No matter how boring filesystems seem to an average user, they solve some interesting problems. One important thing to mention here is that there is no such thing as the-best-file-system, because there are various types of storage devices (disks, SSDs, tape drives, flash drives etc.), and each of them has different characteristics. Furthermore, there are different types of user applications, which have different types of access patterns (think read intensive, write intensive etc), and it becomes difficult (or impossible) to design a filesystem which works well in all scenarios. So, filesystems are usually designed for specific storage devices and application access patterns.

File systems are typically categorized into two types: Disk file systems (or Local file systems as I like to call them), and Distributed file systems.

The disk file systems are further divided into many categories, out of which Extent file systems and Log Structured file systems are the most interesting.

Extent File Systems

These file systems are much more efficient than conventional file systems in block allocation and reads/writes because they make use of something called extents.

To understand extents, let’s see how old file systems (FAT, ext2, ext3 etc.) work. These file systems split a file into blocks (usually 4KB) and store metadata for each and every block. So for a 128MB file, 32 blocks will be allocated (each having its own metadata structure), and no attempt will be made to save these blocks in a contiguous area. So writes will be pretty slow, because block allocations will need to be done as the file is being written. Furthermore, reads will also be slow because of high “seek” times. In addition to these problems, the file system will also become fragmented over time.

An extent file system allocates extents for a file rather than single blocks.  An extent is a contiguous area of storage in a file system. So instead of allocating a block one by one, the file system allocates a contiguous sequence of blocks in one go. This not only improves the read/write performance (because seek time is reduced), but also helps in preventing the file system from becoming fragmented. Of course, the file system can still become fragmented because it is not always possible to allocate blocks contiguously, but the situation is much better than old file systems. Another big benefit of using extents is that very little space is required to store the metadata of blocks on disk, because metadata is stored for each extent, and not for each block. Due to the above mentioned reasons, most of the modern file systems (ext4, Btrfs, XFS, Reiser4 etc.) make use of extents for block allocation. Obviously modern file systems do much more than allocating blocks, but nonetheless it is one of the most fundamental aspect in file system design.

That’s it for now. In the next posts, I will write about Log Structured file systems, and Distributed file systems. Yesterday, I gave a little overview of some popular file systems to my colleagues at SICS. Here are the slides.

If you are interested in knowing more about file systems, BUY MY BOOK. Just kidding.

At the moment, I am trying to understand the techniques used by file systems (like ext3, ext4, Btrfs, XFS, ReiserFS, Haystack (Facebook), Cosmos (Microsoft), CloudStore) for inodes/block management, reliability and efficiency. Reading is boring, but it will give me a good idea of things to keep in mind while I am writing code.

I will also give a talk about these file systems next week, and will also post it here.