Porting the HAMMER File System to Linux

10 Seconds of Fame

2009-06-09T18:18:00.000-07:00

This blog is actually linked to by the Wikipedia article on the HAMMER File System, and it wasn't me. That's actually very cool, thank you :)

Error-Driven Development

2009-05-11T02:49:00.001-07:00

1. Compile
2. Fix errors
3. Goto 1

I like it.

daniel@daniel:~/linux-2.6.29.1$ make ARCH=um 2>&1 | grep 'error: ' | sed -e 's/.*error: //g' | sort | uniq
'EFTYPE' undeclared (first use in this function)
'FREAD' undeclared (first use in this function)
'FSCRED' undeclared (first use in this function)
'FWRITE' undeclared (first use in this function)
'LK_EXCLUSIVE' undeclared (first use in this function)
'LK_RETRY' undeclared (first use in this function)
...

hammerread.c + FUSE = Read HAMMER Filesystem on Linux

2009-04-12T08:12:00.000-07:00

I hacked together a FUSE module using hammerread.c, so you can now mount a HAMMER device in read-only mode on Linux. I haven't done any extensive testing, yet, so things are probably still very flaky. You can get the FUSE module from here: fusehammer-0.1.tar.gz and test with hammerdisk.raw.bz2.

Here's a walkthrough:

daniel@daniel-desktop:~$ wget http://hammerfs.googlepages.com/fusehammer-0.1.tar.gz
daniel@daniel-desktop:~$ tar zxf fusehammer-0.1.tar.gz
daniel@daniel-desktop:~$ cd fusehammer
daniel@daniel-desktop:~/fusehammer$ make
daniel@daniel-desktop:~/fusehammer$ ./fusehammer
HAMMER filesystem for FUSE (readonly)
Usage: ./fusehammer <dev> <mount_point> [<FUSE library options>]
daniel@daniel-desktop:~/fusehammer$ mkdir mnt
daniel@daniel-desktop:~/fusehammer$ ./fusehammer ../hammerdisk.raw mnt
daniel@daniel-desktop:~/fusehammer$ ls -l mnt
total 0
drwxr-xr-x 0 root root 0 1970-01-01 01:00 a
-rw-r--r-- 0 root root 12 1970-01-01 01:00 test
daniel@daniel-desktop:~/fusehammer$ cat mnt/test
Hello
World
daniel@daniel-desktop:~/fusehammer$ fusermount -u mnt

Note that the file attributes are wrong (mtime, ctime ..). I need to investigate that.

POSIX File System Test Suite

2009-04-12T05:33:00.000-07:00

Just found this: POSIX File System Test Suite

"The list of system calls tested is: chmod, chown, link, mkdir, mkfifo, open, rename, rmdir, symlink, truncate, unlink. There are currently 3601 regression tests"

Sounds cool, maybe something to look into at a later date.

In an unrelated matter, I read Amit Singh's (MacFUSE fame) blog on AncientFS, where he mentions:

"In future, it [the UnixFS layer] may also evolve into a "LinuxFS" layer that could make it even easier and faster to systematically make Linux kernel-based file systems work on other platforms with very few code changes."

Now that sounds interesting.

#define student 04302402740280438773

2009-04-11T16:08:00.000-07:00

#define student 04302402740280438773

in blag included from the intertubes.

hammerread.c ported to Linux

2009-04-11T15:07:00.000-07:00

I ported hammerread.c to Linux. I was surprised how little I had to change in the source code. The Makefile contains the following switches:

# -D_BSD_SOURCE for DT_FIFO, DT_CHR .. in <dirent.h>
# -D_FILE_OFFSET_BITS=64 for large ino_t in <dirent.h>
# -D_XOPEN_SOURCE=500 for pread64 in <unistd.h>

You can test the application yourself. Grab a disk image from here: hammerdisk.raw.bz2. The contents of the image are:

$ find . -type f -ls
6400704     0 -rw-r--r--    1 root   wheel  12 Mar 23 20:04 ./test
12801152    0 -rw-r--r--    1 root   wheel   4 Apr 11 00:56 ./a/b/c/bar

Now download hammerread-0.1.tar.gz and use it as follows:

daniel@daniel-desktop:~$ wget http://hammerfs.googlepages.com/hammerread-0.1.tar.gz
daniel@daniel-desktop:~$ tar zxf hammerread-0.1.tar.gz
daniel@daniel-desktop:~$ cd hammerread
daniel@daniel-desktop:~/hammerread$ make
daniel@daniel-desktop:~/hammerread$ ./hammerread ../hammerdisk.raw
signature: valid
name: HAMMER
daniel@daniel-desktop:~/hammerread$ ./hammerread ../hammerdisk.raw /
signature: valid
name: HAMMER
/ 0/0 40755 0
test 8 10061aac0
a 4 1007a1650
daniel@daniel-desktop:~/hammerread$ ./hammerread ../hammerdisk.raw /test
signature: valid
name: HAMMER
/test 0/0 100644 12
Hello
World

STOP!

2009-04-09T15:27:00.000-07:00

Hammer time!

My work on the hammer file system is completed. You may download the module here: hammerfs-0.1.tar.gz. Here's a short walkthrough:

root@daniel-desktop:~# wget http://hammerfs.googlepages.com/hammerfs-0.1.tar.gz
root@daniel-desktop:~# tar zxf hammerfs-0.1.tar.gz
root@daniel-desktop:~# cd hammerfs
root@daniel-desktop:~/hammerfs# make
root@daniel-desktop:~/hammerfs# insmod hammer.ko
root@daniel-desktop:~/hammerfs# mount -t hammerfs nodev /mnt
root@daniel-desktop:~/hammerfs# cd /mnt
root@daniel-desktop:/mnt# ls -l
total 0
-rw-r--r-- 1 root root 434 2009-04-10 00:26 hammertime
root@daniel-desktop:/mnt# touch hammertime
touch: setting times of „hammertime“: Operation not permitted # can't touch this
root@daniel-desktop:/mnt# cat hammertime

                             
            .d:.            
       ...,;l0kc.           
       .l;oddO0xOd:.        
        .'dxOKOd;..         
         ;0kdo:.            
       .xNNXKKl             
       dNNNNNNX;            
       dNNNNNNNx            
       'XNKo0NNk            
       :XK, .ONK.           
       ;Xl   .kK.           
       .Oc    '0.           
        ..    .d;

The special file hammertime will display a new frame each time it is cated. To watch the entire animation, type watch -n0.1 cat hammertime

fat free

2009-04-06T12:23:00.000-07:00

fat_free, sb_breadahead ..

kernel identifiers are funny. ha-ha.

Sample filesystems

2009-04-06T08:29:00.000-07:00

Seems, there is quite a few of them:

Debugging Linux kernels with VMWare

2009-04-03T05:36:00.000-07:00

Still trying to find something, since UML didn't work for me. This here works with VMWare Fusion, too:
Debugging the virtual world.: Debugging Linux kernels with Workstation 6.0

I had to edit the .vmx file and add debugStub.listen.guest32=1 to it, then the log said:

Apr 03 14:33:57.701: vmx| VMware Fusion is listening for debug connection on port 8832.
Apr 03 14:33:57.701: vmx| target remote localhost:8832

yay. I can't load the debug symbols of the kernel, though.

(gdb) file vmlinuz-2.6.27-11-generic
"/Users/daniel/vmlinuz-2.6.27-11-generic": not in executable format: File format not recognized

Update: Seems like I need to get the uncompressed kernel (vmlinux) for the debug symbols, which involves getting the appropriate .debs from ddebs.ubuntu.com. I didn't follow that further, as I got UML working which I find more comfortable anyway.

The Linux Kernel Module Programming Guide

2009-04-01T07:50:00.000-07:00

So, you want to write a kernel module. You know C, you've written a few normal programs to run as processes, and now you want to get to where the real action is, to where a single wild pointer can wipe out your file system and a core dump means a reboot.

Yes, I do.

Volume header and endianness

2009-03-26T08:19:00.000-07:00

The volume header is defined as follows (vfs/hammer/hammer_disk.h):

/*
* Volume header types
*/
#define HAMMER_FSBUF_VOLUME 0xC8414D4DC5523031ULL /* HAMMER01 */
#define HAMMER_FSBUF_VOLUME_REV 0x313052C54D4D41C8ULL /* (reverse endian) */

and

struct hammer_volume_ondisk {
u_int64_t vol_signature;/* Signature */
..
char vol_name[64]; /* Name of volume */
..
}

now

-bash-4.0# dd if=/dev/ad1s0e of=dump bs=1B count=1
1+0 records in
1+0 records out
512 bytes transferred in 0.000320 secs (1600211 bytes/sec)
-bash-4.0# hexdump -C dump | head
00000000 31 30 52 c5 4d 4d 41 c8 00 00 04 00 00 00 00 00 |10R.MMA.........|
00000010 00 00 14 00 00 00 00 00 00 00 24 00 00 00 00 00 |..........$.....|
00000020 00 00 40 1f 00 00 00 00 00 00 00 00 00 00 00 00 |..@.............|
00000030 ab 3a fa 2a dc 17 de 11 92 2e 01 0c 29 6d 7b af |.:.*........)m{.|
00000040 ac 63 dc 61 38 6e dc 11 85 13 01 30 1b b8 a9 f5 |.c.a8n.....0....|
00000050 48 41 4d 4d 45 52 00 00 00 00 00 00 00 00 00 00 |HAMMER..........|
00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000090 00 00 00 00 01 00 00 00 01 00 00 00 bb d0 4b bd |..............K.|
000000a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

The volume signature (10R.MMA.) is backwards and not even correct (partly due to endianness), while the volume name appears correctly.

Maybe something to investigate.

Kernel Hacking with UML

2009-03-26T05:48:00.001-07:00

Kernel Hacking with UML

Fails inside Linux running on VMWare Fusion on my Mac, but works on a physical machine. Seems to be the way to go. I chose Debian 4.0 from the pre-made filesystems.

B+Tree

2009-03-26T05:06:00.000-07:00

Ubiquitous B-Tree, hammer_btree.h, hammer_btree.c

Project Proposal

2009-03-24T10:55:00.000-07:00

For the project proposal, I found a very good template/guideline.

Here's the proposal I handed in: proposal-lorch.pdf.

hammerread.c

2009-03-24T09:10:00.000-07:00

hammerread.c is a self-contained library for the bootloader to read the hammer filesystem. It can be compiled as a command line tool using -DTESTING.

First get the dragonfly sources with cvsup, then:

-bash-4.0# cd /usr/src/lib/libstand
-bash-4.0# cc -DTESTING -std=c99 hammerread.c -o hammerread
-bash-4.0# ./hammerread
usage: hammerread
-bash-4.0# ./hammerread /dev/ad1s0e
signature: valid
name: HAMMER
-bash-4.0# ./hammerread /dev/ad1s0e /test
signature: valid
name: HAMMER
/test 0/0 100644 12
Hello
World

Discovering HAMMER

2009-03-24T08:55:00.000-07:00

Time to play. I installed a hammer filesystem.

-bash-4.0# echo Hello > test
-bash-4.0# echo World >> test
-bash-4.0# hammer history test
test 000000010061aac0 clean {
00000001007a1520 23-Mar-2009 20:04:11
00000001007a1580 23-Mar-2009 20:04:43
}
-bash-4.0# cat test@@0x00000001007a1520
Hello
-bash-4.0# cat test@@0x00000001007a1580
Hello
World
-bash-4.0# cat test
Hello
World

Snapshot creation:

-bash-4.0# hammer snapshot /mnt /mnt/snap
/mnt/snap
-bash-4.0# ls -l snap
lrwxr-xr-x 1 root wheel 25 Mar 23 20:07 snap -> /mnt/@@0x00000001007a15c0
-bash-4.0# ls snap/
test

Getting the Code

2009-03-20T06:42:00.000-07:00

development(7): git clone -o crater git://crater.dragonflybsd.org/dragonfly.git dragonfly.git

Got the code. There is gitweb, too.

Hello

2009-03-20T06:33:00.000-07:00

I am going to port the HAMMER File System from DragonflyBSD to Linux as part of a semester project and thought it might be useful to keep a record of what is being done.