Having a car accident makes you have a lot of free time.
So I decided to create the ultimate Linux filesystems benchmark, took a spare hard disk I had available and tested all the filesystems writable by my Linux system, a 2.6.33-gentoo-r2 kernel.
For the tests I used bonnie++ as root, with the command line "bonnie++ -d /mnt/iozone/test/ -s 2000 -n 500 -m hpfs -u root -f -b -r 999".
My Linux system suppots a quite big number of writable filesystems, being: AFFS, BTRFS, ext2, ext3, ext4, FAT16, FAT32, HFS, HFS+, HFSX, HPFS, JFS, NTFS, Reiser, UFS, UFS2, XFS and ZFS.
I will divide the filesystems in "foreign unsupported", "foreign supported" and "native" filesystems.
A "foreign unsupported" is a filesystem designed and created outside Linux, for another operating system, but it's lacking features needed by Linux to run flawlessly, like the POSIX permissions, links and so on. These are AFFS, FAT16, FAT32, HFS and HPFS.
A "foreign supported" is a filesystem designed and created outside Linux, for another operating system, but supporting all features needed by Linux to run flawlessly, even if the current Linux implementation do not implement them fully. These are HFS+, HFSX, JFS, NTFS, UFS, UFS2, XFS and ZFS.
A "native" is a filesystem designed by the Linux community of developers, with Linux in mind, and usually never found outside Linux systems. These are BTRFS, ext2, ext3, ext4 and Reiser.
AFFS, or Amiga Fast File System is the old AmigaOS 2.0 filesystem, still in use by AROS, AmigaOS and MorphOS operating systems. It does not support the features required by Linux and is not an option for disks containing anything but data.
BTRFS, or B-Tree File System is a filesystem specifically designed to substitute old Linux native filesystems, including online migration of data from the extX filesystems.
ext2, or Extended 2 File System is an evolution of the original Linux ext filesystem, designed to overcome the limits of the Minix filesystems used in first Linux versions. ext3 is an evolution that added journaling and ext4 is another evolution that changes some structures to newer and faster technologies.
FAT, or File Allocation Table, is the original Microsoft filesystem, introduced with Microsoft Disk BASIC in the 80s and extensively used for data interchange, mobile phones, photo cameras, so on. FAT16 is an evolution of the original FAT (retroactively called FAT12) to allow for bigger disks, and FAT32 is the ultimate evolution introduced in Windows 95 OSR2 to overcome all the limitations. However Microsoft abandoned it in favor of NTFS for hard disks and ExFAT (currently not writeable by Linux) for data interchange.
HFS, or Hierarchical File System, is the original Apple filesystem introduced with Mac OS 5.0. It's design is heavily influenced by Mac OS design itself, requiring support for resource forks, icon and window position, etc.
HFS+, or Hierarchical File System Plus is an evolution of HFS to modernize it including support for UNIX features, longer file names, multiple alternate data streams (not only the resource forks). Current Linux implementation of HFS+ does not support all features from it, like for example journaling. HFSX is a variant of HFS+ that adds case-sensitive behavior, so that files "foo" and "Foo" are different ones.
HPFS, or High Performance File System is a filesystem designed by Microsoft for OS/2 to overcome the limitations of the FAT filesystem.
JFS, or Journaled File System is a filesystem designed by IBM for AIX, a UNIX variant, and so it supports all the features required by Linux. Really it's name is JFS2, as JFS is an older AIX filesystem that was deprecated before JFS2 was ported to Linux.
NTFS, or New Technology File System, is a filesystem designed by Microsoft for Windows NT to be the server-perfect filesystem. It was designed to support all features needed by all the operating systems it pretended to serve: OS/2, Mac OS, DOS, UNIX. Write support is not available directly in the Linux kernel but in userspace as NTFS-3G, and it does not implement all the features of the filesystem, neither all those required by Linux.
Reiser filesystem is a filesystem designed by Hans Reiser to overcome the performance problems of the ext filesystems and to add journaling. Its last version, Reiser4, is not supported by the Linux kernel because of reasons outside the scope of this benchmark.
UFS, or UNIX File System, also called BSD Fast File System is a filesystem introduced by the BSD 4 UNIX variant. UFS2 is a revision introduced by FreeBSD to add journaling and overcome some of the limitations. Support for both filesystem has been "experimental" in Linux for more than a decade.
XFS, or eXtended File System, is a filesystem introduced by Silicon Graphics for IRIX, another UNIX variant, to overcome the limitations of their previous filesystem, EFS. Linux's XFS is slightly different from IRIX's XFS, however both are interchangeable.
ZFS, or Zettabyte File System, is a filesystem introduced by Sun Microsystems for their Solaris UNIX variant. It introduces a lot of new filesystem design concepts and is still being polished. Support on Linux is provided by an userspace driver and not in the kernel itself.
The tests check the speed to write a 2 Gb file, rewrite it, read it, re-read it randomly, create 500 files, read them and delete them.
AFFS, FAT and HPFS filesystems failed the create/read/delete files test with some I/O error. In their native operating systems they have no problem to create 500 files so it's a problem in the Linux's implementation.
The output from Bonnie++ is:
|Version 1.96||Sequential Output||Sequential Input||Random
|Sequential Create||Random Create|
|Size||Per Char||Block||Rewrite||Per Char||Block||Num Files||Create||Read||Delete||Create||Read||Delete|
|K/sec||% CPU||K/sec||% CPU||K/sec||% CPU||K/sec||% CPU||K/sec||% CPU||/sec||% CPU||/sec||% CPU||/sec||% CPU||/sec||% CPU||/sec||% CPU||/sec||% CPU||/sec||% CPU|
The table is quite long so comparing graphs following:
This is the table for a single file sequential writing speed. The UFS and AFFS denote an abnormally slow speed, indicating a big need for optimization.
The fastest ones are, without a big difference, ext4, HFSX and XFS.
This is the table for a single file sequential reading speed. AFFS needs optimization also here.
The fastest one with a big difference is BTRFS and the second fastest one is JFS.
This two tables indicate the speed on writing and reading big files. If this is your intended usage, the recommended ones are BTRFS, ext4, HFSX, HFS+, XFS and JFS.
However if you need the maximum portability between systems, HFS+ is the choice because it's readable and writable by more operating systems than the others.
The following graphs show a more normal filesystem usage, where a lot of files are created, read and deleted at the same time.
This test shows surprising results. Reiser takes the lead on native filesystems, while even the newest BTRFS is quite behind all the other native ones.
In foreign filesystems, HPFS takes the lead. However, because of HPFS design limitations, the choice is NTFS, even being run on userspace.
In randomly reading 500 files, the results invert. NTFS loses its lead to UFS2. ext3 shows a huge improvement to ext2, and inversely, BTRFS shows a huge lose of performance.
Finally deleting all files, the lead is taken again by HPFS, second by HFS+ and HFSX.
Unless you want to write files and never delete them, the obvious choice is HFS+ or HFSX.
Surprisingly, even not being on its native implementation, HFS+ provides the best combination in all tests.
It supports all of the features required by Linux, gives a good sequential speed, and maintains its speed when handling a big number of files.
The only bad thing is that Linux does not support the journaling. Not a problem if you take care on unmounting it cleanly, without "oops! what happen to the electricity". Of course you can always spam the Linux kernel developers so they implement that function of the HFS+ filesystem that exists from 2003, 8 years ago.