DiscImageChef and the Lisa operating system source code recovery

In 1983, Apple launched the Lisa computer, one of the first personal computers with a graphical user interface, setting up the first stone to user interfaces as we know them today.
On 24th December 2017, Al Kossow announced he recovered the source code (that is, the computer code used to create the Lisa operating system and user interface), using the DiscImageChef tool.

What is DiscImageChef?
DiscImageChef is a multifunctional manager for all kind of dump images of computer storage. It can do the most common functions usually done with dumps of floppies, hard disks, optical discs, etc. It also allows dumping such media to images.

What was the function of DiscImageChef in the Lisa source code recovery?
One of the functions of DiscImageChef is, for supported filesystems, to extract the contents of a dump. And in the case of the Lisa, DiscImageChef is the only tool supporting its filesystem thanks to my reverse engineering of it.

Reverse engineering?
Yes, that is the process to analyze some software to learn its inner workings, to be able to document that, or to create another software that is interoperable with its documents. In the case of the Lisa filesystem, there was no documentation of it; it was only known that it had some structures called MDDF and S-Record, but there was no information about how those structures were formed. So for several years I studied all known dumps, and created lots of test dumps using Ray Arachelian's emulator, LisaEm. Then, through testing and analysis of these dumps, I finally got enough knowledge of the Lisa filesystem structures to be able to add support for it on DiscImageChef.

So that's why you created DiscImageChef?
No, indeed, DiscImageChef started in 2011 as a personal tool designed to identify and show information about filesystems called FileSystemIDAndChk, that I used to check the integrity and correctness in my collection of operating system dumps. From that, it evolved organically. As dumps came in different formats, I added support for format plugins. In 2014 I renamed it the a more appropriate DiscImageChef. And from then on it has gained the ability of extracting files, creating dumps, comparing two dumps even with different formats, etc.

Why that name?
When talking about storage media, we say that their contents are raw when they represent what is in the media itself, while the contents are cooked when they contain only the data a user has wrote to them. And a chef is someone that takes a bunch of raw ingredients, and not only cooks, but makes magic with them. And disc image is because at first it was oriented for optical discs only, that by convention are spelled with C, contrary to other media that is spelled with K, as in disk. Also, "swiss army knife" is a term that's used so much, it has known became a common thing, losing its uniqueness. So join all, and you have The Disc Image Chef.

So, is it DiscImageChef or Disc Image Chef?
Both can be used. I just prefer to use DiscImageChef when no article is used, and separate when it is, like The Disc Image Chef.

And what can it do today?
Well, the last stable version, it can analyze, checksum, compare and verify 46 different image formats, identify 73 filesystems, list and extract files from 5 of them, and get information and create a dump from almost all devices, as long as they are attached to a ATA (aka IDE), Serial ATA, FireWire, USB, SCSI or SDHCI bus, from 3 of the major operating systems: Windows, Linux and FreeBSD.

Why does it not support macOS?
It does! The only thing it is not able to do is to dump media from a device connected to macOS. All the other functions work perfectly. The problem with macOS is that it doesn't allow any application to arbitrarily read devices attached to it; only from CD and DVD recorders. And I intend to add support for doing that soon.

So what's the future of DiscImageChef?
Well, for start I'm adding support for writing in any of the dump formats currently supported. That will allow people to convert a dump between two formats, or create a new dump from real media in their favorite format. The next thing I'll add is support for dumping floppies, with a standard floppy drive, or using one of the flux dumping hardware cards, like SuperCardPro, DiscFerret or KryoFlux. Also I intend to add the ability to list of extract for as many filesystems as possible.

Is a graphical interface coming?
Yes. I will add a graphical interface to DiscImageChef when all the other functions are stable. With barebones support for floppies, and only 5 filesystems fully supported, I feel it is not currently the time for adding a graphical interface. Also I'm pretty bad designing graphical interfaces, so if any designer is up to it, help is accepted.

About help, how many people work on DiscImageChef?
From time to time a collaborator sends new code, or someone asks for a feature, but it's mostly me alone, and I will be very grateful to have some help, specially when designing new parts of DiscImageChef, a little discussion of different ideas is always welcome.

To end, where can we get more information, or news, about DiscImageChef, or ask for a feature?
You can follow my work on Twitter, @DiscImageChef, or you can follow the project on GitHub, https://github.com/claunia/DiscImageChef.

Natalia Portillo,
31st December 2017

ZFS on read-only devices

On my last post I described my experiments to get along the best way to archive ROM sets with the biggest space saving.
In a nutshell, ZFS got a best-for-money status, except for the RAM usage.
My idea after the tests was to check if it was possible to mount a ZFS pool recorded in a read-only device, like a CD/DVD/BD.
Why here?
They’re cheap, small, and usually, more durable (quite depends on disc quality, but most of the failures are bitwise not full disc, and this can be solved with error recovery algorithms).
But ZFS was not designed with these discs in mind, so some magic was expected.
All of this is done on Linux. Should be doable on BSD, Solaris and Mac OS X with not many differences.
First of all, I did not test ZFS on packet writing, if it works at all optical drives seeking time and speed will be a hell.
So, first of all, I created a dummy file to act as the writable zpool before recording it to disc, with the following command:

dd if=/dev/zero of=zpool.bin bs=2048 conv=sparse count=<sectors>

<sectors> being size as following:

• 333,000 for 74-min CD-Rs and CD-RWs
• 360,000 for 80-min CD-Rs and CD-RWs
• 2,298,496 for single-layer DVD-Rs
• 4,171,712 for double-layer DVD-Rs
• 2,295,104 for single-layer DVD+Rs
• 4,173,824 for double-layer DVD+Rs
• 12,219,392 for single-layer Blu-rays
• 24,438,784 for double-layer Blu-rays
• 48,878,592 for triple-layer Blu-rays
• 62,500,864 for quad-layer Blu-rays

Once the dummy file is created you need to create the zpool. While you could create a RAIDZ pool for error recovery, it would require you to have 3 optical drives with 3 discs inserted, at the same time, at least, so I suggest for another solution (I’m still searching one, you can comment your ideas).
For the zpool creation:

zpool create -o comment="Put some disc identification here" -o ashift=11 -o failmode=continue -O mountpoint=/mnt/myzfsdvd -O checksum=sha256 -O compression=gzip-9 -O dedup=on -O atime=off -O devices=off -O exec=off -O setuid=off myzfsdvd zpool.bin

Explaining each options:

• comment=“Put some disc identification here”, as it’s for a removable disc you need some description of what you’re going to store on it. You can set it up later
• ashift=11, this means that the underlying block device (that is, the optical disc) uses 2048 bytes/sector
• failmode=continue, not a good idea for the system to panic or stop when something happens to the optical disc!
• mountpoint=/mnt/myzfsdvd, good to tell a mountpoint. You can change this on mount, but setting a default prevents security risks
• checksum=sha256, this is just personal taste
• compression=gzip-9, optical discs are only written once, so let that writing time be used the best
• dedup=on, same as above, deduplicate blocks
• atime=off, no sense to store access time on read-only discs
• devices=off, to disable support of device nodes
• exec=off, to prevent executing code from the disc, you may want to do, but I didn’t
• setuid=off, to prevent setuid

Once this is done, ZFS will automatically mount the pool on the specified mountpoint, so you can start copying things.
You may be tempted to change recordsize (block size) to equal hardware sector for optical discs (2048 bytes), DO NOT DO IT, it will make ZFS be marginally slower, and compression and deduplication work worse.
And for RAM usage on deduplication, don’t worry, with the disc out RAM will get freed, and even using quad-layer Blu-rays, the deduplication table will never be bigger than 1Gb.
Time to unmount the ZFS file so we can record it to disc
zpool export myzfsdvd
Now recording it. If you’re going to record on CD (really? sure? ok, as you wish) be sure that it is recorded single-session, mode 1, data, finished. Following line works for CD, DVD and BD:
cdrecord -v driveropts=burnfree -dao -data zpool.bin
Once it is recorded, you’ll be tempted to mount it, and if you do, you’ll see the failure:

~ # zpool import -a

~ # zpool list

NAME         SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT

~ # 

So, where’s the pool? Let’s try it specifying our pool name:

~ # zpool import zfstestdvd

cannot import 'zfstestdvd': no such pool available

~ #

It took me a couple of minutes to find what the problem of that unspecific error was, as it clearly was trying to mount it. It is trying to write!

~ # zpool import -o readonly=on -a -d /dev/

~ # zpool list

NAME       SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT

myzfsdvd  4.34G  1.90G  2.45G    43%  2.09x  ONLINE  -

~ #

I used /dev because sometimes it decided to not mount it from the DVD-ROM drive (/dev/sr0).

Now the pool works perfectly, except for some secondary effects:

• zpool scrub denies to work, because pool is read-only, instead of checking without correcting.
• zdb is unable to find the pool at all.

Cold-storage and live-storage of ROM and ISO sets, or "compression vs deduplication"

This weekend I’ve been thinking about cold storage of my software archive, that include some ROM and ISO sets. The idea was to find the better way to store them in Blu-ray recordable discs.

Considering how usually two sets can contain almost the same data (for example “Tomb Raider” for Saturn and PlayStation can contain same audio, movies and maps, differing only on the small executables), or even the same set (all of the “Super Mario World” clones differ only in a small part of the ROM), both solid-mode compression and deduplication offer really high promises.

So, for the purposes of checking how promises behold, I took two systems, Nintendo Game Boy and Nintendo Game Boy Color, along with all the sets I have from them: Cowering (aka GoodTools), TOSEC, no-intro and NonGood.

For compression I used torrent7z, that given the same files (and filenames) will get the same compressed file exactly using 7zip (and, from it, the LZMA algorithm), and GoodMerge, a tool that takes all the clones from the Cowering set and compresses them together (merges) with 7zip at maximum, to get the maximum space savings.

For the deduplication, first I got theoretical estimations using an application I developed specifically for that (DedupStat, you can get it here, GPL, open source) and then created ZFS (a highly complex and feature-able filesystem available on Solaris, FreeBSD, Linux and Mac OS X, but not Windows, at all) pools with both deduplication and compression enabled.

Cowering set contains 7,930 ROMs, TOSEC contains 135 ROMs, no-intro.org contains 2,980 ROMs and NonGood contains 138 ROMs.

They make 11,183 files for a total of 7,598,579,214 bytes (~7,246Mb).

Test computer is an Intel Core 2 E6400 @ 2.13Ghz, Linux 3.9.0-server (Sabayon), 8Gb DDR2-800 RAM, torrent7z 0.9.1beta (7zip 4.65), ZFSonLinux 0.6.2-r3, on a Maxtor/Seagate STM3320820A.

First of all, let’s test compression.

Cowering (GoodMerge7z + torrent7z) + TOSEC (torrent7z) + no-intro (torrent7z)
5,394 files for a total of 1,407,450,696 bytes (~1,342Mb, 18.52%), took 4,813.013 seconds, approx. 0.27887927590956 Mb/sec

Cowering (GoodMerge7z + torrent7z) + TOSEC (torrent7z) + no-intro (torrent7z), deduplicated on 4096 bytes/block (typical filesystem)
5,394 files for a total of 946,782,208 bytes (~902Mb, 12.46%), took 4,972.689598 seconds, approx. 0.269924264109135 Mb/sec

This of course gets us a lot of space savings, compression alone makes the sets be only about 18% of the original size. But making this (only compression speed is tested, but decompression speed is not blazing fast either) takes a lot of time, almost an hour and a half. Deduplication over the compressed sets can still give more savings, because there is repeated data between sets (compression is per-ROM), and in the same set (only Cowering merges clones and hacks).

The next test I did was deduplication estimations only, using my tool.

Cowering + TOSEC + no-intro, deduplicated on 2048 bytes/block (CD/DVD/BD blocksize)
11,186 files for a total of 1,746,548,736 bytes (~1,665Mb, 22.99%), took 480.059208 seconds, approx. 15.0939714919498 Mb/sec

Cowering + TOSEC + no-intro, deduplicated on 4096 bytes/block (typical filesystem blocksize)
11,186 files for a total of 1,871,421,440 bytes (~1,784Mb, 24.63%), took 513.864285 seconds, approx. 14.1009994496893 Mb/sec

Cowering + TOSEC + no-intro, deduplicated on 131072 bytes/block (typical ZFS blocksize)
11,186 files for a total of 3,480,354,816 bytes (~3,319Mb, 45.80%), took 425.701966 seconds, approx. 17.0212979472122 Mb/sec

On the smallest blocksize, more duplicate blocks are found, giving biggest savings, but still quite far from compression. However, in any case, it’s about 10 times faster.

As Cowering contains lots of hacks, clones, and bad dumps, that may not be interesing for some people, I tested both compression and deduplication with the other sets alone.

TOSEC + no-intro, uncompressed
3,256 files for a total of 2,685,937,612 bytes (~2561Mb)

TOSEC (torrent7z) + no-intro (torrent7z)
3,256 files for a total of 795,382,502 bytes (~758Mb, 29.61%)

TOSEC + no-intro, deduplicated on 2048 bytes/block (CD/DVD/BD)
3,256 files for a total of 1,412,812,800 bytes (~1,347Mb, 52.60%), took 181.490064 seconds, approx. 14.1109653253525 Mb/sec

TOSEC + no-intro, deduplicated on 4096 bytes/block (typical filesystem)
3,256 files for a total of 1,491,755,008 bytes (~1,422Mb, 55.54%), took 168.831435 seconds, approx. 15.1689760855258 Mb/sec

TOSEC + no-intro, deduplicated on 131072 bytes/block (typical ZFS)
3,256 files for a total of 2,127,953,920 bytes (~2,029Mb, 79.23%), took 248.102553 seconds, approx. 10.3223444057023 Mb/sec

Things seem worse here for deduplication, being twice as big as compressed sets.

But all of this, is highly theoretical, so better test with a real life scenario. I created three ZFS pools, all of them with deduplication and compression enabled, for the three block sizes (2048 as in CD/DVD/BD, 4096 as typical by other filesystems and real sector size of Advanced Format hard disks and 131072, default ZFS blocksize). Note: ZFS calls its block size “recordsize”, and dynamically uses a bigger or smaller than the configured one, depending on the stored data.

Cowering + TOSEC + no-intro, ZFS device, dedup=on, compression=gzip-9, recordsize=2048
11,188 files for a total of 1,904,936,848 bytes (~1,817Mb, 25.07%), took 10,986.604 seconds, approx. 0.659580853138233 Mb/sec

Cowering + TOSEC + no-intro, ZFS device, dedup=on, compression=gzip-9, recordsize=4096
11,188 files for a total of 1,515,458,176 bytes (~1,445Mb, 19.94%), took 8,865.023 seconds, approx. 0.817432017876539 Mb/sec

Cowering + TOSEC + no-intro, ZFS device, dedup=on, compression=gzip-9, recordsize=131072
11,188 files for a total of 1,493,678,664 bytes (~1,424Mb, 19.66%), took 1,185.156 seconds, approx. 6.114430201097515 Mb/sec

While compression is called gzip, it should really be called deflate, as that’s the name of the algorithm, used in ZIP files, and theoretically, marginally faster and less powerful than LZMA (the 7zip algorithm). “-9” marks maximum compression.

Clearly ZFS does not behave very well when the recordsize is made smaller than the default, getting worse deduplication, worse compression, and marginally slower speeds (8 to 10 times slower).

Before achieving a conclusion, a thing about RAM usage must be noted. 7-zip RAM usage depends on dictionary size (and so, compression power), and only happens once (on compression). On decompression, RAM usage is marginally lower. On the contrary, deduplication ram usage depends on total blocks (unique and duplicate, with duplicate counting only as 1 block), and its usage is more or less permanent as long as the deduplicated volume is attached.

In case of ZFS, each block takes 320 bytes of RAM. For this set, meaning:

3710244 blocks of 2048 bytes each, taking 1Gb of RAM.

1855122 blocks of 4096 bytes each, taking 566Mb of RAM.

57972 blocks of 131072 bytes each, taking 18Mb of RAM.

Fortunately you can use a SSD to store the deduplication table, so instead of 1Gb/566Mb/18Mb of RAM it would be 1Gb/566Mb/18Mb of SSD.

More than a conclusion, a list of advantages and disadvantages should be noted:

Advantages of 7zip

• Biggest space savings
• RAM usage is not permanent, only when working with the archives

Disadvantages of 7zip

• Slow as floppy drives
• Merged archives need to be traversed and uncompressed fully before accessing one of the ROMs it contains
• Not supported by any emulator afaik, and not completely supported by any ROM manager (Romcenter does not support them, Romulus says it does but it does not work, CLRMAME Pro does if not solid-compressed so defeating merging benefits)

Advantages of deduplication

• Faster
• Depending on deduplication software used (if it provides a live volume) it can allow ROMs to be accessed by emulators and managers

Disadvantages of deduplication

• RAM usage MAY be permanent, depending on deduplication software

 

Advantages of ZFS with deduplication and compression enabled

• Best of both worlds, space savings almost as high as 7zip and speeds almost as fast as deduplication alone
• Supported in practically every operating system (but Windows)
• Allows live access, so supported by emulators and ROM managers
• RAM usage can be moved to SSD, that’s several times cheaper and bigger.

Disadvantages of ZFS with deduplication and compression enabled

• Contrary to what deduplication alone can give, smaller blocks give less space savings and speed, and higher RAM and CPU usage.
• Not supported at all by Windows
• RAM usage is permanently 320 bytes per block

Comparing advantages vs disadvantages, ZFS seems the best solution for archival right now. While other filesystems may get same features in the future (btrfs for example), currently they don’t support them or are experimental or vendor-locked (btrfs is Linux-only)

Microsoft returns customers to the past, ReFS, the Resilient File System

Microsoft has just announced the name and features of his newest operating system, to be introduced with Windows 8 Server.

You can read the announcement in their official blog.

Here I will introduce some concepts and why this new filesystem, the Resilient File System (or ReFS) is not a next generation one, but a return to the past. Also I will compare current and "next generation" offerings from Microsoft (that's NTFS and ReFS) with competition from other filesystems (btrfs, HFS+, ZFS, UFS2, HAMMER).

Metadata: Data about the data, usually refers to timestamps, filename and the location of blocks where the data it's associated to resides.

Alternate Data Streams (ADS), Extended Attributes (xattrs): This is the name given to arbitrary data associated with the main data, mainly used for describing information about the data. It is extensively used by Mac OS X clients, as well as Windows Explorer. Some antivirus software also use this associated data to store information about last scanning of files. Please note that what UNIX/Linux and Macintosh call Extended Attributes are Alternate Data Streams in Microsoft nomenclature.

Extended Attributes (Microsoft nomenclature): These are a kind of alternate data streams limited to 64 Kbytes per file and consisting of name=value pairs. Introduced and extensively used by OS/2, they could be implemented as ADS or xattrs.

Resource Fork: Essential data for pre-Mac OS X applications and documents, contains data that is usually needed to describe the data part of the file (called Data Fork) or is the only data present for the file (like some document formats exclusive to Macintosh systems).

Finder Info: Essential metadata about files and folders for all Mac OS versions (up to and including X 10.7 Lion).

NOTE about the Resource Fork and the Finder Info as well as xattrs on Mac OS X: Traditionally Windows Server systems used ADS to store the Resource Fork and the Finder Info on volumes shared with Macintosh clients. The same is done for the Resource Fork, Finder Info and xattrs for all NTFS solutions existing on Mac OS X (Paragon NTFS, Tuxera NTFS and Apple's NTFS). When Mac OS X encounters a filesystem that does not support storing this information (like FAT32, or ReFS), it creates separate files for storing it, being "filename" the data fork, "._filename" the Resource Fork and Finder Info for "filename" and ".DS_Store" for the folder's Finder Info.

Object IDs, Catalog Node IDs, inode numbers: Is the name given in Windows, Macintosh and UNIX (incl. Linux) worlds to an unique identification inside a filesystem that corresponds to a single file, so in the case it is moved it can still be accessed using the same identification wherever it's new location (as long as it's inside the same volume).

Transparent file compression: This means the filesystem automatically compress the file data. While it is a slow operation when writing, is a lot faster while reading, and for files that are accessed mostly for reading and almost never for writing (that is, operating system and application files, but not documents) it can provide a speed up and alleviate space congestion, as the file takes really less space in disk (and so the disk reads it faster) and the decompression is still faster than the difference in reading the uncompressed data directly from disk.

Transparent file encryption: This allows files to be transparently encrypted with a key known only to a single user, so any other cannot access its contents. This is specially useful in shared volumes and servers where files can be stored in locations accessible by more people than those authorized to access a determined file.

Sparse files: A sparse file is a file where data is dynamically allocated. For example, an application can prevent it will need 1Gb for a file, and so it request that to the filesystem. If the filesystem supports sparse files the file will only take on disk the space that is really used, and not the whole reserved. Sparse files can also contain holes, so if a piece (cluster/block) of the file is never filled (is always 0s), it is never allocated with real space.

Symbolic links, Alias, Shadows: A symbolic link is a file that contains the location of another file. Mac OS calls them Alias and OS/2 called them Shadows.

Hard links: A hard links happens when to files are really, in disk, the same file, so the changes done to one affect the others, that can have a different location, name, timestamps, so on.

Differences between symbolic and hard links: Usually a symbolic link can point to files that reside in another volume, and they use not only the same blocks for metadata as the hard links, but also the blocks needed for the location of the real file (usually 4 or 8 Kbytes). Also if the real file is deleted, all the symbolic links associated with it become useless, while in the case of hard links, all of the links are "the real file" and it only disappears when all of the links are deleted.

Quotas: Quotas allow system administrators to specify that a user cannot exceed a determined allocation, so for example, if the system disk's is 2000Gb, the maximum each user can ever fill (his quota) is 100Gb, with 200Gb reserved to the administrator. This can usually be set per-user or per-group, and is really important in big corporations with hundreds of users sharing a same disk.

Data deduplication: Data deduplication is a feature that's offered by some filesystems and some third party enhancements for filesystems (it can be more or less implemented using hard links) where files that contain the same data are stored only once

Filesystem conversion: A feature almost never offered by any filesystem. It allows to automatically convert between two filesystems without losing any data in the process, neither having to copy/backup it to another drive and then again back. Was offered from HPFS->NTFS and is currently offered from FAT32->NTFS and from ext3/4->btfs.

Checksum: A checksum is a mathematically obtained number from a calculation over the data (or the metadata) that changes as long as any single bit (smallest piece of computer data possible) and is unique to that data. It allows us to find corruption and deduplicate identical files.

Journaled filesystem: A journaled filesystem is a filesystem where any change to the metadata is first done in the journal, which is then slowly applied to the real location of the metadata. If anything prevents the change to be written in the journal the old metadata still is present, and if anything prevents the change to be written in its real position the journal still says the change must be applied, so we never loss anything (we get the change, or we get nothing, no middle points, no corruption).

User defined transactions: Similar to how a journal works with metadata, a user defined transaction allows changes to the data be only finally committed when specifically marked as finished by an application.

Bootable: Means that the operating system can be installed and boots from this filesystem, taking full advantage of its features.

 

What is then offered by ReFS?:

ReFS adds copy-on-write allocation for metadata, that is, metadata is written to another location, and if time allows, the old location is deprecated. The same that a journal does without the journal.

It adds mandatory checksum to metadata and optional checksum for user data, all of them "scrubbed" to check if they are still valid, recovering from correct copies (if they exist) or sending a fail event to the system log (if they don't exist) on a failure.

Integrates with upper layer Windows 8's Storage Spaces to stripe (part, separate, piece) data for enhanced performance.

It removes support that was present in NTFS for: Alternate Data Streams (xattrs, Resource Forks and Finder Info) as well as OS/2's Extended Attributes, transparent file compression and encryption, sparse files, hard links, quotas, filesystem conversion and boot abilities.

Oh, and it removes support for storing separate "8.3" filenames for DOS network clients (no one cares).

It also allows for bigger volumes, files and directories.

 

Comparing with competing filesystems:

Please note in this comparison bootable means only on the native system, and while some features where added lately (like transactions to NTFS and journaling to HFS+) the comparison counts all of the currently as of 17th January 2011 implemented ones (stable or beta, but not just "planned"). Also partially implemented features (NTFS sparse files for example) are counted as a Yes.

	btrfs	ext4	HAMMER	HFS+	NTFS	ReFS	ZFS
Manufacturer	Oracle	Theodore Ts'o	Matthew Dillon	Apple	Microsoft	Microsoft	Sun (Oracle)
Native system	Linux	Linux	DragonFly BSD	Mac OS X	Windows	Windows	Solaris
Year of introduction	2007	2006	2008	1998	1993	2012	2005
ADS/xattrs	Yes	Yes	Yes	Yes	Yes	No	Yes
Object Ids	Yes	Yes	Yes	Yes	Yes	No	Yes
File compression	Yes	No	No	Yes	Yes	No	Yes
File encryption	Planned	No	No	No	Yes	No	Yes
Sparse files	Yes	Yes	Yes	No	Yes	No	Yes
Symbolic links	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Hard links	Yes	Yes	Yes	Yes	Yes	No	Yes
Quotas	Yes	Yes	Yes	Yes	Yes	No	Yes
Deduplication	Yes	No	Yes	No	No	No	Yes
Conversion	Yes	Yes	No	No	Yes	No	No
Metadata checksum	Yes	No	Yes	No	No	Yes	Yes
Data checksum	Yes	No	Yes	No	No	Yes	Yes
Transactions	Yes	No	No	No	Yes	No	Yes
Journaled	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Bootable	Yes	Yes	Yes	Yes	Yes	No	Yes
Integrated data striping and recovery from failure	Planned	No	No	With XSAN	No	With Storage Spaces	With Zpools
Max file size	16 EiB	16 TiB	1 EiB	8 EiB	16 EiB	16 EiB	16 EiB
Max volume size	16 EiB	1 EiB	1 EiB	8 EiB	16 EiB	16 EiB	16 EiB

Conclusion:

Microsoft's new "Resilient File System" is nothing that cannot be implemented on top of NTFS. It's just a huge lost of features that's giving nothing new and taking a lot of features that users, developers and administrators use, plan to use, love and want.

When I read the official MSDN blog post about it I though "yeah Microsoft good joke" while I checked the calendar to see if today was April's Fool. Seeing it's not, my conclusion is that I will stay with NTFS as long as possible and if I somehow want the only interesting feature about ReFS (resilience) I will move to a serious filesystem like ZFS, btrfs or even, HFS+.

Comparison of NTFS drivers for Mac OS X

In this article I will compare the three current NTFS drivers for Mac OS X Lion as well as give a little definition about filesystems and about the major ones that can be used for interoperatibility between Windows and Mac OS X.

What's a filesystem?

A filesystem is the way that files are organized in a disk. Every major operating system comes with its own one, incompatible with the others. Windows comes with NTFS, Mac OS X comes with HFS+ and Linux with ext4.

This articles centers on comparing how to interchange files between Windows and Mac OS X so only the filesystems that can be commonly used by both will be discussed. Those are NTFS, FAT32, exFAT and HFS+.

There is also another filesystem, UDF (from Universal Disk Format) that was created to be universally compatible with all operating systems. It's the one used by DVD and Bluray discs, but its usage outside this kind of disks (like for example in hard disks or flash drives) requires deep knowledge of the operating system, as for example, Windows won't allow to read a UDF formatted hard disk unless tricked to do so, and Mac OS X doesn't offer an option to format anything this way easily.

What's FAT32?

FAT32 (also known as File Allocation Table, 32 bit) is an evolution of the original MS-DOS filesystem, introduced in 1995 with Windows 95 OSR2. It's a fragile filesystem with severe limitations (maximum filesize is 4Gb, Windows cannot format drives bigger than 32Gb on it, it does not support user security/permissions, fragments heavily, etc) but however almost every operating system out there can read and write it perfectly. It's also used extensively in any kind of device using mass storage (phones, digital cameras, DVRs, etc).

What's NTFS?

NTFS (also known as NT File System) is the native filesystem introduced with Windows NT. It's a very clever filesystem, that introduces compression, alternate data streams, security, etc. However its specification are a secret, and just only in the recent years enough of it has been reverse engineered to allow drivers to fully support it.

What's HFS+?

HFS+ (also known as Hierarchical File System Plus and Mac OS Extended) is the native Mac OS X filesystem, first introduced with Mac OS 8.1. It supports features unique to Mac OS like FinderInfo and the Resource Fork. It added new features from UNIX (permissions, links) and alternate data streams, security, etc, over its predecesor, HFS.

What's exFAT?

exFAT (also known as Extended File Allocation Table) is a new filesystem designed for Microsoft to be used on flash drives. It overcomes the most important FAT32 limitations without the complexity of NTFS, and is mandatory to any device claiming to support the SDXC and Memory Stick XC cards to support it. However, its specification is secret, patented and require a paid license to implement a driver (I still need confirmation from a lawyer that this applies in the EU). Currently only Mac OS X and Windows Vista or 7 support it out-of-the-box.

A little filesystem terms glossary

Alternate Data Stream (aka Fork): An ADS or Fork is a piece of data that comes alongside the file's data, but is independent of it. It should not be confused with metadata, that can be stored inside the data, and can be stored as ADSs. It can contain things like image thumbnails, URL where the file was downloaded from, the instant messaging contact's name who send it, so on.

Resource Fork: The Resource Fork is a special ADS that Mac OS previously depended extensively on (not anymore on Mac OS X, but files from previous versions and its applications can be simply broken if they lose it). It's specifically designed as a database of data types, like application pieces, rich text style, so on. NTFS stores it as an ADS called "AFP_Resource", and HFS+ stores it independently. Mac OS X stores them in filesystems that don't support it in separate files prepending "._" to the filename.

Extended Attributes (aka EA or xattr): Extended attributes are small alternate data streams usually limited in size. Its concept was first introduced in OS/2 to store things like the file icon, the filename, etc. FAT versions precious to FAT32 supports them natively (only on OS/2 and Windows NT), NTFS stores them as ADSs and Mac OS X 10.4 calls its ADSs so (but it doesn't limit the size). They are also supported by most Linux filesystems.

Finder Info: Finder Info is a collection of attributes that were required by Mac OS versions previous to Mac OS X. They store the icon positions in a folder, provide support for the Aliases, and stores which application created the file and what kind of file is it (Mac OS previous to X required this to know that a JPEG is a JPEG and it should be opened with Photoshop, X can live without this). NTFS stores it as an ADS called "AFP_AfpInfo".

Alias: The Alias is a Mac OS feature, where it creates a shortcut to a file that works even if the file is still moved to another place (it stores the file and volume unique identifiers to always find the file wherever it is). They are implemented using the resource fork, and OS/2 provided a similar feature called Shadow using EAs.

Symbolic link: A UNIX feature, a file is created that shorcuts to the exact location of another file, so if you move the original file the link is broken.

Hard link: A UNIX feature where a file information is duplicated but not its content. That's, there are two files that are really just the same (you modify one, both get modified), and until all hard links of the same file are not deleted the file isn't either.

The tests, explained

In this article I'll test the three NTFS offerings for Mac OS X that are currently available and supported under Mac OS X Lion, as well as the exFAT, FAT32 and HFS+ filesystems, for features and speed.

While Mac OS X Lion comes with an NTFS driver of itself, designed in-house by Apple, opensource and programmed by the guy that did most of the NTFS reverse engineering, it's as of today, unsupported. Drives are not writeable unless you do some command line tricks (that I will not explain in this article), and even if you do this nothing is warranted to work. If it makes your hard disk burn, or you lose all your data, or an alien abducts you for using it, Apple will say "we told you it's not supported". However, it passed all my tests without a single corruption, so, at your own risk!!!

Paragon Software offers us, among other filesystem drivers, with Paragon NTFS for Mac, compatible with Lion from 9.0, however they do some things I don't like. They exaggerated their features comparison saying that the Apple driver doesn't support NTFS features it does however supported even before Paragon did.

Finally Tuxera offers us Tuxera NTFS for Mac, a commercial solution that evolved from the opensource NTFS-3G widely used for NTFS support on Linux. They still support PowerPC macs (my PowerBook G4 is happy) and were the first to support storing the Resource Fork how NTFS is designed to store them, inside an ADS and not a "._" file.

For the test I basically did two things. The first was creating a separate folder inside the volume, copying different files. One is a Mac OS 7 application, that heavily relies on its Resource Fork (it's data being 0 bytes indeed) and its Finder Info. Then I copied a disk image file that contains four different Mac OS X extended attributes. Then I created some plain text files with names that are not supported by Windows (but they are supported by NTFS, see below). Then I created a symbolic link of a JPEG file and a hard link of it.

Finally for the speed tests I used Bonnie++, with this command line: bonnie++ -d /Volumes/Test/test/ -s 4095 -n 100 -m fat32 -u claunia -f -b

Test machine is a MacBookPro5,4 with 4Gb of DDR1066 RAM, Mac OS X Lion 10.7.2 (11C26), and a Samsung 2.5" IDE hard disk inside an Alcor Micro USB 2.0 case. Tests were done with all applications closed and through a network shell so the GUI is in background and not consuming important foreground memory and cycles.

Note on filenames

NTFS supports ANY character in a filename but the NULL one (byte 0x00). Windows however does not support characters from 0x00 to 0x31, "<", ">", ":", " " " (no spaces, double quotes character), "/", "\", "|", "?" and "*". It also doesn't support files that are end on a space or a dot. More information here. While those files can exist in NTFS volumes, no Windows application will ever be able to do anything with them (access, rename, move, copy, delete, open, close, say it, won't work).

UNIX system supports ANY but NULL and "/".

Mac OS supports ANY but NULL and ":". Mac OS X stores files containing "/" as ":" in HFS+ but shows it to you as "/" (tricked!). Also Mac OS includes the Apple logo character "" (you won't see it unless you're reading this article under any Mac OS version), that's unsupported anywhere else.

Microsoft designed Windows NT to be interoperable in networks containing Mac OS clients, and to serve files to them (with the now deprecated Services for Macintosh product), so they created a way to store filenames unsupported by Windows on NTFS, so Windows will be able to access ithem, and Mac OS will be able also, showing the correct filename to Mac OS network clients, and a fake one to Windows machines. It's described here.

Feature comparison

	HFS+	Apple's NTFS	Paragon NTFS	Tuxera NTFS	FAT32	exFAT
Resource fork	Supported	Supported	Partial	Supported	Partial	Partial
Extended attributes	Supported	Supported	Partial	Supported	Partial	Partial
FinderInfo	Supported	Supported	Partial	Supported	Partial	Partial
Hard links	Supported	Supported	Supported	Supported	Unsupported	Unsupported
Symbolic links	Supported	Partial	Supported	Supported	Unsupported	Unsupported
Filenames	Supported	Supported	Supported	Supported	Supported	Unsupported
Cost	Free	Free	$19.95 / version	$36.25	Free	Free

Partial support means it's stored in "._" files, supported that it's stored as the filesystem designed them to be stored, and unsupported that they simply, won't work.

On links there is a curious difference between the three NTFS implementations.

The Tuxera symbolinc link seems to use the POSIX interface of NTFS, and it's not directly accepted by Windows explorer. However, it's the way NTFS was designed to. Paragon does the same. And Services for UNIX will recognize it as a symbolic link.

Apple's implementation however stores it a a simple file containing the path to the destination, and neither Windows explorer or Services for UNIX recognize it as a link.

Tests

So the Bonnie++ tests outputs shows the following.

Version 1.93c			Sequential Output						Sequential Input				Random Seeks			Sequential Create						Random Create
Concurrency		Size	Per Char		Block		Rewrite		Per Char		Block				Num Files	Create		Read		Delete		Create		Read		Delete
			K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	/sec	% CPU		/sec	% CPU	/sec	% CPU	/sec	% CPU	/sec	% CPU	/sec	% CPU	/sec	% CPU
HFS+	1	4095M			24722	4	11657	2			35292	2	274.3	3	100	18243	77	44878	99	9321	58	253	2	52737	99	107	1
HFS+	Latency				344ms		518ms				35023us		257ms		Latency	99931us		1438us		16793us		858ms		1187us		1066ms
Apple's NTFS	1	4095M			24663	6	12276	2			35382	3	288.9	17	100	401	60	1558	22	839	16	327	68	172	2	71	1
Apple's NTFS	Latency				425ms		619ms				58872us		291ms		Latency	99838us		73278us		346ms		468ms		46196us		692ms
Paragon NTFS	1	4095M			24760	4	11675	2			35400	3	295.0	5	100	206	81	3203	33	455	12	186	77	275	3	42	1
Paragon NTFS	Latency				296ms		530ms				22544us		161ms		Latency	587ms		36047us		119ms		424ms		76842us		596ms
Tuxera NTFS	1	4095M			20470	5	11094	6			31946	4	170.2	4	100	410	68	3875	18	1120	15	95	19	22	0	12	0
Tuxera NTFS	Latency				180ms		382ms				193ms		4658ms		Latency	79166us		138ms		1395ms		36988ms		37577ms		848ms
FAT32	1	4095M			24342	2	11708	2			35293	3	264.3	7	100	56	89	440	93	119	47	69	87	619	90	79	43
FAT32	Latency				528ms		586ms				74162us		3436ms		Latency	286ms		34503us		343ms		368ms		20362us		440ms
exFAT	1	4095M			24665	2	11305	2			35095	3	254.2	23	100	51	79	305	85	69	53	49	79	399	80	56	50
exFAT	Latency				363ms		514ms				262ms		1456ms		Latency	323ms		69381us		333ms		356ms		55743us		443ms

Lots of data to interpret so let's put it in pretty graphs.

This is the bare speed test. Here all the filesystems work quite at the same speed, with a little slowdown on Tuxera NTFS when writing or reading files. Not too much to discuss it, they are all as fast so it's just your choice of flavour.

Things get different when you're treating with folders filled with files. When you copy a folder full of files (say, the iTunes folder, your photo library, so on) you're creating it. When you open it in the Finder, you're reading it. And when you empty the trash you're deleting it.

So this matters more than a single file speed, shown in previous test, because you will do these operations more commonly.

Clearly, HFS+ comes to a huge advantage here.

Of the three NTFS offerings, Apple's one clearly needs more optimization, and Tuxera's gets the lead, being the fastest when multiple file operations are done. Nowhere near of HFS+ however.

FAT32 and exFAT clearly show their design flaws here, being desperately slow in multiple files operation.

Conclusion

This is not a choice that can be simply left to speed tests, it's a complex matter.

If you have a Mac computer, you have bootcamp, and it includes HFS+ drivers for Windows, so clearly this should be the choice isn't it?

However, your friends, university, college, so on, probably won't, cannot, don't want, to have this driver installed. So you're forced to choose between one of the natively supported by Windows filesystems (NTFS, exFAT or FAT32).

FAT32 is not an option, it's not just slow, you will never be able to store any file bigger than 4Gb, that's it, forget about it, simply, erase from your mind, let it be only used by your digital camera and bye bye.

exFAT may seem the future, however it's just an evolution of FAT32 (supports files bigger than 4Gb, but anyway, same internal design), but it's still slow. However it's free to use on Mac OS X and if you don't care about the annoying "._" files that will be created all the way, just use it.

On NTFS, clearly Apple should end it and make it easy to use, integrated, free, writeable. Ok, we've been waiting years for this to come, so sit and wait, and on the meantime decide between Paragon and Tuxera.

Tuxera seems more expensive (but you pay just once per all versions in the future), slower on single files, but handles multiple files faster, and stores things the way they should be stored.

Paragon seems cheaper (you end paying more as every release you pay again), faster on single files, slower on multiple files, and creates the annoying "._" files everywhere.

So Tuxera and Paragon are quite equalled, it's up to you which one to chose.

Personally, I chose Tuxera, because I want to have the confidence that what I write on a NTFS drive using Mac OS X will be correctly identified by Windows applications as what they are (not only Services for Macintosh, Windows explorer recognize AFP_* ADS and inform you will lose them when moving to a filesystem that does not support them) without the annoying "._" files.