Donate-large

If you like this blog, please make a little donation.

It's a secure process, and with your generosity I will be able to review more hardware and software.

Just click the "Donate" button below and follow the easy instructions, and I will thank you eternally.

Tuesday, August 30, 2011

Comparison of NTFS drivers for Mac OS X

In this article I will compare the three current NTFS drivers for Mac OS X Lion as well as give a little definition about filesystems and about the major ones that can be used for interoperatibility between Windows and Mac OS X.

What's a filesystem?

A filesystem is the way that files are organized in a disk. Every major operating system comes with its own one, incompatible with the others. Windows comes with NTFS, Mac OS X comes with HFS+ and Linux with ext4.

This articles centers on comparing how to interchange files between Windows and Mac OS X so only the filesystems that can be commonly used by both will be discussed. Those are NTFS, FAT32, exFAT and HFS+.

There is also another filesystem, UDF (from Universal Disk Format) that was created to be universally compatible with all operating systems. It's the one used by DVD and Bluray discs, but its usage outside this kind of disks (like for example in hard disks or flash drives) requires deep knowledge of the operating system, as for example, Windows won't allow to read a UDF formatted hard disk unless tricked to do so, and Mac OS X doesn't offer an option to format anything this way easily.

What's FAT32?

FAT32 (also known as File Allocation Table, 32 bit) is an evolution of the original MS-DOS filesystem, introduced in 1995 with Windows 95 OSR2. It's a fragile filesystem with severe limitations (maximum filesize is 4Gb, Windows cannot format drives bigger than 32Gb on it, it does not support user security/permissions, fragments heavily, etc) but however almost every operating system out there can read and write it perfectly. It's also used extensively in any kind of device using mass storage (phones, digital cameras, DVRs, etc).

What's NTFS?

NTFS (also known as NT File System) is the native filesystem introduced with Windows NT. It's a very clever filesystem, that introduces compression, alternate data streams, security, etc. However its specification are a secret, and just only in the recent years enough of it has been reverse engineered to allow drivers to fully support it.

What's HFS+?

HFS+ (also known as Hierarchical File System Plus and Mac OS Extended) is the native Mac OS X filesystem, first introduced with Mac OS 8.1. It supports features unique to Mac OS like FinderInfo and the Resource Fork. It added new features from UNIX (permissions, links) and alternate data streams, security, etc, over its predecesor, HFS.

What's exFAT?

exFAT (also known as Extended File Allocation Table) is a new filesystem designed for Microsoft to be used on flash drives. It overcomes the most important FAT32 limitations without the complexity of NTFS, and is mandatory to any device claiming to support the SDXC and Memory Stick XC cards to support it. However, its specification is secret, patented and require a paid license to implement a driver (I still need confirmation from a lawyer that this applies in the EU). Currently only Mac OS X and Windows Vista or 7 support it out-of-the-box.

A little filesystem terms glossary

Alternate Data Stream (aka Fork): An ADS or Fork is a piece of data that comes alongside the file's data, but is independent of it. It should not be confused with metadata, that can be stored inside the data, and can be stored as ADSs. It can contain things like image thumbnails, URL where the file was downloaded from, the instant messaging contact's name who send it, so on.

Resource Fork: The Resource Fork is a special ADS that Mac OS previously depended extensively on (not anymore on Mac OS X, but files from previous versions and its applications can be simply broken if they lose it). It's specifically designed as a database of data types, like application pieces, rich text style, so on. NTFS stores it as an ADS called "AFP_Resource", and HFS+ stores it independently. Mac OS X stores them in filesystems that don't support it in separate files prepending "._" to the filename.

Extended Attributes (aka EA or xattr): Extended attributes are small alternate data streams usually limited in size. Its concept was first introduced in OS/2 to store things like the file icon, the filename, etc. FAT versions precious to FAT32 supports them natively (only on OS/2 and Windows NT), NTFS stores them as ADSs and Mac OS X 10.4 calls its ADSs so (but it doesn't limit the size). They are also supported by most Linux filesystems.

Finder Info: Finder Info is a collection of attributes that were required by Mac OS versions previous to Mac OS X. They store the icon positions in a folder, provide support for the Aliases, and stores which application created the file and what kind of file is it (Mac OS previous to X required this to know that a JPEG is a JPEG and it should be opened with Photoshop, X can live without this). NTFS stores it as an ADS called "AFP_AfpInfo".

Alias: The Alias is a Mac OS feature, where it creates a shortcut to a file that works even if the file is still moved to another place (it stores the file and volume unique identifiers to always find the file wherever it is). They are implemented using the resource fork, and OS/2 provided a similar feature called Shadow using EAs.

Symbolic link: A UNIX feature, a file is created that shorcuts to the exact location of another file, so if you move the original file the link is broken.

Hard link: A UNIX feature where a file information is duplicated but not its content. That's, there are two files that are really just the same (you modify one, both get modified), and until all hard links of the same file are not deleted the file isn't either.

The tests, explained

In this article I'll test the three NTFS offerings for Mac OS X that are currently available and supported under Mac OS X Lion, as well as the exFAT, FAT32 and HFS+ filesystems, for features and speed.

While Mac OS X Lion comes with an NTFS driver of itself, designed in-house by Apple, opensource and programmed by the guy that did most of the NTFS reverse engineering, it's as of today, unsupported. Drives are not writeable unless you do some command line tricks (that I will not explain in this article), and even if you do this nothing is warranted to work. If it makes your hard disk burn, or you lose all your data, or an alien abducts you for using it, Apple will say "we told you it's not supported". However, it passed all my tests without a single corruption, so, at your own risk!!!

Paragon Software offers us, among other filesystem drivers, with Paragon NTFS for Mac, compatible with Lion from 9.0, however they do some things I don't like. They exaggerated their features comparison saying that the Apple driver doesn't support NTFS features it does however supported even before Paragon did.

Finally Tuxera offers us Tuxera NTFS for Mac, a commercial solution that evolved from the opensource NTFS-3G widely used for NTFS support on Linux. They still support PowerPC macs (my PowerBook G4 is happy) and were the first to support storing the Resource Fork how NTFS is designed to store them, inside an ADS and not a "._" file.

For the test I basically did two things. The first was creating a separate folder inside the volume, copying different files. One is a Mac OS 7 application, that heavily relies on its Resource Fork (it's data being 0 bytes indeed) and its Finder Info. Then I copied a disk image file that contains four different Mac OS X extended attributes. Then I created some plain text files with names that are not supported by Windows (but they are supported by NTFS, see below). Then I created a symbolic link of a JPEG file and a hard link of it.

Finally for the speed tests I used Bonnie++, with this command line: bonnie++ -d /Volumes/Test/test/ -s 4095 -n 100 -m fat32 -u claunia -f -b

Test machine is a MacBookPro5,4 with 4Gb of DDR1066 RAM, Mac OS X Lion 10.7.2 (11C26), and a Samsung 2.5" IDE hard disk inside an Alcor Micro USB 2.0 case. Tests were done with all applications closed and through a network shell so the GUI is in background and not consuming important foreground memory and cycles.

Note on filenames

NTFS supports ANY character in a filename but the NULL one (byte 0x00). Windows however does not support characters from 0x00 to 0x31, "<", ">", ":", " " " (no spaces, double quotes character), "/", "\", "|", "?" and "*". It also doesn't support files that are end on a space or a dot. More information here. While those files can exist in NTFS volumes, no Windows application will ever be able to do anything with them (access, rename, move, copy, delete, open, close, say it, won't work).

UNIX system supports ANY but NULL and "/".

Mac OS supports ANY but NULL and ":". Mac OS X stores files containing "/" as ":" in HFS+ but shows it to you as "/" (tricked!). Also Mac OS includes the Apple logo character "" (you won't see it unless you're reading this article under any Mac OS version), that's unsupported anywhere else.

Microsoft designed Windows NT to be interoperable in networks containing Mac OS clients, and to serve files to them (with the now deprecated Services for Macintosh product), so they created a way to store filenames unsupported by Windows on NTFS, so Windows will be able to access ithem, and Mac OS will be able also, showing the correct filename to Mac OS network clients, and a fake one to Windows machines. It's described here.

Feature comparison

 

HFS+

Apple's NTFS

Paragon NTFS

Tuxera NTFS

FAT32

exFAT

Resource fork

Supported

Supported

Partial

Supported

Partial

Partial

Extended attributes

Supported

Supported

Partial

Supported

Partial

Partial

FinderInfo

Supported

Supported

Partial

Supported

Partial

Partial

Hard links

Supported

Supported

Supported

Supported

Unsupported

Unsupported

Symbolic links

Supported

Partial

Supported

Supported

Unsupported

Unsupported

Filenames

Supported

Supported

Supported

Supported

Supported

Unsupported

Cost

Free

Free

$19.95 / version

$36.25

Free

Free

Partial support means it's stored in "._" files, supported that it's stored as the filesystem designed them to be stored, and unsupported that they simply, won't work.

On links there is a curious difference between the three NTFS implementations.

The Tuxera symbolinc link seems to use the POSIX interface of NTFS, and it's not directly accepted by Windows explorer. However, it's the way NTFS was designed to. Paragon does the same. And Services for UNIX will recognize it as a symbolic link.

Apple's implementation however stores it a a simple file containing the path to the destination, and neither Windows explorer or Services for UNIX recognize it as a link.

Tests

So the Bonnie++ tests outputs shows the following.

Version 1.93c Sequential Output Sequential Input Random
Seeks
Sequential Create Random Create
Concurrency Size Per Char Block Rewrite Per Char Block Num Files Create Read Delete Create Read Delete
K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU K/sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU /sec % CPU
HFS+ 1 4095M 24722 4 11657 2 35292 2 274.3 3 100 18243 77 44878 99 9321 58 253 2 52737 99 107 1
HFS+ Latency 344ms 518ms 35023us 257ms Latency 99931us 1438us 16793us 858ms 1187us 1066ms
Apple's NTFS 1 4095M 24663 6 12276 2 35382 3 288.9 17 100 401 60 1558 22 839 16 327 68 172 2 71 1
Apple's NTFS Latency 425ms 619ms 58872us 291ms Latency 99838us 73278us 346ms 468ms 46196us 692ms
Paragon NTFS 1 4095M 24760 4 11675 2 35400 3 295.0 5 100 206 81 3203 33 455 12 186 77 275 3 42 1
Paragon NTFS Latency 296ms 530ms 22544us 161ms Latency 587ms 36047us 119ms 424ms 76842us 596ms
Tuxera NTFS 1 4095M 20470 5 11094 6 31946 4 170.2 4 100 410 68 3875 18 1120 15 95 19 22 0 12 0
Tuxera NTFS Latency 180ms 382ms 193ms 4658ms Latency 79166us 138ms 1395ms 36988ms 37577ms 848ms
FAT32 1 4095M 24342 2 11708 2 35293 3 264.3 7 100 56 89 440 93 119 47 69 87 619 90 79 43
FAT32 Latency 528ms 586ms 74162us 3436ms Latency 286ms 34503us 343ms 368ms 20362us 440ms
exFAT 1 4095M 24665 2 11305 2 35095 3 254.2 23 100 51 79 305 85 69 53 49 79 399 80 56 50
exFAT Latency 363ms 514ms 262ms 1456ms Latency 323ms 69381us 333ms 356ms 55743us 443ms

Lots of data to interpret so let's put it in pretty graphs.

Speedtest

This is the bare speed test. Here all the filesystems work quite at the same speed, with a little slowdown on Tuxera NTFS when writing or reading files. Not too much to discuss it, they are all as fast so it's just your choice of flavour.

Filestest

Things get different when you're treating with folders filled with files. When you copy a folder full of files (say, the iTunes folder, your photo library, so on) you're creating it. When you open it in the Finder, you're reading it. And when you empty the trash you're deleting it.

So this matters more than a single file speed, shown in previous test, because you will do these operations more commonly.

Clearly, HFS+ comes to a huge advantage here.

Of the three NTFS offerings, Apple's one clearly needs more optimization, and Tuxera's gets the lead, being the fastest when multiple file operations are done. Nowhere near of HFS+ however.

FAT32 and exFAT clearly show their design flaws here, being desperately slow in multiple files operation.

Conclusion

This is not a choice that can be simply left to speed tests, it's a complex matter.

If you have a Mac computer, you have bootcamp, and it includes HFS+ drivers for Windows, so clearly this should be the choice isn't it?

However, your friends, university, college, so on, probably won't, cannot, don't want, to have this driver installed. So you're forced to choose between one of the natively supported by Windows filesystems (NTFS, exFAT or FAT32).

FAT32 is not an option, it's not just slow, you will never be able to store any file bigger than 4Gb, that's it, forget about it, simply, erase from your mind, let it be only used by your digital camera and bye bye.

exFAT may seem the future, however it's just an evolution of FAT32 (supports files bigger than 4Gb, but anyway, same internal design), but it's still slow. However it's free to use on Mac OS X and if you don't care about the annoying "._" files that will be created all the way, just use it.

On NTFS, clearly Apple should end it and make it easy to use, integrated, free, writeable. Ok, we've been waiting years for this to come, so sit and wait, and on the meantime decide between Paragon and Tuxera.

Tuxera seems more expensive (but you pay just once per all versions in the future), slower on single files, but handles multiple files faster, and stores things the way they should be stored.

Paragon seems cheaper (you end paying more as every release you pay again), faster on single files, slower on multiple files, and creates the annoying "._" files everywhere.

So Tuxera and Paragon are quite equalled, it's up to you which one to chose.

Personally, I chose Tuxera, because I want to have the confidence that what I write on a NTFS drive using Mac OS X will be correctly identified by Windows applications as what they are (not only Services for Macintosh, Windows explorer recognize AFP_* ADS and inform you will lose them when moving to a filesystem that does not support them) without the annoying "._" files.

9 comments:

  1. What versions of Tuxera and Paragon were you testing? I tried both of them and found no differences except in the long run that they are quite sluggish.

    I tested mine with Tuxera 2011.4.2 and Paragon 9.5 on Lion 10.7.2 by the way.

    ReplyDelete
  2. Thanks Tsuki.
    This was really helpful.
    First place I found that clear explained why OSX aliases won't copy to NTFS drives.
    I have a Win2008 File Server with spanned (aka dynamic) drives that I was going to use to backup my OSX machines but the alias issue is the only thing preventing me from doing that.

    ReplyDelete
  3. This really is helpful. I would love to see such a clear detailed explanation and comparison of HFS+ file system components for windows 7. I use bootcamp, but I would like to rely mostly on my HFS Mac file system, since I prefer to stay in Mac OS X as much as possible.

    Cheers -

    ReplyDelete
  4. thanks for the detailed analysis and explanation on various NTFS drivers for MAC. I recently moved to MAC but still had to work with external drives which has NTFS for backup/old-archive.

    Which version of Paragon and Tuxera did you use for your benchmark? It appears that both have released newer versions recently, so wondering if any of the analysis changes.

    Thanks again.

    ReplyDelete
  5. I have tested both Tuxera 2011.4.2 and Paragon 8.0 / 9.5 with a Seagate GoFlex drive (1.5GB, 5400rpm) and found that with the Firewire 800 interface, Paragon is *twice* as fast as Tuxera. I ran tests on Mac OS X Lion 10.7.2, both by copying via the Finder (single large files and multiple smaller files), and with the following "raw" tests via the command line:

    echo "Writing 1GB file...."
    time dd if=/dev/zero bs=1024k of=tstfile count=1024
    echo "Reading 1GB file...."
    time dd if=tstfile bs=1024k of=/dev/null

    I get about 53MB/s write and 67MB/s read with Paragon, and 25MB/s write, 37MB/s read with Tuxera. This is about twice as fas. With a USB 2.0 interface on the same drive, Paragon gets 28MB/s write + 37MB/s read, Tuxera get 20MB/s write + 28MB/s read. Still quite noticeable.

    I found similar differences with another NTFS USB2.0 drive.

    I already bought Tuxera before discovering Paragon, but in my experience Paragon is a lot faster. I haven't tried the drive natively with HFS+ as I don't wish to reformat it.

    BTW, the SeaGate GoFlex drive actually ships with a free Paragon NTFS 8.0 driver, but this only works for SeaGate drives. My other NTFS drives appear as read-only, so I bought Paragon 9.5 which works with all drives.

    ReplyDelete
    Replies
    1. That huge difference is really interesting, however, your test method is flawed (specially using /dev/zero) as it's being all handled by the cache.

      When you copy a file, it gets cached (and if the source file is /dev/zero easier yet), and copying the same file from the same source a second time is several orders of magnitude faster.

      Filesystem benchmarking tools are specially designed to prevent caching as much as possible.

      However, both Paragon and Tuxera have created new versions of their software (Paragon specially correcting the behavior with supported features) and I will update this comparison with the newer versions as soon as possible, with a faster drive and interface.

      Delete
  6. Thank you for giving me the unique features of HFS+. I've yet to install the genuine OS X ... and wonder if I should.

    Legal use of Mac OS X is also to non-Mac boxes. Genuine Ms Win7 NTFS may also be installed onto any hard disk or partition. I'm surprised that you seem to not realize this.

    I multi-boot my HP Pavilion desktop-replacement notebook (2x 750 GB HDDs, etc) with many 'buntu op sys, including an OS X imitation, as well as Win7-64. All my data is on MS NTFS-compressed partitions, which is reliably & quickly used by all my op systems.

    ReplyDelete
  7. Hi! Thanks so much for your post! I bought an external drive (for backups) for my wife, but she wanted it to be readable in PCs too. Finally, I chose NTFS/Tuxera mix. Let's see how it goes...

    ReplyDelete
  8. Update: everything going great with the mix: Tuxera / NTFS / Western Digital My Passport 1 TB USB 3.0 Portable Hard Drive / Mac.

    ReplyDelete