In December 2010, Engineering Computing moved to a new file server.
ECFile1 is now a 64 bit, dual quad core computer with 72 GB of RAM, running FreeBSD with Sun’s ZFS file system and serving CIFS/SMB (using Samba) to primarily Windows clients and NFS to Unix servers.
ZFS offers a number of advantages over other file systems like BSD’s UFS or Linux’s EFS. Thanks to its inclusion in the OpenSolaris project (which has been shut down since), the open source community had access to the complete source code to this revolutionary file system.
ZFS allows almost unlimited growth – well, up to 16 exabytes or 2^64 bytes per file. That’s about 100,000,000,000,000,000,000 bytes.
ZFS also focuses on data integrity with technology to detect problems better than standard RAID, and to actually heal the data. It actually works better with non-hardware-based RAID, where ZFS can test the data from multiple disks against a pre-computed checksum to determine the most reasonable results.
Modern disk drives use approximately 25% of their disk surfaces for error detection codes. Enterprise SAS drives have an undetected error rate of one in every 10^16 bits. Without the advanced coding of ZFS, enterprise systems were experiencing systems reliability and data errors due to the disk error rates.
Another great feature of ZFS is the ability to use solid state devices as intermediary devices between the disk drives and the computer to speed up both reads and writes – significantly reducing the load on the physical disks while gaining the speed advantage. Writes are typically buffered for about 30 seconds, and then flushed to disk at once. ECfile1 uses two SSDs.
Currently ECFile1 has 36 SAS disks, some are hot spares. We have 12 terabytes of space dedicated to user storage, but only 20% of it is in use for user data, and some additional space is used for snapshots. When we put it in place, we doubled most users’ disk quotas. We should be able to increase them again.
An advantage of a copy-on-write system like ZFS is that old data can be retained. This allows us to have snapshots – allowing us to recover files from previous states. When doing backups, we take a snapshot of the system (it takes a second or two), and then back up the snapshot.
ECFile1 is connected through a 10 gigabit per second connection to the Engineering network.
Backups go to a second similar box (but with SATA drives) in a different building. In the event of a catastrophic failure (like a fire taking out all the disk drives on ECFile1), the other box could easily be brought up as ECFile1 with only a few hours of data loss.
More frequent snapshots could move the catastrophic data loss down to an hour or so. And work on clustering might eliminate it entirely. These are all options for the future.
For now, we have a fast, reliable file server.
