Stop Letting Your RAID Card Ruin Your ZFS Pool

Meta description: Learn why standard RAID cards break ZFS and how to flash IT Mode firmware onto LSI controllers for direct disk access and maximum data integrity.

Building a high-performance storage server using ZFS requires a fundamental shift in how you view storage controllers. For decades, hardware RAID was the gold standard for data redundancy, but in the modern era of software-defined storage, these expensive cards often become a liability. If you are reading this guide, you likely understand that ZFS demands direct, unmediated access to every physical disk in your array to perform its checksumming and self-healing magic. Standard RAID firmware introduces an abstraction layer that can mask disk errors, delay write operations, and ultimately lead to catastrophic pool failure. This guide is aimed at advanced users who are comfortable working in a UEFI shell and understand the risks of low-level hardware modification.

To solve the conflict between hardware and software management, we must transition our controllers into IT Mode. This firmware state strips away the RAID logic of the card, effectively turning a sophisticated RAID controller into a simple Host Bus Adapter (HBA). By flashing your LSI-based card to IT Mode, you ensure that ZFS sees the “bare metal” of your drives. This enables features like S.M.A.R.T. monitoring and native command queuing that are often blocked by traditional RAID firmware. Without this step, your high-integrity filesystem is essentially building its house on a foundation of sand, relying on a third-party controller to tell the truth about the state of your data.

The Fundamental Conflict Between Hardware RAID and ZFS

The primary reason to avoid hardware RAID when using ZFS is the “Write Hole” and the way caching is handled. Most hardware RAID cards utilize a volatile cache to speed up write operations. If power is lost before that cache is flushed to the physical disks, the data on the disks can become inconsistent. While high-end cards use Battery Backed Write Cache (BBWC) to mitigate this, ZFS prefers to manage its own Intent Log (ZIL) on stable storage. When ZFS issues a flush command, it expects the data to be physically on the platter or NAND. A RAID card might lie to the operating system, reporting a successful write while the data is still sitting in the card’s RAM.

Furthermore, ZFS is designed to detect and repair silent data corruption. It does this by storing checksums for every block of data. If a block is read and the checksum doesn’t match, ZFS uses redundancy (like RAID-Z) to reconstruct the correct data and overwrite the corrupt block. Hardware RAID cards often attempt their own error recovery. If a disk has a bad sector, the RAID card might hang the entire bus while it tries to re-read the sector dozens of times. ZFS would rather the disk simply report a failure quickly so that it can handle the recovery at the filesystem level. By using IT Mode, you bypass the card’s internal error recovery logic, allowing ZFS to take the lead.

Another critical issue is the abstraction of drive identity. In a standard RAID configuration, the operating system sees a single logical volume rather than the individual serial numbers and hardware identifiers of the disks. This makes troubleshooting a failing drive significantly harder. If a drive starts reporting high latency or increased seek errors, a card in IT Mode will pass those raw statistics directly to the OS. A RAID card might simply report the volume as “Degraded,” leaving you to hunt through proprietary management utilities like MegaCLI to figure out which physical disk is actually the culprit.

Technical Architecture: IR Mode vs IT Mode

When dealing with LSI (now Broadcom) controllers, you will typically encounter two types of firmware: IR (Integrated RAID) and IT (Initiator Target). Most cards ship with IR firmware because it allows for basic RAID 0, 1, or 10 configurations within the card’s BIOS. While this is convenient for simple servers, it is a bottleneck for advanced storage architectures. IR firmware introduces overhead and can limit the number of drives the controller can address efficiently. It also lacks the transparency required for the “Direct Path I/O” that modern hypervisors and filesystems expect.

IT Mode is the “passthrough” state that turns the controller into a transparent bridge. In this mode, the card does not attempt to manage disk geometry, parity, or striping. It simply passes SCSI commands directly from the operating system to the drives. For SAS controllers using the 2008, 2308, or 3008 chipsets, this transition is a permanent firmware change. Once flashed, the card functions exactly like a native HBA. This is particularly important for SAS3 (12Gbps) environments where bandwidth and IOPS are high enough that any firmware-level processing can introduce measurable latency.

For those running virtualized environments, such as Proxmox or ESXi with a TrueNAS virtual machine, IT Mode is mandatory for PCI Passthrough. If the card is in IR mode, the hypervisor might struggle to hand over the raw disk signatures to the guest VM. By flashing to IT Mode, the entire HBA can be passed through to the storage VM, ensuring that the guest OS has the same low-level hardware access it would have on a physical server. This setup is the gold standard for reliable virtualized storage.

Identifying Hardware and Sourcing Firmware

Before attempting to flash a controller, you must identify the exact chipset. The most common cards used in the community are the LSI 9211-8i (SAS2008), the 9207-8i (SAS2308), and the 9300-8i (SAS3008). Many OEM cards, such as the Dell PERC H310, H710, and IBM ServeRAID M1015, are actually rebranded LSI cards. These OEM cards are often locked into IR mode by default and require “cross-flashing” to become a standard LSI HBA. Using the wrong firmware file for your chipset will result in a bricked card, so verification is the most critical step of the process.

– LSI SAS 2008 Chipset: Found in the 9211-8i, Dell H310, and IBM M1015. Use SAS2Flash utility.
– LSI SAS 2308 Chipset: Found in the 9207-8i and Dell H710. This chipset is faster than the 2008 and supports PCIe 3.0.
– LSI SAS 3008 Chipset: Found in the 9300-8i and Dell H330. This is the 12Gbps SAS3 standard. Use SAS3Flash utility.

You should source your firmware directly from the Broadcom support portal or reputable community repositories that provide the P20 (Phase 20) firmware. The Phase 20 firmware is widely considered the most stable version for ZFS deployments. When downloading, ensure you get both the firmware file (.bin) and the BIOS file (.rom). The BIOS file is optional; if you don’t need to boot from the drives attached to the card, omitting the BIOS file will actually speed up your server’s POST time significantly.

The Flashing Workflow: A Step-by-Step Guide

The flashing process is generally performed in a UEFI shell because modern motherboards often have memory address conflicts with DOS-based flashing utilities. You will need a USB drive formatted to FAT32 containing the flashing utility (sas2flash.efi or sas3flash.efi), the firmware file, and the BIOS file. Before you begin, it is vital to record the SAS address of your card. This is a 16-digit hex code usually found on a green sticker on the back of the PCB. If you erase the card and forget this address, you will have to invent a dummy one, which can cause conflicts if you ever put two such cards in the same machine.

1. Boot into the UEFI shell and identify your controller using the command: `sas2flash.efi -list`. Record the SAS Address listed.
2. Erase the existing firmware and BIOS. This is the “point of no return.” Use the command: `sas2flash.efi -o -e 6`. The -e 6 flag erases the entire flash memory except for the manufacturing area.
3. Flash the new IT Mode firmware and BIOS. Use the command: `sas2flash.efi -o -f 2118it.bin -b mptsas2.rom`. Replace the filenames with the ones specific to your card and version.
4. Program the SAS address back into the card. Use the command: `sas2flash.efi -o -sasadd 500605bxxxxxxxxx`. Use the address you recorded in step one.
5. Verify the flash by running `sas2flash.efi -list` again. Ensure the “Firmware Product ID” reflects IT Mode and the version is correct.

If you are using a SAS3 card, the utility name changes to sas3flash.efi, and the firmware files will have different naming conventions (e.g., 9300it.bin). The logic remains identical. If you encounter an error stating “Failed to initialize PAL,” this usually means there is a conflict between the UEFI shell and the motherboard’s mapping of the PCIe space. Try moving the card to a different PCIe slot or using a different motherboard for the flashing process. Some older enterprise servers are notoriously difficult to flash, and using a consumer-grade desktop motherboard for the flash process is a common workaround.

Common Mistakes and Troubleshooting Advanced Firmware Issues

One of the most frequent errors in advanced storage builds is mismatched firmware and driver versions. For example, if you are running TrueNAS Core (based on FreeBSD), the OS includes a driver (mpt2sas or mpr) that expects a specific firmware phase on the card. If your card is on Phase 16 but the driver is on Phase 20, you may see “Command Timeout” errors in your logs. Always ensure that your firmware matches the driver version recommended by your operating system’s documentation. The TrueNAS community maintains extensive hardware compatibility lists that specify which firmware versions are tested for long-term stability.

Another mistake is failing to provide adequate cooling to the HBA after flashing. RAID cards and HBAs are designed for server chassis with high-velocity airflow. These chips run extremely hot, often exceeding 80 degrees Celsius in a stagnant desktop environment. When a card overheats, it may drop drives from the pool or cause the entire system to lock up. If you are using these cards in a workstation or home lab tower, it is highly recommended to zip-tie a small 40mm fan directly to the card’s heatsink. This simple modification can prevent “Silent Data Corruption” caused by controller thermal throttling.

– Issue: UEFI shell does not see the USB drive. Solution: Ensure the drive is formatted as FAT32 and use a USB 2.0 port if available.
– Issue: SAS address lost during erase. Solution: Look for the sticker on the card; if missing, use a random 16-digit hex code starting with 500605b.
– Issue: Card not detected in OS after flash. Solution: Check if the card was accidentally flashed with the wrong chipset firmware. You may need to use the “clean” erase command (-e 7) and retry.

Performance Tuning and Post-Flash Verification

Once the card is successfully in IT Mode and recognized by your OS, you should perform verification tests before moving production data onto the disks. Use tools like `smartctl` to ensure you can read the S.M.A.R.T. data of every individual drive. If you can see the serial numbers, temperature, and error logs of each disk, the passthrough is working correctly. You should also run a basic benchmark using `fio` or `iozone` to ensure that the controller is not bottlenecking the drives. A SAS3008 chipset should easily handle eight SATA SSDs at their full rated speed without sweat.

For advanced performance tuning, consider the depth of the request queue. In IT Mode, the HBA has a specific queue depth (often 256 or 600) that determines how many I/O operations it can handle simultaneously. If you are using a high number of SSDs, you may hit this limit. You can monitor this in Linux using `iostat -x` and looking at the “avgqu-sz” column. If the queue is constantly full, you might need to spread your drives across multiple HBAs to increase total throughput. This is a common strategy in “All-Flash” ZFS arrays where the controller, not the disk, becomes the primary bottleneck.

Finally, verify that your ZFS pool handles a disk failure as expected. With the card in IT Mode, pulling a drive should result in an immediate “REMOVED” or “FAULTED” status in `zpool status`. If the system hangs or the status takes minutes to update, there may be a firmware-level setting or a driver conflict that needs addressing. Testing the “hot-swap” capability of your backplane and HBA combination is a vital final step in commissioning any advanced storage server.