Shortly after rebuilding my Raspberry NAS last year, it stopped working again. The system wouldn’t boot right, even after changing the micro-SD card with a new build of Raspbian. A few weeks ago, I bought a new Pi 3 b and rebuilt the NAS again. When the system powered on, one of the USB drives wasn’t working. It looked like the file system was corrupt. During attempts to recover the USB drive, it stopped being readable. Both my Windows and Linux computers could see the USB drive, but not the file system, or the data. I re-formatted the disc drive, and it started working fine, it just had no data on it.

Since it was USB drive 2, the backup drive, I didn’t think it would be a problem. The NAS is configured to copy the data from the active drive to the backup drive every day. It is how the walkthrough I followed written. Not sure why the setup didn’t do a proper mirror though RAID 1. Anyway, when I went to access the NAS the next day, I couldn’t access the shared folders from any of my devices, even though they were still showing as shared.

When I logged into the Raspberry Pi, it showed USBHHD1, which is the active drive, as mounted; but it didn’t show the USBHDD2, the backup drive. However, I couldn’t access the mounted drive from the command line on the NAS either. I tried to remount the device, but it wouldn’t unmount or mount. Issuing the command

worked, and the drive was accessible after the NAS came back online. But the next day, the same problem with the NAS.

I pulled the drive from the NAS and plugged it into my laptop to make a copy. While trying to copy the data, the USB drive would remount itself repeatedly, each time the copy would fail. Eventually, Windows stopped recognizing the device, and it wouldn’t show even in Device Manager when it was plugged in. But the USB drive still worked on my Linux system as a read/write disc.

At first, I tried to copy the data over, and it kept failing. Then I tried rsync, and was having the same problem as I was on Windows. While trying to find a way to get rsync to work, I found out that the symptoms were an indication of the drive failing. A lot of the advice I found was to use ddrescue. Others on Twitter said to make an image and try to get the data off.

Using ddrescue, I still had the same problem with drive disconnecting every time the system accessed a bad sector. The ddrescue manual said to use direct disc access, which is a flag option. Using those flags didn’t help. Ddrescue would access a bad sector and the drive would disconnect from the computer.

A few places said to try and get around that with loop to wrap around ddrescue to recover the data. Doing so actually helped. The command I used was

The while runs forever, it tries to do direct disc access to the drive and copy the data while writing a log. When the command stops, in this case, because the disc disappeared, it would wait for 10 seconds and then restart the ddrescue command.

The log helps ddrescue know where the copying left off at and tracks the bad sectors for later attempts.

I did have an issue that occasionally the drive would have a new device identifier. It kept swapping between /dev/sdb1 and /dev/sdc1. So I did have to monitor the process. If the command kept failing, I had to stop the while loop and then change the drive location.

During the recovery, the drive became read-only in Linux as well, but I was able to recover most of the data. I did end up loosing less than 0.01% of the data. Some of it was corrupt, but it was an interesting experience in troubleshooting. The problem is it took a bit to go through the recovery and new build.

I’ve replaced the 256 GB flash drives with 1 TB Western Digital USB Passport drives in a powered USB hub. So far, those are working fine.

After I clear some other projects off my plate, I think I’ll get a Raspberry Pi 4, and rebuild with a proper RAID 1 configuration on 4 TB drives. I might play a bit in that process and try a few of the GUI programs like OwnCloud and OpenMediaVault. I’ve used and liked FreeNAS in the past. The only reason I wouldn’t use one of those is that I prefer the CLI over the GUI.