Repair a damaged XFS filesystem within a LVM2 setup
Today one of the Debian workstations at work started to do some strange things, and those things helped us finding a problem with the XFS filesystem mounted on /.
If I tried to delete some files located in such partition I got a message complaining about some problems in the filesystem hierarchy that should be fixed before proceeding.
This machine has the usual LVM2 setup, being one of the logical volumes the partition that contains the data mounted under the / mount point.
You can't check/repair a XFS filesystem when you have such filesystem mounted, so first I tried to boot Debian into single user mode, but (sincerely, I still don't know why) it kept itself mounting the root partition as read-write (rw) even if I tried to force it to mount it read-only (ro). So I got tired about it (I don't like Debian at all anyway) and I just got a Slax LiveCD, put it in the CD unit and started the computer with it.
In the boot menu (where you have multiple options for booting the Slax system) I selected Slax Text mode, because I don't need all the X-stuff to check some partitions.
Once it finished booting up, I got the usual login screen from Slax, something like this:
I logged in as root and I checked that Slax had recognized the hard drive (/dev/sda in my case). I was able to see two partitions for the drive:
brw-rw---- 1 root disk 8, 0 sep 23 12:20 /dev/sda brw-rw---- 1 root disk 8, 1 sep 23 12:20 /dev/sda1 brw-rw---- 1 root disk 8, 2 sep 23 12:20 /dev/sda2
sda1 is an ext2 partition outside the LVM (mounted as /boot) and sda2 is the partition containing the LVM.
So... so far I had the drive and the phisical partitions recognized, but I couldn't check the root partition yet, I needed access to the LVM logical volumes.
One of the good things about Slax is that it comes with LVM/LVM2 support out of the box, so I just used some tools to get the job done.
First I used vgscan to search for LVM information in all the available hard drives:
vgscan -v --mknodes
The --mknodes parameter tells vgscan to create all the needed entries in /dev so you can use the LVM stuff.
It recognized the LVM information and created all the needed entries under /dev/mapper. The one that interested me the most was /dev/mapper/japones-root (in my case, in yours it will depend on the name you gave the volume) because that was the logical volume mounted as the root partition.
Ok, now I've found the LVM volumes, I needed to activate them, which could be done using vgchange:
vgchange -a y
With that, I did activate all the logical volumes found.
Finally, I used the xfs_check and xfs_repair tools to perform all the needed tasks in the XFS filesystem and repair it:
xfs_check /dev/mapper/japones-root
xfs_repair /dev/mapper/japones-root
I found that the filesystem was quite corrupt and _xfs_repair had to do a lot of work on some parts of it, but the system worked just fine after rebooting back into Debian.
Once the _xfs_repair tool finished cleaning the filesystem I mounted it (to see if everything was working fine):
mkdir /mnt/japones && mount /dev/mapper/japones-root /mnt/japones
And after checking everything was ok I just rebooted the machine, removed the Slax CD and tried Debian again.
Everything worked fine.