Repair a damaged XFS filesystem within a LVM2 setup
August 2022
Sun Mon Tue Wed Thu Fri Sat
  1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30 31      
This site is an effort to share some of the base knowledge I have gathered through all this years working with Linux, FreeBSD, OpenBSD, Python or Zope, among others. So, take a look around and I hope you will find the contents useful.
Recent Entries
Recent Comments
Recent Trackbacks
OpenBSD (9 items)
BSD (0 items)
FreeBSD (19 items)
Linux (3 items)
Security (3 items)
Python (22 items)
Zope (13 items)
Daily (144 items)
e-shell (9 items)
Hacks (14 items)
PostgreSQL (3 items)
OSX (8 items)
Nintendo DS (0 items)
enlightenment (0 items)
Apache (3 items)
Nintendo Wii (1 items)
Django (24 items)
Music (12 items)
Plone (7 items)
Varnish (0 items)
Lugo (2 items)
Sendmail (0 items)
europython (7 items)
Cherokee (1 items)
self (1 items)
Nature (1 items)
Hiking (0 items)
uwsgi (0 items)
nginx (0 items)
cycling (10 items)
Networking (1 items)
DNS (0 items)

Syndicate this site (XML)

RSS/RDF 0.91

23 septiembre

Repair a damaged XFS filesystem within a LVM2 setup

This could be a little bit tricky, and you should have a Slax Live CD at hand

Today one of the Debian workstations at work started to do some strange things, and those things helped us finding a problem with the XFS filesystem mounted on /.

If I tried to delete some files located in such partition I got a message complaining about some problems in the filesystem hierarchy that should be fixed before proceeding.

This machine has the usual LVM2 setup, being one of the logical volumes the partition that contains the data mounted under the / mount point.

You can't check/repair a XFS filesystem when you have such filesystem mounted, so first I tried to boot Debian into single user mode, but (sincerely, I still don't know why) it kept itself mounting the root partition as read-write (rw) even if I tried to force it to mount it read-only (ro). So I got tired about it (I don't like Debian at all anyway) and I just got a Slax LiveCD, put it in the CD unit and started the computer with it.

The boot options available in Slax

In the boot menu (where you have multiple options for booting the Slax system) I selected Slax Text mode, because I don't need all the X-stuff to check some partitions.

Once it finished booting up, I got the usual login screen from Slax, something like this:

The login screen in the Text mode

I logged in as root and I checked that Slax had recognized the hard drive (/dev/sda in my case). I was able to see two partitions for the drive:

brw-rw---- 1 root disk 8, 0 sep 23 12:20 /dev/sda
brw-rw---- 1 root disk 8, 1 sep 23 12:20 /dev/sda1
brw-rw---- 1 root disk 8, 2 sep 23 12:20 /dev/sda2

sda1 is an ext2 partition outside the LVM (mounted as /boot) and sda2 is the partition containing the LVM.

So... so far I had the drive and the phisical partitions recognized, but I couldn't check the root partition yet, I needed access to the LVM logical volumes.

One of the good things about Slax is that it comes with LVM/LVM2 support out of the box, so I just used some tools to get the job done.

First I used vgscan to search for LVM information in all the available hard drives:

vgscan -v --mknodes

The --mknodes parameter tells vgscan to create all the needed entries in /dev so you can use the LVM stuff.

It recognized the LVM information and created all the needed entries under /dev/mapper. The one that interested me the most was /dev/mapper/japones-root (in my case, in yours it will depend on the name you gave the volume) because that was the logical volume mounted as the root partition.

Ok, now I've found the LVM volumes, I needed to activate them, which could be done using vgchange:

vgchange -a y

With that, I did activate all the logical volumes found.

Finally, I used the xfs_check and xfs_repair tools to perform all the needed tasks in the XFS filesystem and repair it:

xfs_check /dev/mapper/japones-root
xfs_repair /dev/mapper/japones-root

I found that the filesystem was quite corrupt and _xfs_repair had to do a lot of work on some parts of it, but the system worked just fine after rebooting back into Debian.

Once the _xfs_repair tool finished cleaning the filesystem I mounted it (to see if everything was working fine):

mkdir /mnt/japones && mount /dev/mapper/japones-root /mnt/japones

And after checking everything was ok I just rebooted the machine, removed the Slax CD and tried Debian again.

Everything worked fine.

Posted by wu at 22:21 | Comments (0) | Trackbacks (0)
<< Addictive | Main | sshd: Corrupted MAC on input. >>
Re: Repair a damaged XFS filesystem within a LVM2 setup

Heh, anything is easy to dislike if one's lacking the basic knowledge about basic tools... such as mount.

Posted by: asdf at marzo 16,2014 04:05
Re: Repair a damaged XFS filesystem within a LVM2 setup

Heh, talking about lack of knowledge is easy too, but give examples and help is quite more difficult, isn't it?

Posted by: Wu at marzo 17,2014 10:49
Re: Repair a damaged XFS filesystem within a LVM2 setup

Wonderful! Followed your step by step advice and my biggest problem was solved.

Thank you for your article.

Posted by: Nad at agosto 26,2015 04:13
Re: Repair a damaged XFS filesystem within a LVM2 setup

Wow, I'm amazed this old article was still useful for someone else. Thank you for reading!

Posted by: Wu at agosto 28,2015 17:28
Re: Repair a damaged XFS filesystem within a LVM2 setup

Yes, it's still useful :) However, I am stuck after xfs_repair /dev/mapper/my-root

The command doesn't have any output. It looks like it hangs and error like this "blocked for more than 120 seconds"

Any idea?


Posted by: Hung at mayo 19,2017 22:09
Re: Repair a damaged XFS filesystem within a LVM2 setup

Finally found a working guide to fix a xfs file system on a lvm2 volume.
Only thing is xfs_check is deprecated and replaced with "xfs_repair -n", otherwise still top notch!

Posted by: Konrad at agosto 20,2018 19:16
Re: Repair a damaged XFS filesystem within a LVM2 setup

Should mention this scenario: ALERT: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair.

One odd case I have...if a thick LV was converted to a thin LV, there's a external origin that's kept around. this always gets this ALERT but I think it might be inherent due to the read-only nature of the external origin...Not exactly sure, but I did nothing to force it..left it alone.

Posted by: kevin at febrero 25,2020 00:55
Please send trackback to:
There are no trackbacks.
Post a comment