24 дек. 2013 г.

Restore deleted virtual disks under XEN or Oracle VM

We have a high availabile Oracle VM hosts based on Oracle VM server release 3.2.2. FS - OCFS2.

Human error occured and all virtual disks from repository was deleted with 'rm' command.

All VM's remain active and continue working.

Our first proposition: find files in the /proc/PID/fd directory and copy them. Failed. We found very small amount of deleted files there.

The second way after some google-fu:

All virtual images are mounted as loopback devices, so we can see them with 'losetup -a' command

[root@t2ru-ovhost-03 recovery]# losetup -a
/dev/loop0: [fc02]:2624777 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop1: [fc02]:2624776 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop2: [fc02]:2624779 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop3: [fc02]:2624786 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop4: [fc02]:2624789 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop5: [fc02]:2624790 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop6: [fc02]:2624793 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/ISOs/0004fb*)
/dev/loop7: [fc02]:2624792 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop8: [fc02]:2624795 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop9: [fc02]:2624798 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop10: [fc02]:2624801 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop11: [fc02]:2624802 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop12: [fc02]:2624806 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop13: [fc02]:2624805 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop14: [fc02]:2624809 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop15: [fc02]:2624813 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop16: [fc02]:2624812 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop17: [fc02]:2624827 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop18: [fc02]:2624826 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop19: [fc02]:2624836 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop20: [fc02]:2624816 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop21: [fc02]:2624823 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop22: [fc02]:2624839 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop23: [fc02]:2624837 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop24: [fc02]:2624783 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop25: [fc02]:2624785 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop26: [fc02]:2624782 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop27: [fc02]:2624842 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop28: [fc02]:2624844 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop29: [fc02]:2624845 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop32: [fc02]:2624849 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/VirtualDisk*)
/dev/loop33: [fc02]:184537605 (/OVS/Repositories/0004fb00000300005652fcdd8e980259/ISOs/0004fb*)

Although losetup has truncated its output so we weren't able to see actual names.

You have 2 options to work it round:

1. run losetup -a /dev/loopNUM with strace

2. go through /sys/block and for each loopNUM directory and do 'cat loop/backing_file' to see actual filename, 'cat size' to see size and echo $base to see loopback filename.

Very fast written script to do this job:

for i in loop* ;\
do echo -n  ${i}" "; \
echo `cat ${i}/loop/backing_file | cut -f6 -d/`;\
echo -n `cat ${i}/size`; echo " ";\
done

And its result:

loop0 0004fb000012000063d6575b3f48b463.img (deleted) 62914560  +
loop1 0004fb0000120000704f06e81dea0133.img (deleted) 67108864 
loop10 0004fb0000120000c93f3e2ce72a21ea.img (deleted) 85995520 
loop11 0004fb00001200008b7ef69e390fc9db.img (deleted) 25165824  +
loop12 0004fb0000120000c7d60e7cd3729ace.img (deleted) 62914560 
loop13 0004fb00001200007884393989c691aa.img (deleted) 67108864 
loop14 0004fb00001200006fedafd936080b23.img (deleted) 25165824  +
loop15 0004fb0000120000681f43d49fcc2937.img (deleted) 62914560 
loop16 0004fb0000120000d10e9e629238e991.img (deleted) 67108864 
loop17 0004fb00001200002d4f0e853511327c.img (deleted) 31457280 
loop18 0004fb000012000085b9c867ce5e0a30.img (deleted) 25165824 
loop19 0004fb00001200000016659316b0e98f.img (deleted) 31457280 
loop2 0004fb0000120000731c98a3fa0f4f8b.img (deleted) 314572800   .
loop20 0004fb00001200008bfdb6bfe0351c53.img (deleted) 25165824 
loop21 0004fb000012000075a678efe2faefa8.img (deleted) 104857600 
loop22 0004fb0000120000262dcdd5cd17e79d.img (deleted) 146800640 +
loop23 0004fb00001200001c3ccc5104a634f8.img (deleted) 20971520 
loop24 0004fb00001200005133cf345cebce21.img (deleted) 62914560 
loop25 0004fb0000120000861c7f6996570b6e.img (deleted) 104857600 
loop26 0004fb0000120000e193db30a91f2cf1.img (deleted) 25165824 
loop27 0004fb000012000090515a0546d1eb8e.img (deleted) 104857600 
loop28 0004fb00001200001b39e9dc716b8f1b.img (deleted) 25165824   +
loop29 0004fb00001200001f205725a33a5014.img (deleted)314572800   +
loop3 0004fb0000120000f49caf272ed15c86.img (deleted) 25165824    +
loop30 cat: loop30/loop/backing_file: Нет такого файла или каталога 0 
loop31 cat: loop31/loop/backing_file: Нет такого файла или каталога 0 
loop32 0004fb000012000040a4a2afd39aa69c.img (deleted) 314572800 
loop33 0004fb0000150000710f803f79f72695.iso 1489320 
loop4 0004fb0000120000e45a30ed3d3aa2a9.img (deleted) 20971520   .
loop5 0004fb00001200006686d4461e733db8.img (deleted) 83886080   .
loop6 0004fb0000150000d7bbc124876ed5b1.iso 7178372 
loop7 0004fb00001200002e0c01393728be30.img (deleted) 8388608    .
loop8 0004fb00001200002bda771dfc94384b.img (deleted) 25165824 
loop9 0004fb00001200008b1629ee7493f640.img (deleted) 13125105 
[root@t2ru-ovhost-03 block]# 

Next time to do monkey job of block copy loopback devices. We've got a new mount point 1.4Tb and copy all loopback files into it:

[root@t2ru-ovhost-03 recovery]# dd if=/dev/loop28 of=./loop28 bs=512k 
24576+0 записей считано
24576+0 записей написано
 скопировано 12884901888 байт (13 GB), 324.303 секунд, 39.7 MB/s

After everything was copied out just cp recovered files to the old place (from backing_file) and restart VMs. Of cource, consistency of database files or FS are not guaranted.