Yrurtj

Tonight my server stopped answering to ssh, ping, so I drove to the server room.
Server did not respond to keyboard (even Ctrl-Alt-Del), there was no data on screen (it was black, no symbols on it), so I decided to reboot him with button on the case. After several attempts to start the system (it does not boot more than GRUB and black screen right after it), I opened BIOS setup, and saw that one disk of array (one of 4 in software mdadm RAID10) is crashed.
In BIOS it was:

SATA Port 1         INTEL SSDSC2CW060A3 (60.0GB)

SATA Port 2         ST1000DM003-1CH162 (1000.2GB)

SATA Port 3         ST1000DM003-1CH162 (1000.2GB)

SATA Port 4         Not Present

SATA Port 5         ST1000DM003-1CH162 (1000.2GB)

I've disconnect this disk, and system had started, but, without array.
Now I get new same disk, that is installed on the place of faulted one. But RAID 10 is not rebuilds automatically (as I thought to). When booting I see

The disk drive for UUID=a3462d0b-7261-48da-8e13-6de332e4de33 is not ready yet or not present

keys:Continue to wait, or Press S to skip mounting or M for manual recovery



The disk drive for /var/log is not ready yet or not present

keys:Continue to wait, or Press S to skip mounting or M for manual recovery



The disk drive for /mnt/vm is not ready yet or not present

keys:Continue to wait, or Press S to skip mounting or M for manual recovery 

/etc/mdadm/mdadm.conf

CREATE owner=root group=disk mode=0660 auto=yes

HOMEHOST <system>

MAILADDR root

ARRAY /dev/md/0 metadata=1.2 name=rockshock:0 UUID=02f2f445:de9ba929:f6a3e9da:249f8e42

/etc/fstab

UUID=6760d8f4-2816-4b54-8938-ab11b329705b /               ext4    noatime,errors=remount-ro 0       1

UUID=7593-A7EC  /boot/efi       vfat    defaults        0       1

UUID=a3462d0b-7261-48da-8e13-6de332e4de33   none    swap    sw  0   0



/dev/raid/logs  /var/log    ext4    noatime,noexec  0   2

/dev/raid/vm    /mnt/vm     ext4    noatime,noexec  0   2

Here is some data that I habe (should I provide any more?):

$ cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 

md0 : inactive sdb[3](S) sdd[0](S) sde[1](S)

   2929894536 blocks super 1.2



unused devices: <none>

$ blkid

/dev/sdb: UUID="02f2f445-de9b-a929-f6a3-e9da249f8e42" UUID_SUB="917412df-140c-dbc6-68ad-57e981cbcff9" LABEL="rockshock:0" TYPE="linux_raid_member" 

/dev/sdc1: UUID="7593-A7EC" TYPE="vfat" 

/dev/sdc2: UUID="6760d8f4-2816-4b54-8938-ab11b329705b" TYPE="ext4" 

/dev/sdd: UUID="02f2f445-de9b-a929-f6a3-e9da249f8e42" UUID_SUB="0904482c-1a9a-1174-6936-805c8a466493" LABEL="rockshock:0" TYPE="linux_raid_member" 

/dev/sde: UUID="02f2f445-de9b-a929-f6a3-e9da249f8e42" UUID_SUB="9badcfe0-f4a4-cd0c-baa8-3d4ebc46c56e" LABEL="rockshock:0" TYPE="linux_raid_member" 

$ sudo mdadm --detail /dev/md0

mdadm: md device /dev/md0 does not appear to be active.

$ sudo mdadm --examine --scan -v

ARRAY /dev/md/0 level=raid10 metadata=1.2 num-devices=4 UUID=02f2f445:de9ba929:f6a3e9da:249f8e42 name=rockshock:0

   devices=/dev/sde,/dev/sdd,/dev/sdb

$ sudo mdadm --examine /dev/sd[abde]

 /dev/sdb:

         Magic : a92b4efc

       Version : 1.2

   Feature Map : 0x0

    Array UUID : 02f2f445:de9ba929:f6a3e9da:249f8e42

          Name : rockshock:0  (local to host rockshock)

 Creation Time : Wed Jan  1 16:40:06 2014

    Raid Level : raid10

  Raid Devices : 4



Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)

    Array Size : 1953262592 (1862.78 GiB 2000.14 GB)

 Used Dev Size : 1953262592 (931.39 GiB 1000.07 GB)

   Data Offset : 262144 sectors

  Super Offset : 8 sectors

         State : clean

   Device UUID : 917412df:140cdbc6:68ad57e9:81cbcff9



   Update Time : Sat Jul 25 04:18:34 2015

      Checksum : 1f73fe53 - correct

        Events : 727



        Layout : near=2

    Chunk Size : 512K



   Device Role : Active device 3

   Array State : AAAA ('A' == active, '.' == missing)



 /dev/sdd:

         Magic : a92b4efc

       Version : 1.2

   Feature Map : 0x0

    Array UUID : 02f2f445:de9ba929:f6a3e9da:249f8e42

          Name : rockshock:0  (local to host rockshock)

 Creation Time : Wed Jan  1 16:40:06 2014

    Raid Level : raid10

  Raid Devices : 4



Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)

    Array Size : 1953262592 (1862.78 GiB 2000.14 GB)

 Used Dev Size : 1953262592 (931.39 GiB 1000.07 GB)

   Data Offset : 262144 sectors

  Super Offset : 8 sectors

         State : clean

   Device UUID : 0904482c:1a9a1174:6936805c:8a466493



   Update Time : Sat Jul 25 04:25:38 2015

      Checksum : 269e217e - correct

        Events : 731



        Layout : near=2

    Chunk Size : 512K



   Device Role : Active device 0

   Array State : AA.A ('A' == active, '.' == missing)



 /dev/sde:

         Magic : a92b4efc

       Version : 1.2

   Feature Map : 0x0

    Array UUID : 02f2f445:de9ba929:f6a3e9da:249f8e42

          Name : rockshock:0  (local to host rockshock)

 Creation Time : Wed Jan  1 16:40:06 2014

    Raid Level : raid10

  Raid Devices : 4



Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)

    Array Size : 1953262592 (1862.78 GiB 2000.14 GB)



    Raid Level : raid10

  Raid Devices : 4



Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)

    Array Size : 1953262592 (1862.78 GiB 2000.14 GB)

 Used Dev Size : 1953262592 (931.39 GiB 1000.07 GB)

   Data Offset : 262144 sectors

  Super Offset : 8 sectors

         State : clean

   Device UUID : 9badcfe0:f4a4cd0c:baa83d4e:bc46c56e



   Update Time : Sat Jul 25 04:25:38 2015

      Checksum : 4100486e - correct

        Events : 731



        Layout : near=2

    Chunk Size : 512K



   Device Role : Active device 1

   Array State : AA.A ('A' == active, '.' == missing)

$ sudo fdisk -l

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes

255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk identifier: 0x00000000



Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes

255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk identifier: 0x00092197



   Device Boot      Start         End      Blocks   Id  System

Disk /dev/sdc: 60.0 GB, 60022480896 bytes

255 heads, 63 sectors/track, 7297 cylinders, total 117231408 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes



I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00000000



  Device Boot      Start         End      Blocks   Id  System

/dev/sdc1               1   117231407    58615703+  ee  GPT



Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes

255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk identifier: 0x0003765c



  Device Boot      Start         End      Blocks   Id  System



Disk /dev/sde: 1000.2 GB, 1000204886016 bytes

255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk identifier: 0x000ca97d



  Device Boot      Start         End      Blocks   Id  System



/dev/sdc1               1   117231407    58615703+  ee  GPT



Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes

255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk identifier: 0x0003765c



  Device Boot      Start         End      Blocks   Id  System



Disk /dev/sde: 1000.2 GB, 1000204886016 bytes

255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk identifier: 0x000ca97d



  Device Boot      Start         End      Blocks   Id  System

So I'm asking expirienced with mdadm: turn me in right direction, what should I do in this situation.

1. Can I get data from the array before doing something? ()


2. For what reason "mdadm --detail /dev/md0" tell me md0 is not active? Is it normal behaviour with RAID10?


3. How to say to mdadm to forget about old disk in array not by logical name?


4. And how to correctly add new disk as a replacement of failed one in array?

Thank you.

/dev/sdb: Events : 727

/dev/sdd: Events : 731

/dev/sde: Events : 731

It looks like you had a two-drive failure: /dev/sdb suffered a transient fault around 04:18:34 and was removed from the array, then /dev/sdc failed around 04:25:38 in a way that caused the entire array to freeze up. Since the event count for /dev/sdb is lower than that of /dev/sdd and /dev/sde, mdadm will not include it when starting the array, and the two remaining disks are not enough to form a complete RAID 10 array (presumably they're mirrors of each other).

The Linux RAID wiki procedure for recovering a failed array should work for RAID 10 just as it works for RAID 5/6. The key step is telling mdadm to ignore the event count mismatch using the --force flag; the purpose of the rest of the procedure is to find out in advance whether doing this will get your data back, or destroy it.