lmfdb-disks.txt

lmfdb.warwick.ac.uk disk setup

Physically there are 12 Physical Disks (PD), each of 4T capacity.
They are in 3 rows of 4 numbered as follows:

    0  1  2  3
    4  5  6  7
    8  9 10 11

Hardware RAID configuration: there are 11 Drive Groups (DG) and 11
Virtual Drives (VD) whose numbering is not quite natural:

DG0 includes PD0+PD1 in VD1, under RAID-1.
DG1 includes PD2     in VD0 (!) under RAID-0
DG2 includes PD3     in VD2     under RAID-0
DG{n} includes PD{n+1} in VD{n} under RAID-0 for n=2,3,...,10.

DG0 = VD1 is the boot drive and contains all the Operating System.
because of the RAID1 it appears to the O/S as a single 4T (actually
3.6T) drive.  This holds both the root partition (3.6T) containing the
operating system and home directories, and also the /boot partitiion
(229M).

Use "lsblk" to see the situation.

Warning: /boot fills up regularly.  The unattended system package
upgrades result in new kernel versions being put in there, which take
up around 25M each.  These need to be cleared out regularly --  see
the file server-notes.txt for hints about this.

DG1-DG10 (the other 10 disks, VD0, VD2,...,VD10) use no hardware RAID,
i.e. they are set as RAID-0.  There is a single partition /data on
device /dev/sdg, which is configured under BTRFS.  This allows each of
the 10 physical disks to be brough online / offline individually.  At
present (July 2017) 9 of these are in use, as revealed by:

jec@lmfdb:~/system$ sudo btrfs fi show
        Total devices 9 FS bytes used 11.87TiB
        devid    1 size 3.64TiB used 2.65TiB path /dev/sda
        devid    2 size 3.64TiB used 2.65TiB path /dev/sdc
        devid    3 size 3.64TiB used 2.65TiB path /dev/sdd
        devid    4 size 3.64TiB used 2.65TiB path /dev/sde
        devid    5 size 3.64TiB used 2.65TiB path /dev/sdf
        devid    6 size 3.64TiB used 2.65TiB path /dev/sdg
        devid    7 size 3.64TiB used 2.54TiB path /dev/sdh
        devid    8 size 3.64TiB used 2.65TiB path /dev/sdi
        devid    9 size 3.64TiB used 2.65TiB path /dev/sdj

Also:

$ sudo btrfs fi df /data
Data, RAID1: total=12.71TiB, used=11.86TiB
System, RAID1: total=32.00MiB, used=1.89MiB
Metadata, RAID1: total=22.00GiB, used=19.40GiB

"Keep in mind, that the free values shown by the normal df are not
reliable, as it is hard to predict how disk usage will behave in a
copy-on-write and snapshotting filesystem like btrfs."

The O/S (i.e. mount) sees only one partition, the last one above (so
currently device /dev/sdh: see /etc/fstab where the relevant entry is

UUID=495a3b08-5ae4-45e5-a994-d2aa0ca1c480 /data btrfs defaults,noatime,compress,user_subvol_rm_allowed 0 1

The O/S thinks (using "df -h /data") it has size 26T of which 24T are
currently used, but this is not accurate since btrfs manages its own
RAID on that.  In fact 11.98T are used (see above output) but (in
effect) duplicated by btrfs.

Note that the physical disk block devices used by this partition are
/dev/sd{x} for x=a,c,d,e,f,g,h,i,j (NOT b!).  I think these letters
correspond to the VDs, so /dev/sda=VD0, /dev/sdc=VD2, ...,
/dev/sdg=VD6 and the reason for /dev/sdb being missing is that it is
the other DG, i.e. DG0 (=VD1).

To add a new device, use "sudo btrfs device add", for example (the
only one not yet added):

"sudo btrfs device add /dev/sdk /data" to add VD10

followed by "sudo btrfs balance /data".  This takes a very long time;
you can use "sudo btrfs fi show" to see how it is going and should run
this under screen/tmux,

Another useful command is "sudo btrfs fi df /data":

Data, RAID1: total=12.71TiB, used=11.96TiB
System, RAID1: total=32.00MiB, used=1.89MiB
Metadata, RAID1: total=22.00GiB, used=19.46GiB

Thu Jan 7 16:35:24 GMT 2016:
    I did "sudo btrfs device add /dev/sdh /data"
Tue Sept 20 2016:
    I did "sudo btrfs device add /dev/sdi /data"
    I did "sudo btrfs device add /dev/sdj /data"
    and then did a balance followed vby a scrub (on 22 Sept).