This tutorial shows how to set up network RAID1 with the help of DRBD on two Ubuntu 11.10 systems. DRBD stands for Distributed Replicated BlockDevice and allows you to mirror block devices over a network. This is useful for high-availability setups (like a HA NFS server) because if one node fails, all data is still available from the other node.
I do not issue any guarantee that this will work for you!
1 Preliminary Note
I will use two servers here (both running Ubuntu 11.10):
- server1.example.com (IP address 192.168.0.100)
- server2.example.com (IP address: 192.168.0.101)
Both nodes have an unpartitioned second drive (/dev/sdb) with identical size (30GB in this example) that I want to mirror over the network (network RAID1) with the help of DRBD.
It is important that both nodes can resolve each other, either through DNS or through /etc/hosts. If you did not create DNS records for server1.example.com and server2.example.com, you can modify /etc/hosts on both nodes as follows:
server1/server2:
vi /etc/hosts
127.0.0.1 localhost.localdomain localhost 192.168.0.100 server1.example.com server1 192.168.0.101 server2.example.com server2 # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters |
I'm running all the steps in this tutorial with root privileges, so make sure you're logged in as root:
sudo su
2 Synchronize Time
server1/server2:
It is very important that both nodes have the same time. Therefore we install the ntp packages:
apt-get install ntp ntpdate
3 Partition /dev/sdb
server1/server2:
Right now, our partitioning looks as follows:
fdisk -l
root@server1:~# fdisk -l
Disk /dev/sda: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders, total 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000712c1
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 499711 248832 83 Linux
/dev/sda2 501758 62912511 31205377 5 Extended
/dev/sda5 501760 62912511 31205376 8e Linux LVM
Disk /dev/sdb: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders, total 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sdb doesn't contain a valid partition table
Disk /dev/mapper/server1-root: 31.4 GB, 31415336960 bytes
255 heads, 63 sectors/track, 3819 cylinders, total 61358080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/mapper/server1-root doesn't contain a valid partition table
Disk /dev/mapper/server1-swap_1: 536 MB, 536870912 bytes
255 heads, 63 sectors/track, 65 cylinders, total 1048576 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/mapper/server1-swap_1 doesn't contain a valid partition table
root@server1:~#
Disk /dev/sda: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders, total 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000712c1
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 499711 248832 83 Linux
/dev/sda2 501758 62912511 31205377 5 Extended
/dev/sda5 501760 62912511 31205376 8e Linux LVM
Disk /dev/sdb: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders, total 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sdb doesn't contain a valid partition table
Disk /dev/mapper/server1-root: 31.4 GB, 31415336960 bytes
255 heads, 63 sectors/track, 3819 cylinders, total 61358080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/mapper/server1-root doesn't contain a valid partition table
Disk /dev/mapper/server1-swap_1: 536 MB, 536870912 bytes
255 heads, 63 sectors/track, 65 cylinders, total 1048576 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/mapper/server1-swap_1 doesn't contain a valid partition table
root@server1:~#
As you see, /dev/sdb is not partitioned. We change that now and create one big partition on it, /dev/sdb1:
fdisk /dev/sdb
root@server1:~# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xf7ab5969.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): <-- n
Command action
e extended
p primary partition (1-4)
<-- p
Partition number (1-4, default 1): <-- 1
First sector (2048-62914559, default 2048): <-- ENTER
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-62914559, default 62914559): <-- ENTER
Using default value 62914559
Command (m for help): <-- t
Selected partition 1
Hex code (type L to list codes): <-- 83
Command (m for help): <-- w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
root@server1:~#
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xf7ab5969.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): <-- n
Command action
e extended
p primary partition (1-4)
<-- p
Partition number (1-4, default 1): <-- 1
First sector (2048-62914559, default 2048): <-- ENTER
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-62914559, default 62914559): <-- ENTER
Using default value 62914559
Command (m for help): <-- t
Selected partition 1
Hex code (type L to list codes): <-- 83
Command (m for help): <-- w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
root@server1:~#
Now run
fdisk -l
again, and you should find /dev/sdb1 in the output:
root@server1:~# fdisk -l
Disk /dev/sda: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders, total 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000712c1
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 499711 248832 83 Linux
/dev/sda2 501758 62912511 31205377 5 Extended
/dev/sda5 501760 62912511 31205376 8e Linux LVM
Disk /dev/sdb: 32.2 GB, 32212254720 bytes
64 heads, 51 sectors/track, 19275 cylinders, total 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xf7ab5969
Device Boot Start End Blocks Id System
/dev/sdb1 2048 62914559 31456256 83 Linux
Disk /dev/mapper/server1-root: 31.4 GB, 31415336960 bytes
255 heads, 63 sectors/track, 3819 cylinders, total 61358080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/mapper/server1-root doesn't contain a valid partition table
Disk /dev/mapper/server1-swap_1: 536 MB, 536870912 bytes
255 heads, 63 sectors/track, 65 cylinders, total 1048576 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/mapper/server1-swap_1 doesn't contain a valid partition table
root@server1:~#
Disk /dev/sda: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders, total 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000712c1
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 499711 248832 83 Linux
/dev/sda2 501758 62912511 31205377 5 Extended
/dev/sda5 501760 62912511 31205376 8e Linux LVM
Disk /dev/sdb: 32.2 GB, 32212254720 bytes
64 heads, 51 sectors/track, 19275 cylinders, total 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xf7ab5969
Device Boot Start End Blocks Id System
/dev/sdb1 2048 62914559 31456256 83 Linux
Disk /dev/mapper/server1-root: 31.4 GB, 31415336960 bytes
255 heads, 63 sectors/track, 3819 cylinders, total 61358080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/mapper/server1-root doesn't contain a valid partition table
Disk /dev/mapper/server1-swap_1: 536 MB, 536870912 bytes
255 heads, 63 sectors/track, 65 cylinders, total 1048576 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/mapper/server1-swap_1 doesn't contain a valid partition table
root@server1:~#
4 Install And Configure DRBD
Now install DRBD on both nodes as follows:
apt-get install drbd8-utils
Load the DRBD kernel module:
modprobe drbd
To check if it is loaded, run:
lsmod | grep drbd
Output should be similar to this one:
root@server1:~# lsmod | grep drbd
drbd 273002 0
lru_cache 14896 1 drbd
root@server1:~#
drbd 273002 0
lru_cache 14896 1 drbd
root@server1:~#
Now we back up the original /etc/drbd.conf file and create a new one on both nodes as follows:
cp /etc/drbd.conf /etc/drbd.conf_orig
cat /dev/null > /etc/drbd.conf
vi /etc/drbd.conf
cat /dev/null > /etc/drbd.conf
vi /etc/drbd.conf
global { usage-count no; } common { syncer { rate 100M; } } resource r0 { protocol C; startup { wfc-timeout 15; degr-wfc-timeout 60; } net { cram-hmac-alg sha1; shared-secret "secret"; } on server1.example.com { device /dev/drbd0; disk /dev/sdb1; address 192.168.0.100:7788; meta-disk internal; } on server2.example.com { device /dev/drbd0; disk /dev/sdb1; address 192.168.0.101:7788; meta-disk internal; } } |
Make sure you use the correct node names in the file (instead of server1.example.com and server2.example.com) - please make sure you use the node names that the command
uname -n
shows on both nodes. Also make sure you fill in the correct IP addresses in the address lines and the correct disk in the disk lines (if you don't use /dev/sdb1).
Now we initialize the meta data storage. On both nodes run:
drbdadm create-md r0
root@server1:~# drbdadm create-md r0
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
root@server1:~#
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
root@server1:~#
Then start DRBD on both nodes:
/etc/init.d/drbd start
root@server1:~# /etc/init.d/drbd start
* Starting DRBD resources [ d(r0) s(r0) n(r0) ] [ OK ]
root@server1:~#
* Starting DRBD resources [ d(r0) s(r0) n(r0) ] [ OK ]
root@server1:~#
The next step has to be carried out on server1 only:
server1:
Now make server1 the primary node:
drbdadm -- --overwrite-data-of-peer primary all
Afterwards, data will start to synchronize between server1 and server2.
server2:
Take a look at
cat /proc/drbd
to see the synchronization progress:
root@server2:~# cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: DA5A13F16DE6553FC7CE9B2
0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
ns:0 nr:10166400 dw:10166400 dr:0 al:0 bm:620 lo:1 pe:7407 ua:0 ap:0 ep:1 wo:f oos:21288860
[=====>..............] sync'ed: 32.4% (20788/30716)Mfinish: 0:03:53 speed: 91,180 (86,152) want: 102,400 K/sec
root@server2:~#
version: 8.3.11 (api:88/proto:86-96)
srcversion: DA5A13F16DE6553FC7CE9B2
0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
ns:0 nr:10166400 dw:10166400 dr:0 al:0 bm:620 lo:1 pe:7407 ua:0 ap:0 ep:1 wo:f oos:21288860
[=====>..............] sync'ed: 32.4% (20788/30716)Mfinish: 0:03:53 speed: 91,180 (86,152) want: 102,400 K/sec
root@server2:~#
(You can run
watch cat /proc/drbd
to get an ongoing output of the process. To leave watch, press CTRL+C.)
Wait until the synchronization has finished - output should be as follows:
root@server2:~# cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: DA5A13F16DE6553FC7CE9B2
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:0 nr:31455260 dw:31455260 dr:0 al:0 bm:1909 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
root@server2:~#
version: 8.3.11 (api:88/proto:86-96)
srcversion: DA5A13F16DE6553FC7CE9B2
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:0 nr:31455260 dw:31455260 dr:0 al:0 bm:1909 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
root@server2:~#
The snippet ro:Secondary/Primary tells you that this node is the secondary node.
server1:
On server1, the output of
cat /proc/drbd
is as follows (after the synchronization has finished):
root@server1:~# cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: DA5A13F16DE6553FC7CE9B2
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:31455260 nr:0 dw:0 dr:31455924 al:0 bm:1920 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
root@server1:~#
version: 8.3.11 (api:88/proto:86-96)
srcversion: DA5A13F16DE6553FC7CE9B2
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:31455260 nr:0 dw:0 dr:31455924 al:0 bm:1920 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
root@server1:~#
The snippet ro:Primary/Secondary tells you that this is the primary node.
Now that we have our new network RAID1 block device /dev/drbd0 (which consists of /dev/sdb1 from server1 and server2), let's create an ext4 filesystem on it and mount it to the /data directory. This has to be done only on server1!
mkfs.ext4 /dev/drbd0
mkdir /data
mount /dev/drbd0 /data
mkdir /data
mount /dev/drbd0 /data
Afterwards you should see /dev/drbd0 in the outputs of...
mount
root@server1:~# mount
/dev/mapper/server1-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
/dev/drbd0 on /data type ext4 (rw)
root@server1:~#
/dev/mapper/server1-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
/dev/drbd0 on /data type ext4 (rw)
root@server1:~#
... and:
df -h
root@server1:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/server1-root
29G 1017M 27G 4% /
udev 238M 4.0K 238M 1% /dev
tmpfs 99M 228K 99M 1% /run
none 5.0M 4.0K 5.0M 1% /run/lock
none 247M 0 247M 0% /run/shm
/dev/sda1 228M 24M 193M 11% /boot
/dev/drbd0 30G 172M 28G 1% /data
root@server1:~#
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/server1-root
29G 1017M 27G 4% /
udev 238M 4.0K 238M 1% /dev
tmpfs 99M 228K 99M 1% /run
none 5.0M 4.0K 5.0M 1% /run/lock
none 247M 0 247M 0% /run/shm
/dev/sda1 228M 24M 193M 11% /boot
/dev/drbd0 30G 172M 28G 1% /data
root@server1:~#
5 Test
server1:
Now let's create some files or directories in the /data directory and check whether they get replicated to server2.
touch /data/test1.txt
touch /data/test2.txt
touch /data/test2.txt
ls -l /data/
root@server1:~# ls -l /data/
total 16
drwx------ 2 root root 16384 2011-10-28 14:12 lost+found
-rw-r--r-- 1 root root 0 2011-10-28 14:13 test1.txt
-rw-r--r-- 1 root root 0 2011-10-28 14:13 test2.txt
root@server1:~#
total 16
drwx------ 2 root root 16384 2011-10-28 14:12 lost+found
-rw-r--r-- 1 root root 0 2011-10-28 14:13 test1.txt
-rw-r--r-- 1 root root 0 2011-10-28 14:13 test2.txt
root@server1:~#
Now let's unmount the /data directory on server1:
umount /data
Then assign the secondary role to server1:
drbdadm secondary r0
Now we go to server2, make it the primary node and check if we can see the files/directories we created on server1 in the /datadirectory on server2.
server2:
First we assign the primary role to server2:
drbdadm primary r0
Check the output of
cat /proc/drbd
... and you should see that server2 is the primary node now:
root@server2:~# cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: DA5A13F16DE6553FC7CE9B2
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:0 nr:31691444 dw:31691444 dr:664 al:0 bm:1909 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
root@server2:~#
version: 8.3.11 (api:88/proto:86-96)
srcversion: DA5A13F16DE6553FC7CE9B2
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:0 nr:31691444 dw:31691444 dr:664 al:0 bm:1909 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
root@server2:~#
Next we create the /data directory and mount /dev/drbd0 to it:
mkdir /data
mount /dev/drbd0 /data
mount /dev/drbd0 /data
Let's check the contents of the /data directory:
ls -l /data/
If everything went fine, it should contain the files/directories that we created on server1:
root@server2:~# ls -l /data/
total 16
drwx------ 2 root root 16384 2011-10-28 14:12 lost+found
-rw-r--r-- 1 root root 0 2011-10-28 14:13 test1.txt
-rw-r--r-- 1 root root 0 2011-10-28 14:13 test2.txt
root@server2:~#
total 16
drwx------ 2 root root 16384 2011-10-28 14:12 lost+found
-rw-r--r-- 1 root root 0 2011-10-28 14:13 test1.txt
-rw-r--r-- 1 root root 0 2011-10-28 14:13 test2.txt
root@server2:~#
server1:
Now that we have switched roles, the output of
cat /proc/drbd
on server1 should show you that server1 has the secondary role:
root@server1:~# cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: DA5A13F16DE6553FC7CE9B2
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:31691444 nr:185568 dw:421752 dr:31457005 al:83 bm:1920 lo:1 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
root@server1:~#
version: 8.3.11 (api:88/proto:86-96)
srcversion: DA5A13F16DE6553FC7CE9B2
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:31691444 nr:185568 dw:421752 dr:31457005 al:83 bm:1920 lo:1 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
root@server1:~#
6 Links
- DRBD: http://www.drbd.org/
- Ubuntu: http://www.ubuntu.com/
No comments:
Post a Comment