My Job As Sys Admin

Raidtools has been the standard software RAID management package for Linux since the inception of the software RAID driver. Over the years, raidtools have proven cumbersome to use, mostly because they rely on a configuration file (/etc/raidtab) that is difficult to maintain, and partly because its features are limited. In August 2001, Neil Brown, a software engineer at the University of New South Wales and a kernel developer, released an alternative. His mdadm (multiple devices admin) package provides a simple, yet robust way to manage software arrays.

mdadm has five major modes of operation. The first two modes, Create and Assemble, are used to configure and activate arrays. Manage mode is used to manipulate devices in an active array. Follow or Monitor mode allows administrators to configure event notification and actions for arrays. Build mode is used when working with legacy arrays that use an old version of the md driver. I will not cover build mode in this article. The remaining options are used for various housekeeping tasks and are not attached to a specific mode of operation, although the mdadm documentation calls these options Misc mode. mdadm by default comes with the newer versions of Red hat Linux. Alternately, you can download one from,

ftp://ftp.kernel.org/pub/linux/utils/raid/mdadm/

Creating an Array:

Create (mdadm --create) mode is used to create a new array. In this example I use mdadm to create a RAID-0 at /dev/md0 made up of /dev/sdb1 and /dev/sdc1

#mdadm --create --verbose /dev/md0 --level=0 --raid-devices=2 /dev/sdb1 /dev/sdbc1

The --level option specifies which type of RAID to create. Valid choices are 0,1,4 and 5 for RAID-0, RAID-1, RAID-4, RAID-5 respectively. Linear (--level=linear) is also a valid choice for linear mode.

In general, mdadm commands take the format:

mdadm [mode] [options]

Each of mdadm's options also has a short form that is less descriptive but shorter to type. For example, the following command uses the short form of each option but is identical to the example I showed above,

#mdadm -Cv /dev/md0 -l0 -n2 -c128 /dev/sdb1 /dev/sdc1

C selects Create mode, and I have also included the -v option here to turn on verbose output. -l and -n specify the RAID level and number of member disks.You can change the default chunk size (64KB) using the --chunk or -c option. In this previous example I changed the chunk size to 128KB. mdadm also supports shell expansions, so you don't have to type in the device name for every component disk if you are creating a large array. In this example, I'll create a RAID-5 with five member disks and a chunk size of 128KB:

#mdadm -Cv /dev/md0 -l5 -n5 -c128 /dev/sd{a,b,c,d,e}1
mdadm: layout defaults to left-symmetric
mdadm: array /dev/md0 started.

This example creates an array at /dev/md0 using SCSI disk partitions /dev/sda1, /dev/sdb1, /dev/sdc1, /dev/sdd1, and /dev/sde1. Notice that I have also set the chunk size to 128 KB using the -c128 option. When creating a RAID-5, mdadm will automatically choose the left-symmetric parity algorithm, which is the best choice

Use the --stop or -S command to stop running array:
#mdadm -S /dev/md0

/etc/mdadm.conf:

/etc/mdadm.conf is mdadms' primary configuration file. Unlike /etc/raidtab, mdadm does not rely on /etc/mdadm.conf to create or manage arrays. Rather, mdadm.conf is simply an extra way of keeping track of software RAIDs. Using a configuration file with mdadm is useful, but not required. Having one means you can quickly manage arrays without spending extra time figuring out what array properties are and where disks belong. For example, if an array wasn't running and there was no mdadm.conf file describing it, then the system administrator would need to spend time examining individual disks to determine array properties and member disks.

Unlike the configuration file for raidtools, mdadm.conf is concise and simply lists disks and arrays. The configuration file can contain two types of lines each starting with either the DEVICE or ARRAY keyword. Whitespace separates the keyword from the configuration information. DEVICE lines specify a list of devices that are potential member disks. ARRAY lines specify device entries for arrays as well as identifier information. This information can include lists of one or more UUIDs, md device minor numbers, or a listing of member devices.

A simple mdadm.conf file might look like this:

DEVICE /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
ARRAY /dev/md0 devices=/dev/sda1,/dev/sdb1
ARRAY /dev/md1 devices=/dev/sdc1,/dev/sdd1

In general, it's best to create an /etc/mdadm.conf file after you have created an array and update the file when new arrays are created. Without an /etc/mdadm.conf file you'd need to specify more detailed information about an array on the command in order to activate it.That means you'd have to remember which devices belonged to which arrays, and that could easily become a hassle on systems with a lot of disks. mdadm even provides an easy way to generate ARRAY lines. The output is a single long line, but I have broken it here to fit the page:

# mdadm --detail --scan
ARRAY /dev/md0 level=raid0 num-devices=2 \
UUID=410a299e:4cdd535e:169d3df4:48b7144a

If there were multiple arrays running on the system, then mdadm would generate an array line for each one. So after you're done building arrays you could redirect the output of mdadm --detail --scan to /etc/mdadm.conf. Just make sure that you manually create a DEVICE entry as well. Using the example I've provided above we might have an /etc/mdadm.conf that looks like:

DEVICE /dev/sdb1 /dev/sdc1
ARRAY /dev/md0 level=raid0 num-devices=2 \
UUID=410a299e:4cdd535e:169d3df4:48b7144a

Starting an Array:

Assemble mode is used to start an array that already exists. If you created an /etc/mdadm.conf you can automatically start an array listed there with the following command:

#mdadm -As /dev/md0
mdadm: /dev/md0 has been started with 2 drives

The -A option denotes assemble mode. You can also use --assemble. The -s or --scan option tells mdadm to look in /etc/mdadm.conf for information about arrays and devices. If you want to start every array listed in /etc/mdadm.conf, don't specify an md device on the command line

#mdadm -As

If you didn't create an /etc/mdadm.conf file, you will need to specify additional information on the command line in order to start an array. For example, this command attempts to start /dev/md0 using the devices listed on the command line:

#mdadm -A /dev/md0 /dev/sdb1 /dev/sdc1

Since using mdadm -A in this way assumes you have some prior knowledge about how arrays are arranged, it might not be useful on systems that have arrays that were created by someone else. So you may wish to examine some devices to gain a better picture about how arrays should be assembled. The examine options (-E or --examine) allows you to print the md superblock (if present) from a block device that could be an array component.

#mdadm -E /dev/sdc1
/dev/sdc1:
Magic : a92b4efc
Version : 00.90.00
UUID : 84788b68:1bb79088:9a73ebcc:2ab430da
Creation Time : Mon Sep 23 16:02:33
Raid Level : raid0
Device Size : 17920384 (17.09 GiB 18.40 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0

Update Time : Mon Sep 23 16:14:52 2002
State : clean, no-errors
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Checksum : 8ab5e437 - correct
Events : 0.10

Chunk Size : 128K

Number Major Minor RaidDevice State
this 1 8 33 1 active sync /dev/sdc1
0 0 8 17 0 active sync /dev/sdb1
1 1 8 33 1 active sync /dev/sdc1
2 2 8 49 2 active sync /dev/sdd1
3 3 8 65 3 active sync /dev/sde1

mdadm's examine option displays quite a bit of useful information about component disks. In this case we can tell that /dev/sdc1 belongs to a RAID-0 made up of a total of four member disks. What I want to specifically point out is the line of output that contains the UUID. A UUID is a 128-bit number that is guaranteed to be reasonably unique on both the local system and across other systems. It is a randomly generated using system hardware and timestamps as part of its seed. UUIDs are commonly used by many programs to uniquely tag devices. See the uuidgen and libuuid manual pages for more information.

When an array is created, the md driver generates a UUID for the array and stores it in the md superblock. You can use the UUID as criteria for array assembly. In the next example I am going to activate the array to which /dev/sdc1 belongs using its UUID

#mdadm -Av /dev/md0 --uuid=84788b68:1bb79088:9a73ebcc:2ab430da /dev/sd*

This command scans every SCSI disk (/dev/sd*) to see if it's a member of the array with the UUID 84788b68:1bb79088:9a73ebcc:2ab430da and then starts the array, assuming it found each component device. mdadm will produce a lot of output each time it tries to scan a device that does not exist. You can safely ignore such warnings.

Managing Arrays
Using Manage mode you can add and remove disks to a running array. This is useful for removing failed disks, adding spare disks, or adding replacement disks. Manage mode can also be used to mark a member disk as failed.

For example, to add a disk to an active array,

#mdadm /dev/md0 --add /dev/sdc1

Or, to remove /dev/sdc1 from /dev/md0 try

#mdadm /dev/md0 --fail /dev/sdc1 --remove /dev/sdc1

Notice that I first mark /dev/sdc1 as failed and then remove it. It's fine to combine add, fail, and remove options on a single command line as long as they make sense in terms of array management. So you have to fail a disk before removing it, for example.

Monitoring Arrays:

Follow, or Monitor, mode provides some of mdadm's best and most unique features. Using Follow/Monitor mode you can daemonize mdadm and configure it to send email alerts to system administrators when arrays encounter errors or fail. You can also use Follow mode to arbitrarily execute commands when a disk fails. For example, you might want to try removing and reinserting a failed disk in an attempt to correct a non-fatal failure without user intervention.

The following command will monitor /dev/md0 (polling every 300 seconds) for critical events. When a fatal error occurs, mdadm will send an email to sysadmin. You can tailor the polling interval and email address to meet your needs.

# mdadm --monitor --mail=sysadmin --delay=300 /dev/md0

When using monitor mode, mdadm will not exit, so you might want to wrap it around nohup and ampersand:

# nohup mdadm --monitor --mail=sysadmin --delay=300 /dev/md0 &

Follow/Monitor mode also allows arrays to share spare disks, a feature that has been lacking in Linux software RAID since its inception. That means you only need to provide one spare disk for a group of arrays or for all arrays. It also means that system administrators don't have to manually intervene to shuffle around spare disks when arrays fail. Previously this functionality was available only using hardware RAID. When Follow/Monitor mode is invoked, it polls arrays at regular intervals. When a disk failure is detected on an array without a spare disk, mdadm will remove an available spare disk from another array and insert it into the array with the failed disk. To facilitate this process, each ARRAY line in /etc/mdadm.conf needs to have a spare-group defined.

DEVICE /dev/sd*
ARRAY /dev/md0 level=raid1 num-devices=3 spare-group=database \
UUID=410a299e:4cdd535e:169d3df4:48b7144a
ARRAY /dev/md1 level=raid1 num-device=2 spare-group=database \
UUID=59b6e564:739d4d28:ae0aa308:71147fe7

In this example, both /dev/md0 and /dev/md1 are part of the spare group database. Just assume that /dev/md0 is a two-disk RAID-1 with a single spare disk. If mdadm is running in monitor mode (as I showed earlier), and a disk in /dev/md1 fails, mdadm will remove the spare disk from /dev/md0 and insert it into /dev/md1.

Here is the actual testing I did:
================================

Created a raid 1 array with two devices (/dev/sda8, /dev/sdb1) of 10GB each.

[root@lnx-rhel5-vm ~]# fdisk /dev/sda

The number of cylinders for this disk is set to 2610.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 16 128488+ 83 Linux
/dev/sda2 17 408 3148740 83 Linux
/dev/sda3 409 800 3148740 83 Linux
/dev/sda4 801 2610 14538825 5 Extended
/dev/sda5 801 1061 2096451 83 Linux
/dev/sda6 1062 1192 1052226 83 Linux
/dev/sda7 1193 1323 1052226 82 Linux swap / Solaris
/dev/sda8 1324 2540 9775521 fd Linux raid autodetect

[root@lnx-rhel5-vm ~]# fdisk /dev/sdb

The number of cylinders for this disk is set to 1435.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sdb: 11.8 GB, 11811160064 bytes
255 heads, 63 sectors/track, 1435 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 1217 9775521 fd Linux raid autodetect

#partprobe /dev/sda
#partprobe /dev/sdb

Creating an arry,
----------------

#mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sda8 /dev/sdb1

This process will take some time to build an raid-1 and then execute to see the status,

[root@lnx-rhel5-vm ~]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[1] sda8[0]
9775424 blocks [2/2] [UU]

unused devices:

Creating a File system,

#mkfs.ext3 /dev/md0
#mkdir /raid1

Mounting,
#mount -t ext3 /dev/md0 /raid1

[root@lnx-rhel5-vm ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 3.0G 273M 2.5G 10% /
/dev/sda6 996M 73M 872M 8% /var
/dev/sda5 2.0G 36M 1.9G 2% /home
/dev/sda2 3.0G 2.0G 838M 71% /usr
/dev/sda1 122M 12M 105M 10% /boot
tmpfs 252M 0 252M 0% /dev/shm
/dev/md0 9.2G 159M 8.6G 2% /raid1

[root@lnx-rhel5-vm ~]# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.03
Creation Time : Wed Oct 20 02:02:46 2010
Raid Level : raid1
Array Size : 9775424 (9.32 GiB 10.01 GB)
Used Dev Size : 9775424 (9.32 GiB 10.01 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Wed Oct 20 02:36:56 2010
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

UUID : c42a6efb:849197a4:4935bd46:a423f66f
Events : 0.40

Number Major Minor RaidDevice State
0 8 8 0 active sync /dev/sda8
1 8 17 1 active sync /dev/sdb1

Now, we can copy some data into /raid1 and then fail the one of the device (/dev/sdb1) in array to see if it works,

Failing a disk or partition on array,

[root@lnx-rhel5-vm raid1]# mdadm --manage /dev/md0 --fail /dev/sda8
mdadm: set /dev/sda8 faulty in /dev/md0

[root@lnx-rhel5-vm raid1]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[1] sda8[2](F)
9775424 blocks [2/1] [_U]

unused devices:

Above [_U] means, one of the disk is in failed state.

[root@lnx-rhel5-vm raid1]# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.03
Creation Time : Wed Oct 20 02:02:46 2010
Raid Level : raid1
Array Size : 9775424 (9.32 GiB 10.01 GB)
Used Dev Size : 9775424 (9.32 GiB 10.01 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Wed Oct 20 02:40:03 2010
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 1
Spare Devices : 0

UUID : c42a6efb:849197a4:4935bd46:a423f66f
Events : 0.42

Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 17 1 active sync /dev/sdb1

2 8 8 - faulty spare /dev/sda8

You could also see the disk failure messages in /var/log/messages:

#tail /var/log/messages

Oct 20 02:39:43 lnx-rhel5-vm kernel: md: cannot remove active disk sda8 from md0 ...
Oct 20 02:40:03 lnx-rhel5-vm kernel: raid1: Disk failure on sda8, disabling device.
Oct 20 02:40:03 lnx-rhel5-vm kernel: Operation continuing on 1 devices
Oct 20 02:40:03 lnx-rhel5-vm kernel: RAID1 conf printout:
Oct 20 02:40:03 lnx-rhel5-vm kernel: --- wd:1 rd:2
Oct 20 02:40:03 lnx-rhel5-vm kernel: disk 0, wo:1, o:0, dev:sda8
Oct 20 02:40:03 lnx-rhel5-vm kernel: disk 1, wo:0, o:1, dev:sdb1
Oct 20 02:40:03 lnx-rhel5-vm kernel: RAID1 conf printout:
Oct 20 02:40:03 lnx-rhel5-vm kernel: --- wd:1 rd:2
Oct 20 02:40:03 lnx-rhel5-vm kernel: disk 1, wo:0, o:1, dev:sdb1

Now, we are going to remove the failed disk from the array,

[root@lnx-rhel5-vm raid1]# mdadm --manage /dev/md0 --remove /dev/sda8
mdadm: hot removed /dev/sda8
[root@lnx-rhel5-vm raid1]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[1]
9775424 blocks [2/1] [_U]

unused devices:

Check the status after removal,

[root@lnx-rhel5-vm raid1]# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.03
Creation Time : Wed Oct 20 02:02:46 2010
Raid Level : raid1
Array Size : 9775424 (9.32 GiB 10.01 GB)
Used Dev Size : 9775424 (9.32 GiB 10.01 GB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Wed Oct 20 02:41:20 2010
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

UUID : c42a6efb:849197a4:4935bd46:a423f66f
Events : 0.44

Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 17 1 active sync /dev/sdb1

You can just shutdown the server and pull the failed disk and replaced with the brand new disk of same size. Once the server back online, issue the following command to add new disk under raid array,

[root@lnx-rhel5-vm raid1]# mdadm --manage /dev/md0 --add /dev/sda8
mdadm: re-added /dev/sda8

[root@lnx-rhel5-vm raid1]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sda8[0] sdb1[1]
9775424 blocks [2/1] [_U]
[>....................] recovery = 1.0% (98752/9775424) finish=6.5min speed=24688K/sec

unused devices:

As you can see above, the reconstruction of array has started working after adding a new one.

References,
http://www.gagme.com/greg/linux/raid-lvm.php
http://scotgate.org/2006/07/03/growing-a-raid5-array-mdadm/
http://lkml.org/lkml/2006/3/20/105

There are several ways to get Perl modules from CPAN installed on your unix-based system. Keep in mind that there is always more than one way to do it with Perl, and this is no different. Before embarking upon any installation, it's a good idea to download the module, unzip it and check out the documentation. In general, though, most modules are installed in the same method.
The simplest way to get Perl modules installed is to use the CPAN module itself. If you are the system administrator and want to install the module system-wide, you'll need to switch to your root user. To fire up the CPAN module, just get to your command line and run this:

perl -MCPAN -e shell

If this is the first time you've run CPAN, it's going to ask you a series of questions - in most cases the default answer is fine. Once you find yourself staring at the cpan> command prompt, installing a module is as easy as install MODULE::NAME - for example, to install the HTML::Template module you'd type:

cpan> install HTML::Template

CPAN should take it from there and you'll wind up with the module installed into your Perl library.
Let's say you're on your system command line and you just want to install a module as quickly as possible - you can run the Perl CPAN module via command line perl and get it installed in a single line:

perl -MCPAN -e 'install HTML::Template'

As I mentioned earlier, it's always advisable to download a module yourself, especially if you're having problems installing with CPAN. If you're on the command line, you can use something like wget to grab the file. Next you'll want to unzip it with something like:

tar -zxvf HTML-Template-2.8.tar.gz

This will unzip the module into a directory, then you can move in and poke around - look for the README or INSTALL files. In most cases, installing a module by hand is still pretty easy, though (although not as easy as CPAN). Once you've switched into the base directory for the module, you should be able to get it installed by typing:

perl Makefile.PL
make
make test
make install

My Job As Sys Admin

Thursday, October 21, 2010

Growing a RAID5 array – MDADM

Sunday, October 17, 2010

mdadm: A Tool For Linux Software RAID Management

Saturday, October 2, 2010

How to connect VM's from a host OS

Tuesday, September 8, 2009

Kernel Panic and system hang senarios

JAVA OUT OF MEMORY ERROR

Install Mysql Server on Windows

Installing Perl Modules using CPAN

Followers

Blog Archive

About Me