Hardware RAID

Overview

In many new compute machines, disks have been formatted into RAID oriented format in manufacturer, so admin must create raid arrays using these disks manually before provisioning OS. How to configure raid arrays in unattended way for hundreds of machines turns to be a problem.

IBM has offered a tool iprconfig to configure raid for IBM power machine. To leverage this tool, xCAT enabled it in xCAT genesis.ppc64 so that admin can use genesis.ppc64 to configure RAID arrays.

There are two commands (diskdiscover and configraid) shipped in xCAT-genesis-scripts package to support RAID arrays configuration using iprconfig, you can use the runcmd facility to configure raid in the hardware discovery procedure using them, runcmd is a facility which will be run in xcat genesis system. You can also use separated manual steps to use configraid in xcat genesis system shell.

  • diskdiscover : Scan disk devices in xcat genesis system, give out disks and RAID arrays information.
  • configraid : Delete RAID arrays, create RAID arrays in xcat genesis system.

Following sections show how to use diskdiscover and configraid, we assume cn1 is compute node in all examples.

Discovering disk devices

Command diskdiscover scans disk devices, it can get the overview of disks and RAID arrays information from compute node; The outputs contain useful information for configraid to configure RAID arrays, user can get pci_id, pci_slot_name, disk names, RAID arrays and other informations from the outputs. It should be ran in xcat genesis system. It can be executed without input parameter or with pci_id, pci_id includes PCI vendor and device ID. For example, power8 SAS adapter pci_id is 1014:034a, 1014 is vendor info, 034a is PCI-E IPR SAS Adapter, more info about pci_id refer to http://pci-ids.ucw.cz/read/PC/1014/.

Here are steps to use diskdiscover:

  1. Start xCAT genesis system in compute node, let compute node cn1 enter xCAT genesis system shell:

    nodeset cn1 shell
rpower cn1 reset

    Note: If user modify diskdiscover or configraid scripts, he needs to run the mknb <arch> command before nodeset command to update network boot root image.

  2. On xcat management node, executing xdsh to use diskdiscover:

    xdsh cn1 diskdiscover

    Or:

    xdsh cn1 'diskdiscover <pci_id>'

    The outputs format is as following:

    # xdsh cn1 diskdiscover
cn1: --------------------------------------------------------------------------
cn1: PCI_ID     PCI_SLOT_NAME  Resource_Path  Device  Description   Status
cn1: ------     -------------  -------------  ------  -----------   ----------------
cn1: 1014:034a  0001:08:00.0   0:0:0:0        sg0     Function Disk Active
cn1: 1014:034a  0001:08:00.0   0:0:1:0        sg1     0 Array Member Active
cn1: -------------------------------------------------------------------
cn1: Get ipr RAID arrays by PCI_SLOT_NAME: 0001:08:00.0
cn1: -------------------------------------------------------------------
cn1: Name   PCI/SCSI Location         Description               Status
cn1: ------ ------------------------- ------------------------- -----------------
cn1: sda    0001:08:00.0/0:2:0:0       RAID 0 Disk Array         Optimized

Configuring hardware RAID

Command configraid introduction

We can use configraid to delete RAID arrays or create RAID arrays:

configraid delete_raid=[all|"<raid_array_list>"|null]
           stripe_size=[16|64|256]
           create_raid="rl#<raidlevel>|[pci_id#<num>|pci_slot_name#<pci_slot_name>|disk_names#<sg0>#..#<sgn>]|disk_num#<number>" ...

Here are the input parameters introduction:

  1. delete_raid : List raid arrays which should be removed.

    • If its value is all, all raid arrays detected should be deleted.
    • If its value is a list of raid array names, these raid arrays will be deleted. Raid array names should be separated by #.
    • If its value is null or there is no delete_raid, no raid array will be deleted.
    • If there is no delete_raid, the default value is null.

  2. stripe_size : It is optional used when creating RAID arrays. If stripe size is not specified, it will default to the recommended stripe size for the selected RAID level.

  3. create_raid : To create a raid array, add a line beginning with create_raid, all attributes keys and values are separated by #. The formats are as followings:

    • rl means RAID level, RAID level can be any supported RAID level for the given adapter, such as 0, 10, 5, 6. rl is a mandatory attribute for every create_raid. Supported RAID level is depend on physical server’s RAID adapter.

    • User can select disks based on following attributes value. User can find these value based on diskdiscover outputs as above section described.

      1. pci_id is PCI vendor and device ID.
      2. pci_slot_name is the specified PCI location. If using pci_slot_name, this RAID array will be created using disks from it.
      3. disk_names is a list of advanced format disk names. If using disk_names, this RAID array will be created using these disks.

    • disk_num is the number of disks this RAID array will contain, default value is all unused disks in its pci slot.

More examples of input parameters:

  1. Delete all original RAID arrays, create one RAID 10 array from pci_id 1014:034a, it uses the first two available disks:

    delete_raid=all create_raid="rl#10|pci_id#1014:034a|disk_num#2"

  2. Delete original RAID arrays sda and sdb on compute node, create one RAID 0 array from pci slot 0001:08:00.0, its RAID level is 0, it uses first two disks:

    delete_raid="sda#sdb" create_raid="rl#0|pci_slot_name#0001:08:00.0|disk_num#2"

  3. Create one RAID array from pci_id 1014:034a, RAID level is 0, stripe_size is 256kb, using first two available disks:

    stripe_size=256 create_raid="rl#0|pci_id#1014:034a|disk_num#2"

  4. Create two RAID arrays, RAID level is 0, one array uses one disks from pci_id 1014:034a, the other array uses two disks from pci_slot_name 0001:08:00.0:

    create_raid="rl#0|pci_id#1014:034a|disk_num#1" create_raid="rl#0|pci_slot_name#0001:08:00.0|disk_num#2"

  5. Create two RAID arrays, RAID level is 0, one array uses disks sg0 and sg1, the other array uses disks sg2 and sg3:

    create_raid="rl#0|disk_names#sg0#sg1" create_raid="rl#0|disk_names#sg2#sg3"

Configuring RAID arrays process

Command configraid is running in xcat genesis system, its log is saved under /tmp on compute node genesis system.

Configuring RAID in hardware discovery procedure

  1. Using runcmd facility to configure raid in the hardware discovery procedure, after configuring RAID, compute node enter xcat genesis system shell. In the following example, configraid deletes all original RAID arrays, it creates one RAID 0 array with first two disks from pci_id 1014:034a:

    nodeset cn1 runcmd="configraid delete_raid=all create_raid=rl#0|pci_id#1014:034a|disk_num#2",shell
rpower cn1 reset

  2. Using rcons to monitor the process:

    rcons cn1

Configuring RAID manually in xcat genesis system shell

  1. Starting xCAT genesis system in compute node, let compute node cn1 enter xCAT genesis system shell:

    nodeset cn1 shell
rpower cn1 reset

  2. On xcat management node, executing xdsh to use configraid to configure RAID:

    xdsh cn1 'configraid delete_raid=all create_raid="rl#0|pci_id#1014:034a|disk_num#2"'

Monitoring and debuging RAID configuration process

  1. Creating some RAID level arrays take very long time, for example, If user creates RAID 10, it will cost tens of minutes or hours. During this period, you can use xCAT xdsh command to monitor the progress of raid configuration.

    xdsh cn1 iprconfig -c show-config

  2. Logs for configraid is saved under tmp in compute node genesis system. User can login compute node and check configraid logs to debug.

  3. When configuring RAID in hardware discovery procedure, user can use rcons command to monitor or debug the process:

    rcons cn1