Intel® Rapid Storage Technology enterprise (Intel® RSTe) for Linux OS
Intel® Rapid Storage Technology enterprise (Intel® RSTe) for Linux OS Software User’s Guide June 2012 Document Number: 327602-001US I NFORMATI ON I N THI S DOCUMENT I S PROVI DED I N CONNECTI ON WI TH I NTEL® PRODUCTS. NO LI CENSE, EXPRESS OR
I MPLI ED, BY ESTOPPEL OR OTHERWI SE, TO ANY I NTELLECTUAL PROPERTY RI GHTS I S GRANTED BY THI S DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND I NTEL DI SCLAI MS ANY EXPRESS OR I MPLI ED WARRANTY, RELATI NG TO SALE AND/ OR USE OF I NTEL PRODUCTS I NCLUDI NG LI ABI LI TY OR WARRANTI ES RELATI NG TO FI TNESS FOR A PARTI CULAR PURPOSE, MERCHANTABI LI TY,
OR I NFRI NGEMENT OF ANY PATENT, COPYRI GHT OR OTHER I NTELLECTUAL PROPERTY RI GHT. I nt el product s are not int ended for
use in m edical, life saving, or life sust aining applicat ions.I nt el m ay m ake changes t o specificat ions and product descript ions at any t im e, wit hout not ice.
Designers m ust not rely on t he absence or charact erist ics of any feat ures or inst ruct ions m arked " reserved" or " undefined." I nt el
reserves t hese for fut ure definit ion and shall have no responsibilit y w hat soever for conflict s or incom pat ibilit ies arising from fut ure
changes t o t hem . The I nt el® Mat rix St orage Manager m ay cont ain design defect s or errors known as errat a which m ay cause t he product t o deviat e from published specificat ions. Current charact erized errat a are available on request .Cont act your local I nt el sales office or your dist ribut or t o obt ain t he lat est specificat ions and before placing your product order. I nt el, I nt el® Mat rix St orage Manager, I nt el® Mat rix St orage Technology, I nt el® Rapid Recover Technology, and t he I nt el logo are t radem arks or regist ered t radem arks of I nt el Corporat ion or it s subsidiaries in t he Unit ed St at es and ot her count ries.
- Ot her nam es and brands m ay be claim ed as t he propert y of ot hers. Copyright © 2012, I nt el Corporat ion. All right s reserved.
Contents
Figures
Figure 1. Mat rix RAI D ...................................................................................... 15 Figure 2. User Prom pt ...................................................................................... 26
Tables
Table 1. RAI D 0 Overview ................................................................................ 11 Table 2. RAI D 1 Overview ................................................................................ 12 Table 3. RAI D 5 Overview ................................................................................ 13 Table 4. RAI D 10 Overview ............................................................................... 14 Table 5 m dadm m onit or Param et ers .................................................................. 49 Table 6 Monit oring Event s ................................................................................ 50 Table 7 SGPI O Ut ilit y Opt ions ........................................................................... 60 Table 8 ledct l opt ions ...................................................................................... 61
Revision History
Document Revision Description Revision Date
Number Number327602 001 I nit ial Developer Release. June 2012
§
1 Introduction
The purpose of t his docum ent is t o enable a user t o properly set up and configure a syst em using t he Linux MDADM applicat ion for I nt el Mat rix St orage. I t provides st eps for set up and configurat ion, as w ell as a brief overview on Linux MDADM feat ures.
Note: The inform ation in this docum ent is only relevant on system s with a supported I ntel
chipset t hat include a support ed I nt el chipset , wit h a support ed operat ing syst em . Support ed I nt el chipset s -
Support ed operat ing syst em s -
Note: The m ajority of the inform ation in this docum ent is related to either software
configurat ion or hardw are int egrat ion. I nt el is not responsible for t he soft ware w rit t en by t hird part y vendors or t he im plem ent at ion of I nt el com ponent s in t he product s of t hird part y m anufact urers. Cust om ers should alw ays cont act t he place of purchase or syst em / soft w are m anufact urer w it h support quest ions about t heir specific hardw are or soft ware configurat ion.
1.1 Terminology Term Description
AHCI Advanced Host Cont roller I nt erface: an int erface specificat ion t hat allow s t he st or age dr iver t o enable advanced Serial ATA feat ures such as Nat iv e Com m and Queuing, nat ive hot plug, and pow er m anagem ent .
Cont inuous Updat e When a recovery volum e is using t his policy , dat a on t he m ast er dr ive Policy is copied t o t he recovery dr ive aut om at ically as long as bot h driv es
are connect ed t o t he syst em .
I nt el® Mat r ix St orage A code m odule built int o t he syst em BI OS t hat provides boot suppor t Manager Opt ion ROM for RAI D volum es as w ell as a user int erface for configuring and m anaging RAI D volum es. Mast er Dr ive The hard dr ive t hat is t he designat ed source dr ive in a r ecovery volum e.Mat rix RAI D Tw o independent RAI D volum es w it hin a single RAI D ar ray. Mem ber A hard dr ive used w it hin a RAI D ar ray. Hot - Plug* The unannounced r em oval and inser t ion of a Serial ATA hard dr ive
w hile t he syst em is pow ered on.
NCQ Nat ive Com m and Queuing: a com m and pr ot ocol in Ser ial ATA t hat allow s m ult iple com m ands t o be out st anding w it hin a hard drive at t he sam e t im e. The com m ands are dynam ically reor dered t o increase hard drive perfor m ance.
On Request Updat e When a recovery volum e is using t his policy , dat a on t he m ast er dr ive Policy is copied t o t he recovery dr ive w hen you request it . Only changes since t he last updat e process are copied. OS Operat ing Syst em Port 0 A serial ATA port ( connect or ) on a m ot herboard ident ified as Port 0.
Port 1 A serial ATA port ( connect or ) on a m ot herboard ident ified as Port 1. Port 2 A serial ATA port ( connect or ) on a m ot herboard ident ified as Port 2. Port 3 A serial ATA port ( connect or ) on a m ot herboard ident ified as Port 3. POST Pow er- On Self Test SAS Serial At t ached SCSI SCU SAS Cont roller Unit
Term Description RAI D Redundant Ar r ay of I ndependent Driv es: allow s dat a t o be dist ribut ed across m ult iple hard dr ives t o pr ovide dat a r edundancy or t o enhance dat a st or age perfor m ance. RAI D 0 ( st riping) The dat a in t he RAI D volum e is st r iped across t he array's m em bers.
St riping divides dat a int o unit s and dist r ibut es t hose unit s across t he m em ber s w it hout creat ing dat a r edundancy, but im proving read/ writ e perform ance. RAI D 1 ( m irror ing) The dat a in t he RAI D volum e is m irror ed acr oss t he RAI D array's m em ber s. Mirr or ing is t he t er m used t o describe t he key feat ure of
RAI D 1, w hich w rit es duplicat e dat a t o each m em ber ; t herefore, creat ing dat a r edundancy and incr easing fault t olerance. RAI D 5 ( st riping w it h The dat a in t he RAI D volum e and parit y are st riped across t he ar ray's par it y ) m em ber s. Par it y infor m at ion is w r it t en w it h t he dat a in a r ot at ing sequence across t he m em ber s of t he ar ray . This RAI D level is a preferred configurat ion for efficiency, fault - t olerance, and perform ance. RAI D 10 ( st riping and The RAI D lev el w here inform at ion is st r iped across a t wo disk ar ray for m irror ing) syst em perfor m ance. Each of t he dr ives in t he ar ray has a m irror for fault t olerance. RAI D 10 pr ovides t he perfor m ance benefit s of RAI D 0 and t he redundancy of RAI D 1. How ever , it requir es four hard dr ives. RAI D Array A logical gr ouping of physical hard drives. RAI D Volum e A fixed am ount of space across a RAI D ar ray t hat appears as a single physical hard drive t o t he operat ing syst em . Each RAI D volum e is creat ed w it h a specific RAI D level t o provide dat a redundancy or t o enhance dat a st or age perfor m ance. Recovery Dr ive The hard dr ive t hat is t he designat ed t arget dr ive in a r ecovery volum e.
Recovery Volum e A volum e ut ilizing I nt el( R) Rapid Recover Technology.
10 Kilobyt e Unit m ount for 1024 byt es or 2 byt es 20 Megabyt e Unit am ount for 2 byt es m dadm m dadm is a Linux ut ilit y creat ed by Neil Brown t o m anage soft wareRAI D dev ices on Linux. I t is available under t he GPL license version 2 or lat er .
1.2 Reference Documents Document Document No./Location
m dadm m anpages Linux m anpages
Ledm on m anpages Linux m anpages
SMP Ut ils m anpages Linux m anpages
2 Intel® Matrix Storage Manager
Features2.1 Feature Overview
The I nt el® Mat rix St orage Manager soft w are package provides high- perform ance Serial ATA and Serial ATA RAI D capabilit ies for support ed operat ing syst em s. Support ed operat ing syst em s -
The key feat ures of t he I nt el® Mat rix St orage Manager are as follow s: RAI D 0 RAI D 1 RAI D 5 RAI D 10 Matrix RAI D I ntel® Rapid Recover Technology Advanced Host Controller I nterface (AHCI ) support SAS Controller Unit (SCU) support
2.2 RAID 0 (Striping)
RAI D 0 uses t he read/ writ e capabilit ies of t w o or m ore hard drives working in parallel t o m axim ize t he st orage perform ance of a com put er syst em . provides an overview of t he advant ages, t he level of fault - t olerance provided, and t he t ypical usage of RAI D 0.
Table 1. RAID 0 Overview 2- 6
Hard Drives Required: Highest t ransfer r at es
Advantage: None – if one disk fails all data will be lost
Fault- tolerance: Typically used in deskt ops and w or kst at ions for m axim um perform ance for
Application: t em porar y dat a and high I / O rat e. 2 - drive RAI D 0 available in specific m obile configurat ions.
Refer t o t he following w eb sit e for m ore inform at ion on RAI D 0:
2.3 RAID 1 (Mirroring)
A RAI D 1 array cont ains t w o hard drives w here t he dat a bet w een t he t w o is m irrored in real t im e t o provide good dat a reliabilit y in t he case of a single disk failure; w hen one disk drive fails, all dat a is im m ediat ely available on t he ot her w it hout any im pact t o t he int egrit y of t he dat a.
provides an overview of t he advant ages, t he level of fault - t olerance provided, and t he t ypical usage of RAI D 1.
Table 2. RAID 1 Overview Hard Drives Required:
2 Advantage: 100% redundancy of dat a. One disk m ay fail, but dat a w ill cont inue t o be accessible. A r ebuild t o a new disk is recom m ended t o m aint ain dat a redundancy .
Fault- tolerance: Excellent
- – disk m irroring m eans that all data on one disk is duplicated on anot her disk.
Application: Typically used for sm aller syst em s w here capacit y of one disk is sufficient and for any applicat ion( s) requiring very high availabilit y. Available in specific m obile configurat ions.
Refer t o t he following w eb sit e for m ore inform at ion on RAI D 1:
2.4 RAID 5 (Striping with Parity)
A RAI D 5 array cont ains t hree or m ore hard drives where t he dat a and parit y are st riped across all t he hard drives in t he array. Parit y is a m at hem at ical m et hod for recreat ing dat a t hat w as lost from a single drive, which increases fault - t olerance. I f t here are N disks in t he RAI D 5 volum e, t he capacit y for dat a w ould be N – 1 disks. For exam ple, if t he RAI D 5 volum e has 5 disks, t he dat a capacit y for t his RAI D volum e consist s of four disks.
Linux MDRAI D support s four t ypes of parit y layout . How ever, I nt el I MSM only support s t he left - asym m et ric parit y layout . provides an overview of t he advant ages, t he level of fault - t olerance provided, and t he t ypical usage of RAI D 5.
Table 3. RAID 5 Overview 3- 6
Hard Drives Required: Higher percent age of usable capacit y and high read perfor m ance as w ell as
Advantage: fault - t olerance.
Excellent - par it y infor m at ion allow s dat a t o be rebuilt aft er replacing a failed Fault- tolerance: hard drive w it h a new drive.
St orage of large am ount s of crit ical dat a. Not available in m obile Application: configurat ions.
Refer t o t he following w eb sit e for m ore inform at ion on RAI D 5:
2.5 RAID 10 A RAI D 10 array uses four hard drives t o creat e a com binat ion of RAI D levels 0 and 1.
I t is a st riped set whose m em bers are each a m irrored set . provides an overview of t he advant ages, t he level of fault - t olerance provided, and t he t ypical usage of RAI D 10.
Table 4. RAID 10 Overview Hard Drives Required:
4 Advantage: Com bines t he read perfor m ance of RAI D 0 w it h t he fault - t olerance of RAI D
1. Fault-
- – disk m irroring m eans that all data on one disk is duplicated on anot her disk.
tolerance: Excellent
Application: High- perfor m ance applicat ions requiring dat a pr ot ect ion, such as video edit ing. Not av ailable in m obile configurat ions.
Refer t o t he following w eb sit e for m ore inform at ion on RAI D 10:
2.6 Matrix RAID Mat rix RAI D allow s you t o creat e t w o RAI D volum es on a single RAI D array.
As an exam ple, on a syst em w it h an I nt el® 82801GR I / O cont roller hub ( I CH7R) , I nt el® Mat rix St orage Manager allow s you t o creat e bot h a RAI D 0 volum e as w ell as a RAI D 5 volum e across four Serial ATA hard drives. An im port ant requirem ent t he Mat rix RAI D has is t hat in a Mat rix RAI D cont ainer, t he volum es inside t he cont ainer m ust span t he sam e set of m em ber disks. Refer t o
Figure 1. Matrix RAID
Refer t o t he following w eb sit e for m ore inform at ion on m at rix RAI D:
2.7 Advanced Host Controller Interface
Advanced Host Cont roller I nt erface ( AHCI ) is an int erface specificat ion t hat allows t he st orage driver t o enable advanced Serial ATA feat ures such as Nat ive Com m and Queuing and Nat ive Hot - Plug. I nt el chipset s t hat support AHCI :
2.7.1 Native Command Queuing
Nat ive Com m and Queuing ( NCQ) is a feat ure suppor t ed by AHCI t hat allow s Serial ATA hard drives t o accept m ore t han one com m and at a t im e. NCQ, w hen used in conj unct ion w it h one or m ore hard dr ives t hat support NCQ, increases st orage perform ance on random w orkloads by allowing t he drive t o int ernally opt im ize t he order of com m ands.
Note: To take advantage of NCQ, you need the following:
Chipset t hat support s AHCI I nt el® Mat rix St orage Manager One or m ore Serial ATA ( SATA) hard drives t hat support NCQ
2.7.2 Hot-Plug
Hot - Plug, also referred t o as hot swap, is a feat ure support ed by AHCI t hat allows Serial ATA hard drives t o be rem oved or insert ed w hile t he syst em is pow ered on and running. As an exam ple, Hot - Plug m ay be used t o replace a failed hard drive t hat is in an ext ernally- accessible drive enclosure.
Note: To take advantage of Hot-Plug, you need the following:
Chipset t hat support s AHCI
I nt el® Mat rix St orage Manager Hot - Plug capabilit y correct ly enabled in t he syst em BI OS by t he OEM/ m ot her board m anufact urer
2.8 SAS Controller Unit
SCU is t he I nt el® Serial At t ached SCSI Cont roller Unit t hat is part of t he C600 fam ily Plat form Cont roller Hub. The Linux SCU driver ( isci) has been upst ream ed t o t he Linux kernel since kernel version v3.0. How ever, t he lat est Linux kernel is alw ays recom m ended t o get t he lat est bug fixes and feat ur es.
2.8.1 SCU OEM Parameters
The SCU driver requires proper OEM param et ers t o be loaded in order t o set t he correct PHY set t ings. The appropriat e OEM param et ers shall be loaded from t he plat form eit her from t he OROM region if boot ing legacy or via EFI variable m echanism if boot ing EFI . Below is an exam ple of what you m ay see from t he isci driver load. The correct driver m essage displayed should be t hat t he OEM param et er is loaded from “platform”. This indicates the driver has found good OEM parameter from the OROM or EFI . isci: Intel(R) C600 SAS Controller Driver - version 1.1.0 isci 0000:03:00.0: driver configured for rev: 5 silicon
isci 0000:03:00.0: OEM parameter table found in OROM
isci 0000:03:00.0: OEM SAS parameters (version: 1.1) loaded (platform)isci 0000:03:00.0: SCU controller 0: phy 3-0 cables: {short, short, short, short} scsi6 : isci isci 0000:03:00.0: SCU controller 1: phy 3-0 cables: {short, short, short, short} scsi7 : isci isci 0000:03:00.0: irq 110 for MSI/MSI-X isci 0000:03:00.0: irq 111 for MSI/MSI-X isci 0000:03:00.0: irq 112 for MSI/MSI-X isci 0000:03:00.0: irq 113 for MSI/MSI-X
2.8.2 Linux libsas Sysfs Components
Linux provides driver inform at ion t hrough sysfs, a virt ual file syst em . The exam ple below provides som e inform at ion on som e of t he libsas relat ed com ponent s t hat can be useful or inform at ional. The sas relat ed ent ries can be found in / sys/ class sysfs direct ory.
ls -1 /sys/class/ | grep sas
sas_device sas_end_device sas_expander sas_host sas_phy sas_port The sas_host direct ory cont ains all HBA at t ached t o t he com put er syst em :
ls -l /sys/class/sas_host/
t ot al 0 lrw xrw xrw x 1 root root 0 May 18 13: 45 host 6 - > ../ ../ devices/ pci0000: 00/ 0000: 00: 01.0/ 0000: 01: 00.0/ 0000: 02: 08.0/ 0000: 03: 00.0/ ho st 6/ sas_host / host 6 lrw xrw xrw x 1 root root 0 May 18 13: 45 host 7 - > ../ ../ devices/ pci0000: 00/ 0000: 00: 01.0/ 0000: 01: 00.0/ 0000: 02: 08.0/ 0000: 03: 00.0/ ho st 7/ sas_host / host 7 Generally t he /sys/class/sas_* direct ories cont ain sym bolic links. Due t o t hose sym bolic links can be quit e long, in t he follow on exam ples t hey w ill be om it t ed and only link nam es w ill be show n.
ls -1 /sys/class/sas_expander/
expander - 6: 0 expander - 7: 0 I n t he expander-6:0 direct ory, 12 disks are shown t o be at t ached. This can be validat ed by looking int o sas_end_device direct ory where all SAS end devices are list ed.
Below show s t he devices at t ached t o expander-6:0:
ls -1 /sys/class/sas_end_device/ | grep end_device-6
end_device- 6: 0: 10 end_device- 6: 0: 11 end_device- 6: 0: 12 end_device- 6: 0: 13 end_device- 6: 0: 14 end_device- 6: 0: 15 end_device- 6: 0: 24 end_device- 6: 0: 4 end_device- 6: 0: 5 end_device- 6: 0: 6 end_device- 6: 0: 7 end_device- 6: 0: 8 end_device- 6: 0: 9 The exam ple above show s t hat t he first four PHYs in t he expander are m issing , and
t h 24 phy is an ext ra virt ual phy t hat is used by t he expander int ernally .
The sas_phy direct ory cont ains all phys in t he syst em , and t he sas_port cont ains all port s in t he syst em .
To see t he connect ion bet w een com ponent s, it is bet t er t o t ransverse from t he sas_host direct ory. What is connect ed t o host 6 can be seen her e:
ls -l /sys/class/sas_host/host6/device/
t ot al 0 drw xr- xr- x 3 root root 0 May 18 09: 27 bsg drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 1 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 2 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 3 drw xr- xr- x 5 root root 0 May 18 09: 27 port - 6: 0 drw xr- xr- x 2 root root 0 May 18 14: 37 pow er drw xr- xr- x 3 root root 0 May 18 09: 27 sas_host drw xr- xr- x 3 root root 0 May 18 09: 27 scsi_host lrw xrw xrw x 1 root root 0 May 18 09: 27 subsyst em - > ../ ../ ../ ../ ../ ../ ../ bus/ scsi
- rw - r- - r- - 1 root root 4096 May 18 09: 27 uevent And m ore:
ls -l /sys/class/sas_host/host6/device/port-6\:0/
t ot al 0 drw xr- xr- x 45 root root 0 May 18 09: 27 expander- 6: 0 lrw xrw xrw x 1 root root 0 May 18 14: 36 phy - 6: 0 - > ../ phy - 6: 0 lrw xrw xrw x 1 root root 0 May 18 14: 36 phy - 6: 1 - > ../ phy - 6: 1 lrw xrw xrw x 1 root root 0 May 18 14: 36 phy- 6: 2 - > ../ phy - 6: 2 lrw xrw xrw x 1 root root 0 May 18 14: 36 phy - 6: 3 - > ../ phy - 6: 3 drw xr- xr- x 2 root root 0 May 18 14: 36 pow er drw xr- xr- x 3 root root 0 May 18 09: 27 sas_port
- rw - r- - r- - 1 root root 4096 May 18 09: 27 uevent Host 6 has 4 phys t hat are configured as a wide port , and t his wide port is a connect ion bet w een t he host and t he expander ( expander-6:0) .
Below are t he cont ent s of t he expander-6:0 direct ory:
ls -l /sys/class/sas_host/host6/device/port-6\:0/expander-6\:0/
t ot al 0 drw xr- xr- x 3 root root 0 May 18 09: 27 bsg drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 0 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 1 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 10 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 11 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 12 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 13 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 14 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 15 drw xr- xr- x 4 root root 0 May 18 09: 27 phy- 6: 0: 16 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 17 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 18 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 19 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 2 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 20 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 21 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 22 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 23 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 24 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 3 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 4 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 5 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 6 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 7 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 8 drw xr- xr- x 4 root root 0 May 18 09: 27 phy - 6: 0: 9 drw xr- xr- x 5 root root 0 May 18 09: 27 port - 6: 0: 10 drw xr- xr- x 5 root root 0 May 18 09: 27 port - 6: 0: 11 drw xr- xr- x 5 root root 0 May 18 09: 27 port - 6: 0: 12 drw xr- xr- x 5 root root 0 May 18 09: 27 port - 6: 0: 13 drw xr- xr- x 5 root root 0 May 18 09: 27 port - 6: 0: 14 drw xr- xr- x 5 root root 0 May 18 09: 27 port - 6: 0: 15 drw xr- xr- x 4 root root 0 May 18 09: 27 port - 6: 0: 16 drw xr- xr- x 5 root root 0 May 18 09: 27 port - 6: 0: 24 drw xr- xr- x 5 root root 0 May 18 09: 27 port - 6: 0: 4 drw xr- xr- x 5 root root 0 May 18 09: 27 port - 6: 0: 5 drw xr- xr- x 5 root root 0 May 18 09: 27 port - 6: 0: 6 drw xr- xr- x 5 root root 0 May 18 09: 27 port - 6: 0: 7 drw xr- xr- x 5 root root 0 May 18 09: 27 port - 6: 0: 8 drw xr- xr- x 5 root root 0 May 18 09: 27 port - 6: 0: 9 drw xr- xr- x 2 root root 0 May 18 14: 37 pow er drw xr- xr- x 3 root root 0 May 18 09: 27 sas_device drw xr- xr- x 3 root root 0 May 18 09: 27 sas_expander
- rw - r- - r- - 1 root root 4096 May 18 09: 27 uevent The expander has 24 phys and 12 narr ow port s. The cont ent s of port-6:0:4 reveals t hat som e end devices are connect ed t o t hat port / phy:
ls -l /sys/class/sas_host/host6/device/port-6\:0/expander-6\:0/port- 6\:0\:4
t ot al 0 drw xr- xr- x 7 root root 0 May 18 09: 27 end_device- 6: 0: 4 lrw xrw xrw x 1 root root 0 May 18 14: 51 phy - 6: 0: 4 - > ../ phy - 6: 0: 4 drw xr- xr- x 2 root root 0 May 18 14: 51 pow er drw xr- xr- x 3 root root 0 May 18 09: 27 sas_port
- rw - r- - r- - 1 root root 4096 May 18 09: 27 uevent
The Linux disk nam e can be found a few levels deeper:
ls -l /sys/class/sas_host/host6/device/port-6\:0/expander-6\:0/port- 6\:0\:4/end_device-6\:0\:4/target6\:0\:0/6\:0\:0\:0/block/
t ot al 0 drw xr- xr- x 7 root root 0 May 18 09: 27 sdb I t can also be found in a different sysfs locat ion:
ls -l /sys/block/ | grep port-6:0:4
lrw xrw xrw x 1 root root 0 May 18 09: 27 sdb - > ../ devices/ pci0000: 00/ 0000: 00: 01.0/ 0000: 01: 00.0/ 0000: 02: 08.0/ 0000: 03: 00.0/ host 6/ port - 6: 0/ expander- 6: 0/ port - 6: 0: 4/ end_device- 6: 0: 4/ t ar get 6: 0: 0/ 6: 0: 0: 0/ block/ sdb This is an Expander At t ached ( EA) configurat ion. I n a Direct At t ached ( DA) configurat ion t he cont ent s of host 6 m ay look like:
ls -l /sys/class/sas_host/host6/device/
t ot al 0 drw xr- xr- x 3 root root 0 May 18 09: 39 bsg drw xr- xr- x 4 root root 0 May 18 09: 17 phy - 6: 0 drw xr- xr- x 4 root root 0 May 18 09: 17 phy - 6: 1 drw xr- xr- x 4 root root 0 May 18 09: 17 phy - 6: 2 drw xr- xr- x 4 root root 0 May 18 09: 17 phy - 6: 3 drw xr- xr- x 5 root root 0 May 18 09: 06 port - 6: 0 drw xr- xr- x 5 root root 0 May 18 09: 02 port - 6: 1 drw xr- xr- x 5 root root 0 May 18 09: 08 port - 6: 2 drw xr- xr- x 5 root root 0 May 18 09: 02 port - 6: 3 drw xr- xr- x 2 root root 0 May 18 09: 39 pow er drw xr- xr- x 3 root root 0 May 18 09: 39 sas_host drw xr- xr- x 3 root root 0 May 18 09: 39 scsi_host lrw xrw xrw x 1 root root 0 May 18 09: 09 subsyst em - > ../ ../ ../ ../ ../ ../ ../ bus/ scsi
- rw - r- - r- - 1 root root 4096 May 18 09: 09 uevent
There are 4 phys and 4 narrow port s, and t his m eans t he 4 end devices are connect ed direct ly t o t he HBA. This can be show n:
ls -l /sys/class/sas_end_device/ | grep end_device-6
lrw xrw xrw x 1 root root 0 May 18 09: 16 end_device- 6: 0 - > ../ ../ devices/ pci0000: 00/ 0000: 00: 01.0/ 0000: 01: 00.0/ 0000: 02: 08.0/ 0000: 03: 00.0/ ho st 6/ port - 6: 0/ end_device- 6: 0/ sas_end_device/ end_device- 6: 0 lrw xrw xrw x 1 root root 0 May 18 09: 16 end_device- 6: 1 - > ../ ../ devices/ pci0000: 00/ 0000: 00: 01.0/ 0000: 01: 00.0/ 0000: 02: 08.0/ 0000: 03: 00.0/ ho st 6/ port - 6: 1/ end_device- 6: 1/ sas_end_device/ end_device- 6: 1 lrw xrw xrw x 1 root root 0 May 18 09: 16 end_device- 6: 2 - > ../ ../ devices/ pci0000: 00/ 0000: 00: 01.0/ 0000: 01: 00.0/ 0000: 02: 08.0/ 0000: 03: 00.0/ ho st 6/ port - 6: 2/ end_device- 6: 2/ sas_end_device/ end_device- 6: 2 lrw xrw xrw x 1 root root 0 May 18 09: 16 end_device- 6: 3 - > ../ ../ devices/ pci0000: 00/ 0000: 00: 01.0/ 0000: 01: 00.0/ 0000: 02: 08.0/ 0000: 03: 00.0/ ho st 6/ port - 6: 3/ end_device- 6: 3/ sas_end_device/ end_device- 6: 3 Or by:
ls -l /sys/block/ | grep end_device-6
lrw xrw xrw x 1 root root 0 May 18 09: 09 sdb - > ../ devices/ pci0000: 00/ 0000: 00: 01.0/ 0000: 01: 00.0/ 0000: 02: 08.0/ 0000: 03: 00.0/ host 6/ port - 6: 0/ end_device-6:0/ t arget 6: 0: 4/ 6: 0: 4: 0/ block/ sdb lrw xrw xrw x 1 root root 0 May 18 09: 09 sdc - > ../ devices/ pci0000: 00/ 0000: 00: 01.0/ 0000: 01: 00.0/ 0000: 02: 08.0/ 0000: 03: 00.0/ host 6/ port - 6: 1/ end_device-6:1/ t arget 6: 0: 1/ 6: 0: 1: 0/ block/ sdc lrw xrw xrw x 1 root root 0 May 18 09: 09 sdd - > ../ devices/ pci0000: 00/ 0000: 00: 01.0/ 0000: 01: 00.0/ 0000: 02: 08.0/ 0000: 03: 00.0/ host 6/ port - 6: 2/ end_device-6:2/ t arget 6: 0: 5/ 6: 0: 5: 0/ block/ sdd lrw xrw xrw x 1 root root 0 May 18 09: 09 sde - > ../ devices/ pci0000: 00/ 0000: 00: 01.0/ 0000: 01: 00.0/ 0000: 02: 08.0/ 0000: 03: 00.0/ host 6/ port - 6: 3/ end_device-6:3/ t arget 6: 0: 3/ 6: 0: 3: 0/ block/ sde The st ruct ure of t he sysfs filesyst em is quit e com plex and full of links point ing t o links.
Full descript ion of sysfs is out side of t he scope of t his m anual. This chapt er discussed only very briefly som e m aj or sysfs direct ories relat ed t o t he SCU driver .
3 RAID BIOS / EFI Configuration
3.1 Overview
To inst all t he I nt el® Mat rix St orage Manager, t he syst em BI OS m ust include t he I nt el® Mat rix St orage Manager opt ion ROM or EFI driver. The I nt el® Mat rix St orage Manager opt ion ROM / EFI driver is t ied t o t he cont roller hub. For det ailed docum ent at ion please see t he
3.2 Enabling RAID in BIOS
To enable RAI D in t he syst em BI OS, refer t o t he m ot herboard or syst em docum ent at ion or cont act t he m ot herboard or syst em m anufact urer or place of purchase for specific inst ruct ions.
4 Intel® Matrix Storage Manager
Option ROM4.1 Overview
The I nt el® Mat rix St orage Manager opt ional ROM is a PnP opt ion ROM t hat provides a pre- operat ing syst em user int erface for RAI D configurat ions. I t also provides BI OS and DOS disk services ( I nt 13h) .
4.2 User Interface
To ent er t he I nt el® Mat rix St orage Manager opt ional ROM user int erface, press t he < Ct rl> and < i> keys sim ult aneously w hen prom pt ed during t he Pow er- On Self Test ( POST) . Refer t o
Figure 2. User Prompt NOTE: The hard drive(s) and hard drive inform ation listed for your system can differ from the follow ing exam ple.
4.3 Version Identification
To ident ify t he specific version of t he I nt el® Mat rix St orage Manager opt ion ROM int egrat ed int o t he syst em BI OS, ent er t he opt ion ROM user int erface. The version num ber is locat ed in t he t op right corner w it h t he following form at : vX.Y.W.XXXX, w here X and Y are t he m aj or and m inor version num bers.
4.4 RAID Volume Creation
Use t he follow ing st eps t o creat e a RAI D volum e using t he I nt el® Mat rix St orage Manager user int erface:
Note: The following procedure should only be used with a newly-built system or if you are
reinst alling your operat ing syst em . The following procedure should not be used t o m igrat e an exist ing syst em t o RAI D 0. I f you wish t o creat e m at rix RAI D volum es aft er t he operat ing syst em soft w are is loaded, t hey should be creat ed using t he MDADM t ool in t he Linux dist ribut ion.
1. Press the < Ctrl> and < i> keys sim ultaneously when the following window appears during POST:
2. Select option 1. Create RAID Volume and press the < Enter> key.
3. Type in a volum e nam e and press the < Enter> key, or press the < Enter> key to accept t he default nam e.
4. Select the RAI D level by using the < > or < > keys to scroll through the available values, t hen press t he < Ent er> key.
5. Press the < Enter> key to select the physical disks. A dialog sim ilar to the following w ill appear:
6. Select the appropriate num ber of hard drives by using the < > or < > keys to scroll t hrough t he list of available hard drives. .Press t he < Space> key t o select a drive. When you have finished select ing hard drives, press t he < Ent er> key.
7. Unless you have selected RAI D 1, select the strip size by using the < > or < > keys t o scroll t hrough t he available values, t hen press t he < Ent er> key.
8. Select the volum e capacity and press the < Enter> key. The default value indicates t he m axim um volum e capacit y using t he select ed disks. I f less t han t he m axim um volum e capacit y is chosen, creat ion of a second v olum e is needed t o ut ilize t he rem aining space ( i.e. a m at rix RAI D configurat ion) .
9. At the Create Volum e prom pt, press the < Enter> key to create the volum e. The following prom pt w ill appear:
10. Press the < Y> key to confirm volum e creation.
11. To exit the option ROM user interface, select option 5. Exit and press the < Enter> key.
12. Press the < Y> key again to confirm exit.
Note: To change any of the inform ation before the volum e creation has been confirm ed, you
m ust exit t he Creat e Volum e process and rest art it . Press t he < Esc> key t o exit t he Creat e Volum e process.
5 Volume Creation
RAI D volum es can be creat ed using t he m dadm com m and line ut ilit y. Mdadm support s t he I nt el Mat rix St orage Manager ( I MSM) m et a dat a form at w hen specified wit h t he
I MSM m et a dat a opt ion.
5.1 RAID Volume Creation
Warning: Creating a RAI D volum e will perm anently delete any existing data on the selected
hard drives. Back up all im port ant dat a before beginning t hese st eps. Below is an exam ple t o creat e a RAI D5 volum e w it h 6 disks: First a cont ainer of I nt el I MSM m et adat a m ust be creat ed.
1. mdadm -C /dev/md0 /dev/sd[b-g] –n 6 –e imsm y
Continue creating array? mdadm: container /dev/md0 prepared. The com m and creat es a RAI D cont ainer w it h I nt el® Mat rix St orage Manager m et adat a for m at . The device node for t he cont ainer w ill be /dev/md0. I n t his exam ple disks sdb, sdc, sdd, sde, sdf, and sdg are used for t his RAI D set , and t he t ot al num ber of disks is 6. The w ildcard expression /dev/sd[b-g] can be used t o specify t he range of disks. Alt hough individual disks can be used t o list out all t he disks. i.e. /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg
Next a RAI D 5 volum e is creat ed.
2.
mdadm -C /dev/md/Volume0 /dev/md0 –n 6 –l 5
The com m and creat es a RAI D 5 volum e / dev/ m d/ Volum e0 wit hin t he /dev/md0 cont ainer. The following com m and param et ers m ay also be used in conj unct ion t o give finer cont rol for t he creat ion of t he RAI D volum e.
- n : Num ber of act ive RAI D devices t o be used in t he volum e.
- x : Specifies t he num ber of spare dev ices in t he init ial array.
- c : Specifies t he chunk ( st ripe) size in Kilobyt es. The default is 512KiB.
- l : Specifies t he RAI D level. The opt ions are 0, 1, 5, 10.
- z : Specifies t he size ( in Kilobyt es) of space dedicat ed on each disk t o t he RAI D volum e. This m ust be a m ult iple of t he chunk size. For exam ple:
mdadm –C /dev/md/Volume0 /dev/md0 –n 3 –l 5 –z $((100*1024*1024))
The com m and above creat es a RAI D volum e inside t he / dev/ m d0 cont ainer wit h 100GB of disk space used on each disk m em ber.
5.2 Filesystem Creation on RAID Volume
Aft er t he RAI D volum e has been creat ed, a filesyst em can be creat ed in order t o allow t he m ount ing of t he RAI D volum e.
mkfs.ext4 /dev/md/Volume0
Once t he filesyst em has been creat ed, it can be m ount ed:
mount /dev/md/Volume0 /mnt/myraidvolume
5.3 RAID Volume Creation Examples
To creat e a RAI D 0 volum e, use t he following exam ple:
mdadm –C /dev/md/Volume0 /dev/md0 –n 2 –l 0
To creat e a RAI D 1 volum e, use t he following exam ple:
mdadm –C /dev/md/Volume0 /dev/md0 –n 2 –l 1
To creat e a RAI D 5 volum e, use t he following exam ple:
mdadm –C /dev/md/Volume0 /dev/md0 –n 3 –l 5
To creat e a RAI D 10 volum e, use t he following exam ple:
mdadm –C /dev/md/Volume0 /dev/md0 –n 4 –l 10
Note: To creat e m ult iple RAI D volum es in t he sam e cont ainer, t hey MUST span equal
num ber of disks. For exam ple, in order t o have a RAI D 0 volum e and a RAI D 5 volum e in t he sam e cont ainer, four disks m ust be used for bot h volum es.
5.4 Adding a Spare Disk
Adding a spare disk allows im m ediat e reconst ruct ion of t he RAI D volum e w hen a device failure is det ect ed. Mdraid will mark the failed device as “bad” and start reconst ruct ion w it h t he first available spare disk. The spare disk can also be used t o grow t he RAI D volum e. The spare disks sit idle during norm al operat ions. When using m dadm w it h I MSM m et a dat a, t he spare disk added t o a cont ainer is dedicat ed t o t hat specific cont ainer. The follow ing com m and adds a spare disk t o t he designat ed cont ainer.
mdadm -a /dev/md0 /dev/sde
5.5 Creating RAID Configuration File
A configurat ion file can be creat ed t o r ecord t he exist ing RAI D volum es. The inform at ion can be ext ract ed from t he exist ing RAI D set up. The configurat ion file is t ypically st ored at t he default locat ion of / et c/ m dadm .conf. This allow s a consist ent assem ble of t he appropriat e RAI D volum es.
mdadm -E –s –-config=mdadm.conf > /etc/mdadm.conf
5.6 RAID Volume Initialization / Resync
I m m ediat ely aft er a RAI D volum e has been creat ed, init ializat ion ( or resync) com m ences if t he RAI D level is 1, 10, or 5. During t his t im e, any dat a st ored on RAI D level 5 volum es are not guarant eed t o be safe if failure occurs. I f a disk failure happens during t he init ializat ion t im e, recovery w ill not be possible. This scenario is also t rue during RAI D volum e rebuilds.
6 Volume Operations
m dadm provides various opt ions t o assem ble, m onit or, exam ine, or st op RAI D volum es.
6.1 Erasing RAID Metadata
Having incorrect and bad RAI D m et adat a can cause RAI D volum es t o be assem bled incorrect ly. The m et adat a can be erased wit h t he following com m and t o m ake sure t he disk is clean. This operat ion does not at t em pt t o w ipe exist ing user dat a.
mdadm --zero-superblock /dev/sdb Mult iple disks can be specified t o clear t he superblock at t he sam e t im e.
6.2 Volume Assemble
RAI D volum es can be creat ed via OROM user int erface or m dadm . I nact ive RAI D volum es t hat are creat ed can be act ivat ed using t he assem ble opt ion wit h m dadm . The following com m and scans for t he m dadm configurat ion file at /etc/mdadm.conf in order t o assem ble t he RAI D volum es. I f t he configurat ion file is not found, it scans all available disks for RAI D m em ber disks and assem bles all t he RAI D volum es:
mdadm –A –s
To m anually assem ble and act ivat e RAI D volum es w it hout t he configurat ion file, t he following exam ple can be used:
mdadm –A /dev/md0 –e imsm /dev/sda /dev/sdb /dev/sdc /dev/sdd
mdadm- –I /dev/md0
The first com m and assem bles t he cont ainer wit h t he nam e /dev/md0 using t he provided list of disks. The second com m and assem bles and act ivat es appropriat e volum es wit h t he device nodes.
6.3 Stopping the Volumes
To st op all act ive RAI D volum es, t he following com m and can be used. Mdadm w ill scan for and st op all running RAI D volum es and cont ainers.
mdadm –S –s How ever, RAI D volum e nam es can be specified t o st op t he volum e direct ly. mdadm –S /dev/md/Volume0 And t o st op a cont ainer, t he following com m and can be used. mdadm –S /dev/md0
6.4 Reporting RAID Information
Use t he follow ing com m and, t o print out det ails about a RAI D cont ainer or volum e:
mdadm –D /dev/md0
/dev/md0: Version : imsm Raid Level : container Total Devices : 5 Working Devices : 5 UUID : b559b502:b199f86f:ee9fbd40:cd10e91d Member Arrays : Number Major Minor RaidDevice 0 8 32 - /dev/sdc 1 8 48 - /dev/sdd 2 8 80 - /dev/sdf 3 8 96 - /dev/sdg 4 8 16 - /dev/sdb
To display det ails about a RAI D volum e:
mdadm –D /dev/md/Volume0
/ dev/ m d/ Volum e0: Container : /dev/md0, member 0 Raid Level : raid5 Array Size : 39999488 (38.15 GiB 40.96 GB) Used Dev Size : 9999872 (9.54 GiB 10.24 GB) Raid Devices : 5 Total Devices : 5 Update Time : Thu Jun 17 07:40:23 2010 State : clean Active Devices : 5 Working Devices : 5 Failed Devices : 0 Spare Devices : 0 Layout : left-asymmetric Chunk Size : 128K UUID : 084d2b20:09897744:36757c5b:77e0e945 Number Major Minor RaidDevice State 4 8 96 0 active sync /dev/sdg 3 8 48 1 active sync /dev/sdd 2 8 32 2 active sync /dev/sdc 1 8 16 3 active sync /dev/sdb 0 8 80 4 active sync /dev/sdf
To print out RAI D det ails about a m em ber disk:
mdadm –E /dev/sdb /dev/sdb: Magic : Intel Raid ISM Cfg Sig. Version : 1.2.04 Orig Family : e0935e91 Family : a830104b Generation : 00000037 UUID : b559b502:b199f86f:ee9fbd40:cd10e91d Checksum : 3ba66395 correct MPB Sectors : 2 Disks : 5 RAID Devices : 1 Disk02 Serial : 9QMCLMDM State : active Id : 00000000 Usable Size : 976768654 (465.76 GiB 500.11 GB) [vol0]: UUID : 084d2b20:09897744:36757c5b:77e0e945 RAID Level : 5 Members : 5 Slots : [UUUUU] This Slot : 3 Array Size : 79998976 (38.15 GiB 40.96 GB) Per Dev Size : 19999744 (9.54 GiB 10.24 GB) Sector Offset : 0 Num Stripes : 19531
Chunk Size : 128 KiB Reserved : 0 Migrate State : idle Map State : normal Dirty State : clean Disk00 Serial : 9QMCLYES State : active Id : 00000000 Usable Size : 976768654 (465.76 GiB 500.11 GB) Disk01 Serial : 9QMCLYB9 State : active Id : 00000000 Usable Size : 976768654 (465.76 GiB 500.11 GB) Disk03 Serial : 9QMCM7XY State : active Id : 00000000 Usable Size : 976768654 (465.76 GiB 500.11 GB) Disk04 Serial : 9QMCF38Z State : active Id : 00000000 Usable Size : 976768654 (465.76 GiB 500.11 GB)
To get t he m ost current st at us on all RAI D volum es, t he file /proc/mdstat can be exam ined. This file is a special file t hat is updat ed cont inuously t o show t he st at us of all t he cont ainers, and RAI D volum es. I n t he exam ple below , t he st at us show s t hat current ly available RAI D support s are level 4, 5, and 6. m d126 is t he act ive RAI D volum e wit h RAI D level 5 and 128k st ripe size. The RAI D volum e cont ains 5 disks t hat are all in norm al ( UP) st at us. m d127 is t he I MSM cont ainer for t he RAI D volum e.
cat /proc/mdstat