11g r2模拟OCR和voting disk不可用，完整恢复过程，以及一些注意事项
环境：RHEL5.8 RAC 11.2.0.3.0
1：查看ORC和voting disk信息：
In 11g Release 2 your voting disk data is automatically backed up in the OCR whenever there is a configuration change.
所以恢复时恢复备份OCR即可，这里和10g是不同的，不需要备份voting disk，备份OCR即可
2：使用cluvfy 工具检查OCR完整性
[grid@rac1 ~]$ cluvfy comp ocr -n all
Verifying OCR integrity
Checking OCR integrity...
Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations
ASM Running check passed. ASM is running on all specified nodes
Checking OCR config file "/etc/oracle/ocr.loc"...
OCR config file "/etc/oracle/ocr.loc" check successful
Disk group for ocr location "+CRSDATA" available on all the nodes
NOTE:
This check does not verify the integrity of the OCR contents. Execute 'ocrcheck' as a privileged user to verify the contents of OCR.
OCR integrity check passed
Verification of OCR integrity was successful.
3：使用ocrcheck检测OCR内容的完整性
[grid@rac1 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3016
Available space (kbytes) : 259104
ID : 1236405787
Device/File Name : +CRSDATA
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user --如果使用root用户执行ocrcheck时，会显示Logical corruption check succeeded
4：检测voting disk的信息
[grid@rac1 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 2b1bd0c122584f5abf72033b2b2d26bd (/dev/asm-b_crs) [CRSDATA]
2. ONLINE 2bc03776cdd94f5cbfb9165c473fdb0e (/dev/asm-c_crs) [CRSDATA]
3. ONLINE 3b43c39513a64f2dbf7083a9510ada89 (/dev/asm-d_crs) [CRSDATA]
Located 3 voting disk(s).
从上面看出，OCR和voting disk都位于+CRSDATA磁盘组 ，注意+CRSDATA磁盘组还有ASM的启动参数文件，ASM启动是根据磁盘头的kfdhdb.spfile指向ASM上的此磁盘的UA NUMBER从而读取spfile文件
5：手动备份一份OCR信息：
[root@rac1 grid]# ocrconfig -export /tmp/ocr_20130717.dmp
[root@rac1 grid]# ll /tmp/ocr_20130717.dmp -h
-rw------- 1 root root 102K Jul 17 14:45 /tmp/ocr_20130717.dmp
6：查看OCR自动备份信息
[grid@rac1 ~]$ ocrconfig -showbackup
rac1 2013/07/16 15:45:24 /u01/app/11.2.0.3/grid/cdata/ad-cluster/backup00.ocr
rac2 2013/07/16 08:13:38 /u01/app/11.2.0.3/grid/cdata/ad-cluster/backup01.ocr
rac2 2013/07/16 04:14:09 /u01/app/11.2.0.3/grid/cdata/ad-cluster/backup02.ocr
rac2 2013/07/16 00:14:38 /u01/app/11.2.0.3/grid/cdata/ad-cluster/day.ocr
rac2 2013/07/07 04:40:11 /u01/app/11.2.0.3/grid/cdata/ad-cluster/week.ocr
PROT-25: Manual backups for the Oracle Cluster Registry are not available
7：保存一份ASM参数文件，如果提前没保存，可以到$CRS_HOME/dbs/init.ora获取一份，后面此启动参数的详细内容
[grid@rac1 dbs]$ sqlplus / as sysasm
SQL> create pfile='/tmp/asm_pfile_130717.txt' from spfile;
File created.
8：破坏保存OCR信息的磁盘组+CRSDATA
[root@rac1 dev]# dd if=/dev/zero of=/dev/asm-b_crs bs=1024 count=1000
[root@rac1 dev]# dd if=/dev/zero of=/dev/asm-c_crs bs=1024 count=1000
9：破坏了磁盘b和c后，都检测通过，没报错，在rac1和rac2停止crs
[root@rac1 dev]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
.....................
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac2 dev]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
.....................
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac1 dev]# ps -ef |grep ora_
root 16189 32265 0 16:26 pts/0 00:00:00 grep ora_
[root@rac1 dev]# ps -ef |grep asm_
root 16195 32265 0 16:26 pts/0 00:00:00 grep asm_
10:再启动CRS,报错
[root@rac1 dev]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac1 ~]# tail -50f /u01/app/11.2.0.3/grid/log/rac1/alertrac1.log
[cssd(16559)]CRS-1637:Unable to locate configured voting file with ID 2b1bd0c1-22584f5a-bf72033b-2b2d26bd; details at (:CSSNM00020:) in /u01/app/11.2.0.3/grid/log/rac1/cssd/ocssd.log
2013-07-17 16:28:15.947
[cssd(16559)]CRS-1637:Unable to locate configured voting file with ID 2bc03776-cdd94f5c-bfb9165c-473fdb0e; details at (:CSSNM00020:) in /u01/app/11.2.0.3/grid/log/rac1/cssd/ocssd.log
2013-07-17 16:28:15.947
[cssd(16559)]CRS-1705:Found 1 configured voting files but 2 voting files are required, terminating to ensure data integrity; details at (:CSSNM00021:) in /u01/app/11.2.0.3/grid/log/rac1/cssd/ocssd.log
2013-07-17 16:28:15.948
[cssd(16559)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in
