RMAN_RAC恢复至单机时redo日志引发的“恐慌”
其实这个问题确切来说由于自己的疏忽所造成，之前做过RAC到单机的恢复实验，生产环境并无涉足，当时测试环境为Redhat5.4+Oracle1
一、概括
其实这个问题确切来说由于自己的疏忽所造成，之前做过RAC到单机的恢复实验，生产环境并无涉足，当时测试环境为RedHat5.4+Oracle11g+RAC，存储方式为ASM；生产环境为hpux+Oracle10g+RAC,存储方式为裸设备；就目前来看，，跟数据文件的存储方式有关。
二、 Hpux中，recover完成，执行’alter database open resetlogs’时报以下错误
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-19502: write error on file "/dev/vgdb/rredo2_2a_256m", blockno 105473
(blocksize=1024)
ORA-27072: File I/O error
HPUX-ia64 Error: 2: No such file or directory
Additional information: 4
Additional information: 105473
Additional information: 433152
查看alert日志
ORA-1589 signalled during: alter database open...
Wed Oct 29 08:15:21 2014
alter database open resetlogs
Wed Oct 29 08:15:21 2014
Errors in file /oracle/admin/pmssd/udump/pmssd1_ora_11597.trc:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: '/dev/vgdb/rredo1_1b_256m'
ORA-27037: unable to obtain file status
HPUX-ia64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 1 thread 1: '/dev/vgdb/rredo1_1ax`_256m'
ORA-27037: unable to obtain file status
HPUX-ia64 Error: 2: No such file or directory
Additional information: 3
Wed Oct 29 08:15:21 2014
查看视图v$logfile
SQL> select * from v$logfile;
GROUP# STATUS TYPE MEMBER IS_
---------- ------- ------- ---------------------------- ---
1 ONLINE /dev/vgdb/rredo1_1a_256m NO
2 ONLINE /dev/vgdb/rredo1_2a_256m NO
3 ONLINE /dev/vgdb/rredo2_1a_256m NO
4 ONLINE /dev/vgdb/rredo2_2a_256m NO
1 ONLINE /dev/vgdb/rredo1_1b_256m NO
2 ONLINE /dev/vgdb/rredo1_2b_256m NO
3 ONLINE /dev/vgdb/rredo2_1b_256m NO
4 ONLINE /dev/vgdb/rredo2_2b_256m NO
5 ONLINE /dev/vgdb/rredo1_3a_256m NO
5 ONLINE /dev/vgdb/rredo1_3b_256m NO
6 ONLINE /dev/vgdb/rredo2_3a_256m NO
6 ONLINE /dev/vgdb/rredo2_3b_256m NO
根下目录dev为设置目录，我们需要修改redo日志目录，以完成数据库open操作，语句如下：
SQL> alter database rename file '/dev/vgdb/rredo1_1a_256m' to '/oracle/oradata/pmssd/redo01.log';
我并没有按照以上语句操作，而是重新创建了控制文件，这个问题是自己犯二了。修改完redo日志目录后就可以把数据库open了。
下面是我在linux ASM模式下的操作（’alter database open resetlogs’)。如以下日志，数据库自动重建了redo日志，并指定到了相应目录
Wed Oct 29 09:27:19 2014
alter database open resetlogs
RESETLOGS after incomplete recovery UNTIL CHANGE 783268
Errors in file /oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_ora_3900.trc:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: '+DATA/racdb/onlinelog/group_1.257.859438553'
ORA-17503: ksfdopn:2 Failed to open file +DATA/racdb/onlinelog/group_1.257.859438553
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-29701: unable to connect to Cluster Synchronization Service
ORA-00312: online log 1 thread 1: '+DATA/racdb/onlinelog/group_1.273.859438547'
ORA-17503: ksfdopn:2 Failed to open file +DATA/racdb/onlinelog/group_1.273.859438547
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-29701: unable to connect to Cluster Synchronization Service
Errors in file /oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_ora_3900.trc:
…………………………………………………………….
WARNING: ASM communication error: op 16 state 0x40 (15077)
ERROR: slave communication error with ASM
WARNING: Cannot delete Oracle managed file +DATA/racdb/onlinelog/group_4.275.859442003
WARNING: ASM communication error: op 16 state 0x40 (15077)
ERROR: slave communication error with ASM
WARNING: Cannot delete Oracle managed file +DATA/racdb/onlinelog/group_4.271.859442011
Wed Oct 29 09:28:15 2014
Clearing online redo logfile 4 complete
Resetting resetlogs activation ID 857045326 (0x3315794e)
Online log /oracle/oradata/racdb/RACDB/onlinelog/o1_mf_1_b50jmr0p_.log: Thread 1 Group 1 was previously cleared
Online log /oracle/backup/archive_racdb/RACDB/onlinelog/o1_mf_1_b50jmsl4_.log: Thread 1 Group 1 was previously cleared
Online log /oracle/oradata/racdb/RACDB/onlinelog/o1_mf_2_b50jn5rw_.log: Thread 1 Group 2 was previously cleared
Online log /oracle/backup/archive_racdb/RACDB/onlinelog/o1_mf_2_b50jn6jm_.log: Thread 1 Group 2 was previously cleared
Online log /oracle/oradata/racdb/RACDB/onlinelog/o1_mf_3_b50jnkqn_.log: Thread 2 Group 3 was previously cleared
Online log /oracle/backup/archive_racdb/RACDB/onlinelog/o1_mf_3_b50jnltx_.log: Thread 2 Group 3 was previously cleared
Online log /oracle/oradata/racdb/RACDB/onlinelog/o1_mf_4_b50jnz10_.log: Thread 2 Group 4 was previously cleared
Online log /oracle/backup/archive_racdb/RACDB/onlinelog/o1_mf_4_b50jo0jb_.log: Thread 2 Group 4 was previously cleared
Wed Oct 29 09:28:16 2014
Setting recovery target incarnation to 2
Wed Oct 29 09:28:17 2014
Assigning activation ID 859863386 (0x3340795a)
LGWR: STARTING ARCH PROCESSES
Wed Oct 29 09:28:17 2014
ARC0 started with pid=20, OS id=3924
ARC0: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
Thread 1 opened at log sequence 1
Current log# 1 seq# 1 mem# 0: /oracle/oradata/racdb/RACDB/onlinelog/o1_mf_1_b50jmr0p_.log
Current log# 1 seq# 1 mem# 1: /oracle/backup/archive_racdb/RACDB/onlinelog/o1_mf_1_b50jmsl4_.log
Successful open of redo thread 1
ARC0: STARTING ARCH PROCESSES
Wed Oct 29 09:28:17 2014
ARC1 started with pid=21, OS id=3926
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Oct 29 09:28:17 2014
SMON: enabling cache recovery
Wed Oct 29 09:28:17 2014
ARC2 started with pid=22, OS id=3928
Wed Oct 29 09:28:17 2014
ARC3 started with pid=23, OS id=3930
ARC1: Archival started
ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC2: Becoming the heartbeat ARCH
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Redo thread 2 internally disabled at seq 1 (CKPT)
ARC0: Archiving disabled thread 2 sequence 1
Archived Log entry 27 added for thread 2 sequence 1 ID 0x0 dest 1:
[3900] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:4182896 end:4184866 diff:1970 (19 seconds)
Dictionary check beginning
File #7 is offline, but is part of an online tablespace.
data file 7: '+DATA/racdb/datafile/test2.260.859451699'
File #9 is offline, but is part of an online tablespace.
data file 9: '+DATA/racdb/datafile/test4.258.859451727'
Wed Oct 29 09:28:24 2014
Errors in file /oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_dbw0_3814.trc:
ORA-01157: cannot identify/lock data file 201 - see DBWR trace file
ORA-01110: data file 201: '+DATA/racdb/tempfile/temp.286.859438605'
ORA-17503: ksfdopn:2 Failed to open file +DATA/racdb/tempfile/temp.286.859438605
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-29701: unable to connect to Cluster Synchronization Service
Errors in file /oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_dbw0_3814.trc:
ORA-01186: file 201 failed verification tests
ORA-01157: cannot identify/lock data file 201 - see DBWR trace file
ORA-01110: data file 201: '+DATA/racdb/tempfile/temp.286.859438605'
File 201 not verified due to error ORA-01157
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Re-creating tempfile +DATA/racdb/tempfile/temp.286.859438605 as /oracle/oradata/racdb/RACDB/datafile/o1_mf_temp_b50jor8m_.tmp
Database Characterset is ZHS16GBK
Wed Oct 29 09:28:27 2014
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Wed Oct 29 09:28:31 2014
QMNC started with pid=24, OS id=3937
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Wed Oct 29 09:28:42 2014
db_recovery_file_dest_size of 5727 MB is 13.97% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Wed Oct 29 09:28:45 2014
Starting background process CJQ0
Wed Oct 29 09:28:45 2014
CJQ0 started with pid=27, OS id=3957
Wed Oct 29 09:28:46 2014
Completed: alter database open resetlogs
在执行’alter database open resetlogs’之前，修改redo日志目录,open时redo日志就会在指定的目录生成。
alter database rename file '+DATA/racdb/onlinelog/group_1.273.859438547' to '/oracle/oradata/racdb/redo1_a.log';
alter database rename file '+DATA/racdb/onlinelog/group_1.257.859438553' to '/oracle/oradata/racdb/redo1_b.log';
alter database rename file '+DATA/racdb/onlinelog/group_2.256.859438561' to '/oracle/oradata/racdb/redo2_a.log';
alter database rename file '+DATA/racdb/onlinelog/group_2.291.859438567' to '/oracle/oradata/racdb/redo2_b.log';
alter database rename file '+DATA/racdb/onlinelog/group_3.277.859441989' to '/oracle/oradata/racdb/redo3_a.log';
alter database rename file '+DATA/racdb/onlinelog/group_3.276.859441997' to '/oracle/oradata/racdb/redo3_b.log';
alter database rename file '+DATA/racdb/onlinelog/group_4.275.859442003' to '/oracle/oradata/racdb/redo4_a.log';
查看alert日志
lter database open resetlogs
RESETLOGS after incomplete recovery UNTIL CHANGE 783268
Errors in file /oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_ora_3683.trc:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: '/oracle/oradata/racdb/redo1_b.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
ORA-00312: online log 1 thread 1: '/oracle/oradata/racdb/redo1_a.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_ora_3683.trc:
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1: '/oracle/oradata/racdb/redo2_b.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
……………………………..
Additional information: 3
ORA-00312: online log 2 thread 1: '/oracle/oradata/racdb/redo2_a.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Clearing online redo logfile 2 /oracle/oradata/racdb/redo2_a.log
Clearing online log 2 of thread 1 sequence number 30
Errors in file /oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_ora_3683.trc:
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1: '/oracle/oradata/racdb/redo2_b.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
………………………………..
Clearing online redo logfile 4 complete
Resetting resetlogs activation ID 857045326 (0x3315794e)
Online log /oracle/oradata/racdb/redo1_a.log: Thread 1 Group 1 was previously cleared
Online log /oracle/oradata/racdb/redo1_b.log: Thread 1 Group 1 was previously cleared
Online log /oracle/oradata/racdb/redo2_a.log: Thread 1 Group 2 was previously cleared
Online log /oracle/oradata/racdb/redo2_b.log: Thread 1 Group 2 was previously cleared
Online log /oracle/oradata/racdb/redo3_a.log: Thread 2 Group 3 was previously cleared
Online log /oracle/oradata/racdb/redo3_b.log: Thread 2 Group 3 was previously cleared
Online log /oracle/oradata/racdb/redo4_a.log: Thread 2 Group 4 was previously cleared
Online log /oracle/oradata/racdb/redo4_b.log: Thread 2 Group 4 was previously cleared
Wed Oct 29 09:18:08 2014
Setting recovery target incarnation to 2
Wed Oct 29 09:18:09 2014
Assigning activation ID 859838345 (0x33401789)
LGWR: STARTING ARCH PROCESSES
Wed Oct 29 09:18:09 2014
ARC0 started with pid=20, OS id=3730
ARC0: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Thread 1 opened at log sequence 1
Current log# 1 seq# 1 mem# 0: /oracle/oradata/racdb/redo1_a.log
Current log# 1 seq# 1 mem# 1: /oracle/oradata/racdb/redo1_b.log
Successful open of redo thread 1
Wed Oct 29 09:18:10 2014
ARC1 started with pid=21, OS id=3732
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Oct 29 09:18:10 2014
SMON: enabling cache recovery
Wed Oct 29 09:18:10 2014
ARC2 started with pid=22, OS id=3734
Wed Oct 29 09:18:10 2014
ARC3 started with pid=23, OS id=3736
ARC1: Archival started
ARC2: Archival started
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
ARC2: Becoming the heartbeat ARCH
Redo thread 2 internally disabled at seq 1 (CKPT)
ARC0: Archiving disabled thread 2 sequence 1
Archived Log entry 27 added for thread 2 sequence 1 ID 0x0 dest 1:
Wed Oct 29 09:18:22 2014
[3683] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:3581386 end:3583566 diff:2180 (21 seconds)
Dictionary check beginning
File #7 is offline, but is part of an online tablespace.
data file 7: '+DATA/racdb/datafile/test2.260.859451699'
File #9 is offline, but is part of an online tablespace.
data file 9: '+DATA/racdb/datafile/test4.258.859451727'
Wed Oct 29 09:18:23 2014
Errors in file /oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_dbw0_3455.trc:
ORA-01157: cannot identify/lock data file 201 - see DBWR trace file
ORA-01110: data file 201: '+DATA/racdb/tempfile/temp.286.859438605'
ORA-17503: ksfdopn:2 Failed to open file +DATA/racdb/tempfile/temp.286.859438605
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-29701: unable to connect to Cluster Synchronization Service
Errors in file /oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_dbw0_3455.trc:
ORA-01186: file 201 failed verification tests
ORA-01157: cannot identify/lock data file 201 - see DBWR trace file
ORA-01110: data file 201: '+DATA/racdb/tempfile/temp.286.859438605'
File 201 not verified due to error ORA-01157
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
Wed Oct 29 09:18:23 2014
SMON: enabling tx recovery
Re-creating tempfile +DATA/racdb/tempfile/temp.286.859438605 as /oracle/oradata/racdb/RACDB/datafile/o1_mf_temp_b50j2zdh_.tmp
Database Characterset is ZHS16GBK
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Wed Oct 29 09:18:32 2014
QMNC started with pid=24, OS id=3743
Wed Oct 29 09:18:33 2014
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Wed Oct 29 09:18:45 2014
Starting background process CJQ0
Wed Oct 29 09:18:45 2014
CJQ0 started with pid=27, OS id=3763
Wed Oct 29 09:18:45 2014
Completed: alter database open resetlogs