rman备份出现ORA-19502/ORA-27063错误

两个通道在写入rman备份到磁盘中的时候,在05/15/2012 04:39:58发现磁盘空间不足,两个通道分别准备写入30480897/30753665号块的时

RMAN> 2> 3> 4> 5> 6> 7> 8> 9> 10> 11> 12> 13> 14> 15> 16> 17> 18> 19> 20>

allocated channel: t11

channel t11: sid=824 instance=ncdb1 devtype=DISK

allocated channel: t12

channel t12: sid=838 instance=ncdb1 devtype=DISK

allocated channel: t13

channel t13: sid=809 instance=ncdb1 devtype=DISK

allocated channel: t14

channel t14: sid=886 instance=ncdb1 devtype=DISK

allocated channel: t15

channel t15: sid=620 instance=ncdb1 devtype=DISK

allocated channel: t16

channel t16: sid=599 instance=ncdb1 devtype=DISK

allocated channel: t17

channel t17: sid=482 instance=ncdb1 devtype=DISK

allocated channel: t18

channel t18: sid=506 instance=ncdb1 devtype=DISK

一共开通8个通道

channel t12: starting full datafile backupset

channel t12: specifying datafile(s) in backupset

input datafile fno=00008 name=/dev/rnc32g_39

input datafile fno=00016 name=/dev/rnc32g_47

input datafile fno=00024 name=/dev/rnc32g_57

input datafile fno=00032 name=/dev/rnc32g_25

input datafile fno=00040 name=/dev/rnc32g_33

input datafile fno=00048 name=/dev/rnc32g_3

input datafile fno=00056 name=/dev/rnc32g_11

input datafile fno=00064 name=/dev/rnc32g_19

input datafile fno=00072 name=/dev/rnc32g_67

input datafile fno=00080 name=/dev/rnc32g_106

input datafile fno=00088 name=/dev/rnc32g_114

input datafile fno=00096 name=/dev/rnc32g_87

input datafile fno=00104 name=/dev/rnc32g_95

input datafile fno=00112 name=/dev/rnc32g_103

input datafile fno=00120 name=/dev/rnc32g_75

input datafile fno=00003 name=/dev/rnc50_sysaux

input datafile fno=00130 name=/dev/rnc32g_119

channel t12: starting piece 1 at 14-MAY-12

--通道12备份数据文件

channel t17: starting full datafile backupset

channel t17: specifying datafile(s) in backupset

input datafile fno=00002 name=/dev/rnc32g_22

input datafile fno=00013 name=/dev/rnc32g_44

input datafile fno=00021 name=/dev/rnc32g_54

input datafile fno=00029 name=/dev/rnc32g_62

input datafile fno=00037 name=/dev/rnc32g_30

input datafile fno=00045 name=/dev/rnc32g_38

input datafile fno=00053 name=/dev/rnc32g_8

input datafile fno=00061 name=/dev/rnc32g_16

input datafile fno=00069 name=/dev/rnc32g_64

input datafile fno=00077 name=/dev/rncundo_33g_4

input datafile fno=00085 name=/dev/rnc32g_111

input datafile fno=00093 name=/dev/rnc32g_84

input datafile fno=00101 name=/dev/rnc32g_92

input datafile fno=00109 name=/dev/rnc32g_100

input datafile fno=00117 name=/dev/rnc32g_72

input datafile fno=00006 name=/dev/rnc50_4g_1

channel t17: starting piece 1 at 14-MAY-12

--通道17备份数据文件

channel t15: finished piece 1 at 15-MAY-12

piece handle=/rman/db_mpnb04jl_1_1 tag=TAG20120514T204954 comment=NONE

channel t15: backup set complete, elapsed time: 06:07:59

channel t11: finished piece 1 at 15-MAY-12

piece handle=/rman/db_mlnb04jk_1_1 tag=TAG20120514T204954 comment=NONE

channel t11: backup set complete, elapsed time: 06:17:25

channel t16: finished piece 1 at 15-MAY-12

piece handle=/rman/db_mqnb04jm_1_1 tag=TAG20120514T204954 comment=NONE

channel t16: backup set complete, elapsed time: 06:34:49

channel t14: finished piece 1 at 15-MAY-12

piece handle=/rman/db_monb04jl_1_1 tag=TAG20120514T204954 comment=NONE

channel t14: backup set complete, elapsed time: 06:40:05

channel t18: finished piece 1 at 15-MAY-12

piece handle=/rman/db_msnb04jn_1_1 tag=TAG20120514T204954 comment=NONE

channel t18: backup set complete, elapsed time: 06:43:38

channel t13: finished piece 1 at 15-MAY-12

piece handle=/rman/db_mnnb04jl_1_1 tag=TAG20120514T204954 comment=NONE

channel t13: backup set complete, elapsed time: 07:40:56

--这里可以看出rman的备份完成了通道11/13/14/15/16/18，也就是说目前为止通道12/17未完成.

RMAN-03009: failure of backup command on t12 channel at 05/15/2012 04:39:58

ORA-19502: write error on file "/rman/db_mmnb04jl_1_1", blockno 30481025 (blocksize=8192)

ORA-27063: number of bytes read/written is incorrect

IBM AIX RISC System/6000 Error: 28: No space left on device

Additional information: -1

Additional information: 1048576

ORA-19502: write error on file "/rman/db_mmnb04jl_1_1", blockno 30480897 (blocksize=8192)

ORA-27063: number of bytes read/written is incorrect

IBM AIX RISC System/6000 Error: 28: No space left on device

channel t12 disabled, job failed on it will be run on another channel

--通道12报错(硬盘空间不足)

channel t11: starting full datafile backupset

channel t11: specifying datafile(s) in backupset

input datafile fno=00008 name=/dev/rnc32g_39

input datafile fno=00016 name=/dev/rnc32g_47

input datafile fno=00024 name=/dev/rnc32g_57

input datafile fno=00032 name=/dev/rnc32g_25

input datafile fno=00040 name=/dev/rnc32g_33

input datafile fno=00048 name=/dev/rnc32g_3

input datafile fno=00056 name=/dev/rnc32g_11

input datafile fno=00064 name=/dev/rnc32g_19

input datafile fno=00072 name=/dev/rnc32g_67

input datafile fno=00080 name=/dev/rnc32g_106

input datafile fno=00088 name=/dev/rnc32g_114

input datafile fno=00096 name=/dev/rnc32g_87

input datafile fno=00104 name=/dev/rnc32g_95

input datafile fno=00112 name=/dev/rnc32g_103

input datafile fno=00120 name=/dev/rnc32g_75

input datafile fno=00003 name=/dev/rnc50_sysaux

input datafile fno=00130 name=/dev/rnc32g_119

channel t11: starting piece 1 at 15-MAY-12

--在通道12报错后,通道11已经完成了上次备份,所以启动备份通道12出错的数据文件

RMAN-03009: failure of backup command on t17 channel at 05/15/2012 04:39:58

ORA-19502: write error on file "/rman/db_mrnb04jm_1_1", blockno 30753793 (blocksize=8192)

ORA-27063: number of bytes read/written is incorrect

IBM AIX RISC System/6000 Error: 28: No space left on device

Additional information: -1

Additional information: 1048576

ORA-19502: write error on file "/rman/db_mrnb04jm_1_1", blockno 30753665 (blocksize=8192)

ORA-27063: number of bytes read/written is incorrect

IBM AIX RISC System/6000 Error: 28: No space left on device

channel t17 disabled, job failed on it will be run on another channel

--通道17也因为磁盘空间报错

channel t13: starting full datafile backupset

channel t13: specifying datafile(s) in backupset

input datafile fno=00002 name=/dev/rnc32g_22

input datafile fno=00013 name=/dev/rnc32g_44

input datafile fno=00021 name=/dev/rnc32g_54

input datafile fno=00029 name=/dev/rnc32g_62

input datafile fno=00037 name=/dev/rnc32g_30

input datafile fno=00045 name=/dev/rnc32g_38

input datafile fno=00053 name=/dev/rnc32g_8

input datafile fno=00061 name=/dev/rnc32g_16

input datafile fno=00069 name=/dev/rnc32g_64

input datafile fno=00077 name=/dev/rncundo_33g_4

input datafile fno=00085 name=/dev/rnc32g_111

input datafile fno=00093 name=/dev/rnc32g_84

input datafile fno=00101 name=/dev/rnc32g_92

input datafile fno=00109 name=/dev/rnc32g_100

input datafile fno=00117 name=/dev/rnc32g_72

input datafile fno=00006 name=/dev/rnc50_4g_1

channel t13: starting piece 1 at 15-MAY-12

--通道13也尝试备份通道17失败的数据文件

RMAN-03009: failure of backup command on t11 channel at 05/15/2012 04:39:59

ORA-19504: failed to create file "/rman/db_mtnb104u_1_1"

ORA-27044: unable to write the header block of file

IBM AIX RISC System/6000 Error: 28: No space left on device

Additional information: 3

Addition

--因为当前没有空闲空间,通道11终止,

--这个时候rman异常终止,导致后续的通道13终止记录未打印到日志

阅读完rman日志,很好理解因为存放rman备份的磁盘空间不足导致了一系列错误

检查磁盘剩余空间

Filesystem GB blocks Free %Used Iused %Iused Mounted on

/dev/hd4 10.00 9.75 3% 6548 1% /

/dev/hd2 10.00 4.55 55% 84383 8% /usr

/dev/hd9var 5.00 4.04 20% 6290 1% /var

/dev/hd3 5.00 3.87 23% 1551 1% /tmp

/dev/hd1 10.00 9.91 1% 382 1% /home

/proc - - - - - /proc

/dev/hd10opt 5.00 4.89 3% 3502 1% /opt

/dev/archalv 99.00 82.98 17% 96 1% /archa

/dev/fslv01 40.00 19.49 52% 72324 2% /ora10

/dev/fslv00 1800.00 467.25 75% 10 1% /rman

这下让人迷糊了,磁盘空间还剩余467.25G,怎么会报错呢？

分析原因

RMAN-03009: failure of backup command on t12 channel at 05/15/2012 04:39:58

ORA-19502: write error on file "/rman/db_mmnb04jl_1_1", blockno 30481025 (blocksize=8192)

ORA-27063: number of bytes read/written is incorrect

IBM AIX RISC System/6000 Error: 28: No space left on device

Additional information: -1

Additional information: 1048576

ORA-19502: write error on file "/rman/db_mmnb04jl_1_1", blockno 30480897 (blocksize=8192)

ORA-27063: number of bytes read/written is incorrect

IBM AIX RISC System/6000 Error: 28: No space left on device

channel t12 disabled, job failed on it will be run on another channel

RMAN-03009: failure of backup command on t17 channel at 05/15/2012 04:39:58

ORA-19502: write error on file "/rman/db_mrnb04jm_1_1", blockno 30753793 (blocksize=8192)

ORA-27063: number of bytes read/written is incorrect

IBM AIX RISC System/6000 Error: 28: No space left on device

Additional information: -1

Additional information: 1048576

ORA-19502: write error on file "/rman/db_mrnb04jm_1_1", blockno 30753665 (blocksize=8192)

ORA-27063: number of bytes read/written is incorrect

IBM AIX RISC System/6000 Error: 28: No space left on device

channel t17 disabled, job failed on it will be run on another channel