Pacemaker Resource Agent的错误处理
Pacemaker Resource Agent的错误处理
1.前言 Pacemaker通过调用各个resource agent提供的操作（比如start，stop）实现对资源的控制，当这个方法执行出错时，Pacemaker会根据执行的操作和错误类型进行不同的错误处理。2. 错误类型
Pacemaker将错误分成3类：soft，hard和fatal，后两种属于环境或配置问题，如果没有人工干预是不可能自动修复的。一般的故障都采用OCF_ERR_GENERIC作为返回值，比如，服务进程crash，网络不通等，OCF_ERR_GENERIC属于soft类型。
B.3.How are OCF Return Codes Interpreted?
The first thing the cluster does is to check the return code against the expected result. If the result does not match the expected value, then the operation is considered to have failed and recovery action is initiated.There are three types of failure recovery:TableB.3.Types of recovery performed by the cluster
| Type | Description | Action Taken by the Cluster |
|---|---|---|
| soft | A transient error occurred | Restart the resource or move it to a new location |
| hard | A non-transient error that may be specific to the current node occurred | Move the resource elsewhere and prevent it from being retried on the current node |
| fatal | A non-transient error that will be common to all cluster nodes (eg. a bad configuration was specified) | Stop the resource and prevent it from being started on any cluster node |
Assuming an action is considered to have failed, the following table outlines the different OCF return codes and the type of recovery the cluster will initiate when it is received.
B.4.OCF Return Codes
TableB.4.OCF Return Codes and their Recovery Types
| RC | OCF Alias | Description | RT |
|---|---|---|---|
| 0 | OCF_SUCCESS | Success. The command completed successfully. This is the expected result for all start, stop, promote and demote commands. | soft |
| 1 | OCF_ERR_GENERIC | Generic "there was a problem" error code. | soft |
| 2 | OCF_ERR_ARGS | The resource’s configuration is not valid on this machine. Eg. refers to a location/tool not found on the node. | hard |
| 3 | OCF_ERR_UNIMPLEMENTED | The requested action is not implemented. | hard |
| 4 | OCF_ERR_PERM | The resource agent does not have sufficient privileges to complete the task. | hard |
| 5 | OCF_ERR_INSTALLED | The tools required by the resource are not installed on this machine. | hard |
| 6 | OCF_ERR_CONFIGURED | The resource’s configuration is invalid. Eg. required parameters are missing. | fatal |
| 7 | OCF_NOT_RUNNING | The resource is safely stopped. The cluster will not attempt to stop a resource that returns this for any action. | N/A |
| 8 | OCF_RUNNING_MASTER | The resource is running in Master mode. | soft |
| 9 | OCF_FAILED_MASTER | The resource is in Master mode but has failed. The resource will be demoted, stopped and then started (and possibly promoted) again. | soft |
| other | NA | Custom error code. | soft |
Although counterintuitive, even actions that return 0 (aka.OCF_SUCCESS) can be considered to have failed.
3. 错误处理
每个资源的操作（operation）有一个on-fail属性，用于控制如何进行出错处理。
http://clusterlabs.org/doc/en-US/Pacemaker/1.1-plugin/html-single/Pacemaker_Explained/index.html#_monitoring_resources_for_failure
Table5.3.Properties of an Operation
| Field | Description |
|---|---|
| id | Your name for the action. Must be unique. |
| name | The action to perform. Common values: monitor, start, stop |
| interval | How frequently (in seconds) to perform the operation. Default value: 0, meaning never. |
| timeout | How long to wait before declaring the action has failed. |
| on-fail | The action to take if this action ever fails. Allowed values:* ignore - Pretend the resource did not fail* block - Don’t perform any further operations on the resource* stop - Stop the resource and do not start it elsewhere* restart - Stop the resource and start it again (possibly on a different node)* fence - STONITH the node on which the resource failed* standby - Move all resources away from the node on which the resource failedThe default for the stop operation is fence when STONITH is enabled and block otherwise. All other operations default to stop. |
| enabled | If false, the operation is treated as if it does not exist. Allowed values: true, false |
但是，实际测试验证后，发现不管如何设置on-fail，效果都不会变，也就是说永远是缺省行为。
以下是让Resource Agent的各个操作返回OCF_ERR_GENERIC时资源管理器的处理：
| 操作 | 错误处理 | 对应的on-fail值 |
|---|---|---|
| start |
设置fail-count=1000000 在本节点上调用stop 在其它节点上start该资源 |
restart |
| stop |
设置fail-count=1000000 阻止该资源的进一步操作，该资源成为unmanaged FAILED状态，如下 dummy(ocf::heartbeat:Dummy2):Started srdsdevapp69 (unmanaged) FAILED |
block |
| monitor |
设置fail-count+=1 在本节点上依次调用stop，start，monitor。如果monitor依然出错，重复stop，start，monitor，直到fail-count达到migration-threshold后，保持资源为stop状态。 |
restart |
| promote |
设置fail-count+=1 在本节点上依次调用demote，stop，start。 在其它节点上调用promote以提升其它节点上的资源为master |
restart |
| demote |
设置fail-count+=1 在本节点上依次调用stop，start，demote。如果demote依然出错，重复stop，start，demote，直到fail-count达到migration-threshold后，保持资源为stop状态。 |
restart |
| notify | 无视 | ignore |
注1：超时的处理与OCF_ERR_GENERIC相同
注2：Pacemaker不会对已经stop了的资源调用post stop notify。
注3：测试环境Pacemaker 1.1.7-6 ,CentOS 6.3
4.启示
上面关于错误处理的测试结果，可以给Resource Agent编写者提供几点启示：
- 1. 如非确实必要，不要让stop操作返回错误
- 2. monitor和start的判断要保持一致，即不应该出现start成功后立刻执行monitor却失败的情况，否则可能导致循环。
- 3. restart成功后执行demote不应该失败，否则可能导致循环。
- 4. migration-threshold设置为一个比较小的值（默认值是INFINITY，即100000），也可以减少上面的2和3的影响。
上一篇： PHP 单态设计模式