How to resolve Ceph error Possible data damage: 1 pg inconsistent
Ceph Storage Cluster
Ceph is a clustered storage solution that can use any number of commodity servers and hard drives. These can then be made available as object, block or file system storage through a unified interface to your applications or servers. The data can be replicated based on your needs so any single disk or server failure does not effect your data, or the availability of the storage cluster.
Checking the Cluster
We monitor our Ceph cluster health by using Nagios with Ceph plugins and recently had an alert that needed to be resolved.
This issue started after the cluster performed a deep cleaning process. The alert was one of the pg on the cluster was getting active+clean+inconsistent, this is the process I went through to resolve.
First Check Ceph Health
Run the standard command for ceph health detail and lets see the error message, and determine what page is triggering an alert on our cluster.
[root@osd1 ~]# ceph --version
ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
[root@osd1 ~]# ceph health detail
HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent
OSD_SCRUB_ERRORS 1 scrub errors
PG_DAMAGED Possible data damage: 1 pg inconsistent
pg 1.133 is active+clean+inconsistent, acting [7,10]
You can see that we’re getting an inconsistent pg that needs to be repaired. Now using pg 1.133 we get more information on the inconsistent page.
[root@osd1 ~]# rados list-inconsistent-obj 1.133 --format=json-pretty
{
"epoch": 6804,
"inconsistents": [
{
"object": {
"name": "20000028ef9.00000112",
"nspace": "",
"locator": "",
"snap": "head",
"version": 111879
},
"errors": [],
"union_shard_errors": [
"read_error"
],
"selected_object_info": {
"oid": {
"oid": "20000028ef9.00000112",
"key": "",
"snapid": -2,
"hash": 2129486131,
"max": 0,
"pool": 1,
"namespace": ""
},
"version": "6726'111879",
"prior_version": "0'0",
"last_reqid": "client.823767.0:146716624",
"user_version": 111879,
"size": 4194304,
"mtime": "2018-10-16 03:10:28.422335",
"local_mtime": "2018-10-16 03:08:32.081019",
"lost": 0,
"flags": [
"dirty",
"data_digest"
],
"legacy_snaps": [],
"truncate_seq": 0,
"truncate_size": 0,
"data_digest": "0x6c87416b",
"omap_digest": "0xffffffff",
"expected_object_size": 0,
"expected_write_size": 0,
"alloc_hint_flags": 0,
"manifest": {
"type": 0,
"redirect_target": {
"oid": "",
"key": "",
"snapid": 0,
"hash": 0,
"max": 0,
"pool": -9223372036854775808,
"namespace": ""
}
},
"watchers": {}
},
"shards": [
{
"osd": 7,
"primary": true,
"errors": [
"read_error"
],
"size": 4194304
},
{
"osd": 10,
"primary": false,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0x6c87416b"
}
]
}
]
}
We can see that the primary shard is getting a read error but the secondary shard doesn’t have any errors.
Repair Inconsistent Pg
Next we’re going to use the command ceph pg repair with the page number we determined that had the problem.
[root@osd1 ~]# ceph pg repair 1.133 instructing pg 1.133 on osd.7 to repair
I then run a ceph -w to watch the cluster status
[root@osd1 ~]# ceph -w
cluster:
id: ffdb9e09-fdca-48bb-b7fb-cd17151d5c09
health: HEALTH_ERR
1 scrub errors
Possible data damage: 1 pg inconsistent, 1 pg repair
...
2018-10-27 12:32:48.206830 osd.7 [ERR] 1.133 shard 7: soid 1:cc86b77e:::20000028ef9.00000112:head candidate had a read error
2018-10-27 12:37:11.285243 mon.osd1 [INF] Health check cleared: OSD_SCRUB_ERRORS (was: 1 scrub errors)
2018-10-27 12:37:11.285294 mon.osd1 [INF] Health check cleared: PG_DAMAGED (was: Possible data damage: 1 pg inconsistent, 1 pg repair)
2018-10-27 12:37:11.285331 mon.osd1 [INF] Cluster is now healthy
2018-10-27 12:36:36.212157 osd.7 [ERR] 1.133 repair 0 missing, 1 inconsistent objects
2018-10-27 12:36:36.212182 osd.7 [ERR] 1.133 repair 1 errors, 1 fixed
2018-10-27 13:00:00.000164 mon.osd1 [INF] overall HEALTH_OK
After about a minute the health of the cluster will return to HEALTH_OK
During this repair process I referenced information from the Ceph issue tracker.