手上项目在线上正常运行,数据库中有数据更改后,数据未同步到es对应的索引中
将elasticsearch和logstash的日志下载下来后发现es日志中有很多行提示:
2021-10-21T23:53:03.448528162Z {"type": "server", "timestamp": "2021-10-21T23:53:03,448Z", "level": "WARN", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "my-cluster", "node.name": "node-1", "message": "flood stage disk watermark [95%] exceeded on [ts_jvdneTHa3dgd36yACwg][node-1][/usr/data/nodes/0] free: 3.9gb[3.9%], all indices on this node will be marked read-only", "cluster.uuid": "a8BvfadOTr95b2CplpFh7Q", "node.id": "ts_jvdneTHa3dgd36yACwg" }
logstash日志中也有相关错误
2021-10-22T08:10:02.813081623Z [2021-10-22T08:10:02,813][INFO ][logstash.outputs.elasticsearch][indexname][0d06b97eedc01234eb09f649512131204e592b676bb231139b4b62e92ab] retrying failed action with response code: 429 ({"type"=>"cluster_block_exception", "reason"=>"index [indexname] blocked by: [TOO_MANY_REQUESTS/12/index read-only / allow delete (api)];"})
翻看ES官网文档后发现 es在发现磁盘空间大于95%以后,将变成只读状态,此时将不会更新数据,只能查询和删除。查看es索引设置后发现
get localhost:9200/index_name/_settings "settings": { "index": { "number_of_shards": "1", "blocks": { "read_only_allow_delete": "true" }, } } //因日志显示剩余3.9g空间,此时通过 put localhost:9200/index_name/_settings { "index": { "blocks": { "read_only_allow_delete": "false" } } } //关闭只读状态
然后再次查看索引设置,此时只读状态已关闭,过一会发现数据仍然没有同步成功,无果
当磁盘空间大于5%后,es日志不再显示警告,此时还是无法更新索引,需要重复上述关闭只读状态操作