当前 OpenSearch 使用的越来越多, 但是 OpenSearch 生态还不尽完善.
针对如下情况:
我查了下, 官方还没有提供完备的方案.
这里如何监控 K8s 中的 OpenSearch, 包括安装 exporter 插件、采集、展示全环节。
config/opensearch.yml
中配置静态设置和动态设置来配置插件。http(s)://<opensearch-host>:9200/_prometheus/metrics
获得。本文会使用到 2 个资源:
两种方案:
📝Notes:
这里以
opensearch:2.12
版本为例
Dockerfile 内容如下:
FROM opensearchproject/opensearch:2.12.0 LABEL maintainer="cuikaidong@foxmail.com" ARG EXPORTER_PLUGIN_URL="https://github.com/Aiven-Open/prometheus-exporter-plugin-for-opensearch/releases/download/2.12.0.0/prometheus-exporter-2.12.0.0.zip" RUN opensearch-plugin install -b ${EXPORTER_PLUGIN_URL}
📝Notes
如果 docker build 过程下载超时, 可以将对应
EXPORTER_PLUGIN_URL
行替换为相关代理的 URL(这里不详述).
或者, 下载后, 通过COPY
复制进去后再执行:opensearch-plugin install -b file:///path/to/prometheus-exporter-2.12.0.0.zip
构建并推送镜像:
docker build -t xxxxx/opensearch:2.12.0-prometheus-exporter -f ./Dockerfile docker push xxxx/opensearch:2.12.0-prometheus-exporter
📝Notes
您可以通过 CICD Pipeline, 随着 OpenSearch 和 prometheus-exporter-plugin-for-opensearch 的更新, 自动构建新的镜像.
我相信, 随着 OpenSearch 生态的完善, 应该会有已经包含 exporter 的 OpenSearch 镜像.
对于容器化或 K8s 运行的 OpenSearch, 只需要将镜像改为构建后的, 带 prometheus-exporter 的镜像即可.
如:
原来是:
image: opensearchproject/opensearch:2.12.0
修改为:
image: xxxx/opensearch:2.12.0-prometheus-exporter
如果你是在 K8s 中运行 OpenSearch, 也可以考虑使用 OpenSearch 的 Helm Chart, 它包含了安装第三方插件的功能, 具体 values.yaml 如下:
## Enable to add 3rd Party / Custom plugins not offered in the default OpenSearch image. plugins: enabled: true installList: - https://github.com/Aiven-Open/prometheus-exporter-plugin-for-opensearch/releases/download/2.12.0.0/prometheus-exporter-2.12.0.0.zip
📚️参考文档:
OpenSearch Helm Chart
另外, 可以按需修改prometheus-exporter
的配置, 详细配置说明见:
示例配置如下:
在 config/opensearch.yml
, 追加如下内容:
plugins.security.disabled: true prometheus.indices_filter.selected_indices: "log-*,*log,*log*,log*-test" prometheus.indices_filter.selected_option: "STRICT_EXPAND_OPEN_FORBID_CLOSED"
📝声明
plugins.security.disabled: true
可选项, 允许通过 http 协议访问插件 url. 生产不建议使用. 建议只在快速验证时采用
prometheus.indices_filter.selected_indices
仅供参考. 请按需调整.
prometheus.indices_filter.selected_option
使用默认配置. 请阅读细节后按需调整.
修改完配置后, 重启容器正常生效.
指标可直接在以下位置获取:
http(s)://opensearch-host:9200/_prometheus/metrics
作为示例结果,你将得到如下内容:
# HELP opensearch_process_mem_total_virtual_bytes Memory used by ES process # TYPE opensearch_process_mem_total_virtual_bytes gauge opensearch_process_mem_total_virtual_bytes{cluster="develop",node="develop01",} 3.626733568E9 # HELP opensearch_indices_indexing_is_throttled_bool Is indexing throttling ? # TYPE opensearch_indices_indexing_is_throttled_bool gauge opensearch_indices_indexing_is_throttled_bool{cluster="develop",node="develop01",} 0.0 # HELP opensearch_jvm_gc_collection_time_seconds Time spent for GC collections # TYPE opensearch_jvm_gc_collection_time_seconds counter opensearch_jvm_gc_collection_time_seconds{cluster="develop",node="develop01",gc="old",} 0.0 opensearch_jvm_gc_collection_time_seconds{cluster="develop",node="develop01",gc="young",} 0.0 # HELP opensearch_indices_requestcache_memory_size_bytes Memory used for request cache # TYPE opensearch_indices_requestcache_memory_size_bytes gauge opensearch_indices_requestcache_memory_size_bytes{cluster="develop",node="develop01",} 0.0 # HELP opensearch_indices_search_open_contexts_number Number of search open contexts # TYPE opensearch_indices_search_open_contexts_number gauge opensearch_indices_search_open_contexts_number{cluster="develop",node="develop01",} 0.0 # HELP opensearch_jvm_mem_nonheap_used_bytes Memory used apart from heap # TYPE opensearch_jvm_mem_nonheap_used_bytes gauge opensearch_jvm_mem_nonheap_used_bytes{cluster="develop",node="develop01",} 5.5302736E7 ...
(仅作为参考示例, 请按需调整), 在 Prometheus 的 scrape
下面, 追加如下内容:
- job_name: opensearch metrics_path: /_prometheus/metrics relabel_configs: - replacement: '<your-instance-name>' target_label: node static_configs: - targets: ['<your-host-name>:9200']
这里随便举一个简单例子, 现在使用 OpenSearch 的, 之前应该有完备的 ES 相关的 rules 和 alerts. 略作修改即可.
alert: OpenSearchYellowCluster for: 5m annotations: summary: At least one of the clusters is reporting a yellow status. description: '{{$labels.cluster}} health status is yellow over the last 5 minutes' runbook_url: '' labels: severity: warning '': '' expr: | opensearch_cluster_status == 1
可以使用如下 Grafana Dashboard 进行查看:
效果如下:
更多 OpenSearch Dashboard 可以在 https://grafana.com/grafana/dashboards/ 中搜索关键词 “OpenSearch”.
如何监控容器或 K8s 中的 OpenSearch?
以上.
三人行, 必有我师; 知识共享, 天下为公. 本文由东风微鸣技术博客 EWhisper.cn 编写.