背景
ECK,也就是Elastic Cloud on Kubernetes的缩写,可以基于K8s operator在Kubernetes集群来自动化部署、管理、编排Elasticsearch、Kibana、APM Server服务。
ECK目前的features:
Elasticsearch, Kibana 和 APM Server的部署;
TLS证书管理;
安全的Elasticsearch cluster 配置和拓扑变动管理;
使用PV和PVC;
定制K8s node上的系统配置和属性;
在本文中,Gemfield将介绍如何使用ECK来进行部署,以及如何使用一个python的例子来访问部署好的Elasticsearch服务。说来也巧,Gemfield写本文的时候,ECK 1.0正式版刚刚发布了2天。
部署
官方部署是基于K8s operator的方式进行部署的,因此只支持K8s 1.12+的kubernetes版本;
1,安装CustomResourceDefinition和operator
gemfield@civilnet:~/ES$ kubectl apply -f https://download.elastic.co/downloads/eck/1.0.0/all-in-one.yaml
customresourcedefinition.apiextensions.k8s.io/apmservers.apm.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticsearches.elasticsearch.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/kibanas.kibana.k8s.elastic.co created
clusterrole.rbac.authorization.k8s.io/elastic-operator created
clusterrolebinding.rbac.authorization.k8s.io/elastic-operator created
namespace/elastic-system created
statefulset.apps/elastic-operator created
serviceaccount/elastic-operator created
validatingwebhookconfiguration.admissionregistration.k8s.io/elastic-webhook.k8s.elastic.co created
service/elastic-webhook-server created
secret/elastic-webhook-server-cert created
2,查看operator的log
kubectl -n elastic-system logs -f statefulset.apps/elastic-operator
3,查看新安装的CustomResourceDefinition
gemfield@civilnet:~/ES$ kubectl get CustomResourceDefinition
NAME CREATED AT
apmservers.apm.k8s.elastic.co 2020-02-06T11:17:38Z
elasticsearches.elasticsearch.k8s.elastic.co 2020-02-06T11:17:38Z
kibanas.kibana.k8s.elastic.co 2020-02-06T11:17:38Z
4,查看新安装的service
gemfield@civilnet:~/ES$ kubectl -n elastic-system get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elastic-webhook-server ClusterIP 10.103.189.140 443/TCP 5m1s
gemfield@civilnet:~/ES$ kubectl -n elastic-system get po
NAME READY STATUS RESTARTS AGE
elastic-operator-0 1/1 Running 0 5m48s
gemfield@civilnet:~/ES$ kubectl -n elastic-system get statefulset
NAME READY AGE
elastic-operator 1/1 6m11s
gemfield@civilnet:~/ES$ kubectl -n elastic-system get ServiceAccount
NAME SECRETS AGE
default 1 7m43s
elastic-operator 1 7m43s
gemfield@civilnet:~/ES$ kubectl -n elastic-system get ValidatingWebhookConfiguration
NAME CREATED AT
elastic-webhook.k8s.elastic.co 2020-02-06T11:17:38Z
gemfield@civilnet:~/ES$ kubectl -n elastic-system get Secret
NAME TYPE DATA AGE
default-token-bx7vs kubernetes.io/service-account-token 3 8m20s
elastic-operator-token-jptwl kubernetes.io/service-account-token 3 8m20s
elastic-webhook-server-cert Opaque 2 8m20s
5,查看elasticsearch资源
当然,目前还没有
gemfield@civilnet:~/ES$ kubectl get elasticsearch
No resources found.
于是新建gemfield-elasticsearch.yaml文件,内容如下:
gemfield@civilnet:~/ES$ cat gemfield-elasticsearch.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: gemfieldes
spec:
version: 7.5.2
nodeSets:
kubectl apply -f gemfield-elasticsearch.yaml
查看状态,这个时候就有了:
gemfield@civilnet:~/ES$ kubectl get elasticsearch
NAME HEALTH NODES VERSION PHASE AGE
gemfieldes unknown 7.5.2 ApplyingChanges 97s
…
…
gemfield@civilnet:~/ES$ kubectl get elasticsearch
NAME HEALTH NODES VERSION PHASE AGE
gemfieldes green 1 7.5.2 Ready 48s
直到HEALTH的状态为GREEN。
6,查看elasticsearch service
然后查看service:
gemfield@civilnet:~/ES$ kubectl get service gemfieldes-es-http
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
gemfieldes-es-http ClusterIP 10.108.80.75 9200/TCP 83s
对于Elasticsearch、Kibana 和ApmServer这三种资源来说,K8s operator都会无了对应的service,名字为:-[es|kb|apm]-http,类型默认为ClusterIP。ClusterIP类型的service会把服务暴露在集群内网IP上,因此只能在集群内部的Pod上来访问。那怎么让外网可以访问呢?可以使用下面的选项:
apiVersion: .k8s.elastic.co/v1
kind:
metadata:
name: gemfieldes
spec:
version: 7.5.2
http:
service:
spec:
type: LoadBalancer
重点是这里的type,和K8s service的type一样。
7,ECK的TLS证书
这里的证书说的是HTTP层的,用于在不同node之间Elasticsearch内部通信的传输层的证书是ECK内部管理且不可配置的。
默认情况下,operator会维护一个用自定义的CA然后自签的证书,用于Elasticsearch, Kibana 和APM Server。这个自定义的CA、证书、私钥都存放在独立的Secret中:
gemfield@civilnet:~/ES$ kubectl get secret | grep es-http
gemfieldes-es-http-ca-internal Opaque 2 22h
gemfieldes-es-http-certs-internal Opaque 3 22h
gemfieldes-es-http-certs-public Opaque 2 22h
其中,公开的证书存放在名为-[es|kb]-http-certs-public的Secert中:
gemfield@civilnet:~/ES$ kubectl get secret gemfieldes-es-http-certs-public -o go-template=’{{index .data “tls.crt” | base64decode }}’
-----BEGIN CERTIFICATE-----
MIIDrzCCApegAwIBAgIQbXsHRaaZsLACr7MTtzSNkTANBgkqhkiG9w0BAQsFADAv
…
eHTZaZSMCMx8ZwF4WqFqNO0Q45GWTrk9aS0DcELppPeNl9A=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDKTCCAhGgAwIBAgIPYzCgkK7dz6/rq4rCo9vEMA0GCSqGSIb3DQEBCwUAMC8x
…
5teU1SYvf8JgHJecJNPl/sZMjVmwR0I7fcHeNceaQajBrG7SBgN9j0GJX4CZ
-----END CERTIFICATE-----
你可以显式的禁用TLS:
spec:
http:
tls:
selfSignedCertificate:
disabled: true
验证服务是否已经正常运行
1,获取访问服务的用户名和密码
用户名默认为elastic,密码为bc7fzkplf8kn5266dgggvhs2,通过下面的命令获得。
gemfield@civilnet:~/ES$ kubectl get secret gemfieldes-es-elastic-user -o=jsonpath=’{.data.elastic}’ | base64 --decode
bc7fzkplf8kn5266dgggvhs2
2,使用curl命令访问服务
root@gemfield-mysql-client-5596f7ff6f-5l8c2:/# curl -k https://elastic:bc7fzkplf8kn5266dgggvhs2@gemfieldes-es-http:9200
{
“name” : “gemfieldes-es-default-0”,
“cluster_name” : “gemfieldes”,
“cluster_uuid” : “mzYCVK6GSmuxwXF8SfRXZQ”,
“version” : {
“number” : “7.5.2”,
“build_flavor” : “default”,
“build_type” : “docker”,
“build_hash” : “8bec50e1e0ad29dad5653712cf3bb580cd1afcdf”,
“build_date” : “2020-01-15T12:11:52.313576Z”,
“build_snapshot” : false,
“lucene_version” : “8.3.0”,
“minimum_wire_compatibility_version” : “6.8.0”,
“minimum_index_compatibility_version” : “6.0.0-beta1”
},
“tagline” : “You Know, for Search”
}
如何管理ECK的资源
使用request关键字和limit关键字,和K8s类似,就不细说了。
spec:
nodeSets:
APM Server的Requests是512MB,Limits是512MB;
Elasticsearch的Requests是2GB,Limits是2GB;
Kibana的Requests是1GB,Limits是1GB;
如何卸载ECK
1,先删除所有的elastic资源
这一步会删除所有的Elasticsearch、Kibana和APM Server的资源,包括(Pods、secrets、services等):
kubectl get namespaces --no-headers -o custom-columns=:metadata.name
| xargs -n1 kubectl delete elastic --all -n
2,再卸载operator
kubectl delete -f https://download.elastic.co/downloads/eck/1.0.0/all-in-one.yaml
使用Python访问ES
Gemfield使用的是Python3。
1,安装elasticsearch模块
pip3 install elasticsearch
2,编写脚本如下
import ssl
from datetime import datetime
from elasticsearch import Elasticsearch
#from elasticsearch.connection import create_ssl_context
#ssl_context = create_ssl_context()
#ssl_context.check_hostname = False;
#ssl_context.verify_mode = ssl.CERT_NONE
es = Elasticsearch(hosts=“https://elastic:bc7fzkplf8kn5266dgggvhs2@gemfieldes-es-http:9200/”,ca_certs=False,verify_certs=False)
print("gemfield begin: " + str(datetime.now()))
for key in range(10000000):
print(key)
es.index(index=‘messages’,doc_type=‘message’,body={‘message’: ‘gemfield is a civilnet maintainer’,‘name’:‘gemfield’,‘age’:30,‘id’:key})
print("gemfield end: " + str(datetime.now()))
这个脚本里,Gemfield禁用了https中的证书检查。
3,使用curl命令搜索结果
对ECK的扩展
本文前面部署的ECK只是一个node,在这个章节中,Gemfield将介绍如何扩展ECK。
1,node.store.allow_mmap
在这之前,需要介绍一个对ECK很重要的kernel参数:
gemfield@civilnet:~/ES$ sudo sysctl -a | grep max_map
vm.max_map_count = 65530
按照ECK的官方建议,这个值应该修改为262144:
sudo sysctl -w vm.max_map_count=262144
要持久化这个修改,还需要修改/etc/sysctl.conf文件。
这个修改后,就把gemfield-elasticsearch.yaml文件中的"node.store.allow_mmap: false"去掉,内容如下:
gemfield@civilnet:~/ES$ cat gemfield-elasticsearch.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: gemfieldes
spec:
version: 7.5.2
nodeSets:
2,更改node的数量
你可以通过改变count的值来更改node的数量:
gemfield@civilnet:~/ES$ cat gemfield-elasticsearch.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: gemfieldes
spec:
version: 7.5.2
nodeSets:
通过PodTemplate设置elasticsearch的资源:比如java堆的大小,K8s上cpu和ram资源等;通过http字段设置是否启动https:
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: gemfieldes
spec:
version: 7.5.2
http:
tls:
selfSignedCertificate:
disabled: true
nodeSets: