pod本身有生命周期,其应用容器及生成的数据自身均无法独立于该生命周期之外持久存在,并且同一pod中的容器可共享PID、Network、IPC和UTS名称空间,但mount和user名称空间却各自独立,因而跨容器的进程彼此间默认无法基于共享的存储空间交换文件或数据。因此,借助特定的存储机制甚至是独立于pod生命周期的存储设备完成数据持久化也是必然之需。
存储卷并非kubernetes上一种独立的API资源类型,它隶属于Pod资源,且与所属的特定Pod对象有着相同的生命周期,因而通过API Server管理声明了存储卷资源的Pod对象时也会相应触发存储卷的管理操作,在具体的执行过程中,首选由调度器将该Pod对象绑定到一个工作节点上,若该Pod定义存储卷尚未被挂载,Controller Manager中的AD控制器(Attach/Detach Controller)会先借助相应的存储卷插件把远程的存储设备附加到该目标节点,而由内置在kubelet中的Pod管理器触发本地的存储卷操作实现,它借助存储卷管理器(Volume Manager)调用存储卷插件进行关联并驱动相应存储服务,并完成设备挂载、格式化和卸载等操作。存储卷独立于Pod对象中容器的生命周期,从而使得容器重启或更新之后数据依然可用,但删除Pod对象时也必将删除其存储卷。
root@k8s-master-01:/data/k8s# kubectl explain Deployment.spec.template.spec.volumes KIND: Deployment VERSION: apps/v1 RESOURCE: volumes <[]Object> DESCRIPTION: List of volumes that can be mounted by containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes Volume represents a named volume in a pod that may be accessed by any container in the pod. FIELDS: awsElasticBlockStore <Object> AWSElasticBlockStore represents an AWS Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore azureDisk <Object> AzureDisk represents an Azure Data Disk mount on the host and bind mount to the pod. azureFile <Object> AzureFile represents an Azure File Service mount on the host and bind mount to the pod. cephfs <Object> CephFS represents a Ceph FS mount on the host that shares a pod's lifetime cinder <Object> Cinder represents a cinder volume attached and mounted on kubelets host machine. More info: https://examples.k8s.io/mysql-cinder-pd/README.md configMap <Object> ConfigMap represents a configMap that should populate this volume csi <Object> CSI (Container Storage Interface) represents ephemeral storage that is handled by certain external CSI drivers (Beta feature). downwardAPI <Object> DownwardAPI represents downward API about the pod that should populate this volume emptyDir <Object> EmptyDir represents a temporary directory that shares a pod's lifetime. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir ephemeral <Object> Ephemeral represents a volume that is handled by a cluster storage driver. The volume's lifecycle is tied to the pod that defines it - it will be created before the pod starts, and deleted when the pod is removed. Use this if: a) the volume is only needed while the pod runs, b) features of normal volumes like restoring from snapshot or capacity tracking are needed, c) the storage driver is specified through a storage class, and d) the storage driver supports dynamic volume provisioning through a PersistentVolumeClaim (see EphemeralVolumeSource for more information on the connection between this volume type and PersistentVolumeClaim). Use PersistentVolumeClaim or one of the vendor-specific APIs for volumes that persist for longer than the lifecycle of an individual pod. Use CSI for light-weight local ephemeral volumes if the CSI driver is meant to be used that way - see the documentation of the driver for more information. A pod can use both types of ephemeral volumes and persistent volumes at the same time. This is a beta feature and only available when the GenericEphemeralVolume feature gate is enabled. fc <Object> FC represents a Fibre Channel resource that is attached to a kubelet's host machine and then exposed to the pod. flexVolume <Object> FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. flocker <Object> Flocker represents a Flocker volume attached to a kubelet's host machine. This depends on the Flocker control service being running gcePersistentDisk <Object> GCEPersistentDisk represents a GCE Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk gitRepo <Object> GitRepo represents a git repository at a particular revision. DEPRECATED: GitRepo is deprecated. To provision a container with a git repo, mount an EmptyDir into an InitContainer that clones the repo using git, then mount the EmptyDir into the Pod's container. glusterfs <Object> Glusterfs represents a Glusterfs mount on the host that shares a pod's lifetime. More info: https://examples.k8s.io/volumes/glusterfs/README.md hostPath <Object> HostPath represents a pre-existing file or directory on the host machine that is directly exposed to the container. This is generally used for system agents or other privileged things that are allowed to see the host machine. Most containers will NOT need this. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath iscsi <Object> ISCSI represents an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://examples.k8s.io/volumes/iscsi/README.md name <string> -required- Volume's name. Must be a DNS_LABEL and unique within the pod. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names nfs <Object> NFS represents an NFS mount on the host that shares a pod's lifetime More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs persistentVolumeClaim <Object> PersistentVolumeClaimVolumeSource represents a reference to a PersistentVolumeClaim in the same namespace. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims photonPersistentDisk <Object> PhotonPersistentDisk represents a PhotonController persistent disk attached and mounted on kubelets host machine portworxVolume <Object> PortworxVolume represents a portworx volume attached and mounted on kubelets host machine projected <Object> Items for all in one resources secrets, configmaps, and downward API quobyte <Object> Quobyte represents a Quobyte mount on the host that shares a pod's lifetime rbd <Object> RBD represents a Rados Block Device mount on the host that shares a pod's lifetime. More info: https://examples.k8s.io/volumes/rbd/README.md scaleIO <Object> ScaleIO represents a ScaleIO persistent volume attached and mounted on Kubernetes nodes. secret <Object> Secret represents a secret that should populate this volume. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret storageos <Object> StorageOS represents a StorageOS volume attached and mounted on Kubernetes nodes. vsphereVolume <Object> VsphereVolume represents a vSphere volume attached and mounted on kubelets host machine
spec: volumes: - name: <string> VOL_TYPE <Object> # kubectl explain Deployment.spec.template.spec.volumes containers: - name: ... image: ... volumeMounts: - mountPath: <string> #容器文件系统上的挂载点路径 name: <string> #要挂载的存储卷的名称,必须匹配存储卷列表中某项的定义 mountPropagation <string> #挂载卷的传播模式 readOnly <boolean> #是否挂载为只读模式,默认为否 subPath <string> #挂载存储卷上的一个子目录至指定的挂载点 subPathExpr <string> #挂载由指定模式匹配到的存储卷的文件或目录挂载点
kubernetes支持的存储卷类型中,emptyDir存储卷的生命周期与其所属的Pod的对象相同,它无法脱离pod对象的生命周期提供数据存储功能,因此通常仅用于数据缓存或临时存储。基于emptyDir构建的gitrepo存储可以在Pod生命周期起始时,从对应的git仓库中克隆相应的数据文件到底层的emptyDir中,也就是使得它具有了一定意义上的持久性。
emptyDir存储卷可以理解为pod对象上的一个临时目录,类似于Docker上的挂载卷,在pod对象启动时即被创建,而在pod对象被移除时一并被移除,因此emptyDir存储卷只能用于某些特殊场景中,例如同一pod内的多个容器间的文件共享或作为容器数据的临时存储目录用于数据缓存系统等。
kubectl explain Deployment.spec.template.spec.volumes.emptyDir KIND: Deployment VERSION: apps/v1 RESOURCE: emptyDir <Object> DESCRIPTION: EmptyDir represents a temporary directory that shares a pod's lifetime. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir Represents an empty directory for a pod. Empty directory volumes support ownership management and SELinux relabeling. FIELDS: medium <string> #此目录所在的存储介质的类型,可用值为defalut或Memory,默认为defalut,表示使用节点的默认存储介质;Memory表示使用基于RAM临时文件系统tmpfs,总体可用空间受限于内存,但性能非常好,通常用于容器中的应用提供缓存存储。 What type of storage medium should back this directory. The default is "" which means to use the node's default medium. Must be an empty string (default) or Memory. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir sizeLimit <string> # 当前存储卷的空间限额,默认值为nil 表示不限制;不过在medium字段值为Memory时,建议务必定义此限额。 Total amount of local storage required for this EmptyDir volume. The size limit is also applicable for memory medium. The maximum usage on memory medium EmptyDir would be the minimum value between the SizeLimit specified here and the sum of memory limits of all containers in a pod. The default is nil which means that the limit is undefined. More info: http://kubernetes.io/docs/user-guide/volumes#emptydir
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 1 selector: matchLabels: app: ng-deploy-80 template: metadata: labels: app: ng-deploy-80 spec: containers: - name: ng-deploy-80 image: nginx ports: - containerPort: 80 volumeMounts: - mountPath: /cache name: cache-volume volumes: - name: cache-volume emptyDir: medium: Memory sizeLimt: 16Mi
gitRepo存储卷可以看作是emptyDir存储卷的一种实际应用,使用该存储卷的pod资源可以通过挂载目录访问指定的代码仓库中的数据。使用gitRepo存储卷的pod资源在创建时,会首先创建一个空目录并克隆一份指定的git仓库中的数据至该目录,而后在创建容器并挂载该存储卷。
gitrepo存储卷构建于emptyDir之上,其生命周期与pod资源一样,故使用中不应在此类存储卷中保存由容器生成的重要数据。另外,gitrepo存储插件即将废弃,建议在初始化容器或sidecar容器中运行git命令来完成相应的功能。
kubectl explain Deployment.spec.template.spec.volumes.gitRepo KIND: Deployment VERSION: apps/v1 RESOURCE: gitRepo <Object> DESCRIPTION: GitRepo represents a git repository at a particular revision. DEPRECATED: GitRepo is deprecated. To provision a container with a git repo, mount an EmptyDir into an InitContainer that clones the repo using git, then mount the EmptyDir into the Pod's container. Represents a volume that is populated with the contents of a git repository. Git repo volumes do not support ownership management. Git repo volumes support SELinux relabeling. DEPRECATED: GitRepo is deprecated. To provision a container with a git repo, mount an EmptyDir into an InitContainer that clones the repo using git, then mount the EmptyDir into the Pod's container. FIELDS: directory <string> #目标目标名称,但名称中不能包含“..”字段;"."表示将仓库中的数据直接克隆值存储卷映射的目录中,其它字符则表示将数据克隆至存储卷上以用户指定的字符串为名称的子目录中。 Target directory name. Must not contain or start with '..'. If '.' is supplied, the volume directory will be the git repository. Otherwise, if specified, the volume will contain the git repository in the subdirectory with the given name. repository <string> -required- #git仓库的URL,必选字段 Repository URL revision <string> #特定revision提交的哈希码 Commit hash for the specified revision.
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 1 selector: matchLabels: app: ng-deploy-80 template: metadata: labels: app: ng-deploy-80 spec: containers: - name: ng-deploy-80 image: nginx ports: - containerPort: 80 volumeMounts: - mountPath: /usr/share/nginx/html name: html volumes: - name: cache-volume gitRepo: repository: https://github.com/kubernetes/kubernetes.git directory: . revision: "master"
hostPath存储卷插件是将工作节点上某文件系统的目录或文件关联到pod上的一种存储卷类型,其数据具有相同工作节点生命周期一样的持久性。hostPath存储卷使用的是工作节点本地的存储空间,所以仅适用于特定情况下的存储卷使用需求,例如将工作节点上的文件系统关联为pod存储卷,从而让容器访问节点文件系统上的数据,或者排布分布式存储系统的存储设备等。hostPath存储卷在运行有管理任务的系统级pod资源,以及pod资源需要访问节点上的文件时尤为有用。
对于由Deployment或StatefulSet等一类控制器管控的、使用了hostPath存储卷的pod对象来说,需要注意在基于资源可用状态的调度器调度pod对象时,并不支持参考目标节点之上hostPath类型的存储卷,在pod对象被重新调度至其他节点时,容器进程此前创建的文件或目录则大多不会存在。一个通常的解决办法是通过pod对象上使用nodeSelector或者nodeAffinity赋予该pod对象指定要绑定到的具体节点来影响调度器决策,但即便如此,管理员仍然不得不手动管理涉及的多个节点之上的目录,低效且易错。因此,hostPath存储卷虽然能持久保存数据,但对于由调度器按需调度的应用来说并不适用。
kubectl explain Deployment.spec.template.spec.volumes.hostPath KIND: Deployment VERSION: apps/v1 RESOURCE: hostPath <Object> DESCRIPTION: HostPath represents a pre-existing file or directory on the host machine that is directly exposed to the container. This is generally used for system agents or other privileged things that are allowed to see the host machine. Most containers will NOT need this. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath Represents a host path mapped into a pod. Host path volumes do not support ownership management or SELinux relabeling. FIELDS: path <string> -required- #用于指定工作节点上的目录路径,必选 Path of the directory on the host. If the path is a symlink, it will follow the link to the real path. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath type <string> #指定节点之上的存储类型 Type for HostPath Volume Defaults to "" More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 1 selector: matchLabels: app: ng-deploy-80 template: metadata: labels: app: ng-deploy-80 spec: containers: - name: ng-deploy-80 image: nginx ports: - containerPort: 80 volumeMounts: - mountPath: /data name: cache-volume volumes: - name: cache-volume hostPath: path: /opt/data
kubernetes的NFS存储卷用于关联某事先存在的NFS服务器上导出的存储空间到pod对象中以供容器使用,该类型的存储卷在pod对象终止后仅是被卸载而非删除。而且NFS是文件系统级共享服务,它支持同时存在的多路径挂载请求,可由多个pod对象同时关联使用。定义NFS存储卷时支持嵌套使用以下几个字段。
apiVersion: v1 kind: Pod metadata: name: volumes-nfs labels: app: redis spec: containers: - name: redis image: redis:alpine ports: - containerPort: 6379 name: redisport securityContext: runAsUser: 999 volumeMounts: - mountPath: /data name: redisdata volumes: - name: redisdata nfs: server: 192.168.0.1 path: /data/redis readOnly: false
Ceph是一个专注于分布式的、弹性可扩展的、高可用的、性能优异的存储系统平台。同时支持提供块设备、文件系统和对象存储3中存储接口。它是一个高度可配置的系统,并提供了一个命令行界面用于监视和空值其存储集群。kubernetes支持通过RBD卷插件和CephFS卷插件,基于ceph存储系统为Pod提供存储卷。要配置Pod对象使用RBD存储卷,需要事先满足以下前提条件。
定义RBD类型的存储卷时需要指定要连接的目标服务器和认证信息等配置,它们依赖以下几个可用嵌套字段。
apiVersion: v1 kind: Pod metadata: name: volumes-rbd spec: containers: - name: redis image: redis:alpine ports: - containerPort: 6379 name: redisport volumeMounts: - mountPath: /data name: redis-rbd-vol volumes: - name: redis-rbd-vol rbd: monitors: - '172.29.200.1:6789' - '172.29.200.2:6789' - '172.29.200.3:6789' pool: kube image: redis-img1 fsType: xfs readOnly: false user: kube keyring: /etc/ceph/ceph.client.kube.keyring
删除pod对象仅会解除它对RBD映像的引用而非级联删除它,因而RBD映像及数据将依然存在,除非管理员手动进行删除。
CephFS是在分布式对象存储RADOS之上构建的POSIX兼容的文件系统,它致力于为各种应用程序提供多用途、高可用和高性能的文件存储。CephFS将文件元数据和文件数据分布存储在各自专用的RADOS存储池中,其中MDS通过元数据子树分区等支持高吞吐量的工作负载,而数据则由客户端直接相关的存储池直接进行读写操作,其扩展能跟随底层RADOS存储的大小进行线性扩展。kubernetes的CephFS存储卷插件以CephFS为存储方案为pod提供存储卷,因而可受益于CephFS的存储扩展和性能优势。
CephFS存储卷插件嵌套定义于pod资源的spec.volumes.cephfs字段中,它支持通过如下字段的定义接入到存储预配服务中。
apiVersion: v1 kind: Pod metadata: name: volumes-cephfs spec: containers: - name: redis image: redis:alpine volumeMounts: - mountPath: "/data" name: redis-cephfs-vol volumes: - name: redis-cephfs-vol cephfs: monitors: - 172.29.200.1:6789 - 172.29.200.2:6789 - 172.29.200.3:6789 path: /kube/namespaces/default/redis1 #需提前创建 user: fsclient secretFile: "/etc/ceph/fsclient.key" readOnly: false
删除pod对象仅会卸载其挂载的CephFS文件系统,因而文件系统及相关数据将依然存在,除非管理员手动进行删除。另外在实践中,应该把认证到CephFS文件系统上的用户的认证信息存储为kubernetes集群上的Secret资源,并通过SecretRef字段进行指定,而非直接使用secretFile字段引用相应的用户秘钥信息文件。
GlusterFS是一个开源的分布式文件系统,是水平扩展存储解决方案Gluster的核心,它具有强大的横向扩展能力,通过扩展能够支持PB级的存储容量和数千个客户端。GlusterFS借助TCP/IP或InfiniBand RDMA网络将物理分布的存储资源聚集在一起,使用单一全局命名空间来管理数据,它基于可堆叠的用户空间设计,可为各种不同的数据负载提供优异的性能,是另一种流行的分布式存储解决方案。kubernetes的GlusterFS存储卷插件依赖于GlusterFS存储集群作为存储方案。要配置pod资源使用GlusterFS存储卷,需要事先满足以下前提条件:
GlusterFS存储卷嵌套定义在pod资源的spec.volumes.glusterfs字段中,常用的配置字段有以下几个。
用于访问GlusterF集群的相关节点信息要事先保存在特定的Endpoint资源中。glusterfs-endpoint资源需要在kubernetes集群中事先创建,而kube-redis需要先于Gluster集群创建。
创建顺序为先创建Endpoint资源,之后创建pod资源,然后创建Gluster集群。
apiVersion: v1 kind: Endpoints metadata: name: glusterfs-endpoints subsets: - addresses: - ip: 192.168.0.100 ports: - port: 24007 name: glusterd - addresses: - ip: 192.168.0.101 ports: - port: 24007 name: glusterd - addresses: - ip: 192.168.0.102 ports: - port: 24007 name: glusterd
apiVersion: v1 kind: Pod metadata: name: volumes-glusterfs labels: app: redis spec: containers: - name: redis image: redis:alpine ports: - containerPort: 6379 name: redisport volumeMounts: - mountPath: /data name: redisdata volumes: - name: redisdata glusterfs: endpoints: glusterfs-endpoints path: kube-redis readOnly: false