Crush map
CRUSH map采用树状结构,最小叶子节点就是真实物理磁盘称为device,中间节点称为bucket,每个bucket可以是device集合,也可以是低一级的bucket集合,最上面还有一个root节点,代表集群的入口
CRUSH hierarchy
生成环境可以考虑这样的层级:数据中心-----机架-----主机-------磁盘(root–rack–host–osd)
默认Crush hierarchy
编辑Crush map
配置文件替换
ceph osd getcrushmap -o map.file #导出集群的map配置,默认是二进制 crushtool -d map.file -o map.txt #二进制反编译成文本文件,进行编辑 crushtool -c map.txt -o map.new #文本编译成二进制文件 ceph osd setcrushmap -i map.new #设置crush map
配置解释
定义bucket中间节点
格式
[bucket-type] [bucket-name] {
id [a unique negative numeric ID]
weight [the relative capacity/capability of the item(s)]
alg [the bucket type: uniform | list | tree | straw ]
hash [the hash type: 0 by default]
item [item-name] weight [weight]
}
rack rack-02 { id -13 # do not change unnecessarily alg straw2 hash 0 # rjenkins1 item ceph1 weight 0.098 item ceph3 weight 0.098 } root default { id -1 # do not change unnecessarily id -4 class hdd # do not change unnecessarily alg straw2 hash 0 # rjenkins1 item rack-02 weight 0.196 }
定义CRUSH rule
配置格式
rule <rulename> { type [ replicated | erasure ] min_size <min-size> max_size <max-size> step take <bucket-type> step [choose|chooseleaf] [firstn|indep] <N> <bucket-type> step emit } rule replicated_rule { id 0 type replicated min_size 1 max_size 3 step take default step chooseleaf firstn 0 type rack step emit }
复杂场景的一个应用
step take ssd << --从ssd根节点出发 step chooseleaf firstn 1 type osd << --从ssd整个系统中选择一个OSD作为主OSD step emit << --输出选择,清除栈空间 step take hdd << --重新从hdd根节点出发 step chooseleaf firstn -1 type osd << --从hdd中选择处剩下的需要的OSD step emit
举例配置故障域
root–rack–host–osd
rack rack-01 { #定义机柜层级 id -11 # do not change unnecessarily # weight 0.391 alg straw2 hash 0 # rjenkins1 item ceph2 weight 0.098 item ceph4 weight 0.098 } rack rack-02 { #定义机柜层级 id -13 # do not change unnecessarily # weight 0.391 alg straw2 hash 0 # rjenkins1 item ceph1 weight 0.098 item ceph3 weight 0.098 } root default { #默认root入口,修改项目为机柜级别定义的 id -1 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 0.391 alg straw2 hash 0 # rjenkins1 item rack-01 weight 0.196 item rack-02 weight 0.196 } # rules rule replicated_rule { id 0 type replicated min_size 1 max_size 3 step take default step chooseleaf firstn 0 type rack #修改故障域级别,默认为host step emit }