ceph存储

所有其他客户端接口使用和部署的基础。由以下组件组成： 
	OSD：Object StorageDevice，提供数据实体存储资源；
	Monitor：维护整个Ceph集群中各个节点的心跳信息，维持整个集群的全局状态；
	MDS：Ceph Metadata Server，文件系统元数据服务节点。MDS也支持多台机器分布式的部署，以实现系统的高可用性。

典型的RADOS部署架构由少量的Monitor监控器以及大量的OSD存储设备组成，它能够在动态变化的基于异质结构的存储设备集群之上提供一种稳定的、可扩展的、高性能的单一逻辑对象存储接口。 

# 安装python3

# 安装docker

# 时间同步服务器

# 安装lvm2

yum -y install python3 lvm2

# 使用curl获取独立脚本的最新版本。
curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm

# 添加权限
chmod a+x cephadm

# 尽管独立脚本足以启动集群，但将cephadm命令安装在主机上会很方便。
# 要安装提供该cephadm命令的软件包，请运行以下命令：
./cephadm add-repo --release octopus
./cephadm install

# 运行which以下命令确认它现在在您的PATH中：
[root@master1 ~]# which cephadm
/usr/sbin/cephadm


# 特定发行版安装
# 在 Ubuntu 中：
apt install -y cephadm

# 在 Fedora 中：
dnf -y install cephadm

# 在 SUSE 中：
zypper install -y cephadm

# 创建新Ceph集群的第一步是在Ceph集群的第一台主机上运行命令。
# 在Ceph集群的第一台主机上运行命令的行为会创建Ceph集群的第一个“监视器守护进程”，并且该监视器守护进程需要一个 IP 地址。您必须将 Ceph 集群的第一台主机的 IP 地址传递给该命令，因此您需要知道该主机的IP地址。

# 如果有多个网络和接口，请确保选择一个可供访问 Ceph 集群的任何主机访问的网络和接口。
# 有关初始化的更多详情，可以运行 cephadm bootstrap -h

cephadm bootstrap --mon-ip 192.168.10.241

此命令将：
	在本地主机上为新集群创建一个监视器和管理器守护进程。
	为 Ceph 集群生成一个新的 SSH 密钥并将其添加到 root 用户的/root/.ssh/authorized_keys文件中。
	将公钥的副本写入/etc/ceph/ceph.pub.
	将最小配置文件写入/etc/ceph/ceph.conf. 需要此文件与新集群通信。
	将client.admin管理（特权！）密钥的副本写入/etc/ceph/ceph.client.admin.keyring.
	将_admin标签添加到引导主机。默认情况下，这个标签的任何主机将（也）获得的副本/etc/ceph/ceph.conf和 /etc/ceph/ceph.client.admin.keyring。

[root@master1 ~]# cephadm bootstrap --mon-ip 192.168.10.241
Creating directory /etc/ceph for ceph.conf
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chronyd.service is enabled and running
Repeating the final host check...
podman|docker (/usr/bin/docker) is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
Cluster fsid: 9cad7e42-3d1a-11ec-ab43-faad9d5a0000
Verifying IP 192.168.10.241 port 3300 ...
Verifying IP 192.168.10.241 port 6789 ...
Mon IP 192.168.10.241 is in CIDR network 192.168.10.0/24
Pulling container image quay.io/ceph/ceph:v15...
Extracting ceph user uid/gid from container image...
Creating initial keys...
Creating initial monmap...
Creating mon...
Waiting for mon to start...
Waiting for mon...
mon is available
Assimilating anything we can from ceph.conf...
Generating new minimal ceph.conf...
Restarting the monitor...
Setting mon public_network...
Creating mgr...
Verifying port 9283 ...
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Wrote config to /etc/ceph/ceph.conf
Waiting for mgr to start...
Waiting for mgr...
mgr not available, waiting (1/10)...
mgr not available, waiting (2/10)...
mgr not available, waiting (3/10)...
mgr not available, waiting (4/10)...
mgr not available, waiting (5/10)...
mgr is available
Enabling cephadm module...
Waiting for the mgr to restart...
Waiting for Mgr epoch 5...
Mgr epoch 5 is available
Setting orchestrator backend to cephadm...
Generating ssh key...
Wrote public SSH key to to /etc/ceph/ceph.pub
Adding key to root@localhost's authorized_keys...
Adding host master1...
Deploying mon service with default placement...
Deploying mgr service with default placement...
Deploying crash service with default placement...
Enabling mgr prometheus module...
Deploying prometheus service with default placement...
Deploying grafana service with default placement...
Deploying node-exporter service with default placement...
Deploying alertmanager service with default placement...
Enabling the dashboard module...
Waiting for the mgr to restart...
Waiting for Mgr epoch 13...
Mgr epoch 13 is available
Generating a dashboard self-signed certificate...
Creating initial admin user...
Fetching dashboard port number...
Ceph Dashboard is now available at:

             URL: https://master1:8443/
            User: admin
        Password: d64o9mtmor

You can access the Ceph CLI with:

        sudo /usr/sbin/cephadm shell --fsid 9cad7e42-3d1a-11ec-ab43-faad9d5a0000 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring

Please consider enabling telemetry to help improve Ceph:

        ceph telemetry on

For more information see:

        https://docs.ceph.com/docs/master/mgr/telemetry/

Bootstrap complete.

# 默认情况下，可以使用cephadm shell

# 要使用ceph命令，可以使用cephadm shell -- ceph -s

# 也可以安装ceph-common包，其中包含所有ceph命令，包括ceph、rbd、mount.ceph（用于挂载 CephFS 文件系统）等

    cephadm add-repo --release octopus
    cephadm install ceph-common
    
# 检查是否安装成功
[root@master1 ~]# ceph -v
ceph version 15.2.15 (2dfb18841cfecc2f7eb7eb2afd65986ca4d95985) octopus (stable)

# 使用以下ceph命令确认该命令可以连接到集群及其状态：
[root@master1 ~]# ceph status
  cluster:
    id:     9cad7e42-3d1a-11ec-ab43-faad9d5a0000
    health: HEALTH_ERR
            Module 'dashboard' has failed: OSError(98, 'Address already in use')
            OSD count 0 < osd_pool_default_size 3
 
  services:
    mon: 1 daemons, quorum master1 (age 9m)
    mgr: master1.jzdjri(active, since 7m)
    osd: 0 osds: 0 up, 0 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:     

# 添加秘钥
ssh-copy-id -f -i /etc/ceph/ceph.pub root@master2
ssh-copy-id -f -i /etc/ceph/ceph.pub root@master3
ssh-copy-id -f -i /etc/ceph/ceph.pub root@node1
ssh-copy-id -f -i /etc/ceph/ceph.pub root@node2
ssh-copy-id -f -i /etc/ceph/ceph.pub root@node3

# 告诉cpeh新增机器
ceph orch host add master2 192.168.10.242
ceph orch host add master3 192.168.10.243
ceph orch host add node1 192.168.10.244
ceph orch host add node2 192.168.10.245
ceph orch host add node3 192.168.10.246

# 自动检测部署
ceph config set mon public_network 192.168.10.1/24

# 特定网络部署
ceph orch apply mon --unmanaged

# 使用任何可用和未使用的设备
ceph orch apply osd --all-available-devices

# 使用指定主机上的设备
ceph orch daemon add osd master1:/dev/vdb
ceph orch daemon add osd master2:/dev/vdb
ceph orch daemon add osd master3:/dev/vdb
ceph orch daemon add osd node1:/dev/vdb
ceph orch daemon add osd node2:/dev/vdb
ceph orch daemon add osd node3:/dev/vdb

# 自动创建
ceph orch apply osd --all-available-devices

[root@master1 ~]# ceph status
  cluster:
    id:     9cad7e42-3d1a-11ec-ab43-faad9d5a0000
    health: HEALTH_OK
 
  services:
    mon: 1 daemons, quorum master1 (age 15m)
    mgr: master1.jzdjri(active, since 8m), standbys: master2.uwtwzp
    osd: 6 osds: 6 up (since 15m), 6 in (since 80m)
 
  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   6.0 GiB used, 114 GiB / 120 GiB avail
    pgs:     1 active+clean
 

# 获取dashborad地址
[root@master1 ~]#  ceph mgr services
{
    "dashboard": "http://master1:8080/",
    "prometheus": "http://master1:9283/"
}

# 创建RADOS 池，一个用于数据，一个用于元数据
[root@master1 ~]# ceph osd pool create cephfs_data
pool 'cephfs_data' created
[root@master1 ~]# ceph osd pool create cephfs_metadata
pool 'cephfs_metadata' created

# 创建文件系统
# 格式 ceph fs new <fs_name> <metadata> <data>
[root@master1 ~]# ceph fs new cephfs cephfs_metadata cephfs_data
new fs with metadata pool 3 and data pool 2

[root@master1 ~]# ceph fs ls
name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]

# 检查文件系统
[root@master1 ~]# ceph mds stat
cephfs:0
# 发现文件系统并未启动，获取详细报错信息
[root@master1 ~]# ceph health detail 
HEALTH_ERR 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds
[ERR] MDS_ALL_DOWN: 1 filesystem is offline
    fs cephfs is offline because no MDS is active for it.
[WRN] MDS_UP_LESS_THAN_MAX: 1 filesystem is online with fewer MDS than max_mds
    fs cephfs has 0 MDS online, but wants 1
    
# 这是因为文件系统需要启动 ceph-mds 
# ceph-mds 用来管理文件系统的元数据，只有文件系统才需要，对象储存和块储存都不需要。
# 在web管理页面，Cluster-->Service-->Create--> type mds 创建mds即可

# 挂载文件系统，name指代用户，需要系统中存在此用户，且有相关的权限才可
mount -t ceph 192.168.10.241:6789:/ /mnt/mycephfs -o name=admin
umount /mnt/mycephfs

# 持久挂载
192.168.10.241:6789:/     /mnt/ceph    ceph    name=admin,noatime,_netdev    0       2

# 若有多个文件系统，则使用以下方式
mount -t ceph :/ /mnt/mycephfs2 -o name=fs,fs=mycephfs2

# 更多权限管理请参考官方文档

# 列出池
ceph osd lspools

# 创建块设备池
[root@master1 ~]# ceph osd pool create ceph-rbd
pool 'ceph-rbd' created


# 初始化池以供RBD使用
rbd pool init ceph-rbd

# 创建块设备用户
ceph auth get-or-create client.rbp mon 'profile rbd' osd 'profile rbd pool=vms, profile rbd-read-only pool=images' mgr 'profile rbd pool=images' > /etc/ceph/ceph.client.rbp.keyring

# 创建设备映像，若不指定pool，则使用默认pool
rbd create --size 1024 ceph-rbd/bar

# 查看块设备映像
rbd ls {poolname}

# 查看池中延迟删除的的块设备
rbd trash ls {poolname}

# 调整块设备映像的大小


# 删除块设备映像
rbd rm {pool-name}/{image-name}

# 创建池
ceph osd pool create volumes
ceph osd pool create images
ceph osd pool create backups
ceph osd pool create vms

# 新创建的池必须在使用前进行初始化
rbd pool init volumes
rbd pool init images
rbd pool init backups
rbd pool init vms

# 查看创建的池
ceph osd lspools

# 配置OpenStack Ceph客户端
# 在ceph集群的控制节点上执行,分别往计算节点，控制节点，镜像节点和存储节点复制文件
ssh {your-openstack-server} sudo tee /etc/ceph/ceph.conf </etc/ceph/ceph.conf


# ceph节点配置认证
## 创建glance用户及权限
ceph auth get-or-create client.glance mon 'profile rbd' osd 'profile rbd pool=images' mgr 'profile rbd pool=images' -o /etc/ceph/ceph.client.glance.keyring

## 创建cinder用户及权限
ceph auth get-or-create client.cinder mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd-read-only pool=images' mgr 'profile rbd pool=volumes, profile rbd pool=vms' -o /etc/ceph/ceph.client.cinder.keyring

## 创建cinder-backup用户及权限
ceph auth get-or-create client.cinder-backup mon 'profile rbd' osd 'profile rbd pool=backups' mgr 'profile rbd pool=backups' -o /etc/ceph/ceph.client.cinder-backup.keyring

## 创建nova用户及权限
ceph auth get-or-create client.nova mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd-read-only pool=images,profile rbd pool=backups' mgr 'profile rbd pool=volumes, profile rbd pool=vms' -o /etc/ceph/ceph.client.nova.keyring

# 安装ceph客户端包
# 此操作在openstack节点执行，所有需要访问ceph集群的节点都需要安装
yum install -y python-rbd ceph-common


ceph auth get-or-create client.glance | ssh 192.168.10.241 sudo tee /etc/ceph/ceph.client.glance.keyring
ssh 192.168.10.241 sudo chown glance:glance /etc/ceph/ceph.client.glance.keyring
ceph auth get-or-create client.cinder | ssh 192.168.10.241 sudo tee /etc/ceph/ceph.client.cinder.keyring
ssh 192.168.10.241 sudo chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring
ceph auth get-or-create client.cinder-backup | ssh 192.168.10.241 sudo tee /etc/ceph/ceph.client.cinder-backup.keyring
ssh 192.168.10.241 sudo chown cinder:cinder /etc/ceph/ceph.client.cinder-backup.keyring

ceph auth get-key client.cinder | ssh 192.168.10.241 tee client.cinder.key

cat > secret.xml <<EOF
<secret ephemeral='no' private='no'>
  <uuid>b68d66f9-2316-4a2c-b2a9-1a525abbbf32</uuid>
  <usage type='ceph'>
    <name>client.cinder secret</name>
  </usage>
</secret>
EOF

sudo virsh secret-define --file secret.xml

sudo virsh secret-set-value --secret b68d66f9-2316-4a2c-b2a9-1a525abbbf32 --base64 $(cat client.cinder.key) && rm client.cinder.key secret.xml