Creating ceph pool and create RBD storage in Proxmox 8

Step 1: Install Ceph on Proxmox 8 Nodes

• You must install Ceph on all nodes in your Proxmox cluster.
• Log into your Proxmox web GUI.
• Select your first Node, navigate to Ceph, and click Install Ceph.
• Choose Ceph Quincy or Ceph Reef (depending on your specific PVE 8 sub-version; Quincy is standard). Click Start Installation.
• When asked for network configuration, select your dedicated Ceph Public Network and Ceph Cluster Network (highly recommended to use a fast, separated 10GbE+ network).
• Repeat this installation wizard on your other cluster nodes.

Step 2: Configure Ceph Monitors and Managers

• Ceph needs at least 3 Monitors (MON) and Managers (MGR) for high availability.
• Go to Datacenter > [First Node] > Ceph > Monitor.
• Under Monitor, click Create to add monitors on your other nodes until you have an odd number (typically 3).
• Under Manager, click Create to add manager daemons on your additional nodes for redundancy.

Step 3: Create Ceph OSDs (Disks)

• OSDs (Object Storage Daemons) consume raw, unformatted physical disks to build the storage cluster.
• Go to [Node] > Ceph > OSD.
• Click Create: OSD.
• Select an unused disk (such as an SSD or NVMe) from the dropdown menu. Leave DB/WAL settings as default unless you are splitting metadata onto faster NVMe drives.
• Click Create.
• Repeat this for all available disks across all your nodes until the cluster health turns to HEALTH_OK.

Step 4: Create a Ceph Pool

• Once your OSDs are up, you need a pool to orchestrate your files or block devices.
• Go to Datacenter or any individual node, navigate to Ceph > Pools.
• Click Create.
• Fill out the configuration parameters:Name: e.g., vm-rbd-pool
• Size (Replication): 3 (stores 3 copies of your data across different nodes)
• Min Size: 2 (allows writes if at least 2 nodes/OSDs are online)
• Crush Rule: Select your target rule (usually replicated_rule).
• PG Autoscale: Keep this set to on so Ceph manages Placement Groups automatically.
• Click Create.

Step 5: Add the Pool as RBD Storage in Proxmox

• To make Proxmox use this new Ceph pool for VM and container disks, you must register it as a virtual storage tier.
• Navigate to Datacenter > Storage.
• Click Add and select RBD from the dropdown menu.
• Configure the following fields:ID: Enter a recognizable name in Proxmox (e.g., ceph-storage).
• Pool: Select the Ceph pool you created in Step 4 (e.g., vm-rbd-pool).
• Content: Select Disk image (for VMs) and Container (if you want to host LXCs).
• Nodes: Leave blank to let all cluster nodes access it (critical for live migration).
• KRBD: Leave unchecked for standard virtual machines (librbd is faster), but check it if you plan to run containers on this storage.
• Click Add.

Your new highly-available RBD storage cluster is now ready! When creating a new VM, simply select your newly defined RBD storage tier in the “Disks” tab.

1. Can I create multiple pools (e.g., SATA pool vs. SSD pool)?

Yes. You can separate your storage into a slow tier (SATA HDDs for backups or cold storage) and a fast tier (SSDs/NVMes for high-performance VMs).
To achieve this, Ceph uses CRUSH Rules to filter disks by device class:

• Verify Device Classes: When you create OSDs in Proxmox under Node > Ceph > OSD, Ceph automatically detects their type and labels them as hdd or ssd.
• Create a Specific CRUSH Rule (via CLI): Proxmox’s GUI usually creates a generic replicated_rule that uses all drives. To separate them, you can quickly create specific rules via any node’s shell CLI):

• For SSDs: ceph osd crush rule create-replicated ssd-rule default host ssd

• For HDDs: ceph osd crush rule create-replicated hdd-rule default host hdd

• Create the Pools:Go to Ceph > Pools > Create.
• Name your first pool ssd-pool and select ssd-rule under Crush Rule.
• Create a second pool named sata-pool and select hdd-rule under Crush Rule.
• Add to Storage: Go to Datacenter > Storage > Add > RBD and map them as separate storage IDs (e.g., ceph-fast and ceph-slow).

2. Can I expand an existing pool by adding new disks and nodes?

Yes. This is one of Ceph’s greatest strengths. You never add disks directly to a pool; instead, you add disks to the overall Ceph cluster as new OSDs. The pools will automatically expand to utilize the new space.
When you add new nodes and 10 new disks:

• Connect the Nodes: Install Proxmox on the new nodes and join them to your existing Proxmox cluster.
• Install Ceph: Go to the new nodes in the GUI, install Ceph, and configure their network settings to match your Ceph cluster network.
• Create New OSDs: On the new nodes, go to Ceph > OSD > Create: OSD and add the 10 new disks.

What happens next automatically:

• As soon as the new OSDs turn “Up” and “In”, Ceph’s CRUSH algorithm instantly recognizes the new hardware.
• Ceph will automatically begin rebalancing data in the background, moving a portion of your existing virtual machine data from the old disks to the new disks across the new nodes.
  Your existing RBD pools will immediately show a larger total capacity.
• Note: Your VMs will remain online and fully operational during this expansion process, though you might notice a slight performance dip while the data is actively migrating.

3. View All Disks and Their Assigned Classes

The easiest way to check how Ceph has classified every drive in your cluster is the tree view command: 

ceph osd tree

What to look for: Look under the CLASS column. You will see hdd, ssd, or nvme next to each OSD ID: 

ID CLASS WEIGHT TYPE NAME
-1 0.07910 root default
-3 0.02637 host pve-node1
0 ssd 0.00879 osd.0 up 1.00000
1 hdd 0.00879 osd.1 up 1.00000

4. List Classes Present in Your Cluster

To see a clean list of all unique drive classes currently active in your cluster, use:

ceph osd crush class ls

Output example: [ “hdd”, “ssd” ]

5. List OSDs Belonging to a Specific Class

If you have a massive cluster and only want to see the disk IDs assigned to your fast tier, you can filter them: 

ceph osd crush class ls-osd ssd

Output example:

[ 0, 2, 4 ] (Returns just the OSD IDs that are SSDs)

6. How to Fix a Misclassified Disk

Sometimes, if a drive is connected through certain RAID controllers or virtual hypervisors, Ceph might mistakenly label an SSD as an hdd (or vice versa). You can manually overwrite this using the

Remove the wrong class from the OSD (replace 0 with your target OSD ID):

ceph osd crush rm-device-class osd.0

Assign the correct class manually:

ceph osd crush set-device-class ssd osd.0