kvm: apply rbd_default_data_pool when creating volumes from templates on RBD by bhouse-nexthop · Pull Request #13361 · apache/cloudstack

bhouse-nexthop · 2026-06-06T03:19:27Z

Description

RBD erasure-coded (EC) pool support (#9808) taught the KVM agent to honor the rbd_default_data_pool storage-pool detail, but only in two places:

KVMPhysicalDisk.RBDStringBuilder() — the qemu-img URI builder
createPhysicalDisk() — blank volume creation (routes RBD creates to the QemuImg path when a data pool is set)

It was not applied to createDiskFromTemplateOnRBD(), which creates ROOT volumes from a template using rados-java (rbd.clone() / rbd.create()) directly. Those calls build a Rados connection that sets mon_host, key and client_mount_timeout but never rbd_default_data_pool, so the resulting image is created without a data pool.

Net effect: every volume cloned from a template onto an EC-backed primary storage has all of its data objects written to the replicated metadata pool instead of the erasure-coded data pool. This silently defeats EC and consumes ~3x raw space for those volumes. Blank data disks on the same pool are correct, which makes the inconsistency easy to miss.

This was observed in production: of the VMs on an EC primary storage, the template-derived ROOT volumes had no data_pool (rbd info shows no data_pool: line and lacks the data-pool feature), while blank DATADISKs and the template base images themselves were correct.

Fix

In createDiskFromTemplateOnRBD, read the destination pool's rbd_default_data_pool detail once and, when present, confSet it on the Rados connection before connect() — in both the same-cluster clone/copy branch and the cross-cluster copy branch. librbd then uses it as the default data pool when the new image is created, so template-derived volumes get data_pool set, exactly like blank volumes already do. This mirrors how RBDStringBuilder injects the same key for the qemu-img path.

String dataPool = (destDetails == null) ? null : destDetails.get(KVMPhysicalDisk.RBD_DEFAULT_DATA_POOL);
...
if (dataPool != null) {
    r.confSet(KVMPhysicalDisk.RBD_DEFAULT_DATA_POOL, dataPool);
}
r.connect();

No behavior change for non-EC pools (dataPool == null → no-op).

Types of changes

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Root cause reproduced live: deploying a VM from a template onto an EC primary storage produced a ROOT clone with no data_pool (parent base snapshot in the metadata pool); the data-pool feature was absent.
Confirmed the working paths (template seed via qemu-img, blank DATADISK create) already set the data pool, isolating the gap to the rados-java clone/create path.
Code-path traced to Add support to RBD erasure code pools #9808, which did not touch createDiskFromTemplateOnRBD.

… on RBD RBD erasure-coded pool support (apache#9808) added handling of the rbd_default_data_pool storage-pool detail to RBDStringBuilder (qemu-img path) and to createPhysicalDisk (blank volumes), but not to createDiskFromTemplateOnRBD. As a result, ROOT volumes created from a template via rados-java rbd.clone()/rbd.create() are created without a data pool: all of their data objects land in the (replicated) metadata pool instead of the erasure-coded data pool, defeating the point of EC and consuming ~3x raw space. Set rbd_default_data_pool on the Rados connection (before connect) in both the same-cluster clone/copy branch and the cross-cluster copy branch of createDiskFromTemplateOnRBD, using the destination pool's detail. librbd then honors it as the default data pool when the new image is created, so template-derived volumes get data_pool set, the same way blank volumes already do. Signed-off-by: Brad House <bhouse@nexthop.ai>

DaanHoogland · 2026-06-15T09:53:43Z

@blueorangutan package

blueorangutan · 2026-06-15T09:56:04Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2026-06-15T10:00:52Z

Codecov Report

❌ Patch coverage is 0% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.94%. Comparing base (10d3e11) to head (c664fb9).
⚠️ Report is 19 commits behind head on 4.22.

Files with missing lines	Patch %	Lines
.../hypervisor/kvm/storage/LibvirtStorageAdaptor.java	0.00%	8 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.22   #13361      +/-   ##
============================================
+ Coverage     17.91%   17.94%   +0.02%     
- Complexity    16109    16146      +37     
============================================
  Files          5928     5928              
  Lines        534964   535182     +218     
  Branches      65463    65497      +34     
============================================
+ Hits          95865    96019     +154     
- Misses       428191   428234      +43     
- Partials      10908    10929      +21

Flag	Coverage Δ
uitests	`4.02% <ø> (-0.01%)`	⬇️
unittests	`19.01% <0.00%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

blueorangutan · 2026-06-15T10:51:39Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18262

DaanHoogland · 2026-06-17T10:22:27Z

@blueorangutan test

blueorangutan · 2026-06-17T10:24:04Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-06-18T01:15:46Z

[SF] Trillian test result (tid-16344)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 50505 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13361-t16344-kvm-ol8.zip
Smoke tests completed. 149 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland

clgtm

bhouse-nexthop · 2026-08-06T14:32:24Z

@vladimirpetrov @sureshanaparti please make sure this gets into 4.22.2

Signed-off-by: Brad House <bhouse@nexthop.ai>

weizhouapache · 2026-09-10T02:03:21Z

@blueorangutan package

weizhouapache

code lgtm, I will apply the change, and run some tests to make sure no regression

blueorangutan · 2026-09-10T02:04:04Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-09-10T03:39:45Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19193

weizhouapache · 2026-09-11T07:32:10Z

I didn't see any regressions in my testing. That said, I don't use RBD erasure code pools, so my testing doesn't cover that case.

@bhouse-nexthop Have you tested this? If so, could you share the "rbd info" for the root and data volumes?

@BryanMLima Would you mind reviewing and testing this? You added support for RBD erasure code pools in #9808.

bhouse-nexthop · 2026-09-11T16:14:13Z

Sure.

I just reverted to the official 4.22.1.1 release in my lab, and created an instance on the EC pool:

root@node1:~# rbd info vm-images-ec-meta/54c2ce78-822f-401f-91fb-653a436e560d
rbd image '54c2ce78-822f-401f-91fb-653a436e560d':
	size 20 GiB in 5120 objects
	order 22 (4 MiB objects)
	snapshot_count: 0
	id: 15009df80d35f7
	block_name_prefix: rbd_data.15009df80d35f7
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
	op_features:
	flags:
	create_timestamp: Fri Sep 11 11:55:44 2026
	access_timestamp: Fri Sep 11 12:04:08 2026
	modify_timestamp: Fri Sep 11 12:07:33 2026
	parent: vm-images-ec-meta/ba450ca9-8259-4351-90f3-234932d860f2@cloudstack-base-snap
	overlap: 3.5 GiB

See no data_pool entry.

I'll follow up after I re-upgrade and show a new instance there.

bhouse-nexthop · 2026-09-11T16:33:26Z

And with fix deployed:

root@node1:~# rbd info vm-images-ec-meta/9c5c2996-666a-4814-848d-05ac5f26f722
rbd image '9c5c2996-666a-4814-848d-05ac5f26f722':
	size 20 GiB in 5120 objects
	order 22 (4 MiB objects)
	snapshot_count: 0
	id: 1558c5c328f8fa
	data_pool: vm-images-ec
	block_name_prefix: rbd_data.17.1558c5c328f8fa
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool
	op_features:
	flags:
	create_timestamp: Fri Sep 11 12:31:04 2026
	access_timestamp: Fri Sep 11 12:31:04 2026
	modify_timestamp: Fri Sep 11 12:31:04 2026
	parent: vm-images-ec-meta/ba450ca9-8259-4351-90f3-234932d860f2@cloudstack-base-snap
	overlap: 3.5 GiB

Note the data_pool entry

weizhouapache · 2026-09-12T09:30:40Z

And with fix deployed:

root@node1:~# rbd info vm-images-ec-meta/9c5c2996-666a-4814-848d-05ac5f26f722
rbd image '9c5c2996-666a-4814-848d-05ac5f26f722':
	size 20 GiB in 5120 objects
	order 22 (4 MiB objects)
	snapshot_count: 0
	id: 1558c5c328f8fa
	data_pool: vm-images-ec
	block_name_prefix: rbd_data.17.1558c5c328f8fa
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool
	op_features:
	flags:
	create_timestamp: Fri Sep 11 12:31:04 2026
	access_timestamp: Fri Sep 11 12:31:04 2026
	modify_timestamp: Fri Sep 11 12:31:04 2026
	parent: vm-images-ec-meta/ba450ca9-8259-4351-90f3-234932d860f2@cloudstack-base-snap
	overlap: 3.5 GiB

Note the data_pool entry

cool, I will verify too

boring-cyborg Bot added the component:kvm label Jun 6, 2026

github-actions Bot mentioned this pull request Jun 6, 2026

[repo-status] Daily Status Report – June 6, 2026 #13364

Closed

DaanHoogland approved these changes Jun 18, 2026

View reviewed changes

DaanHoogland added the needs-review label Jun 18, 2026

weizhouapache added this to the 4.22.2 milestone Jun 29, 2026

DaanHoogland requested review from sureshanaparti and vladimirpetrov August 10, 2026 11:39

DaanHoogland added this to CloudStack Testing Aug 10, 2026

github-project-automation Bot moved this to Backlog in CloudStack Testing Aug 10, 2026

DaanHoogland moved this from Backlog to Ready in CloudStack Testing Sep 1, 2026

DaanHoogland added needs-testing and removed needs-review labels Sep 1, 2026

DaanHoogland assigned nvazquez Sep 1, 2026

nvazquez assigned weizhouapache and unassigned nvazquez Sep 7, 2026

weizhouapache added the component:ceph label Sep 8, 2026

Merge branch '4.22' into fix/kvm-rbd-ec-datapool-template-clone

c664fb9

Signed-off-by: Brad House <bhouse@nexthop.ai>

weizhouapache reviewed Sep 10, 2026

View reviewed changes

bhouse-nexthop force-pushed the fix/kvm-rbd-ec-datapool-template-clone branch from 5283d5a to c664fb9 Compare September 11, 2026 01:26

weizhouapache assigned BryanMLima Sep 11, 2026

Uh oh!

Conversation

bhouse-nexthop commented Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Fix

Types of changes

How Has This Been Tested?

Uh oh!

DaanHoogland commented Jun 15, 2026

Uh oh!

blueorangutan commented Jun 15, 2026

Uh oh!

codecov Bot commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Jun 15, 2026

Uh oh!

DaanHoogland commented Jun 17, 2026

Uh oh!

blueorangutan commented Jun 17, 2026

Uh oh!

blueorangutan commented Jun 18, 2026

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

bhouse-nexthop commented Aug 6, 2026

Uh oh!

weizhouapache commented Sep 10, 2026

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Sep 10, 2026

Uh oh!

blueorangutan commented Sep 10, 2026

Uh oh!

weizhouapache commented Sep 11, 2026

Uh oh!

bhouse-nexthop commented Sep 11, 2026

Uh oh!

bhouse-nexthop commented Sep 11, 2026

Uh oh!

weizhouapache commented Sep 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

bhouse-nexthop commented Jun 6, 2026 •

edited

Loading

codecov Bot commented Jun 15, 2026 •

edited

Loading