Give a system VM one NIC queue per CPU - #14136

Description

A system VM gets one NIC queue no matter how many CPUs it has, so every packet interrupt lands on
CPU0. Adding CPUs to a router therefore does not move more packets, which is the only reason to add
them.

The queue count is already a VM detail, nic.multiqueue.number, and the agent already turns it into
<driver queues='N'/> on the interface. A user VM can set it three ways:

A system VM goes through none of them. It is not created by deployVirtualMachine, it is not a
UserVm so updateVirtualMachine does not apply, and it has no details editor. There is no way
to set it at all, which is why this is a fix rather than a new setting.

So a system VM now gets one queue per CPU when nothing is set:

The 256 is the tap device ceiling in the host kernel, which refuses the interface rather than
trimming to fit. Above the CPU count it would make no difference anyway - the guest driver uses
min(CPUs, queues) and pushes that at probe, so no guest side change is needed.

A queue number already set on the VM still wins.

The default offering is a single CPU, so a default install emits the same domain XML as before.
Only a system VM someone has already resized differs, and it differs in the direction the resize
asked for.

Worth calling out

• There is no way to turn this off. There is no way to turn it on today either, so it is
  symmetric, but it is worth being explicit about on a release branch.
• The cost scales with NIC count, not just queue count. A VPC router has one NIC per tier, so a
  4 CPU router fronting 8 tiers goes from 8 vhost threads to 32. Idle queues are cheap, but the
  thread count is not nothing on a dense host.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

How Has This Been Tested?

KVMGuruTest, Tests run: 35, Failures: 0. Six new cases:

• a 4 CPU system VM gets queues='4'
• a 512 CPU one stops at the tap ceiling of 256
• a single CPU one is left alone, so no attribute is emitted
• other details on the VM are preserved
• a queue number already on the VM wins
• a user VM is never touched

The guest side was checked against the driver source the system VM runs, Linux 6.1
drivers/net/virtio_net.c:

	/* Enable multiqueue by default */
	if (num_online_cpus() >= max_queue_pairs)
		vi->curr_queue_pairs = max_queue_pairs;
	else
		vi->curr_queue_pairs = num_online_cpus();

followed by virtnet_set_queues(vi, vi->curr_queue_pairs) at probe, so the queues come up without
ethtool -L. The 256 comes from MAX_TAP_QUEUES in drivers/net/tun.c.