[GSoC] Dynamic attachment and removal of filesystem volumes

[alicefr]

Description

Direct directory sharing between a virtual machine and the host is made possible by filesystem devices. Pods and virtual machines can share the same Persistent Volume Claim thanks to this virtiofs. For instance, the capability to hotplugging an extra directory might be utilized to obtain diagnostic information from the VM and thereafter examine it.

While attaching a disk to a running virtual machine dynamically is standard for VMs, it is an uncommon operation for pods.
KubeVirt has already implemented the capability for hotplugging disks and LUNs, but it does not yet have the functionality to add or remove filesystems from a virtual machine. The volume hotplug/unplug feature isn’t supported natively by Kubernetes. KubeVirt mechanism relies on an additional pod known as the attachment pod to schedule and attach storage to the specific node where the VM is operating. Then, the storage is hotplug through Libvirt api.

In addition, virtiofs is deployed in a separate container which is usually started together with the pod. However, in the case of hotplug, the pod cannot be dynamically modified with extra containers. This is a further challenge which needs to be taken into account during the design proposal

Expected outcome

The project goal is to propose and develop a solution based on the current approach of the attachment pod which supports filesystem volume types.

Project requirements

Project size: 350 hours
Difficult: Hard
Required skills: Kubernetes knowledge and GoLang programming skills
Mentors: Alice Frosi: [email protected], German Maglione: [email protected], Javier Cano Cano [email protected]

How to start

1. Install KubeVirt and deploy KubeVirt VMs following the getting started guide
2. [Optional] Look for good-first issues and try to solve one to get familiar with the project
3. Try to attach and detach a volume in kubevirt with virtctl add/removevolume with and without the –persistent flag (see kubevirt documentation for hotplugged volumes)
4. Try to understand how the mechanism for the attachment pod works and how the request for adding and removing volume API work
5. Try to use a filesystem volume and understand the mechanics behind it
6. Try to install the guest agent into the guest and issue a command with virsh using

$ kubectl exec -ti  virt-launcher-vm-rjfmh -- bash
bash-5.1$ virsh list
 Id   Name         State
----------------------------
 1    default_vm   running

bash-5.1$ virsh qemu-agent-command default_vm '{"execute": "guest-exec", "arguments": { "path": "ls", "arg": [ "/" ], "capture-output": true }}' --pretty
{
  "return": {
    "pid": 912
  }
}

bash-5.1$ virsh qemu-agent-command default_vm  '{"execute": "guest-exec-status", "arguments": { "pid": 912 }}' --pretty               
{
  "return": {
    "exitcode": 0,
    "out-data": "YmluCmJvb3QKZGV2CmV0Ywpob21lCmxpYgpsaWI2NAptZWRpYQptbnQKb3B0CnByb2MKcm9vdApydW4Kc2JpbgpzcnYKc3lzCnRtcAp1c3IKdmFyCg==",
    "exited": true
  }
}

How to submit the proposal

The preferred way is to create a google doc and share it with the mentors (slack or email work). If for any reason, google doc doesn't work for you, please share your proposal by email. Early submissions have higher chances as they will be reviewed on multiple iterations and can be further improved.

What the proposal should contain

The design and your strategy for solving the challenge should be concisely explained in the proposal. Which components you anticipate touching and an example of an API are good starting points. The updates or APIs are merely a draft of what the candidate hopes to expand and change rather than being final. The details and possible issues can be discussed during the project with the mentors that can help to refine the proposal.

It is not necessary to provide an introduction to Kubernetes or KubeVirt; instead, candidates should demonstrate their familiarity with KubeVirt by describing in detail how they intend to approach the task.

Mentors may find it helpful to have a schematic drawing of the flows and examples to better grasp the solution. They will select a couple of good proposals at the end of the selection period and this will be followed by an interview with the candidate.

The proposal can have a free form or you can get inspired by the KubeVirt design proposals and template. However, it should contain a draft schedule of the project phases with some planned extra time to overcome eventual difficulties.

[alicefr]

/cc @aburdenthehand @germag @jcanocan

[Ashlok2003]

Would it be correct to say that the goal of this project is to extend KubeVirt’s attachment pod approach to enable the dynamic attachment and detachment of filesystem-based Persistent Volumes to running virtual machines ?

[alicefr]

Would it be correct to say that the goal of this project is to extend KubeVirt’s attachment pod approach to enable the dynamic attachment and detachment of filesystem-based Persistent Volumes to running virtual machines ?

Yes, right now you cannot attach filesystems volume while the VM is running. Only at start-up

[Ashlok2003]

Sorry for replying late! 😊 It took me a while to understand how things work in KubeVirt.

Before proposing a solution, I want to confirm my understanding of how block device hotplugging currently works in KubeVirt.

Current Understanding of Block Device Hotplugging in KubeVirt

---

1. A user requests a volume hotplug operation via the Kubernetes API.
2. The KubeVirt controller processes the request and spawns an attachment pod.
3. The attachment pod mounts the Persistent Volume Claim (PVC) as a block device.
4. The attachment pod attaches the block storage to the Virtual Machine (VM) pod.
5. The Virtual Machine detects the newly attached block device and makes it available for use.

How Are libvirt and QEMU Involved in This Process?

---

---

• libvirt: KubeVirt interacts with libvirt to manage virtual machines. When a new block device is attached, libvirt updates the VM's XML definition and issues commands to dynamically attach the device.
• QEMU: QEMU, as the underlying hypervisor, executes the hotplug operation and integrates the new storage device into the running VM without requiring a restart.

Questions Regarding the Attachment Pod

1. Is the current attachment pod in KubeVirt capable of running a Virtio-FS daemon to enable filesystem-based volume hotplugging?
2. Can the attachment pod handle the workload of managing the daemon and facilitating communication between libvirt, QEMU, and the VM for proper filesystem hotplugging?
3. If not, what modifications or enhancements would be required to support this functionality?
4. Does my understanding of block device hotplugging in KubeVirt align with the actual process, or am I missing any key aspects?

Based on my understanding, the attachment pod can be upgraded or extended to incorporate the Virtio-FS daemon, enabling proper filesystem hotplugging within KubeVirt.

To address this, please find my proposed solution outlining the necessary enhancements and implementation details below.

[alicefr]

@Ashlok2003 please don't publish your design publicly. Please, follow the way it was suggested in the issue description either gdoc with the mentor or via a private email.

[Ashlok2003]

@alicefr Sorry, my bad. I deleted the proposal. I will email it to the mentor.

[alicefr]

Sorry for replying late! 😊 It took me a while to understand how things work in KubeVirt.

Before proposing a solution, I want to confirm my understanding of how block device hotplugging currently works in KubeVirt.

Current Understanding of Block Device Hotplugging in KubeVirt

1. A user requests a volume hotplug operation via the Kubernetes API.
2. The KubeVirt controller processes the request and spawns an attachment pod.
3. The attachment pod mounts the Persistent Volume Claim (PVC) as a block device.
4. The attachment pod attaches the block storage to the Virtual Machine (VM) pod.
5. The Virtual Machine detects the newly attached block device and makes it available for use.

How Are libvirt and QEMU Involved in This Process?

• libvirt: KubeVirt interacts with libvirt to manage virtual machines. When a new block device is attached, libvirt updates the VM's XML definition and issues commands to dynamically attach the device.
• QEMU: QEMU, as the underlying hypervisor, executes the hotplug operation and integrates the new storage device into the running VM without requiring a restart.

The schema looks great! It seems you have understood the flow correctly.

Questions Regarding the Attachment Pod

1. Is the current attachment pod in KubeVirt capable of running a Virtio-FS daemon to enable filesystem-based volume hotplugging?

Currently not, but it could be possible to extend it.
This is an option. One problem although is that currently we use a single attachment pod for multiple volumes which is restarted every time the hotplug volume set changes (for addition or removal). This isn't a problem for normal disk since the process running the container isn't doing anything special, but it is simply a placeholder.

Although, if you replace that with virtiofs, it means that you will restart virtiofs and this is a problem since QEMU and virtiofs really don't support well the restart of virtiofs. @germag please keep me honest here.

You might also want to explore ephemeral containers. It will be great if you could include in the proposal the evaluation of one solution over the other.

2. Can the attachment pod handle the workload of managing the daemon and facilitating communication between libvirt, QEMU, and the VM for proper filesystem hotplugging?

Possibly, keep in mind that virtiofs and QEMU communicate via socket, so QEMU will require to have access to it.

3. If not, what modifications or enhancements would be required to support this functionality?

Today, virtiofs is started in a separate container but INSIDE the virt-launcher pod. You could move this to the attachment pod, however, QEMU needs to be able to communicate with virtiofs via a UNIX socket.

6. Does my understanding of block device hotplugging in KubeVirt align with the actual process, or am I missing any key aspects?> > Based on my understanding, the attachment pod can be upgraded or extended to incorporate the Virtio-FS daemon, enabling proper filesystem hotplugging within KubeVirt.

It is one possible approach, yes. I already added a note about the ephemeral containers. This could be an option to evaluate as well.

[alicefr]

@alicefr Sorry, my bad. I deleted the proposal. I will email it to the mentor.

@Ashlok2003 One additional thing. Officially, the projects haven't been announced yet. We should have good chances to be taken, but until the 27th of February this is still uncertain.

[Ashlok2003]

@alicefr Ok Got it !