So let's get started. Thanks everyone for coming. Today I'm going to talk about restartable
confidential guests under Kimo. This project that I've been working on for the last few
months. My name is Annie and I'm a software engineer at Red Hat. So let's get started.
So this is the background of what we're going to discuss today. It usually will give you
background of the work, the context, and then we'll talk about the implementation
choices and the challenges. I'll try to give it a more today. A live one if time permits.
And then I'll talk about the current state of the patzer that I've posted already on the
Kimo devil mailing list and the revealing challenges. So the background. So talking
about the different kinds of guests that runs at a Kimo. We have non-confidential virtual
machines. We have confidential virtual machines. And I'm a confidential virtual machines. We have
the SCV VMs. We have ACV ES. We have ACV ES and PVMs. They're all in the technology and for
Intel we have the TDX VMs. But the only ones that are actually restartable are the non-confidential
VMs and the confidential SCV VMs. And none of the others are currently restartable. And the
reason for this is being is that for SCV SMP and SCV ES and the TDX VMs. They encrypt
their state of the CPU racers. And once it is encrypted it's locked in. And when they get
boots the hypervisor is not able to modify the state of those CPU racers. So in order to do
the reset the hypervisor has to reset the state of the racers which since they have
the racers can't the the the guest has to terminate. For ACV VMs they only encrypt their state
the memory. They don't touch the CPU racers. So they are restartable like just any other
novel guests. So a little bit of the context of why this is actually we're doing this.
First of all we want to bring all guests at part with these other both confidential or different
kinds of confidential guests as well as the confidential guest. But there is also another reason for this
which is the work that we have been driving at red art which is the Fooky work and I gave
a presentation for that last year at fourth of them in the vote dev dev room. So you can take a
look at that and there I presented this particular slide and you see that there are two states
of these confidential VMs. The one on the left is actually running with a stock firmware from the
cloud provider. And then one on the right is actually running with a firmware that the end user brings
with them. And ultimately we want to go from this state of the confidential VM to the state of the
confidential VM. So in order to do that there is a reset process. And this at least we can reset the
guest from this state to the state and do all the administration and everything. An encryption
attestation all of it again for this guy. We are not able to implement this. So the main motivation
for me getting involved with this is to implement this reset here so that we can do this.
So in terms of implementation choices. So as I said the the CPU racer states are actually
encrypted. So one option we could do is ask KVM in the kernel to save the pre encrypted states.
The initial state of the memory, the piece table, the CPU racer, everything. And then upon reset
restore that unencrypted state and then do the whole thing again. The problem with this there are two
problems with this one is that what if we wanted to reset to a different launch state. So for example
we could imagine what we were trying to do with this is that here we are changing the firmware.
So our launch state is actually different from what we initially launched with. So that will not
be possible if you are already always resetting with the same launch state. And the second is that
we potentially need a new KVM API which introduces all kinds of complications. And so we
kind of didn't want to get into that. So the other option that we did go for is this one where
what we do is we throw away the initial states of the encrypted state of the confidential gas
and we closed the old KVM file descriptor and we create a new one. And with when we are creating
a new KVM file descriptor for the virtual machine, KVM predictors are the new guests. It is a different
guest. And then with that it does the whole encryption process and the locking in of the
initial launch state. All of it happens again. And then we transition from an unencrypted
encrypted state. And the good thing about this is that we do not need a new KVM API and it's
just transparent to it's just kind of the user does not use any defense and the old KVM process
keeps running. So this is the implementation that we went for. So basically at the end it kind of
looks like this where we have the initial confidential guest that encrypted memory registers
etc. starts running and then it initializes initiates a reboot. And as a part of the reboot
we do a device reset. And at that time we can bring in, we can use the same IDVM or we can
do like we can bring in a different IDVM which is the idea for Fookie or we can use the same
IDVM to state the initials to set the initial state. And then we close the old KVM VM file descriptor
and then we create a new one and we go to the entire sequence again. So it's like basically
initializing the confidential guest all over again with the new VMLD. So what are the challenges
here? So changing KVM VM file descriptor is not being balanced. They're really not done for
obviously for normal guests it's not needed. So it's like pulling the carpet under your feet
and replacing with another. So here we know that all the so this is to understand the challenges
we try to understand the structure of how the different modules work. So you have these accelerators
and then the KVM accelerator implements all architecture diagnostics stuff in in this in KVM
all dot c and then from there it calls into more architecture specific stuff. And then also as a
part of this creates the VCPUs against the VM file descriptors and those VCPU initialization
calls architecture diagnostics stuff as well as architecture specific stuff here and other devices
and other modules and KVM also does the same thing. They have the new all kinds of idol calls
to the VM file descriptor here to do both architecture specific stuff and non-architecture specific
stuff also they do sometimes initialization is the different states of the VCPUs so they call
into the VCPU file descriptor as well. So this is so basically this is the broad picture of how it
looks like like I said there is a architecture non-system then there is architecture specific
stuff that initializes the confidential competing module which does some general initialization
and then it does some architecture specific initializations here depending on what technology
we're using for the confidential cast and then there is also the VCPU file descriptors that are
created against the VM file descriptors and they do all kinds of initialization. So how to make sure
that all these happen again as a part of the reset process because we're changing the VM file descriptor.
So what we do here we create different notifiers one notifier is before the VM file descriptor change
and then the other is after the VM file descriptor change so basically those module that want to
save the states from KVM into KMO before the VM file descriptor changes can use this notifier
and then this notifier is used after the VM file descriptor changes to do all the initializations
as well as pushing in the states from KMO to KVM again if needed not in all cases it is needed.
Migration for example is not supported today so we don't deal with migration cases but there are
other cases where we need to push in the states from the KMO to the KVM so there we can use
this mechanism also since the VCP initialization needs to be done again we need to actually
have this notifier to notify subsystems that VCP will have been changed and you need to do the
VCP initialization stuff again more architected specific as well as architected elastic stuff.
So that's basically what I just described so how do we actually do this in terms of implementing
and figuring out the issues and actually getting to do this stuff so we use stress explicitly
to figure out all the different ioptocals and the code paths from where they're called
and then we need to cover all the code paths we need to understand the code where they're being
called why they're being called what needs to be done again what need not be done again because
sometimes the initializations happens once the data structures and the tables gets initialized like
IQ routing for example they get initialized once within KMO and that remains and now
only thing we need to do is push that information back into KVM once we have the new KVM VM file descriptor
so we need to understand what's really going on and do the stuff that's needed to be done again
some functions for example are not written in a way that can be called again because they're
meant to be done once because confidential gas signalize once and then you know since reset was not
supported they they were not written in a way so that those functions can be called again so
we need to fix those things there were a lot of debug cycles and many of these debugging can
actually happen in non-copen environment because if you look at this slide here this this part
is all confidential specific but this parts can all be exercised even in non-confinition
environments so all of this we can do the debugging of all of this we can do any any any
host we don't need a confidential hardware or a confidential setup to actually do the investigations
so I'm sorry I'm going in the wrong direction
okay so demo I'll give in the end because in case I run out of time I'm not going to take a risk
but we I have already uploaded some demo videos on my top page so those are the confidential
specific demos because I do not have a confidential hardware available right now so the non-copo
part I'll try to if I have time I'll show you it's it's pretty easy so let's let's
discuss the current state of the process it's actually currently we're only addressing
exit six only and we're not talking about arm or other platforms I just posted the version
three of the parts that earlier last week so it's in the mailing list I have a look at that
we have tested this on all these confidential guests as well as non-confinential hosts
with Zen staff as well key muci pipeline is good although key muci pipeline does not
actually there's not enable all caveum specific functional tests because they're all skipped
because dev caveum is not available so the key muci pipeline only proves that there are no
regressions but it doesn't actually exercise any of the new changes that I'm talking about
but maybe we'll fix this maybe maybe we'll enable the caveum tests in the c i and then we can
we can run this as well the test scripts are public here so we can take a look at those mostly
the delivery scripts and some stress scripts etc there is a functional test also has a part
of the V3 of the my padset and that is what I'm kind of going to show you if I have time I just
practice it into a functional test with Zen and normal cases it's kind of nice to be able to
actually exercise the whole thing on your on your laptop or your desktop or whatever if
if you don't have a confidential host currently we're targeting chemo version 11 if everything
gets done well and fast but if not then it will be 11.1 so whatever the remaining challenges
we need people to actually review our our our padset we need people who have specific knowledge
in subsystems like Zen or other devices to look at the changes and let us know if there is
anything that need to be done or not done because there are a lot of subsystems that we touch
which I do not know that all of it I just try to make the best judgment and so we need eyes on those
things and in general look at the padset and see if there are any issues with it so in terms of
focus going back to the focus question what else remains so let's let's go back to that picture
that I had before this one so first of all if we are talking about IGVM then IGVM needs to work with
this reset process and that so IGVM support is already in chemo today but IGVM does not currently
work with the reset in place so for that we are already doing some work and good has posted some
patches on the mailing list so please have a look at this padset and we need to make sure that
this actually works with the reset in place and then we need to have a mechanism for focus specifically
we need to have a mechanism for end users to be able to bring in their own IGVM implementation
or the some way to actually push the IGVM bundle back to the to the hypervisor so that the new IGVM
can set a new initial state of the launch state of the VM and then reset into that so so that work
is yet to be done and I posted an old version of it here but it's not relevant anymore
I thank very much for follow for guidance throughout the process of this work and God as well
God is also deeply involved with the fuki now so he's also doing a lot of the IGVM set of the work
the chemo community in general the reviewers so I've already reviewed my padset and I tagged them and
we I welcome other reviewers and comments on my patches and red hat for supporting me with this work
and giving me all the resources to be able to continue this work as part of fuki so now I'm just
going to quickly see if I can give demo here okay let's see anyway so this is too big actually
but you can see this is my script what it actually does is it creates a chemo VM and what I'm going
to emphasize is this debug flag right here which enables the VM file descriptor change on resets
and rest of it is you have I use the chemo accelerator and everything and we use a uky and we use
dash kernel mechanism in the chemo this uky packages a kernel and a basket as in it so we'll see
what happens when I kind of run this so let's run this so this is non-confidential case because this
is not a confidential hardware so it will just do the VM file descriptor flip and if we do a reset so
another thing that I actually forgot to mention here is that we need to monitor this guy here
so I'm monitoring this chemo dot log and it should speed out a name function that's
it's called in the reset so let's do this
all right so let's resetting if you go back and hopefully
there is
right so there is there is this log that says virtual fine accelerator file that's with the
Tvm file descriptor has changed and hopefully there is this there is a Tvm reset VMFD is
actually the the function call that I enabled on the tracing so you see this on the tracing log here
so that's it and that's it for me do you have any questions do we have time for questions
a couple minutes couple minutes yeah so yes yes
so basically the idea was okay so the question was that
what did I do for testing whether I had any specific functional test in mind so
the idea was to test all the modules that are actually so all the devices and all the modules
where we are using this extra initialization after the VM file descriptor picture so
the the the intent is to use those devices and those configurations in the chemo and then
try and do a reset so that to make sure that all that code is actually being exercised
through the notifies and everything so I added trace point so basically you can you can
enable tracing on those functions and do a reset and see out okay those functions are getting
called everything is fine so any other questions yes yes yeah yeah yeah yeah yeah yeah
this is just for timing or is it the time to move and in general in the tech issue whether yes
this is in the state where you need this change so okay so the question is the
device that I used in my demo whether it is actually needed in the final state or not so
so normal for normal gets non-confinancial guests you don't need this right for
confidential guests it is it happens automatically because that's that's our solution right so
so if you you want to see if actually this is getting called you can you can actually check the
log that I just show or you can enable tracing to see that those functions are actually
getting called or not so the debuts like that I introduced was only for non-coco case
because that's the only way to force a VMFD change but also make sure that in the normal case
when you are running in production environment that's not used it's only used for your debugging
and testing purpose