Red Hat for the OpenShift and Kubernetes ecosystem platforms.
Today we're going to discuss how we improved the performance of Lama CPP, running on containers
on macOS.
This work was led and implemented by Kevin Pujé, and fortunately, Kevin was
able to come to date, so you guys are stuck with me.
But Kevin is on the matrix channel to answer all the top questions.
So let's get started.
As you guys know, it's not possible to run containers natively on macOS.
The containers are some have a Linux technology, and in order to run containers on Mac OS,
we need a virtualization layer.
So both podman and Docker, and more recently, I think last year, Apple released the new
Apple container framework, and they all use virtual machines to run the containers on top
of them.
I think Docker and podman, they have this one virtual machine to run all the containers
and Apple follows some have a slightly different approach where they have one micro
virtual machine per each of the containers.
But the truth is, we need a think virtualization layer in macOS in order to be able
to run the containers.
So in the case of podman and podman desktop, we need a backend for run this virtual
machines.
For some time now, I think this became defaulting like in podman 5.6, like last summer,
the backend that we use is Libcare Run.
Libcare Run is a dynamic library that allows us to run virtual machines in user space.
Very fast to create these virtual machines is very focused for this use case of running containers
on top of virtual machines.
Libcare Run leverages the native, I provide some framework of each of the platform.
So if you use Libcare Run Linux, you get a KVM acceleration.
And if you run Libcare Run on macOS, you get a Apple hypervisor utilization framework.
Libcare Run is also relevant in other context as well for the story of micro virtual machines
in Linux.
Not only it is a very important piece of the confidential computing history, it has
support for this trusted platform extensions for Intel and AMD.
So very nice tool, and very relevant for this talk, Libcare Run was the first backend
that supported GPUs on macOS, and we are going to discuss how this was possible now.
I think it is important for us to understand why we would like to run AI models on top
of containers.
But containers nowadays are the default for packaging, distributing, and managing applications.
So that is why we want containers for running models.
It also makes much easier to transition from local development to for instance Kubernetes,
clusters or production environments.
So I think the justification is clear.
There are a couple of tools that make running containers and models on top of containers very
convenient.
One of them is the podnone AI lab, it is a catalog of models and they are easy to leverage
to make quick proof of concepts and eventually transition to Kubernetes and open shift
another production environments.
There is also the RamaLama project for people that familiar with Docker and podnone is
very convenient, because we have exactly the same API.
The same Docker commands that we are familiar with, Docker podnone, Docker podnone,
Docker podnone, and we can use them to run models with exactly the same syntaxes.
It also supports this bench subcommand which is very convenient for this specific testing
that we conducted during this work.
So what about the GPU acceleration?
How this works when we have this visualization layer in the middle of the containers and
the host?
So in Linux it is very easy, we just expose the depth, the corresponding depth, files to the container
and we automatically get this GPU acceleration within the container.
The issue with virtual GPU acceleration is that we have the virtual machine in the middle.
There are different strategies that we can follow here.
First strategy would be full device password with the GPU, the whole GPU to the virtual
machine.
Unfortunately, this is not possible in Mac, we only have one GPU and the GPU is being used
to render the screen, so that's not an option here.
Hardware devices have a system device password.
So there are enterprise GPUs, Nokia, and the enterprise GPUs that support like
it's lies in the GPU, so you can deliver as small as lies of the GPU to the virtual machine.
The Apple GPUs that doesn't support this kind of hardware assistance.
Virtual device emulation is out of question for hyperforman inference workloads, so we are
left with a parameter to analyze the device option, and this is the option that leave
the run currently implemented.
So in order to make sure that we have a good performance when running containers with
leave the run, there's this virtual driver called the GPU, and there's a collaboration
between the guest kernel in the virtual machine and the host running the supervisor.
In the right side of the diagram, you can see the virtual machine space.
In the userland, we have Lama CPP.
We have the kernel of the guest and the virtual GPU driver, and in the left side of the right
on the left side of the screen, we can see the host side, so we have the partner next
to publication, manage in the containers, leave the run as the lie type advisor lay you're
running a small virtual machine, and the real renderer that makes the translation between
the GPU calls from the guest and the host.
This is a little bit more detailed, we are using Vulcan as the GPU API, and we have like
this layer that translates between Vulcan and the native FP, framework which is called
APL, APL, metal framework.
So we have this little translation layer in the middle to be able to reach the GPU.
And this is the current state of the art, this is what Kevin found itself like six months
ago when he started with his project.
And let's take a look at the numbers.
So first of all, on an optimistic note, the inference performance on containers on Mac OS
has been growing, increasing and proving over time.
So in 2024, unfortunately, we didn't have this CPU support from Limke Ranson, the only
driver available was the CPU one, and as you can see, we were able to generate like
half token per second.
Early last year, there was this compute framework for GML, GMLS, the token is the kernel
back in the giant for Lama CPP.
And with that, we were able to somehow really improve the performance of inference, and
recently, with a new Vulcan native backend, we were able to reach like 40 times faster
results that with just CPU.
So this is an optimistic evolution of running inference workloads on Lama CPP on containers.
However, there is room for improvement.
In this graph, we can see in the first column, we can see the performance of native Lama
CPP Ranson with FAPIL-METAL framework.
So this is somehow what we would like to achieve.
This is the ideal performance, the maximum performance that we can get.
And in the second, in the third column, we can see the performance using the Vulcan tensor
backend.
As you can see, there's no much difference between running this natively on the Mac OS
as a process or running the solution inside of it or machine.
So the performance between the second and the third column are almost identical.
What we can see here is the virtualization was not the bottleneck.
We suspect the bottleneck may be related to this translation between Vulcan and April
framework instructions.
And there is a new meta implementation that probably could improve these results at the
time, at this effort, started the solution wasn't like finalized, so we don't have
numbers for this, but I understand it sounds promising as well.
So this is the gap that we tried to cover in this work.
So this was the state of the art before this work started.
So how we can close the gap between this Vulcan virtual machine solution and the native
framework?
One first of your solutions, so let's use the meta API framework.
This is the most performant one in Mac OS, right?
Why don't we use it?
Because of several reasons, first of all, Linux doesn't have the drivers for April, for
April GPU.
It's very specific to Mac OS.
I'm not sure, but maybe there could be some legal aspects to supporting them and implement
Windows in the kernel.
So that didn't look like a reasonable solution.
So the approach that was found was to use API remoteing to use the native metal tensor
back end from La Masipiti.
So the host uses the native metal back end and the calls are triggered from La Masipiti.
This is somehow the solution that was found.
So as you can see, now the application itself La Masipiti can send commands and instructions
directly to the back end, which is in the host side, translating these instructions directly
to the Apple Metal framework.
So somehow we avoided the use of the Vulcan tensor back end and we made some how the application
which is La Masipiti aware of the incapable of running these commands on the host side.
So the idea is that we intercept the API calls in the guest.
We further the parameters to the host, the host runs the function in the GPU and retams
the data back to the guest.
It's the same strategy used for the Vulcan tensor communication, but we move the strategy
one layer up to the application to La Masipiti.
This is not a generic solution.
This is a specific to the La Masipiti use case.
The Vulcan solution was generic.
We'll see the result now, but it's not very high.
We pay the light price for the for the for the for the hyper calls latency and the synchronization
between the guest and the host.
But the price is not so high because we use parameterization, the host share memory pages
to avoid the double copying issue and sharing data between the guest and the host.
Here we can take a look at the results, the first column, this is by the way packet
processing, processing, inference workload.
So the blue column represents the native performance, this is what we would like to achieve.
And the red column represents the performance of the new solution using this API remote
strategy.
The third column is the the the classic solution using the the Vulcan framework.
So as you can see with the API remote strategy and for this very particular case of La Masipiti
inference, we were able to achieve almost identical performance as the native one.
We run this results in different hardware, so this is a token generation, a token generation
workload, the first graph represents different hardware from M1 to M4.
The second graph here in the right represents different models.
And finally the third graph in the bottom represents different model sizes.
So as you can see in general the the the performance is very good and very reasonable
very close to the native one.
Obviously there's there's a slight performance overhead, but is is very very acceptable
and within limits.
We observe that this overhead is greater in smaller models, which makes sense because
the overhead of the of the hypercolt is bigger, but in general in general the the performance
is really close to the to the to the native performance considering this is running in a container
within a virtual machine, I think the performance is is amazing actually.
So these the main work from an implementation side to happening in a couple of projects,
the biggest code changes are in Lama CPP to implement this new
attention interface backend.
This was merged earlier this week very timely for the confidence and there are also some changes
required in the real GL renderer for the for the API remote in communication that we are waiting
for for some reviews on on that side.
So there's this article in in the developer fret hat blog which explains this in more detail and
with more technical references the work is is available to try.
So we can try this in in macOS we can try this in Linux with with K run and we are we are working
on enable this this workload on Windows as well and we expect as the the remaining PR get
met to have this implemented in postman desktop through this year probably in the next quarter.
Some security considerations so as I said they provide the ransom on user space the virtual
machine manager so with user privileges and the backend and the GPU on the execute calls that
comes from Lama CPP different from the Vulcan model work where we have like more possibilities
of running and trusted code in the in the GPU of the host.
But there are currently limitations and then they are very related to the multi-tenancy aspect.
So the the backend library leaves in the in the advisor addresses space so
a crash within the backend library that translates the the calls to the GPU between the
guests and the host could bring down the advisor and all the virtual machines that run on the
advisor and also the backend of multiple container running the same process.
So we don't have like per user GPU isolation so we don't consider business sign like multi-tenancy
for production ready I think it's great to run locally workloads on macOS but it's not something
that could be extrapolated to production or or other scenarios.
We also have a nightly CI where we run all these scenarios.
We compare native to Vulcan to to IPI remote performance in a daily basis.
We were able to to find some regressions on Rama Lama or Lama CPP so so the the CI has proven
very useful and we plan to extend it and to maintain it with with other use cases.
And I think that's all from our side there are some extra slides that I will upload with some
cuts snippets some code commands that you can use to replicate the experiment on your local machines.
And this this is also the email of of Kevin and my email review guys have any further
questions thanks a lot for your attention and Kevin is happy to take any questions on matrix.
Thank you very much.
Okay so can everyone hear me? Okay so one good question that was asked is this is an amazing work
which I don't think is a question just a statement which by the way I absolutely agree with.
The question is can some of the same techniques be used on VLOM sort of running on macOS and containers?
Probably yes I I think one of the limitation of this work was very specific to Lama CPP so very targeted
and the downside of Kevin and application level approach is that in order to support other
applications you will need to implement this this work on other on any of LLM or other frameworks as well.
So this is the downside of not having like a generic solution like the Vulcan one very targeted
performance goal here with Lama CPP.
Yeah and at what's pointed out VLLM is largely based on PyTorch-like approaches and again then you would
have to have you know to have to have that back end so you know that's kind of goes without saying.
So no other questions so again thank you so much.