OK, do you hear me, right?
OK, so first of all, thank you very much for being here.
I'm Danieli Mingol, I'm today.
I'm going to talk about one AI, an open source framework
for managing AI models at scale.
A bit of introduction about myself.
I'm Danieli. I work as a software developer for AI
in a top enabled system. I work in the innovation team,
so we try to implement good things into the product.
Regarding my background, I have a machine in the science,
measuring computer science, and I've been working over the last few years
on different things in different industries,
mobile gaming, logistics, delivery food,
machine learning system, maybe test,
a lot of experimentation that engineering things.
Regarding my company, we are the first open source infrastructure
as a service solution. We are more than 15 years of experience
in building enterprise infrastructure software.
We have a bunch of official and Europe,
and thanks to open enabled,
there are more than 5,000 clouds worldwide.
When I talk about managing AI at scale,
I talk about these scales.
The Guardian, my speech,
it will be mainly divided in six parts.
A first part, where it's all about the product,
open enabled, and the concept of AI factory,
because this is related to the framework that we are currently working on.
Then a second part about one AI,
why we start building this framework?
What problem we are trying to solve,
and if you are able to solve these or not,
then the architecture,
I will give you an envelope review of the architecture of our framework,
and the three pillars of the three pillars of our framework is based on
that our marketplace, data,
sorry, inference API.
And then at the end, I will show you,
from theory to fact, so I will show you the command
and what are the expected, how to put.
I will not be able to show you a demo.
I also had a demo, but because my laptop is not working fine,
but yes, it will be clear the same.
So regarding OpenNable, OpenNable is an open cloud edge computing platform.
We act as a virtual infrastructure manager,
so this means that we enable you to run and orchestrate workflow
that are related to your machine and so on.
So it's simple, it's light, it does not require a Navy stock,
it's sensible in the sense that you can import in your cloud
different application and enhance its capabilities.
It's multi-tenancy and multi-vehams out of the box,
so you don't need to do anything regarding this.
And if those of you know that are using Kubernetes,
we also have an appliance in our marketplace.
It's called one key, you can download it,
and it will enable you to deploy openNable,
sorry, Kubernetes cluster on top of openNable.
Now, regarding AI factory and why this concept is important.
Imagine that you have GPU, you have network,
you have storage, you have CPU and so on.
So you have the hardware, right?
And you want to understand how to easily share these resources
with all machine and so on such that you can create,
isolated environment that in this slide are called tenant,
where a person or a team not can do their own things.
Developing AI models, running other service and so on,
without interfering in each other.
So we, in terms of openNable, we are in the middle there
between your hardware and the visualization part on top.
And we are helping you do in this.
So one AI, the framework, it's related to enable you
not to share your resources with different virtual machine
in an efficient way.
So regarding the problem, no, when why we start developing this framework?
No, which problem we are trying to solve.
First of all, we started noticing that there was the first problem
related to model discovery.
Okay, I want to train my model, I want to train my models,
but where I put them, no.
Is there like a sort of collection where I can find all my models,
where I can easily know, put whatever I built there.
Then the storage, okay, I find you in the model,
I train the model, but this model are quite heavy, right?
So we talk about more than one under GB sometimes.
So I cannot download them every time, and then it's sensitive,
and then exposing this model and so on.
It's not efficient, right? A lot of bar.
Regarding the deployment, the question is, okay, I have the GPU,
but then I can leverage this GPU in order to reduce the inference
time, the fine tuning, the training time, sorry, and so on.
The first problem is this one.
I have clients that are already developing.
So I have a code base, for example, I deploy my model,
and then what? Do I need to change my code base,
such that it can communicate now with the new model deploy?
That's why we start working on one AI.
One AI is an open software to run AI models and inference on open nebula.
And it's based mainly on three pillars.
The argument face art marketplace that is a lightweight catalog,
metadata first. What this means is that we already have
this concept of collection of models on argument face rights.
So what you can do, you can easily import this public or
private collection of models from hugging face to your open nebula marketplace.
But it's metadata first because you don't download the model.
You only download the metadata.
The model then will be materialized only when you want to deploy.
It would just once and then you can share the images.
Thanks to the shared file system data store with a lot of VMs.
You download the model once and you reuse these model files a lot of times
and you can share it with all the VMs that you want.
Now that we have the model stored in our shared data store,
it's time to expose it, right?
And you can easily do that because in open nebula we have this apply on BLLM.
That what it does in summary will mount the model into the VMs
and then we'll expose it through an open AI API.
So this means that you don't need to rewrite your clients and so on.
Because if you already use open AI and tropic models on,
you just need to change the URL, then the point and that's it.
The level architecture is this.
The user execute a command that is one AI run.
Later I will show you some of the argument that you can pass to this command.
What this command will do, it will generate a virtual machine template
that is at the description of the virtual machine.
Then the name of the virtual machine, the amount of memory,
the number of CPU and so on, the network etc etc.
Open nebula will read this virtual machine template
and will instantiate the virtual machine.
What I mean when I say instantiate the virtual machine.
In this case, it will instantiate the virtual machine with the VLM image.
Then it will attach all the disk that will contain the model files and so on.
And it will also attach the PCI devices to the virtual machine.
So the GPU and then of course, it will expose the service.
With one command line at the end, you get this.
Now in the picture below, you get an endpoint.
And then you can communicate with your model.
An important thing that is that no extra case run service is needed.
You just run the CLI.
It talks with open nebula and open nebula will schedule the VM to the right host.
In this case, there are all the data, the GPU availability and also the GPU that you want.
Now, regarding the first pillars before I was talking about this hugging phase marketplace.
There is a picture there that show you what this marketplace in open nebula contains.
This market will contain all the models fetched from your public or private collection in hugging phase.
Just the metadata.
So as you can see, these are only metadata.
Very, very easy, just text.
So it's very fetching is also quite rapid.
What happens is that then in one eye, you pass the name of the model that you want to deploy.
And what it will happen is that it will search for this model in the open nebula marketplace.
It will import it into a shared file system data store.
And if it's already present, of course, because if you are ready to load it,
it will just reuse the image.
Materials lies on deploy.
Artifacts are copied only when you deploy.
So you download once, the files, and you reuse multiple times.
When you download the model, it is imported in a shared file system data store.
That in summary, what is this data store that allows you to share what it contains with different virtual machines.
So in the picture is very clear.
You download the things one, download the images once, and then you can reuse multiple times.
This is also cool because a lot of research lab, a lot of company already have their own high-performance storage,
so you can also use them.
The benefit, of course, is that you don't need to download the things every time and every time,
because we are talking about very heavy model.
And then, of course, it also have been reducing the storage cost, right?
The third pillar is the inference, no?
And the inference part, the lattice part, right?
So one eye generates a template.
The template is as a description of the virtual machine.
How it should be insensitive.
What will happen is that openable a create the virtual machine, assign the GPU, and start it.
The inference engine, what we will do is just to add the model from the disk.
Of course, you can also pass different configurations, so the maximum number of tokens in the output,
the temperature of the model in order to manage its creativity and so on.
Of course, these configs are passed as a context variable.
And then, at the end, you get an end point that is reasonable.
You know, if your codebase is already like talking with all the models that we know,
because this is a standard way of communicating with the model.
Okay, I talk a lot.
This is just a practical example of a common that you can run.
One eye run, you specify the model, the model will be imported if it's not present.
Then you can specify the name of the ID or the inference engine.
Right now, we only support the LLM, but of course, in the future, we are planning to support more inference engine.
You also can pass the name of the network to associate to the VM, the number of CPU, the amount of memory, the number of GPU,
the number of end data and the GPU type.
What will happen here is that openable will do all the rest.
So, in fact, you don't need any external customers, scheduler, whatever, because you pass the GPU count and the type.
And the openable will automatically find the host that fulfill these requirements.
Okay, regarding the GPU, that is also cool because before now, the presentation was about me,
that we still don't support, but we are working on it, is that, of course, the VM can see the GPU right on the host.
Because we pass it through the PCI pass through.
So, the LLM can also, of course, leverage the GPU.
Of course, if no GPU is present, it will use the CPU.
Now, in theory here, I should show you like the demo.
I don't know if maybe is there a way, but overall in the demo, what I was planning to show you was this.
The demo was like a six minute demo divided in two parts, where I showed you how currently you can deploy a model together,
not with the inference engine, expose a model using the common open API API in openable arena.
You can already do it, right?
But it's a manual process.
You need to import the marketplace, you need to import the model from the marketplace into the shared data store,
you need to go into the menu, check the virtual machine templates, all the list, find the inference engine,
then you need to instantiate from the inference engine template, virtual machine templates,
you need to instantiate the VM, so you need to assign a name, amount of memory, and so on, all these things,
you need to attach the disk that contains the model files and so on.
It can be done, but it's very manual, right?
But thanks to the common line that we are currently working on, still in the element,
I want to online this, you can do it almost like automatically.
So in summary, one AI is a CLI that lets you discover a model from adding phase.
It's important to say that in our interface, you can have also private collections, okay?
So if you are a company that is developing and tuning training model, whatever,
you can also have your private collection and import it into openable.
You can store the models into a shared data store, so the load once and reduce multiple times,
deploy an inference engine with that model and expose an open, an open-eye compatible API, okay?
Just a list of the main benefits in common, same results.
So this means that if you are into the MCP,
wall, MCP2, MCP servers, and if you are into automation workflow and so,
you can easily use this and put this like in your automation workflow, this common line.
Your control, so this means your model, your infra openable is an open source cloud platform.
So, not a keen, whatever, and all these sort of things.
Reuse storage, one copy of an AI model serves many VMs.
You reduce the storage cost.
And then of course, as I said, in the first slide, openable will,
out of the box will manage all the solution, the resource across the cluster, okay?
So you don't need any other scale, Lorenzo.
One AI, and the sheriff has been sent out, so our car rental in development,
and we'll be available in a future openable release.
So, well, can I, I, no, I know, but I have the debo, so maybe I can,
we can just open for the, because the demo, I can not show it because it's not working.
Yeah, yeah, maybe open for questions, such that.
Thank you very much.
Thank you.
Oh, maybe.
Hello, thank you for our talk.
My question is about the shell effect.
It's based on, um, it's based on some kind of, uh, me, you offset.
Yes.
It's based on virtual effects, yes.
So what happens is that you have, uh, all the file, all the model of the files in your share
file system data store, what happens is that when the virtual machine that contains the
inference engine will mount the disk through virtual effects.
So you're going to get, like, the VM can, no, can get visibility on what is story in the
share, if I, if I, if I, if I, if I, if I, if I, if I, if I, if I, if I, if I answer your
question, yeah, okay.
So I actually also have a question.
So I know that a lot of this type of clusters are kind of legacy and they don't really have
GPU, so they have very underpowered GPUs.
Uh, do you have any way of deploying inference engines other than VLLM?
You know, that would actually be more CPU friendly, like LMSEPP or anything like that.
Very good question.
Thank you very much.
Right now, no, in the sense that we only have the LLM.
Of course, in the future, we are planning to, to implement some other inference
engines that are more CPU prone, but right now, no, but of course, this is like an
issue, right?
Because as you say, not all the people have, not all the component, all the people,
well, they don't have the GPU, so we should enable them also to run models,
maybe not, you know, the top models, but something, you know, that can still
bring value.
Anybody else?
Uh, I just wanted to ask where the GPU is located physically?
Where?
For OpenR, inabula, where's the GPU is located physically?
Well, the GPU is located on the other, in the sense that OpenNembola is a cloud platform
that you can install, no, in the meta, also, we support almost fully providers.
And it will enable you to, it's sensitive, you have to machine orchestrate, we're from
machine and so on.
So, the GPU is like, it's not in OpenNembola servers.
It will be on the else, customers are lost, they install OpenNembola on top, and what
we do in terms of OpenNembola and be sure that they've used all the machines that are
running on OpenNembola, okay?
Can access to the GPU in order not to reduce the inference, inference, time,
functioning, I mean, so.
Okay, this is all I have got.
Thank you.
Come next here, come next here.