Hello everyone, so my name is Samuel Deso, I am a founder of Aerithics and I work on monitoring
on observability and I will present today is a summary of an experience I had last
year. I had to implement LLN on a special environment. On a stage last year, it was a
period of 25. I was not an expert on it, so I had to challenge to implement monitoring
observability on a special environment and I have seen various tools. The black box for me
can I do it or can I push it in or can I implement monitoring observability. So I had work
many many times when I had success already. So what I will present today is a summary of
my experience on how to implement observability for II or work run on an HPC. Because,
the fact when you GPU utilization show 95 percent, it's a good fire, it's so wonderful,
but at the same time when you are 75, I will training a job, it's a thing, you can add a question,
what is the problem? The problem, as I said, when you are 95 percent of a GPU utilization,
you can say, well, it looks great, but is it right? I'm not sure. Because, in fact, when we
talk about a GPU utilization, it is a CPU, a lot of the average of AI workloads, because a GPU
when it is at one of your person, it could be spining on a poorly optimized can,
and it can wait in or memory or it can be blocked by the IO. So, actually, it's not the same thing
as a polyclisting. So, if you are going to sum up the thing, you're not monitoring AI workloads,
you're monitoring hardware, it's not the same thing. So, what are the three blind spots,
or what you are missing in this case? So, the solution, it is a free layer, I have built
in this experience, so it is a course layer collection, which can reveal this, I don't plan,
so it is free layer. The first one is in the first sector. I have to check the health of my hardware,
so it is a GPU utilization, which I run for the power of the temperature, and so on.
The second layer, it is most workload. I have to evaluate the training efficiency by the
front board, if I have a ratio of IO and put out the run ratio on the zone, on the sub layer,
it is the model F. So, I can evaluate the progress of my running, in terms of first,
in terms of gradients, in terms of covariance, on the what we are doing in your,
so I will try to present my demo after of the lab as a criterion.
On when I have worked on this project, I am understood,
especially it is not for additionality. The specificity I have to think of it, the source one is the
scale of a cardinality. The second one is the long run engine, because simulation run for
day and four week on an intuition can prove a massive loss in time, on energy or on money.
On the way over run challenge, it is known that it is introduced to the instance,
especially, because it is a typical cycle for money to run, it is to run from compute.
So, the question is, what do I actually monitor? We can't correlate, but we can't aggregate.
I have put an example of a correlation issue, which we have to see after, or come with
that in draw from it, so I have a draw by G or the node, a GPU matrix, let's try it.
But you have to think on a three question to evaluate of the root cause.
When the training, you have training, which is stagnant, is it the model? Is it the data pipeline?
It is a network or let's try it. That's the first question.
The second one, when you monitor, which consumer cluster resource is, when you put a drop,
it's the second one. Under the third one, why did the business, to increase the training
of the school year, when it was received? So, in the project, I already built.
I have been in the study with the work path, the collection with a very
specter, which collects the GPU matrix, and draw by counting the storage
I have on the node export, for metric system. Under the storage, I have used the Victorian
matrix, because in terms of ICAD90, which is a very performant, I have an extra,
which my company will partner since last week with the Victorian matrix, only we will propose
that training, just an extra. On the internal visualization, we have used the graphana,
in terms of dashboard, in terms of alerting, and to distribute the tracing, we will get our base,
and we have a, I have this, a project on the open-term experience.
So, the question, why did the Victorian matrix for HPC? Because it is an open source, so it's a DB,
which is very beautiful for ICAD90 team. The first advantage of the parameters,
with a better complexion than parameters, we have an addition of a somewhat
per second, which is very good on the advantage, it is a promture compatible, in fact, open source.
It's really good. So, if you do this, I will be in a short, quickly demonstration.
So, if you implement a V-stack, Monday, tomorrow morning, when you will go out of the job,
there's a free thing you can do immediately to improve your monitoring. The first action
is to achieve a time-sort core on your body matrix. You can also monitor storage
higher latency per period, because the cellion is a bit clearer. On the LOSGPS, the latency is
not that destroyed, you need to put a little problem. On the last one, you can try,
just trigger in this routine, the job. So, I'm in a screenshot, I'm going to make a team dive
on the one-to-demo. So, it is an example of how I have organized my monitoring on the
side of the team, in terms of the year. So, you can see the first layer on the time of infrastructure,
I'm checking the GPUTization, the tensile curve, etchorem usage. The second layer, it is a workload
efficiency. So, I look at the two new footwear, the data-loading button, the one-to-one,
it's unique, tiny breakdown on the second rhythm. First, the third layer is a model f. So, I look at the
LOSGPS, the gradient norm. At the end, I have tried to cross, to make a correlation, a cross
correlation between this field part. So, if you think you're going to go as well, I'm going to
pray up. So, as a lab, I have built on how you can, after the tutorial, I have access to
the repository to test. So, I present here the different materials I have used to implement
my lab on the other, I have done the email project to LOSGPS. We're differentiating the different
scenarios you can test. On the other hand, it's a texture. So, it's based on the training simulator
that I have to pull, it's data on the VM Engine, it's in Valhya. And at the end, you can see the results
in Dwarfana. So, I will do quickly, I think I have a minute time, just a minute. So, what I will present
here quickly, so, you will find an explanation of this lab of AIOPSability for HBCM. So, the
current Valhya I have described in this demo on the so on. So, it's more a summary on if I go
on the next tutorial. So, yes. So, what I have shown present with my screenshots. So,
actually, it works a sensor. So, you have this presentation, you have a new review of your training
on the other, I have a spending in the different layers, so, in the first tutorial, you have
seen the GPU utilization, the sensor co-authization. The second one is the workflow efficiency.
As I found the help of my model, I thought, do I have a lot of graphics, we have a
gang on the arm, on the so on, I have a very, also the credentials of the LCD, the long red,
on a ZN, you have a course layer, a coalition, on the last poll. So, yes, it's just the last
at the end of the sentence, and we are good. I will share the presentation after the
for example, to you come all the presentation on the sources of this lab on these values.
Thank you very much.
One question? No. Any questions for Sam?
Yeah, JP.
Is this a spending in media or not?
Is this a vendor at non-stakes, or a market in media?
It is agnostic.
Yeah, sure.