All right, hi, my name is Ian Romanock, it's been a long time since I've been in front
of a graphics dev room, it's good to be back up here, I'm actually awake now, so that's
helped things much better than I was earlier today, so I'm going to talk about as part
of the compiler work that I do on Mesa, I do a lot of running a thing called FossilDB,
so Fossil, I'm going to start at the beginning in case for people who don't know, Fossil,
a Fossil is a serialization format that was created by Valve Software, they use it to capture
shaders and some auxiliary state from through the Vulcan API so that they can save those things
offline and use them to do offline shader compilation, so that when people download a game,
it comes with a shader cache, pre-populated, and the first time the user starts the game,
it doesn't take an hour, now FossilDB is a collection of Fossils that the open source
community has collected over the years, right now, the private, there's a public FossilDB
and a private FossilDB, the private FossilDB unfortunately is the more interesting one,
it has dumped from a little over 700 games and benchmarks and things like that and there's
kind of in the ballpark of two and a half million shaders in it, so it's a big chunk of stuff,
we don't use it for pre-filling shader caches, we use it for testing compiler changes, and
there are, you'll see a lot of commit logs in Mesa where people will log, so some shader
statistics about, you know, ran all the, try to compile all the Fossils and collect metrics
from the shaders before your change, do the same thing after, compare changes in the metrics,
and try to decide, did this change make things better, did it make things worse, we kind of
use this as a proxy for real performance testing, because as long as compiling two and a half
million shaders takes, actually running benchmarks on 700 applications would take orders of magnitude
longer, so it's a pretty good approximation for performance, especially for things that maybe
if you have a long series of changes that you're working on over a period of several years
and you have, this change is going to make a half percent change, and that's going to make a half
percent change, you may not actually be able to measure any of the difference across any of those,
and you don't want to say, well over two years, we made 3% improvement like that, so this is
something that is reproducible that you can actually collect the data in a reasonable way,
but there are additional uses for fossil DB, using it, using it in conjunction with a bunch
of internal validation passes that we have in the compiler that look for inconsistencies in the
intermediate code, sort of as the shader is going through the whole process of compilation,
can find a bunch of corner cases and invariants that get their violated, and we have a lot of
regular functional test cases, but two and a half million shaders that were written by random people
is going to catch a lot more stuff, and over the last year or so we found a whole bunch of extra
bugs that the only thing that found them was running them through fossil DB, like it escaped through
all of the CTA, hundreds of thousands of tests in the Vulcan CTS, all these things slipped right
on by, but there was one shader in fossil DB. There's a couple of these that one shader in fossil DB
hit, like, okay, cool. So it's, we can use it as a proxy for performance testing and also as a
proxy for functional testing. Now, I use fossil DB a little bit differently than the way that most
people do. I think that most people, when they have an MR, they'll collect data before the whole
MR and after the whole MR on a single platform that they care about. I am a bit, especially in the
presence of possible functional regressions, I run it on every platform on every commit.
Every platform that we at least care about a little bit. And so what that means is
I'm going to go through and I'm going to run all two and a half million of the shaders
with the compiler configured for one of six different Intel platforms. So it takes a long time.
It takes a long, long time to compile what? 15 million shaders now?
So I wouldn't even attempt it on this laptop. It would take, I think I figured out it would probably
take about 18 hours to do a single commit on this laptop. And this laptop is not that old.
It gets low wattage, but still. So it turns out that compiling millions of shaders is the
sort of problem that we refer to as embarrassingly parallel. They're all independent. You just
you can completely, just have a bunch of workers, one worker for each CPU that is just going to
pull another piece of work to do off the queue and keep doing that until everything is done.
And this is the exact usage model that the Vulkan API was designed around. And the piece of
software that we use to actually collect the shader metrics that's part of the fossilized
package from Valve called fossilized replay, it already does this internally. So it would seem
that the answer to make this go fast is, you know, we need cores, lots, lots and lots of cores.
So over the years, I have one of my friends in the background says that I'm a, that was a
pack rat, a pigeon, something. Anyway, he calls me something because I find things that are still
useful that other people are getting rid of. And I go repurpose it. And over the years, I've collected
a handful of machines that in their prime would have been ridiculously expensive and would have
been considered like outstandingly powerful. So I have a 36 core 72 thread, Zion E5 V3.
So CPU came out in 2014 and one of those CPUs would have cost $4,000 and the server that someone
was sending off to eWaste has two of them. And like 128 gigs a RAM and it was just going to
I'm like, I think I know what I can use this for. I, you know, I'll be honest, I try not to think
about that. I try not to think about that. But thank you for bringing that up, Luke.
Since then, I also picked up a 44 core 88 thread, Zion E5 V4. So the next generation,
and then most recently a 64 core 128 thread, Zion platinum. Now, doing just like a quick
sort of fat thumb estimate, it seems like even the slowest of these machines should be
five to six times as powerful as like this laptop. Like they should be just like dividing
numbers of cores. Like it should be at least five or six times as powerful. When I started
running on these machines, that was not the speed up that I got. I was getting a little bit less
than a four times speed up. And I kept hearing this voice in the back of my mind saying, why isn't
it faster? It should be faster. But I had already gotten a really huge speed boost and I could
had a big noisy machine somewhere in a room that I didn't have to think about. And those things,
they each one of them sounds like a leaf blower. It's, it's ridiculous. For a while, we had it
had one of them in the back of a big shared cube that we have. And every time it came out, people were like,
come on. Yeah, yeah. So, but I just like, I had this voice in the back of my head wanting to know
why isn't it faster, but I just like so many bad feelings. You just suppressed them.
And until one day, you can't anymore. And what happened was on, I think it was on the 88th
red system. I had H top running. And I noticed a couple of weird things while H top was running
on that. One, there would be times where the system would be really fully loaded. And then there'd
be like two threads active on the machine for a minute or two. And then it would burst back up
to being really loaded. And then there would kind of had like the sinusoidal patterns of
not being very utilized. And over a period of Iran, I don't know if this shows this yet. So you
can see that on an 88th red machine, there's only about 51, like barely half the machine is used.
What, what the heck? So it, I thought about for a second. And I pretty quickly figured out,
you know, one, what was going on. And two, like, well, there's where all my performance is going.
So the fossilized program that actually processes the fossils takes a single fossil at a time.
And the fossil contains the dump of all the shaders from some application. And it'll take
every shader that's in that fossil and it'll throw it at all the CPUs. And they'll just keep
working away on it. But at some point, there's going to be fewer shaders left to work on
than there are CPUs. And so during that time, you are wasting CPU time.
And I don't think that this is something that the people who made fossilized replay ever even considered,
because, you know, not that long ago, 8 or 12th red CPU machines, like that was the most common.
And even still like, you know, raise your hand if you have a 128th red machine.
All right, there's like five. So, you know, if you have an 8th red machine and half of it,
and it's only half loaded for three minutes, like you've wasted 12 minutes of CPU time. Like,
okay, you know what, maybe I should care about that, but I don't. But if you've got a 72th red machine,
and it's only 50% loaded for three minutes, okay, you've wasted two and a half hours of CPU time.
And that starts to feel like when you're talking about a runtime, you know, on this laptop,
that's 18 hours, we're wasting two hours. Okay, all right, you got my intention now.
So, how can I get that back? How can I use that time that's just wasted?
So, my first thought was, I'll just modify fossilized replay.
I'll modify it so that it can process multiple fossils at once. And as one is getting done,
it'll start executing shaders from the other. But kind of the way that Vulcan is structured,
and the way that you enable features and do things, it seemed like that would be very
intrusive to fossilize replay. And it's really far out of the scope that that program was designed for.
So, one, it seemed like it was going to be a big pile of work that I kind of didn't feel like doing.
And two, I wasn't 100% confident that it would get accepted upstream.
So, I just, I put that idea on the back burner while I tried to think of a better solution.
And then kind of around that same time, someone else in the community made a piece of software called
called DQPrunner that runs tests from the DQP test suite all in parallel. That will wait.
I could maybe I could make a separate runner that just detects when, oh look,
this instance of fossilized replay is getting about done. I'll go ahead and start another one
to use up some of those available CPU resources, kind of in the same sense as what DQPrunner does.
Now, the challenges are a little bit different. With DQPrunner, all the tests are inherently
single threaded, so it's really easy to detect. Oh, one gets done, start a new one.
But with sitting on top of fossilized replay, it itself is multi threaded, so it's kind of a
different challenge of trying to figure out, well, when, how do I know when it's time
before one is done to start another one overlapping it? And that, that was a head scratcher for a bit.
And I did some internet searching of like, okay, what are good ways to detect when CPU load has
dropped enough? And the answer I found was, don't, because GNU parallel already exists.
It is literally the fully generalized version of that runner. It is the tool
that people have spent a lot of time working on. And they have the other thing,
one thing I find a little bit annoying about the program is the first time when you run it,
it gives you this nag screen about if you use this in your academic work, please give us a shout out.
So here is I am checking the box. So it is very configurable about how you can, how you can send
work to it, how you can have it parcel out work and detect CPU load and do all kinds of things.
But that's kind of, it's kind of the point where the bad news starts in that it's,
I find it, I find parallel to be a bit like e-max. It is this really great tool that can do everything.
And because it can do everything, like the simple tasks and the massively complicated tasks
are about the same level of complexity. And so it kind of makes it difficult to know,
if I want to accomplish a particular thing, what is actually the right way to do it.
And so I started, I went off on kind of a false path. And that is kind of most of
what I'm going to talk about now. So there was originally this script in the fossil DB
repository that goes through and collects the big list of fossils that are going to be processed.
And it's got its wrapped in this big for loop. And it starts a single instance of fossilized
replay one at a time. So the starting point was I took this script and started making it work with
with parallel. So my first change, my first thought was, well, what I'll do is I'll take
this line of the script. The one that's actually doing the real work. And instead of actually
executing that, I will dump that whole line of text out to a file. And that, each line of that
file is what I'll send to parallel of like these are the pieces of work that I want you to do in
parallel. And that is a super easy way to get started with it. But it doesn't work out very well
in the long run. And it's definitely, I don't think the way it's intended to be used.
The first thing that I tripped over is the old troll in any kind of unix shell scripting of
as soon as you start having file names that have single quotes or double quotes or spaces in them,
how do I get that in my script file so that it can get the quoting can get into the script file
and be preserved. And out of the script file and be preserved. And the answer is, you can't,
it's madness. You have better luck facing Kathulu. And yeah, so the better way to accomplish this
is what you really want to do with parallel and I'll have another example of this in a bit.
If you want to create a list of, a list of the work items that it's going to parcel out,
and then a program that it's going to work on them with. So instead of creating the giant list of
here's a bunch of command lines that I want handled in parallel, I create a list of,
above here of, here's just the files I want to be operated on and then take each of these files
and then this is the program with some extra command line invocation.
This is how to operate on each line from this file. And that's sort of the more natural way for
parallel to work. And I've found that a very easy way to adapt it into some other
previously kind of inherently serial pipelines that that I have and some other things.
Okay, so how much, all this work was to see if we could get some speed up. So how much speed up
could I get? Well, I looked at at five different machines. I looked at the the two Zions. I looked at a
big relatively modern desktop that I have. I looked at this laptop and then another fairly old
desktop that I have. And the most interesting thing was so on the 12 core machine, there was no
change on this laptop because it has performance cores, efficient cores, and low power efficient
cores. There was a huge amount of variation in the run times just based on did a really tough
shader get scheduled on one of the really slow cores. And so there was like, there was so much
variation in the run times that I think there was that it's effectively the same either way, but it's
I would have to do a bunch more 18 hour runs to find out I'm not gonna. But on the other machines,
it was worth the effort. Even on the regular old desktop, it was a 4 percent sent,
4 percent speed up, which is decent. On the massive machine, 42 percent, like that starts to feel like
real time. It still doesn't quite get it up to the what the anticipated speed up, but
it still felt pretty good. There are some areas of for future improvement to this.
I haven't actually spent a lot of time optimizing this. Once I got it working and before I
even measured what the speed up was, I could tell like, oh, it's going by a lot faster. I can get
through a lot more of this done. It's fast enough, right? And so I didn't want to spend any more
of my time to spend less of that time. But there are a few areas where I think it could be improved
that I would be a pick at in the future. Parallel has a whole bunch of different modes where you can
control when it is going to start new work. You can specify how many idle cores, how many
what load percentage you can specify. Once don't start another worker, unless there's at at
least 20 percent of memory available, like there's a whole, it is a massive chunk of the
man page of here's all the different tools that you have to control how this does. So I think perhaps
adjusting that load percentage, especially on the smaller core, on the CPUs with a smaller number
of cores may help on those, but I haven't experimented with it yet. I also think that optimizing
the order of the fossils might help because there are some, there is one from one application where
that single fossil, it has so many shaders and some of them are massive, massive computers,
it takes 20 minutes by itself. And so ordering it of putting some of like putting
into my intuition says if you put a fossil that has some of these really, really long poll shaders
in it followed by a bunch of them that go very fast. So we have that one fossil that takes 20 minutes,
there's a few dozen that take under a second. So by kind of mixing them like that, that maybe that
could help better utilize the CPUs. It seems worth experimenting with.
Parallel also has a mode where kind of like this CC, you can say, I have all these computers.
Go run it on all of them. And since I have all those computers. And so then that kind of opens up
to make it interesting of, well, if I have just a handful of those 24 core desktops, you know,
my current computer and my computer for a couple of years ago and my computer for a couple of
years before that, well, I don't, I just spread it out across like use it as a cluster.
It seems more likely that someone will have that in their house than will have one of those
the ones. And then you can heat up multiple rooms in your home instead of just the one.
Yes, yes, yes. Oh, and then playing around with some scheduling modes or something, I don't,
I don't think Parallel has a way to do this, but if there's a way to try to schedule,
I want this work item to run on the performance cores. Don't put it on deficient cores,
because if it comes at the wrong place in the pipeline, I'm going to be waiting for it to finish
because it's running on on the crummy cores. But once I, I kind of got to know this tool,
it turned into one of those. When you got hammer and you got really good hammer, like everything
starts to look like a nail. And I started finding a whole bunch of other places in my workflow,
where oh, I could, I can use this tool to make this thing go faster. So I had a script that I wrote,
oh my gosh, 10 years ago, maybe, maybe more to, I hate that every digital camera, all the
file names are uppercase. It doesn't use optimized JPEG tables and it sets weird file mode bits.
So I wrote a script that will use JPEG Tran to pull in all the JPEG images off the memory card,
optimize the, the Huffman tables, fix the file names, fix the file file modes. And I've used the
script for years and years and years. But what I, it has always kind of bugged me, but never enough
to do anything about it that it's going to, it's going to do pull in like 80 pictures from my vacation,
it's going to take a while. Like it's because it's going to do one at a time. It's going to take a while.
But if I split this, if I split that one script up into two scripts where I have sort of a
leaf script that does the actual work of running JPEG Tran on the file, figuring out the fixed
file name and fixing the file mode, and then have the other part that finds all the files and just
sends that to parallel. Like it's now basically bandwidth limited on how fast my SD card reader is
and it goes a lot faster. And it was another problem of, you know, I'm sure I could like spawn
a bunch of these in the background, but I have to figure out how many CPUs there about like, no,
this, this change took me five minutes. So my actual scripts, they, I've been having trouble with
internet here today. I think like everyone else, they will be in this GitLab repo for, for shared
DB on the free desktop GitLab. There's a bunch of other scripts in there that I use for things too,
but that's where the parallel fossil DB runner is. If you have some other machines with
weird, you know, big numbers of course, I'd be very curious to hear if you get what kind of
a speed up you get for the serialized version versus the parallel version. And that's it.
Do you have other any questions? Yes, in the back.
I sit in front of the steady. Instead of these. Someone took a look at the
igen. OK. OK. I'm actually used the CC in, long time, OK.
Yes. There's a evaluation if you're looking to spin up across lots of tasks across the
So the suggestion was it's called hypercue.
Hypercue did.
Okay.
I don't want them to stop like a slurring dog.
Right.
Okay.
So hypercue has a way to start a bunch of jobs across a cluster.
Okay.
Does it?
So the one of the nice things about and some of the changes that I would have to make to the scripts that I have to make them work across a cluster using parallel is.
There's a bunch of the sort of temporary file management that I'm still doing myself and parallel can do that.
And when you let parallel do the temporary file management, it knows which files are temporary files and which things are result files.
And so it knows which things to push out to the remote machines and then which things to pull back and which things to just delete.
I have after it's done.
And that seems convenient.
But that does this.
And so I mean it's important for this.
You don't trust them either.
Because you you parallel file systems.
Okay.
Yeah.
I haven't domesticated this at all.
Okay.
It's just a very quite minimal.
Single binary type situations.
It's very quick to do.
Okay.
To try out the very similar to parallel.
But I mean it's a it's a workload management.
So you you would to you that one of these tests.
This is a big big one.
So it gets 20 course.
Okay.
Oh, okay.
I wouldn't even schedule this optimally.
Oh, interesting.
Yeah, because that and that's the thing with.
With using fossilized replay.
Is that it.
You can tell it how many course to use, but it you basically tell it all of them.
Yeah.
I don't know if you ever end them.
This fossilized would have less than.
The 64.
I mean, especially on the on the really large machines.
There are there are some some of the fossils that.
Are quite small that maybe have 30 or 40.
So maybe not.
Such a big win.
Yeah.
Interesting.
We'll go here in front.
And then you'll be next.
At the beginning when when you're looking at the problem.
You just said, well, there is this one runner.
But you don't be worked with jobs that are seeing the threat it.
Right.
Well, so that's so there is.
Yeah, there's another runner thing called DQP runner.
Yeah.
And it's basically.
Are you familiar with it or.
No, no, no.
And my question is.
There's a.
What didn't.
Why didn't you try to basically make your problem fit?
Make your replay to sing and spread it.
And then you don't have to problem of guessing how much.
First, basically.
So the the reason for that is that.
A big part there's a big part of work that happens.
Once per fossil that if you did it.
If you made it so that.
So you would end up with one of two problems.
You would either end up with invoking fossilized replay to compile.
A single shader, which would have a lot of extra overhead.
Because there's a there's a bunch of stuff that it has to do to set up Vulcan so that it can even compile it.
Or you would run fossilized replay on a single thread.
And so then if you have.
If you have the.
The the fossil from the the game I won't name that that takes 20 minutes.
Now that that 20 minutes.
I mean that's that's 20 minutes on the 88 thread machine.
So it would take hours.
And so even if you started that one first now you're going to end up with sort of.
Like the transpose of the original problem of now you've got it.
It's at the end and it's.
So you do have a lot of fossils.
But yeah, this one is too long.
So the long tail at the end is still.
Yeah, yeah, yeah.
Yeah, and it's and it's there is a huge variation in in.
I mean there are ones that take a quarter of a second.
All the way up to the one that takes 20 minutes and every amount of time in between.
And so it makes it.
I see what you're saying.
But there are other factors that that kind of make that more more of a challenging way to try to.
To try to reshape the problem in in that way.
I was going to go with him first and I'll come back to you.
Which is.
Would.
We make a sense of trying.
Focus on the.
Master.
Uh.
A society.
Presence.
Yeah.
So that's that's kind of the idea of of optimizing the order of the.
The fossils of like there's ones that we that I know are.
Yeah, our hard.
that I know are, yeah, are hard, start them first.
Oh, well, so generally, I mean, I'm going to end up running all of them.
So in terms of optimizing runtime, there's sort of two separate things there about picking
a subset or reordering them is reordering them might allow you to run all of them more efficiently.
And if there is a subset that you know is more likely to be problematic, then just running
those, like you're just skipping a whole bunch of work.
Yeah, so usually what, so the kind of the workflow that, let me go back.
The workflow that I have gone through that ended up with almost all of these is when I'm going
to test something of mine, I have to do the before run and when I do get at some point in the before run,
there's some fossil that hits an assertion failure, hits a segment fault, has some kind of unexpected
termination and that's like, okay, my whole, my whole work today is going to be different now than I thought
it was going to be 10 minutes ago.
And I have just one.
I know, like, okay, this one is broken and so then I can do get bisect and instead of running all of
them, I will just run that one until I figure out what, what committed is the, that broke
that and then can usually figure out like, it's this app and this one and that one and that one.
And can usually, I'll usually at least list all of them in the commit message of this change
fixes, fixes this bad, this other bad commit on, on these various fossils.
So in the fact, let's comment on the, on the background between, say, run the mold as parallel as it can versus a,
a small, many of them have more shaders than you have course anyway.
Right.
And I mean, worst case, no, you have a easy one shaders.
And so you run the first eight day and that's one for long period.
I mean, but you could also, you don't have to go to single thread versus, right.
Maximum, you go nine course.
But that one.
Yeah, and that would, that would be, I mean,
so the suggestion is to, instead of going from the one extreme of all the course to the other
extreme of just use one core, like, pick some middle ground.
I think that kind of falls in with some of the other, like, trying to adjust, like the load setting on parallel and some of the other things to sort of adjust and tune.
That could be combined with, if you had a more free and actual scheduler.
Right.
Like it's learned.
Yeah.
But they actually sort of keep track of actually core number one, two three and four.
I currently basically run in this god.
So when I have run it, I had the display here.
I mean, kind of why I didn't go much farther than I did is so on this, all right.
So at this point, I was getting, like, 48 out of 88 threads are in use with just the work that I've done on this machine.
It kind of hovers around 70, 75.
And like, okay, that's, he need 90% utilization.
Like, there's, there's not that much left to ring out of it.
And how much, how much compute time am I going to save for how much time am I going to spend trying to get that extra mile?
And, you know, in another six months, I'm going to find another e-waste machine.
This one will be 256 threads.
That's not as easy as you do.
The big opening would be that this would be a paradise across the entire machine.
Yeah.
We had a small, not too small work units, but right.
Not that this low is young that you started that intervening is still.
Yeah.
And so that's sort of a general, like, when it comes time to schedule the things across multiple machines,
the challenge there, and it's part of why I haven't done that yet, is there is such a variation in the power of the machines,
is that, yeah, if you end up with the wrong work unit or the last work unit or whatever on the slowest machine,
you're wasting less CPUs time in some sense, but it could actually make it take a little bit longer.
Yeah.
And a lot of the way that I end up working is I'll have one of the machines doing some stuff and then some other experiments on one of the other machines.
So I'll be using the machines, but for different, I'm parallelizing them in different ways.
I was going to go over here quick.
Yeah.
So when you talked about phospholite replay, you said that if you want to run to phospholite together, you will have the problem of this matching features.
Can you?
I'm just wondering, can you actually create multiple contacts for the same device?
Yeah.
And so the question is, because of the different feature requirements of the fossils, could you create multiple contexts to account for that?
That is basically what you would do.
The problem, the program that we use to do this kind of thing for OpenGL shaders does exactly that.
It creates a bunch of processes.
It creates one process for each kind of OpenGL API there is.
And then those create multiple threads, and it farms out the work to those in that way.
To do basically that.
And it's quite a bit of added complexity.
And that's kind of the thing that I wasn't.
I didn't feel 100% confident that that work would get accepted upstream.
And I didn't want to have my own branch just kind of hanging around.
But yeah, I think that that is the way that you would have to implement that.
It was just because I think a pop-boss line has been played and was used by Steve playing.
It does.
Yeah.
Up until when I think all type of line captures where it's like, then it just kind of never just turns it off.
Right.
So I was thinking if you have some things that maybe Steve can actually use.
But yeah, it's been done that now.
Yeah.
Well, so they're, I mean, they had utilized it in a different way.
And they were running it on their big servers elsewhere.
So I suspect that they just had.
I'm not sure how they deployed it, but I, I suspect they had different problems.
Then.
Um.
What's your memory use?
It's like I need it.
Oh, so let's see.
If there, if there's learning too many threads, then the schedule works usually just fine.
Yeah.
If you need one after the other, but at some point when you're just forming too many threads, you're running out of memory or crashing your passes.
So more and then it gets slower, okay?
So there is cash issues, but this like.
So this is running for the, it's a serialized version.
But even running in parallel, I have never seen this machine use more than eight gigs.
Okay.
Like, I don't, I don't understand how we're not using more memory than that.
But yeah, that was, that was a thing that for sure surprised me.
And if that were a problem, um, there are additional parameters that you can give to parallel to tell it like don't, don't over commit memory.
In addition, so like, um, one of the things that parallel doesn't know how many new threads or how much memory usage.
Right.
We'll go to use when it starts one next come on.
So there will be at least two.
That's true.
Yeah.
You would, you would have to, you would have to, you may have to do some additional hand tuning and,
and be careful about how you set the, um, so it's not in the slide.
So it's always more grace.
So like, you remember when your machine hit five?
Yeah.
And, and on that machine, you know, when I got it, it had 128 gigs.
Like, I've never even, never even come close to exhausting that.
Yeah.
Send it back.
What's the level of package?
Are you, are you within it?
So I, I closed the level of it.
Um.
Well, on that.
I mean, it's those.
I, I, I, again, that can be a subject of the, I can recommend to two of us.
I think it should work on the over the ones.
Yeah.
I have not looked at, at that.
I mean, I know that they, that is, that is probably another thing.
Another area to, uh, for future optimization is to look at the thermal usage and to see if we're actually.
If overloading it is causing it to use slower clocks and that that's.
I, I, I, I, I, I, I, I, I, I, I, I, I don't want to know how many watts have been spent.
I, I, I, I, I don't want to know how many watts have been spent.
I try not to think about it.
Like, just, just, just those machines being on is a huge, is a huge draw.
And especially when they're going full tilt, it's, it's kind of, it's kind of ridiculous.
All right. Well, thanks, everyone.
Uh, thanks for having me. I get, I guess you're all free to go home now.
This is the, the end of the session.
Thank you.