Okay, so we are ready to continue with the next session as we were talking in the previous
session.
We continue with the DNS topic, the last year we delivered in 3PA, what we call
encrypted DNS, and this is a talk from Yuseppe and Ramon about the performance analysis.
So, guys, you're up.
Thank you very much.
Thanks for coming, and welcome all of you.
So, first, I'm interested in myself, I'm principal solution architect, in Red Hat,
I'm passionate about technology, and I like to speak in events like that, number of
communities, and I'm also a father of a young gamer, and passionate about videos,
so it's our video, so I'm getting old.
Hi, I'm Yuseppe Andreo, I'm senior format Red Hat, well, I remarked about my work that I
spent a lot of time working with engineering teams on real-production problems with these
guys, and well, I also teach classes in university about several topics.
Okay, we start with the origins and motivations of these investigations, well, everybody
have heard this sentence that it's always DNS, and now we can confirm that if it's not
DNS, it's encrypted DNS.
Okay, playing DNS assumes that the network is friendly, and such anyone on the network path
can, can listen to your queries, can see your queries, can know to who you're listening, and
even can change your money for your answers.
And this is not theoretical, this is inherent to an encrypted DNS.
And this is where a encrypted DNS comes to the rescue, and more than that, zero-trust architectures,
because with that, we have ensure that all communications are authenticated, authorized, and
encrypted, including DNS.
Okay, encrypted DNS and zero-trust architectures, and coming from nowhere, there are two main
tri-forces.
One of that is the US-Memonandum, that mandates that every internal network that has
super-for-hybrid warlobes, should adhere to the zero-trust architectures principles, and
this says that internal network no longer means trusted by default, and in European Union,
we have similar direction with the Cyber Resilience Act that translate the set of best practices
into legal requirements, and also ensures that all the security requirements are in place
for hardware and software products with digital elements.
Well, to secure communication, to ensure communication in free-PA, we must provide certificates,
certificates can be provided by the free-PA.ca, or can be your own certificates, this depends
on your trust model, and most importantly, the encryption is not only provided at ruined
time, it's provided from early stages of installation, and also at boot time, so we can
assure that encryption is based on assumption from the zero.
Okay, DNS is an abstract, it's visual, and it's visibility can be tool, and also a risk.
This is a project that is called DNSMAT, and well, it's visualized geographically, all the
DNS traffic around the world, and highlights the fundamental truth that is that anyone
can see a geortrific, and this is where encrypted DNS can limit what attackers can infer
or exploit by limiting all the visibility.
Okay, come on.
So, what is the alternative?
So, we have to move from an encrypted to encrypted, and the implementation that free-PA has
for the buying server is used DNS over TLS.
What is that?
We move from PUR, UDP, or TCP communication in plain text to TCP communication with encrypted messages.
So, we are tunneling all the communication, and nobody can see, except the client on the
server.
What are the alternatives that we have for encrypted DNS?
We have the DNS over HTTPS, that is very used in public clouds and sites that only
allows HTTP traffic.
And there is a new one that is not very mature jet, that is DNS over a quick, that will
use HTTP three with UDP, but both of them have caveats versus the alternative of using
DNS over TLS.
Okay, so we want to encrypt that.
What is the question that we will raise for the first time?
What is the performance impact that we will have doing that?
It is not just doing, but seeing what is the result of it.
So, we find out, what is the procedure that we will have to test what is happening?
When we move from an encrypted to an encrypted first, we want to know where we will do the test.
So, we use some small instances in the cloud, because we didn't want to do something fancy.
We will select a standard performance client from the DNS or work community that is DNS
perf, almost everyone in the field knows this tool.
And then the most important part is that we have to monitor what is happening.
When we do the performance test, because if we don't monitor what is happening, we will only have some numbers,
but we will not have the possibility to dig deep on what is the result.
So, we create a big amount of records.
We configure three for providing this kind of encrypted, an encrypted communication.
We also use a Kubernetes cluster, because it is very interesting right now.
And we modify some patterns, and we do some performance test to try different alternatives,
and try to stress out the most of the bind server, and see what is happening across the chain.
We measure two different things, firstly, the throughput, and another is the latency, the latency.
We did a deep inspection of what is happening at some point, because we don't want like a median at the end.
We want to have a detail of what is happening at every time that we will implement that.
So, we use one second, and this is architecture that we built for this kind of test.
Okay, I will expose the two architectors and the results.
This is the first one.
This is a client that is joined to a 90-end domain, that is listening on the DNS port,
on the DNS standard.
The rest is the collection of metrics, the parameters and how we visualize it via graphana.
Okay, this is the stable compares, the three scenarios.
This scenario is exactly the same, the only thing that changes the transport protocol.
UDP has expected we obtain the highest throughput, and the lowest latency, in that case, even in the order of microseconds.
In TCP, we see a reduction in throughput and increasing latency.
This is due to reliability cost, congestion, return submission, ordering, etc.
And the most penalized is the DNS over TLS about the reduction and also in lead on increase in latency.
This is expected, but the question is not that this is lower, because everyone expects that,
but how much is lower, and if the behavior is predictable under load and sustain.
Okay, we don't slide the numbers into actual cost.
We see a drop of 70% from transition to UDP to TCP.
This is the cost of reliability set, UDP is extremely fast, microseconds, latency response, the latency, when we go to TCP is triple.
This is due to the mechanism of TCP, three-way hand shape, and the establishing of connections, etc.
That takes roughly three milliseconds to establish TCP to DOT.
We see a drop of 23%.
This is overhead for all the cryptographic operations.
And if we see end-to-end, total capacity impact, we see a drop of 37%.
This is less than this sum, because this one is calculated on a gross status fee one than the total.
Okay, this is not 40%.
And latency increases over 20%.
But here context matters, and the topic is that the latency is incremented, but very few.
So only 0.8 milliseconds if we compare to TCP.
So in Sumari, system behaves really well.
It has sustained the performance of 119k BS.
That's an impressive result, and the system is able to absorb all the cryptotacks without any penalty or falling apart.
Okay, this is next scenario.
This is Kubernetes-based setup.
This is really close to how this is really deployed on production environment.
We use a forwarder.
This is a pod, the product on the Kubernetes cluster.
That talks via standard DNS to core DNS.
And core DNS them forwards the queries to free by and back.
Okay, and this communication is playing UDP.
This is always this case.
And the only thing that you can encrypt is the upstream leg.
So you make the test with this leg, encrypted or unencrypt.
And the rest is more or less the same scheme about recollection and gathering and visualization.
Okay, and this is the thing that more or less can be surprised,
because the performance is identically the same in both cases.
This, what we'll explain later, the latency is increased, but marginally only 0.7 milliseconds.
And the latency is the most affected by 4 milliseconds.
If we translate this also, the surprises that both configurations have the same throughput.
And this is due mainly to how this is done, core DNS versus free power.
Core DNS is designed to be extensive, to be integrated with Kubernetes and free-based and auto-itative bind name server.
And as such, is on to maximize throughput and to maximize the throughput of query handling.
And having TLS encryption has practically any performance penalty.
So this is totally marginally, in third or second, free-based care to extremely well.
And the performance scaling factor is the core DNS.
So is completely outside of free power binding domain.
And, but I've seen it delivers the same throughput as the in-secule-based line.
So the security gain is best-lead weight.
The penalty that you can have in TLS.
Okay.
So we have recorded a demo, so you can see it later if you want.
In this video.
So we test on DNS resolution on clients.
90.
Okay.
So you can see here, and we'll see that what we try to do is try to reproduce and make it reproducible.
For even testing in the CACD pipelines, when there are some new features.
So we first use direct communication from the pod to the DNS server.
To double check that everything was working right with the IPA server.
Then we will try to communicate through core DNS with the baseline.
And the core DNS will not forward it to the IPA server because it is not configured for that.
Then we will move from a standard configuration where the core DNS for that zone is forwarding using both an encrypted communication with the IPA server.
And we did some small test and then the performance test.
And we will see also that is how the core DNS is configured for the forwarding.
And then you will see that we did some performance test running different scenarios.
So we can double check where the results either from the client side and from the server side.
So we can know if the server is having some issues with CPU memory or something like that or has some kind of networking issues.
And we will also know what is the client scene from its side.
So we will see with the DNS Perf we are using jobs, Kubernetes jobs and configuration.
So we can check and modify that pretty easily.
So we can do a lot of different scenarios.
And then we upload the results, the login results.
So we can see also in Prometheus and Grafana both both thus boards.
And with that we find out that the result numbers were not just some gross information about what is happening.
But the performance and the stability is quite impressive.
And that's all. So if you have any questions.
Any questions from anyone?
Yeah, please.
So from your question, you're testing to remember running into a bottleneck.
And also the size of the project.
Aspects, where was it mainly into the extra.
Regarding UDP.
When you were testing was done on the bottom of that.
So the question is if what is the main issue when the performance is impacted?
If it's the TCP or it is the encrypted communication.
So we find out first that there are different performance parameters that should be taken into account when using TCP or UDP.
Just by itself. So we have to tune both sides when we were doing the test.
For sure, TCP has some performance impact.
But on the other hand, and especially in cloud vendors, they have other limitations.
In cloud regarding the number of packets per second that they can handle with the virtual networks that they are using.
Depending on the situation and depending on where are the environments.
You come face more impact by TCP or by by by encrypted configuration.
Indeed, in some other tests that we did in other cloud providers.
We face out that the TCP was more performance at UDP because of this because of the penalty impacted in packets per second.
And in the number of contract connections.
But of course, it has to be something related with encrypting.
We didn't try to use hardware accelerator for encrypting.
And we think it will enhance the communication performance on that.
We find out at the end is that if the client is not so performant and we have seen here that code DNS is not so performant.
Even TCP or the encrypted thing is not relevant for the final fears.
So you have to be a man, what is the whole picture?
To know what is the most impacted factor for the whole architecture.
Well, we have done several performance tuning as someone said.
And we will pull these two articles about with stretch really free time.
One of the most interesting thing is that you can have a lot of threats.
But if you are writing in a file log, all of them has to be synchronized.
So we careful about that.
Well, there is not yet.
Can you explain to the difference?
No.
I think we use the usual one.
I think it is 1.2.
But it is with different DLS versions.
So I think we use the default one from Fedora.
That is 1.2.
But it should go on point 3.
Then it is 1.3.
But we don't know exactly what core DNS is using for communicating with each other.
It is also 1.3.
We will double check.
Well, for next investigations in future,
we have been meant to test this setup with post quantum.
So change all cryptographic protocols, etc.
To see how much affects.
Sir, we have signed for one question.
Is there any other one?
Yeah?
Yeah.
I was intentioned here to have a DLT3 services.
Within the environment.
Is this intended?
I used to come back.
So it is going to look up.
DLT3.
Yeah, it is.
So the question if what is the intention of or what is the
configuration exactly that we have tested.
If it is for internal use or for just also for external communication.
So another test that we didn't do is use the bind server as a general app
stream server.
We just used for that soon.
But we think if this bind server has to forward one other.
Up stream server.
Either you can do use the same DLT communication or maybe.
Then you have to also back forward to unencrypted DNS.
So it should not impact a lot in the bind performance.
Because what we have seen is that the main.
The main factor is usually at the input of a DNS server.
Not anything that they have to do because you can put a lot of different
file threads for the output but on the input.
You have the networking staff in the Linux kernel and so on.
And it is harder to parallelize the input than the output.
Now it is that situation.
When courty Ns supports DOT at the input then you can have all of all of the communication
and cryptone secure.
But as far as I know courty Ns is not supporting encrypted communication yet.
So we are all the different providers of DNS servers are working on some kind of solution for that kind of requirement.
Thank you very much.