So it's about deaf-cut and replication, first of all, who's actually running deaf-cut with replication enabled?
Okay, so it's still some.
Anybody already got rid of it? Because of it.
Okay, so at least I'm actually affected by this.
Why are we here today? Roughly two years ago,
deaf-cut actually removed replication from the open source product.
So until deaf-cut 2.3, we had a very nice replication feature, which worked active,
but now it's gone. So there's different assumptions.
Why this happened?
Some assume that in the past there actually have been some difficulties with the replication code,
and that it's just maintenance burn.
You could assume that they actually want to pitch their commercial product,
because this is of course interesting for the enterprise users, who need the highest possible availability.
But availability? Why me?
I'm doing here by net-less fleet management.
I'm doing this for about 10 years now, so very few emails stuff involved here.
Before that, I did study at the University of Constance,
I have a master in computer sciences,
but in between there's actually been two years of main administration at the university's
computer department, and yeah.
Once running, you know,
mail service in a large scale, you end up running mail service in small scale.
So I'm hosting my own mail setup.
You might get it, as I'm otherwise doing Kubernetes fleet management.
It's on top of Kubernetes to its sites, active, active, whatever.
Duff code replication is lovely for this.
So I kind of actually need it, because this is at home in my basement, in my parent's basement,
with a slow links in between, so low availability, high latency,
all the traditional stuff doesn't work here.
Let's look at replication in Duff code until 2.3.
I think the most central part is actually decent.
Duff aid, you can call it manually with Duff aid in decent, which is
bidirectional or unidirectional as you like it,
synchronization between Duff code and another IMAP server.
It's actually used pretty widely also for migration tasks.
For backup, you can do a lot of nice stuff with it.
It has built in conflict resolution, which is helpful.
For example, if you're doing some migration tasks, you start replicating,
like we had in the last talk, a few some files in mailboxes,
with a lot of mates in there.
It takes some time, and so you kind of finish this within some minutes,
window, you want to be in the background.
But this conflict resolution actually makes it work in
even in some more selective active scenarios.
You shouldn't do it, but Duff code is able to
like fix all the inconsistencies eventually.
It has some stay-for replication, so you don't need to
start from scratch and copy over all these
10,000,000 mates for a user.
It really is kind of good finding the differences,
and eventually running in within
subsequent time for replication when for a user.
So this was used as basis for the Duff code replication feature.
It's on layer 7, so I don't have shared anything.
So no shared storage, it works on slower,
or high latency, network links.
Yeah, and it's still there.
It was not removed.
So, actually we just need to run it somehow
at the right point of time to get replication back.
Looking at the replication module a little bit deeper,
so we have in the end two Duff code servers.
And so the grey one is just the other one in this
diagram, actually both would be running pretty much the same
not check here.
And whenever my Duff code A gets some change, for example,
a new mail arrives or a user like deletes a mail,
max it, whatever could happen.
There's an event sent by the internal notification
system to be so-called aggregator, which is another component
running inside Duff code.
There's pretty much some checks with this.
For example, hey did I really change something,
and if so it pushes an event into a queue.
On the other hand, we have the replicator, which is not doing
a lot more than pulling events from this queue,
and calling Duff ADM.
So, it's not really like a super complicated logic.
There's some more details, which I hit here.
For example, we actually have a background controller, which is doing
like paratics, we sync of everything,
so just in case you missed some event or something else broke.
We are just adding all the uses.
For example, once per day to the queue,
they get we synced, we are done.
So, now this is gun.
We have Duff code 2.4.
What can we do?
I guess preferred solution by Duff code
is getting your credit card,
and buying their professional product,
a completely different architecture
towards how they do HAA.
There's a cloud native, whatever storage solution.
We have a metadata database,
which is also replicated,
and statusbackers.
So, they don't have this issue at all.
They don't need the replication module.
Well, it's a closed source, so we had first them.
That's probably not what we are looking into.
Also, with this cloud native storage system,
with Cassandra, it's a little bit more involved in operations for sure.
The officially recommended way of doing this is a shared file system.
This is actually a pretty classic way,
which probably a lot of uses.
I actually do it.
It's totally fine for Duff code.
If you put the maids on some NFS server,
you have two sides.
You replicate stuff,
and you're doing a failover needed.
Works pretty nice.
If data centers structure fits and everything,
you are kind of bound to meet your level gear redundancies,
a chance of cropping over your maids to some place that's further away.
And, well, from my special case,
this is not built for a high latency network links.
So, this was out for me.
Yeah.
Patrick Chen, I hope I'm pronouncing this correct,
actually did some work in,
while taking this patch, which was removed,
and just applying it to 2.4 again.
Yeah.
So, from what he reports,
this was kind of fine and actually worked out pretty well.
There's a minor changes between 2.3 and 2.4,
which requires some adoptions of the patch.
Well, yeah, Captain obvious.
It's well tested.
The code works.
You don't have to change a configuration.
It's kind of a drop-in replacement.
You just take the patch binary.
Throw it into the system, and you'll find.
Hmm.
On the long term, I'm not sure,
because this requires patching the duff-cut source code.
So, whenever duff-cut is changed,
you have to add up the patch.
When you have trouble with duff-cut,
they realize as they say,
yeah, fix it yourself.
We are not analyzing this.
So, and over time,
this will definitely get more complicated in big potting.
I'm pretty sure.
And if duff-cut changes some more stuff,
like the notification system,
which is just for building plugins,
then, yeah,
you might end up learning this is now entirely broken.
So, these things are still around.
We just need something that calls duff-admdysic.
Whenever it is needed,
well,
I'm not sure if the two guys from Heinlein are still around
because they had a talk like two years ago,
and in their slides,
was something like, well, you could do it, crunch up.
I think this was not like,
really what they wanted to do,
and it will not be good enough.
Because we can run this every minute,
but this is too much effort.
We can do it once per day,
and then you're listening the data from once a day.
So, this is not a good thing.
So, let's check out what duff-cut actually
has offering offers to us.
We need a notification system.
So, going through the duff-cut documentation,
we actually see there's a pretty nice,
little documented official duff-cut event,
HPI,
which can send us events for pretty much.
Anything running in duff-cut,
also through an HTTP API,
as JSON documents.
Looks pretty nice.
Whoever looked into this already
realizes duff-cut can emit a lot of events here.
So, we are definitely getting whatever we need.
We need a queue.
So, this is a sort of problem.
We want to have a priority queue,
because, like, if I receive a new mail,
this is probably more important than somebody else,
marking his mail as red.
So, we want to sync new mail first, whatever.
Red is might be an example,
which is like easy to use.
And, how we perform and well tested as queue backend.
And, eventually, we need some controller,
which is just bluing all these pieces together.
So, we're taking these events on any changes that happened.
Put it into the queue, pull it out again,
put it against duff-ADM sync,
see if it worked, otherwise,
it would have been a bit of a queue,
because never could be problem, whatever.
Yeah, this should be some kind of lines of code.
And, there we are.
So, I started the development environment,
and we have duff-ADM.
Thanks to AI, we already have a logo.
You already know that picture pretty much.
So, it's kind of the same.
We have duff-Code, and another duff-Code.
So, now, the duff-Code is completely a separate process, or duff-Code.
So, this is a small demo.
We are now receiving the events through the HTTP event API.
We are queueing it into redness.
I get back to this later.
And, well, some other replication book,
or tasks, pulling these events, and, well,
now we use the duff-ADM HTTP API, which also exists,
to, well, get the sync running.
All you need to do in duff-Code is setting up the event handler,
and allow the duff-ADM HTTP connections.
So, right now, the code can react on IMAP command finished,
which is all the changes to IMAP,
to IMAP, to IMAP, and, of course, we want new names.
And, most important to me, it's unmodified duff-Code,
because I don't want to read the code if something's written.
They included mini redness, which is a small goal library.
So, redness included.
You don't even need redness to use this.
It just works.
And, once you connect the external redness,
the entire demo is stateless.
So, this both skates out.
If needed for a larger setup,
and, of course, you can go to really high availability setups.
And, how the way it is built,
actually it's totally sufficient to have one setup of this,
and you can connect both your duff-Code.
As the sync is bidirectional, it doesn't matter
from where you trigger it.
So, what do we now have?
It's an external demo.
It will totally not crash your duff-Code server,
no matter what happens, it just can't.
Otherwise, we have some bad issues with duff-Code.
As mentioned standard API usage,
if this is breaking to some changes in duff-Code,
then a lot of other things will break, too.
So, pretty much all enterprise integration will break eventually.
The notification receiver right now has three-carved
of IMAP commands.
I was surprised that actually duff-Code replication
did not catch all changes to mayboxes
from in my opinion.
So, if I did not get to the code wrong,
for example, completely misses out on CIF changes
and ACL stuff.
Yeah, and of course, it breaks on mail delivery.
So, it is pretty much covered by a full-interinter suite.
I'm also shooting at works.
And I catch all these events.
Yeah, the queue I already mentioned.
So, better is included, but swapable.
We have a mini-redist in there.
So, if you want to run it, it just works.
And if you have a larger setup,
you could at any time add a full-redist if required.
But I assume that this is actually doing pretty well with pretty much,
pretty large setups from where they did test with,
with IMAP test from Duff-Code.
It can handle quite a lot of changes.
For observability, Duff-Wordness exposing
Prometheus metrics.
So, we get a pretty good insight
and what's actually happening.
I will add some more of these.
We can see stuff like error rates.
Hold on, the replication actually takes for users.
The latency and stuff like this.
And right now, it's available as a Duff-Code match.
As this is GoCode, it's like a 15-minute megabyte,
statically linked binary.
It's just a matter of writing the release code.
And you can also run it as a binary.
As I'm running anything in Kubernetes,
the first thing I built for Duff-Code is a hand shot,
but it's like a simple binary.
Putting this into a system to unit is probably easy.
Just a very small example of the Duff-Code configuration,
which I need to insert, so that's pretty much it.
This is a long list.
I did not fit any on this slide,
but it's mostly these two config items,
and we have replication back.
It's available now.
On GitHub, it's running end production on my mayor of this one user.
I wouldn't crawl it production right now
until this gets some more testing.
Yeah.
There's some smaller to the list.
What's still missing for when I would call it production ready.
I want to check if this is actually working with Duff-Code 2.3.
I think it is, but I have to check out.
I want to do some more load testing,
so to see where some boundaries is,
for example, where you need the external radius,
if really required.
Well, the real-world testing,
so I don't operate huge main servers anymore.
On the other hand, I have a very short line to my users or users,
so I know when something would put break.
Mostly documentation.
And I need to know if this is helpful to you.
If you would love to use something like this, reach out to me.
I want to understand your use case,
to see what's really the credit,
and how we need to improve this.
I'm pretty confident that this will work out
if and stay around for quite some time.
Do you still need prop-free?
I don't know.
I haven't checked it out,
but it's like mostly copy-paste,
and adjust the testing.
Yeah, waiting for your feedback.
I still think we have some time left.
So I have a short live demo running offline,
so I'm not required to use the wireless in here.
What we see is, is it big enough for us, should I enlarge?
So we have a tough cut on top left,
another one on top right.
Over here, we actually have the bottom dimming,
and I can trigger some commands.
And what's happening right now,
we are having this parity sync,
which is I think happening all 30 seconds right now,
because I want to show something.
It should realize that actually not a lot needs to be done.
But over here, we actually see that the free static uses
that I configured have been discovered,
and looped over.
So, okay, it just happened again.
So, it connects to DuffCode,
since the sync commands,
let's see if we actually find it in here.
Is this a laser? I think.
I have two.
No, no, no, no, no, no, no, no, no.
Which one?
No, no, no, no, no.
Yeah, yeah, I'm just looking for the line,
because it's like a hard to look at my back.
I think it's this one.
This is where we receive the message,
and on the right, we should be able to observe.
Yeah, so this is like the DuffADIM connection
at a decent connection happening.
Otherwise, if I actually trigger something
like creating a mailbox,
this is from the end to end just with,
we see how we got the message
for the user create.
Well, we sent this in command
and on the right side,
the mailbox was transmitted again.
Yeah, that's pretty much all I had.
If there's any questions left,
I'm also around for the entire day, I'm leaving.
So can you run multiple of the instances of DuffADIM at the same time?
Yes, you can.
There's actually two reasons for this.
One is they are pulling from the same radius.
Can you run multiple of the instances of DuffADIM at the same time?
Yes, you can.
There's actually two reasons for this.
They are pulling from the same radius queue.
So this is the assumption you actually share in one radius
and say you will pull out once and work on it.
This also allows scaling out.
The other thing is DuffADIM actually takes care of
locking for a user.
So if you do it manually, there's a flag
which allows you to lock each user during synchronization
and then DuffADIM will wait to run the other instance.
Using this way, this actually even should work in parallel with DuffADIM
to point three's replication code.
Because from how I read the code, it's also setting the lock.
So you have, you even do have for my questions.
It's very interesting.
And when I was reading and looking at it, it's great.
I would just use it for different use cases
that replication actually, you know, like it's like in the name,
like DuffADIM can imagine that the actions you take out of the
once you have collected all those events could be all sort of different
use cases, especially if you have telemetry.
It can also extend to, like, not just replication.
Like a foundation.
Okay.
Other use cases.
So the question was about, well, this is about replication,
but you have a lot of other ideas like doing some telemetry stuff.
Actually, using the same event subsystem,
DuffADIM, at least 2.4, I'm not sure about the other ones,
actually has built in telemetry exporters.
So you can pretty much use the same events without an external
demand to have some counters and everything in a premises
compatible fashion.
So it's built in already, you don't need anything extra.
All right.
Any questions?
Everyone.
All right.
We're going to have a three-fold replication.
Yeah, yeah.
So DuffADIM is still built into DuffADIM 2.4, which is the free one.
So you don't need to buy the perversion, which is like a 3.0 release.
This is bringing the removed replication code back to DuffADIM 2.4
in the open source version.
So this is fully working with open source.
I have a three-fold replication code.
Ah, a three.
I haven't heard of, this is like a supported way of operation.
I guess it would actually work, but I don't think a lot of people
run it this way and you might run into these weird edge cases
that might exist.
I wouldn't do it personally.
Yeah, I assume.
Okay.
So, thank you.