Let's welcome Ramon and Ben, who will tell us about precision landing with PX4 and Ros2.
Hello, hello, you can hear me?
Yes?
Okay.
Yes, the applause please.
Thank you.
We're technical difficulties.
Sorry for the presentation.
Wi-Fi is what it is.
But thank you for being here today.
I know it's a packed room for those who are waiting outside to see us.
Thank you.
Yeah, so we're going to talk about drones and how to land precisely.
Let me see if I can start moving this.
Okay.
So let me introduce my co-speaker Benjamin.
Ben.
He's a PX4 maintainer.
He's a maintainer for the Ros2 interface in PX4.
Do you want to?
Hello, everybody.
Yes.
I'm the emino-potsam.
Ben, if you want to.
I have four years of experience in validation.
Like a validation.
No.
God knows navigation control.
For a robotics, I'm the exhumantainer.
Last couple of years have been working in genesis in the line of education.
These are days for AOV calls.
Yeah.
We're going to continue the engineering.
I hope to talk a little.
If I could use the body.
Okay.
And I'm Raymond Roche.
I'm the general manager for Drunker Foundation.
I'm employed by the Linux Foundation to work in open source robotics.
I've been working in drones for the last decade.
A little bit more than that.
And the co-lead together became really my wire of the IRO robotics work group or community work group.
However, whatever the name is right now for the Ross IRO robotics group.
And I'm also leading another project within the within the Linux Foundation called Space Race Linux,
which will build a job to distribution for space devices.
We're interested in space, come talk to me later.
All right.
So brief intro for context for those of you that don't know what this is.
This is Drunker.
We have an autonomy stack.
We've been around for since 2014.
The projects themselves since 2008 or so.
So it's been a while now.
Some numbers, some contributions to put things into perspective.
Last year, we managed to have almost 18,000 contributions.
300,300 pool requests distributed across 100 repositories.
More than 1400 contributors around 220 organizations.
And, like I said, yeah, a lot of repositories.
So it's not just me and my basement.
Companies depend on us to build products.
Fly controllers, which are the real time operating system.
And the embedded controller, which piece for runs on top of this,
piece for runs on top of notexartos.
For these of you that don't know notexartos is a post-excompliant
Artos by the Apache Foundation.
And there's multiple companies building drones from swarming drones
to underwater vehicles and all types of drones.
All right, so I'm assuming you have some context of PX4.
So I'm going to skip the basic introduction.
However, we're going to talk about some basic concepts that you're going to need to
be able to do precision landing with PX4.
And then, stop me if I get something wrong here.
So let's talk about middleware.
Inside PX4, there's a middleware.
So PX4 is not just like a single process running inside the artos.
It's a collection of applications running inside the artos.
As you can see here, and sorry for the people and the screen,
I'm going to have to move.
You got different modules, all of them think of them like apps.
And some of them need to talk to each other because they have valuable information
that they need to share.
So for this, we divide something that's called micro-uart,
which is all middleware.
It's that we got topics, messages, modules, very standard stuff
for a middleware.
It's run in the embedded controller.
Why does this matter to you?
Because we extended this middleware to rawst to through different
I'll tell you in the next slide.
But basically, we extend all of these messages.
Well, some of these messages, the most important ones to you,
so that you can use rawst to be able to modify PX4.
What it actually looks like is something like this.
It's crazy if we have hundreds of modules.
So it's a lot of messages, a lot of things going on in there.
Next concept that it's important to know, it's flight modes.
Whenever you think of a drone that is flying somewhere,
like the one that's in front of you right there,
the very first thing you think is like,
I'm going to control a drone with my RC, right?
Well, you really want to control it with raw power in the RC.
You will have to have the sticks on you and control all the time.
But there's also guided modes that help you alleviate some of those constraints.
We got manual stabilized.
The pilot has the direct control,
but the autopilot stabilizes the inputs position,
which it holds, like it says, the position,
and the pilot commands the velocity.
Have mission.
This is one that is fully autonomous.
You take off, the round goes to some different waypoints
and does mission that you predefined,
and then we have another example, return to landing,
something that you go, you fly,
and then you activate the switch or a mode,
and then it comes back to certain positions that you define.
This is already inside the embedded controller.
There's nothing noble in here.
But we recently added a new feature called external flight modes,
but allows you to do is basically go beyond the flight controller,
for when you do not want to modify the explorer inside the module,
inside the flight controller, inside Ross, in Linux lands.
You connect, and you actually have access to all of the middle
messages like I said, and we have a really nice system that allows you
to define flight modes, similar to what the ones that are defined
in the flight controller, and then it's an orchestration
that allows you to predefined certain movements, certain sets,
and you can match and choose between the internal and external flight modes.
I didn't skip anything in there, right?
Yeah, okay.
I promise you the boring part is going to be over soon.
Now, let's talk about how do we actually move
for middleware to Ross?
So we're using actually the DDS for this.
Micro-XRCDDS is the Ross to Bridge.
We take all of the topics from New York,
and when we have an a macro-XRCDS client running inside PX4,
and then we assume that you have on the mission computer,
somewhere else in Linux land, an agent, we connect both of them,
and then we expose our middleware messages to Ross.
Now, this is the raw messages from PX4.
So you have no context of what's going on in PX4.
You have to go to the source code to be able to know what messages
to use and how to modify the behavior.
Otherwise, you'll be shooting a blanks into the wall,
without knowing what's happening.
We also support CNO for this,
so we have not only have a DDS,
support we also support CNO.
We say it is experimental, not because it's a fragile,
it's because it's you.
We haven't tested it as much, but it is available,
and it's fully supported.
All right.
You want to take over the raw screen interface?
Yes.
Is your other maintainer?
No.
That one.
Yes.
So for making easier for user to interact with Ross Chu
and not to have to subscribe to the internal PX4 topic,
I'll tell you on the company that is deeply involved
in this sort of development as developed
the Ross Chu interface library.
So this library abstracts most of the hinders,
inert PX4 components,
and with a simple API that you can just hold
from your host tuners,
let you particularly explore
the interfaces, the control and navigation,
and the mission interface.
We've seen a bit of set of data
and how we can actually do that.
But in particular, control just set
to create your custom modes,
the custom mode will have activation,
growth routines, update routines,
and combination routines.
And the nice part about it is that
when you have your mission planner,
you're not a mission planner,
you're going to do a station.
So the software normally, you have on the ground,
that controls the domain, make sure everything is going well.
Your new mode will actually appear there,
so you can trigger it when you want.
Of course, you can do it from the last show,
which will move also from other part of the system.
On top of the control part,
you can also have the navigation part,
because indoor navigation,
all the navigation without GPS,
you always have to provide new data
to PX4 position data,
can come from Islam,
map matching,
any sort of feature.
So you keep the heavy intense part on the show,
and just then send a data to PX4,
because it's really happy.
It's easy to get to use all of that.
And you can also do, like,
mission with more, even more high level part of it.
For the position landing example,
we'll see data,
we'll use just the
water navigation,
meant to see control part,
using the control part,
to drive the drone where we actually want it.
Simulation gazebo,
this was put a lot of effort in using,
been directly integrated into gazebo,
from gazebo class,
to a new gazebo.
We do it by passing.
Last show,
we pick so itself in,
because the user,
user transport,
so we have our motor plugin,
sensor plugin,
some of them are,
we actually defined them,
combined and linked to gazebo,
like optical flow plugin.
For example,
and this allows us to just run simulation without having,
actually,
the extra dependency of,
from,
from our shoe.
Of course,
if you also want to connect,
our shoe,
then you can use the as a CDS agent,
or the design interface,
and then you have access to all of your
washrooms.
Thank you.
So I'm guessing the video is not going to play.
Hmm.
That's why.
This video just shows you the takeoff.
Okay, it actually works.
So,
if you've got,
if you saw there,
I moved as a slide,
triggered a takeoff,
this is the,
the contestation,
on this,
side here,
on,
down there,
it triggered,
actually,
this was,
where the pixel simulator was running,
oh,
okay,
and it received a takeoff,
in simulation,
a date takeoff,
and the raw shoe interface,
now we show them as this box of panel.
So we have the raw shoe interface,
connection running,
just a couple of packages to do,
causing information,
and then,
a box of 3D panels showing that
they don't take off,
we expect to,
our predefined map play.
No,
so nothing novel here,
just showing you how the stack works,
and everything is connected.
So click on the side part,
yes?
All right,
so let's talk about px for control,
with raw shoe.
So how do you actually control the fly controller?
Why do we want to control the fly controller from boss?
So we can enable more advanced autonomous behavior,
remember,
running in the embedded controller,
you're a bit constrained on resources,
so moving to Linux land,
you'll be able to,
use other type of hardware that is more,
performance,
you can integrate with raw shoe,
raw shoes as I'm huge,
ecosystem as you're all aware,
you can test new control strategies,
you can go beyond what's inside px force,
you can modify the behavior,
but there's some risk in here,
you have,
you can blue some of the fail saves,
Px4 has a lot of fail saves,
embedded inside of it,
so if you are only using the middleware inside
Ross,
you can blue some of the fail saves,
Px4 has a lot of fail saves,
embedded inside of it,
so if you are only using the middleware inside Ross,
without using the API,
you might lose some of those protections in the fail saves,
if it's unstable,
unstable flight,
if set points are wrong or the rate of,
delivering those is too slow,
so direct control obviously means that you're,
directly responsible for the vehicle behavior,
and yes,
please test in simulation first,
but,
okay,
so when you want to take on this,
so how do we actually control Ross 2,
we have 2 modes,
the old one was off board,
data approach,
we no longer recommend you do,
but it,
because it actually exposes a lot of control,
and it can be a bit dangerous,
so we have made a new API for this.
Yeah, exactly,
so that's all,
thanks for the second one,
the older approach,
maybe you are familiar with marbling,
so all the approach was meant to be used,
by marbling,
and we are the internal of Px4 components,
to do the transition,
so it was by safe,
the moment we will move that
and expose everything to Ross 2,
then it becomes,
machines here,
extremely powerful,
if you know what you're doing,
but at the same time,
by,
by the dangers,
so here they come,
the two depicts for Ross 2 interface,
the custom mode,
so,
you,
when you are working on it,
it just exposes the,
the mode ways,
the mode ways will interact with,
with the bridge to Px4,
again,
DDS or,
or Zeno,
it's suppose,
compare behavior,
autonomous logic,
it's,
let you control Px4 in many different ways,
because Px4 force runs its own control inside,
from,
rate,
attitudes,
stabilization,
velocity position,
emission,
so you decide,
how deep do you want to go,
if you want to control the single propeller,
find by me,
make sure you're publishing that one kilo or so,
and the communication is safe,
and,
and you have access to the tail safe,
and the life cycle,
yes,
of the vehicle,
basically,
that's,
that's a bit,
the interface,
exposes shoe,
to different entities.
One is the mode,
to mention,
to use a bit for,
so they are,
just,
enabling,
so the internal,
px4 modes,
you can switch to them,
you can define,
starting,
running,
stopping condition,
and it's,
basically,
the brain,
control,
the behavior of the dog,
it has,
in that moment,
one mode,
we see later,
we'll be the procedural ending mode,
but you could do,
we have,
like, a precision,
we point,
following,
like,
multiple modes together,
and,
specific logic to translate,
one of them,
okay,
you can create your own,
find a machine,
or,
or similar,
or you can use the,
the,
like,
and,
the,
actually cognitive condition.
So they comes onto the mission flow,
of a,
of,
of your location,
Um,
Yeah,
we should skip forward,
because we're running at the time.
these example,
but,
we see the nice,
Demulator.
Yeah,
owania build
for,
so let's talk about,
perception quickly,
before we jump into the actual,
Okay, let's skip.
Yeah, we can skip.
So this is just a primer and a loop of markers, just in case, I guess we can skip.
All right, this is the pipeline that we're going to follow in the example now, so I cannot
read from here.
Can you read?
Yeah, I can read it.
So, Ding Azebo, we have our own, this demo, down facing camera, so we bridge that with
us, this is the bridge over to us, that we have our last shooting order, the bullet, our
contract, that just simple, open, civil function, it might process, macro detection, and
distortion.
Yeah, for the second mission, we have the pose, we can skip out again over to us network
and maybe also publish just a frame, and you might be attached, a marker on top of it.
All right, so now the group is finally, 20 minutes to get here, yes, we can do it, so, all right,
so we're going to be landing on a Rucom marker, so what do we actually need to do here?
We're obviously we need a camera, we need to be able to look at the Rucom marker, and then
find the position, relative to the drone, and then find the drone in top of it, and
then find to land it, right?
So that's what's basically going to happen.
Do you want to take this one?
So just a bit of a call unit on its form, of course, the Rucom detector, our Rucotlacker
doesn't know anything about the Ruconis, so it's having the Rucom target, the Rucom position,
this part, which we respect for the frame of the drone, sorry, to the frame of the camera,
of the frame of the camera, so we need to do a bit of conversation, we actually want
the target in the world frame, for this example, we are assuming that the fix for to GPS
or anything similar as it knows where it is, it has an option position, therefore, to call
into a form, we can actually get the position of the target in some global reference, or
local, depends on how you consider your map.
Yes, and we'll see at the code example of the actual formula.
So we have some sealed code, because the actual code doesn't fit in the slides, so we
defined the class, called PrecisionLand, and we are extending the mode base that we were
talking about in the API, and this is basically the signature, we have a state, we're
going to switch states from searching to approaching, to descending, to finished, and then
we have an update set point class that is actually going to be switching the stage for
us.
Again, the search is going to help you fly to find the Rucomarker, or approach here is basically
going to spiral around an area to try to find an Rucomarker, once we find it, we're
going to position on top of it, like I said before, and then we're going to try to land.
All right, so yeah, you take this one and I'll take the next one.
Sure.
So for this position landing mode, we form the Rucomarker, we are only interested in the target
calls, probably by the camera tracker, Rucomarker, and also we actually go dive to Rucomarker
in this case, just to get this topic, it's called the vehicle land detected, is one of
the internal effects for topic messages, and it contains information, if the pixel
of things is landed all in air, we're going to use it to detect when we are fully, fully
on the ground, and terminate our mode.
From the Rucomarker interface library standard, we get a trajectory set point, we'll use
this subject to send a set point to click for, vehicle attitude to do the, the, the
information, and vehicle local position, vehicle completion, and vehicle attitude to do the
frequency-concephome and the PID for the position approach.
All right, so now the optics, say, point, let's talk about the search, so we're going to
have a search waypoint, and we're going to update the position of the drone, based on
this search point here, we're not putting in how we're doing this spiral, imagine something
is feeding this spiral, and it's giving you the search waypoint, and then we're sending
that to the API, we'll be able to update position to PX4, and then there's a state machine
inside search, it says if the attack is visible, finally, then you're going to switch
to your approach, and I promise you the code is available later, you can download everything,
it's real code, not serial code.
In the approach, it's very simple, you get the target, which is getting the tag position,
and the approach altitude, which is predifying by you, and then you get the trajectory set
point again, and you say, okay, here's the local marker, again, just send this position
back to PX4, PX4 will position itself into this altitude, they do their fine, and then
you transition to the next one, now this is some magic here, so you get the velocity, and
you have a PID controller here to actually control the velocity, let's jump into that, Ben.
Yeah.
All right, do you want to take this one?
Yes, so of course, we would have just said the typical, so, just stay above the
U.C. marker, and use the internal PX4, but you want to allow the user to actually
resume them to have a better tracking of that specific position in X, Y, so we just implement
a simple PID controller, so computer velocity, proportional gain, inter-ideal, computer velocity,
integral gain, make sure you don't want to go 1,000 meters per second, because you set
the game wrong, and return it, at the point, directly update, we call it a velocity, the API
is, as a kind of distinguish, we can tell it position of velocity over the level, you prefer,
so in this case, we update it well, with velocity, and if at some point we are landed,
we just, we, we, we, we terminate the modern, we go back to it.
Let's show the video now, then, this was just a simple flow, sequence flow, but why don't
you show it like that?
Two minutes, no rush, like them all, time constraint, you can do it, nice, so the
level simulation, apologies for the zoom, yeah, this is, this is supposed to be an
aruko marker on the floor, we attach the X4 to it, we just run in bash script on the other
on this screen, trust me, it will appear, I don't know, yeah, this is up here, I just
want to zoom in a little bit, yeah, then you go, I see a run, yes, this is done, perfect, so now
it's always running and we can actually connect two gans on the floor to it, why
mean I thought yes, I do have time, yes, plenty of time, hoping to question of absolute, so
you don't go to it, we actually connect to our beautiful, what is the last one, what
is the last one, yeah, so we are connected, why to fly, you position mode, why in position
mode, because I actually have a joystick connected to it, so I can fly it manually and
I will do that if I find my, in a minute, yeah, let me go back here, lunching just a bit of
auxiliary mode, what are you doing there, lunching mode, ordering your food dinner,
okay now they lunch another lunch another lunch, I can actually connect to the fox
hello, that's true, give you please fox food, you know, it already failed, it's freedom fox
or fox bloodslave, oh, fox bloodslave failed, oh no, no, it didn't, it didn't, it didn't, yeah, it didn't,
it didn't, it didn't, it didn't, it didn't, it didn't, it didn't, don't worry,
okay show the fox blood then, yeah, I want to show them the fasma, you got me, you got 16 seconds,
16 seconds, why don't you just lunch the drone and said okay, and I just lunch the one without fox
then nine, five, five, five, five, and no question at all, so let me know this,
there we go, oh, you can take off and just use it to see if you take off,
yeah, you can go up, okay, now I'm not doing this a bit out of the photo, I'm at four meters
down there, you can see six meters with too much oil, you know, then if you have questions,
I'm in a stand next to Sefer and then you can come and ask,
but now it's a moment of delaying the thing, so if I go to position here,
you can see the battery resistance, position a custom, this is the custom mode that is running
rush, just click it, moving down, and then, yeah, and then fine set and then it goes in and
lands, yeah, again, but plus again, most importantly, there's a code here, if you want to see the
code, you can scan this in this repository, thank you.
Yeah, oh, I think I think, thank you again for the speaker.