Long-Range Depth Model - OAK-D-LR

[Luxonis-Brandon]

Pre-order on shop here

Start with the why:

While the OAK-D Series 1 and 2 (here) cover medium depth ranges (from ~20cm to 16 meters), and the coming OAK-D-SR (short-range) series (USB #241, PoE #244) specializes in close-in depth (from 0 to 1 or 2 meters), OAK/DepthAI technology is actually capable of much longer depth sensing - and we've tested up to 350 meters (as below) when doing a 30cm stereo baseline:

And the only current way to accomplish this is to use OAK-FFC (e.g. OAK-FFC-3P or OAK-FFC-4P with modular cameras such as the OV9282; 1MP Global Shutter Grayscale, OV9782; 1MP Global Shutter Color, or AR0234; 2.3 MP Global Shutter Color.

This works, but is not "production ready" and dealing with FFC cables is just annoying. So although it is possible to make a wide-stereo-baseline + narrow-FOV stereo pair using the options above (as shown below) such a setup is larger and harder to integrated than is desirable in many cases.

For longer ranges, using the setups above, we've found that a stereo baseline of 15cm (2x the 7.5cm baseline of the OAK-D Series 2), coupled with variable optics, can cover quite a wide range of depth sensing needs. And also from this testing, we've found the AR0234 to be quite beneficial for long-range sensing, given its large pixel size (matching the OV9282) while having a higher resolution of 2.3MP, which effectively doubling the maximum depth sensing range compared to the OV9282.

The AR0234 also provides the benefit of supporting both global shutter grayscale (for maximizing low-light performance) and also global shutter color (for native pixel-aligned RGBD).

The desired maximum depth range various quite a bit per application - with some situations requiring 100 meters, others 200 meters, and some 300+ meters. (The furthers the Luxonis team has tested with the AR0234 is 350 meters.) Supporting M12-mount lenses in the design enables choosing optics with FOVs (fields of view) corresponding to the required sensing distance.

Move to the how:

Leverage existing DepthAI ecosystem support for AR0234 to implement a dual-AR0234 M12-mount OAK-D-LR.

Move to the what: [NEW; based on feedback below]

• triple AR0234 [NEW]
• Camera Spacing {right} <-5cm-> {middle} <-10cm-> {left} [NEW]
• Gives 3 stereo baselines: 5cm, 10cm, and 15cm for short, medium and long-range depth
• M12-mount lenses
• M12 lock rings (to keep focus/alignment)
• Front Gorilla Glass [NEW; Replaceable front for various filter options by unscrewing front]
• USB3.1
• 802.3at PoE [NEW]
• IP67 Sealed [NEW]
• Aluminum Enclosure
• OAK-SoM-Pro based so can support Myriad X and KeemBay equally [NEW]

[ynjiun]

Welcome OAK-D-LR!

I would suggest to include following few features:

add polarizer filter lenses option: since this long range version primarily is for outdoor application, there might be glare caused by the windshield glass, or raining road surface reflection, etc. Adding polarizer might remove or attenuate the glare significantly.

keembay: wish this new product using latest 3rd Gen VPU!

Ship with 2 sets of lenses: I would suggest to ship this product with two sets of M12 rectilinear lenses: One set of Narrow HFOV (e.g. around 40 to 50 deg) and one set of Wide HFOV (e.g. around 80 to 120 deg). Also the unit should store manufacturing calibrated matrixes for these two sets of lenses. The shipping configuration for example could be preinstalled Narrow_FOV lenses and the default LensMode.Narrow_FOV, such that the rectifiedLeft/Right will assume to apply the Narrow_FOV stereo calibrated matrixes. If user would like to change the lenses to Wide_FOV, then in programming, after user changed the lenses themselves and then need to set LensMode.Wide_FOV to produce correct rectifiedLeft/Right output.

The reason of requesting to ship two lenses sets is a convenient product configuration for an end user like me. Such that I don't need to go out and do the rectilinear lenses sourcing myself and sometime cannot get a quality rectilinear lenses with low quantity like 2 lenses ; )

Synced Left/Right requirement: I am not sure the current OAK-D left/right sync requirement in terms of time difference. I would suggest that new product left/right sync frame time difference < 100us. If it can be less than 50us or smaller would be even better. When the left/right exposure start/end time not in sync, the disparity map will not be accurate for moving objects. The problem will be amplified particularly when the ego view is turning (with angular speed): for example, when a car is turning at the cross road, say at 90 deg/ 5.4 sec => 16.6 deg/sec, then at around 25 m away, there will be disparity error by 1 pix for every 1 ms of out of sync left/right exposure. The further the distance, the bigger the disparity error will be (at around 250m, then it will be amplified by 10X..)

Future product roadmap:

PoE: wish will have IP67 enclosure PoE version available soon.

[Luxonis-Brandon]

Thanks @ynjiun !

Add polarizer filter lenses option: since this long range version primarily is for outdoor application, there might be glare caused by the windshield glass, or raining road surface reflection, etc. Adding polarizer might remove or attenuate the glare significantly.

This is the beauty of M12. This makes it possible to simply buy M12 solutions that have this capability built-in.

keembay: wish this new product using latest 3rd Gen VPU!

Great point. We can likely design this using the OAK-SoM-Pro or OAK-SoM-MAX to have KeemBay support.

Ship with 2 sets of lenses: I would suggest to ship this product with two sets of M12 rectilinear lenses: One set of Narrow HFOV (e.g. around 40 to 50 deg) and one set of Wide HFOV (e.g. around 80 to 120 deg). Also the unit should store manufacturing calibrated matrixes for these two sets of lenses. The shipping configuration for example could be preinstalled Narrow_FOV lenses and the default LensMode.Narrow_FOV, such that the rectifiedLeft/Right will assume to apply the Narrow_FOV stereo calibrated matrixes. If user would like to change the lenses to Wide_FOV, then in programming, after user changed the lenses themselves and then need to set LensMode.Wide_FOV to produce correct rectifiedLeft/Right output.

Great suggestion on lens types/etc. and agreed on the FOVs here. One catch is that once the lenses are removed, the calibration is invalidated. So probably what is best here is to have just 2 different purchase options as the default for this model, such that they can come pre-calibrated. And then there's a decent chance that for a given purpose these might be "Good enough". And then a no-lens option for folks who need their own lenses (so as to not waste lenses/etc.)

1. Narrow FOV (e.g. around 40 to 50 deg), lenses pre-installed and lock-ringed in place (likely with breakable staking), factory calibrated.
2. Wide FOV (e.g. around 80 to 120 deg), lenses pre-installed and lock-ringed in place (likely with breakable staking), factory calibrated.
3. No lenses. Lock-rings included in gift box.

We'll then also sell lenses on our site, just like our friends OpenMV do, here. Note they even have the polarization filter as well, which would very likely work here (not yet tested of course, as we haven't even started work on OAK-D-LR yet).

The reason of requesting to ship two lenses sets is a convenient product configuration for an end user like me. Such that I don't need to go out and do the rectilinear lenses sourcing myself and sometime cannot get a quality rectilinear lenses with low quantity like 2 lenses ; )

Agreed. Good points. Doing them as separately-orderable options will save cost. And it's required for factory-calibration to be applied and usable.

Synced Left/Right requirement: I am not sure the current OAK-D left/right sync requirement in terms of time difference. I would suggest that new product left/right sync frame time difference < 100us. If it can be less than 50us or smaller would be even better. When the left/right exposure start/end time not in sync, the disparity map will not be accurate for moving objects. The problem will be amplified particularly when the ego view is turning (with angular speed): for example, when a car is turning at the cross road, say at 90 deg/ 5.4 sec => 16.6 deg/sec, then at around 25 m away, there will be disparity error by 1 pix for every 1 ms of out of sync left/right exposure. The further the distance, the bigger the disparity error will be (at around 250m, then it will be amplified by 10X..)

Yes. We'll do hardware sync here. My understanding of the OAK-D sync is it's likely in the sub-1-microsecond range. It gets very hard to actually test that though in terms of pixels. But as far as our testing has revealed (leveraging MIPI readouts/etc.) we're at least under a couple microseconds. And the same would be true here.

PoE: wish will have IP67 enclosure PoE version available soon.

Agreed. We'll do a PoE as well. I actually think PoE will be more important here. USB is a tad quicker to get out to test optics/etc. so we'll simply do that first - to learn the unknown unknown here in terms of the discovery that always occurs on a new product like this.

Thanks again!
-Brandon

[stephansturges]

I would be very interested in this product as well, especially the PoE version with an IP67 enclosure.

On the subject of filters and mechanics I would love the option to order the camera with a set of two "front plates", with the glass on one of these plates incorporating a polarization layer directly if that's possible.

Or alternatively (but more complicated) having a gasket-sealed "hatch" on the side of the assembly that would allow to slide a filter in front of the 3 lenses, held in place by a groove for example. This could then also allow the use of other filters for specific wavelengths of light or ND filters to reduce exposure without mounting anything to the outside of the sensor.... but I realise this second option is more complicated as the use of filters would depend on the size of the lenses and sensors on the board.

[ynjiun]

@Luxonis-Brandon

Yes. We'll do hardware sync here. My understanding of the OAK-D sync is it's likely in the sub-1-microsecond range. It gets very hard to actually test that though in terms of pixels. But as far as our testing has revealed (leveraging MIPI readouts/etc.) we're at least under a couple microseconds. And the same would be true here.

Question: is that possible to sync all 3 cameras including the center 4K camera with the other two stereo cameras?

Center 4K camera position: for this 15cm baseline version, I would recommend the center 4K camera position at right side with distance to right camera of 5cm and distance to the left camera of 10cm. Plus if possible to sync all three cameras, then we might have a stereo system with 3 baselines: 15cm, 10cm, and 5cm in operation simultaneously. If the software and keembay hardware throughput can handle 3 pairs of stereo disparity computation, then we could have a hybrid system that fuse 3 disparity maps together to generate a much higher accuracy disparity map. What do you think? I would happy to join the prototyping work on this front if you can send me an early prototype ; ))

[Luxonis-Brandon]

On the subject of filters and mechanics I would love the option to order the camera with a set of two "front plates", with the glass on one of these plates incorporating a polarization layer directly if that's possible.

I love this idea! We will do this for sure.

And yes, I do like the idea of like separate front covers for this. So that they can still be sealed.

Thanks,
Brandon

[Luxonis-Brandon]

For this one we were thinking only 2x AR0234 global shutter color 2.3MP. So no 4K rolling shutter at all. Thoughts on that?

[stephansturges]

For my applications rolling sur if any kind is a no-go, and 2MP global shutter sensors would actually be ideal for balance between light sensitivity and quantity of data :) *Stephan Sturges*

…

On Thu, May 26 2022 at 6:04 PM, Luxonis-Brandon < ***@***.*** > wrote: For this one we were thinking only 2x AR0234 global shutter color 2.3MP. So no 4K rolling shutter at all. Thoughts on that? — Reply to this email directly, view it on GitHub ( #247 (comment) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AE3BDJR2UTXF7RIS4HSFUI3VL6OILANCNFSM5V6OC6ZQ ). You are receiving this because you are subscribed to this thread. Message ID: <luxonis/depthai-hardware/issues/247/1138735751 @ github. com>

[gtech888AU]

Brandon,
Have you considered adding a design feature which would allow users to be able to modify the camera angles to face inwards rather than being forced to keep them parallel. This would open up usage of this model across different depth ranges (both near and far) due to its ability to customize the lens into much narrower FOVs if desired - to date, this is something your other cameras besides FFC all lack.

[Luxonis-Brandon]

Hi @gtech888AU ,

So actually for stereo disparity depth having cameras angled-in prevents the disparity depth from working properly. As with disparity depth the key is for the cameras to see the information from the same vantage point, just displaced. And with angled cameras, one camera will see the left side of an object, and the other will see the right - and so feature matching won't work.

That and any angle between the sensors reduces FOV of the depth. So it's a disadvantage to do so all around.

Hi @stephansturges,

Thanks, makes sense. And PS I edited your post to remove private information.

Thanks all,
Brandon

[ynjiun]

For this one we were thinking only 2x AR0234 global shutter color 2.3MP. So no 4K rolling shutter at all. Thoughts on that?

I think no 4K rolling shutter is fine.

Then I would suggest to enhance this unit with an extra AR0234 global shutter (thus total of 3x AR0234) at the following position:

left <- 1.5 cm -> middle <- 13.5cm -> right

So the left/right baseline is 15cm
And middle/right baseline is 13.5cm
And left/middle baseline is 1.5cm

All three cameras are in h/w sync.

[Luxonis-Brandon]

YES! That's a great idea @ynjiun . As that gives both long-range and not-long-range depth out of the same model.

And with that idea I'm now pondering what the "right" distribution of baselines is.

My gut is something like the following might be super useful.

left <- 5cm -> middle <- 10cm -> right

As this gives the 3 following baselines:

1. 15cm
2. 10cm
3. 5cm

The why of this distribution is that the "middle" baseline of 13.5cm and widest baseline of 15cm feel very close in terms of usable range. And likely are. Whereas with something like this spread, the "middle" baseline is a bigger jump from the widest baseline, with 10cm covering a more-different depth range than 15cm.

And then similar to 5cm, that can allow fairly-close in depth compared to both 10cm an 15cm.

Thoughts?

Thanks,
Brandon

[ynjiun]

@Luxonis-Brandon

left <- 5cm -> middle <- 10cm -> right

Agree. This would definitely address those who need different baseline in a meaningful seperation.

[Luxonis-Brandon]

Fascinating. Thanks. I just updated the specs at the very top with the culmination of discussions here. Added [NEW] for things that came from the discussion.

[stephansturges]

I would love to know if there is already a tentative timeline attached to this project?

[Luxonis-Brandon]

Unknown as of yet. We're triaging internally on when we can implement this. We'll circle back when we have a better view of schedule.

We're also triaging interest on this vs. other models. So this helps that there's interest. And if you want to pre-order, we can get you links for that. :-). Helps with customer voting. ;-)

[microaisystems]

@Luxonis-Brandon

Yes, send me the pre-order link.

[Luxonis-Brandon]

Thanks @microaisystems . Will get it up now. :-)

[Luxonis-Brandon]

It's live here:
https://shop.luxonis.com/products/oak-d-lr-pre-order

(The LR will have to be quite a bit more expensive because it has the (expensive) 2.3MP global shutter color and (expensive) large/nice optics, which then also make the whole thing bigger (and more expensive).)

[stephansturges]

This is like computer vision catnip to me... I've ordered 2 units 😄

[ynjiun]

@Luxonis-Brandon

Great! Before I order d-lr, couple questions: would this pre-ordered unit shipped with Myriad X or KeemBay? would this unit have Raspberry Pi Module 4 integrated as well? Thank you.

[Erol444]

Hi @ynjiun,
We will likely design the OAK-D-LR with the OAK-SOM-PRO, so users will be able to choose either the Myriad X (BW2099) version or KeemBay (DM2399) version:)! Thoughts?
Thanks, Erik

[stephansturges]

The option to use either module would be very interesting to me, but especially if they are user-replaceable.
ie: I really want to get working with the Keem Bay architecture but there are things I would like to test immediately which I know will work on the Movidius version without breaking any of my code.
So the ideal scenario would be to have the option to "upgrade" to the leem bay by swapping out a module, if at all possible.

[Erol444]

Hi @stephansturges,
I believe these would indeed be user replaceable! As we designed KeemBay version of OAK-SOM-PRO so it's backwards compatible with the MyriadX version of OAK-SOM-PRO:)
Thanks, Erik

[stephansturges]

That's ideal, thanks.

[stephansturges]

Hey guys! I see on the product page in the store that you are targeting Jan 2023, is this still correct? If there's any update ton the timeline I'd love to know more :)

[Luxonis-David]

Quick update on the project, we have sent the design to FAB house and we are estimating to get back the first prototypes in a month or so. Below are a few renders a sneak peak to what we are working on.
Please note those renders does not necessarily represent the look of the final product and any changes can be made without prior notice.

[stephansturges]

Looking good! I can't wait to use this.

[chad-green]

YES! That's a great idea @ynjiun . As that gives both long-range and not-long-range depth out of the same model.

And with that idea I'm now pondering what the "right" distribution of baselines is.

My gut is something like the following might be super useful.

left <- 5cm -> middle <- 10cm -> right

As this gives the 3 following baselines:

1. 15cm
2. 10cm
3. 5cm

The why of this distribution is that the "middle" baseline of 13.5cm and widest baseline of 15cm feel very close in terms of usable range. And likely are. Whereas with something like this spread, the "middle" baseline is a bigger jump from the widest baseline, with 10cm covering a more-different depth range than 15cm.

And then similar to 5cm, that can allow fairly-close in depth compared to both 10cm an 15cm.

Thoughts?

Thanks, Brandon

@Luxonis-Brandon the renderings look pretty awesome. What did you guys decide on for the baselines?

[Luxonis-David]

@chad-green we used 5cm for shorter and 15cm for longer baseline

[Luxonis-David]

@ynjiun At the moment I don't really have anything that would ease the calibration besides that you can use the parameter -ih to invert/mirror the horizontal axis.
We are also working on the TV calibration where you have three LCD screens being positioned orthogonally with the same origin. This will ease the calibration a lot, but I think we are not there yet so we could calibrate the OAK-D-LR with TV calibration approach.

CC: @themarpe

[chengguizi]

@themarpe Any updates on hardware sync all three AR0234 cameras on the LR units? Is it already supported, or still being worked on? Thanks!

[Erol444]

Hi @chengguizi , could you please create Issue/feature request on depthai-core? Just for tracking purposes for our development team.
Thanks, Erik

[themarpe]

@chengguizi currently available as a temporary change in modular-calibration-lr on depthai repo if you install requirements. But will soon land in develop of Core as well

[Erol444]

Perhaps it has already been merged to develop (commit here)?

[themarpe]

Correct, also in latest develop now:)

[chengguizi]

@Erol444 Ok let me see if it is indeed not done yet!
@themarpe Could you shine some light on what is the change to make it work? Will this change also applies to all other SoM based product, that carries all-AR0234 sensors? Or it is only OAK-D-LR specific, and Luxonis has a way to detect it is a OAK-D-LR?

[themarpe]

@chengguizi at the moment this is an OAK-D-LR specific. A common FSync between all sensors must be available, which then drives the sensors to capture at the same time. We are yet to expose this mechanism in a general way however

[chengguizi]

@themarpe Ok, got it! Would it be possible to expose at all? I guess it is implemented through triggering from VPU's GPIO pin at fixed interval?

[themarpe]

@chengguizi
Yes it'd be possible (yes it's driven by a VPU pin) - however generalizing this isn't currently on our agenda for rvc2 at the moment. Feel free to reachout to our email if commercially/business critical.

[stephansturges]

Hey guys, is there any update on the IP-rated version of this sensor? I'm getting close to the point where I will be able to test this in real-world conditions and IP rating would make that a lot easier!

[Luxonis-David]

@stephansturges yes we did improve the IP rating by existing design adding rubber sealings, improving rain cover...
We are validating the design on which IP5X can we rate the enclosure with these days and next production batch will be made with the latest design improving IP rating.
Will keep you posted!

[stephansturges]

Is there any

@ynjiun we are still working on the depth quality so I wouldn't say that recalibrating the device at the moment would give any huge difference on what you already have in hand WRT depth quality. If you still want to recalibrate the device you can follow this steps:

1. checkout modular-calibration-lr branch
2. Install requirements in requirements.txt
3. Run calibration script setting the parameters according to your target, example:
   ./calibrate.py --board resources/boards/OAK-D-LR.json -s 8.05 --defaultBoard -cd 0 -c 1 --fps 60

Please note that you should use a bit bigger target for OAK-D-LR because of the larger baseline. A bigger TV would also do it for a start.

Is there are specific branch that I should use to try the latest improvements in depth quality for the LR sensor?
I also have one module that is badly calibrated, so any tips / best practices for calibration would be great too.

[themarpe]

CC: @njezersek

@stephansturges we'll be merging modular-calibration soon, that also adds a rescaling of the AR0234 sensors resolution down to 1280x800, which we saw had biggest implications on the calibration quality.

That should help with calibrating

[stephansturges]

Hey guys, is there any update on the IP5X rated-version, or future availability of the current version? I will probably need better IP rating and will have to make my own case anyway, but I need to start planning :)

[Erol444]

Hi @stephansturges , We actually have that in plan already - our next production batch OAK-D-LR will (should) be IP5X rated :)

[stephansturges]

@Erol444 yep I saw that but I think I will need IP63 at minimum so I will need to make my own enclosures probably, which is why I was curious on when the first samples from the final design will be available for me to work with 😄

[jingnanshi]

Any information on when will the camera be in stock? We would like to try this camera in our robotics lab for outdoor depths sensing. @Erol444

[Luxonis-David]

@jingnanshi we will start shipping the cameras shortly.
We were doing extra testing and improved calibration procedure so it took us a bit longer.
Below is a photo of few pcs we just packaged and should be distributed next week, hopefully you get one as well (if you placed an order already).

[jingnanshi]

@Luxonis-David Thanks for the info! We haven't placed an order yet but I'll pass the news to my colleagues. We will probably place an order soon.

[GoldenZephyr]

@Luxonis-David Do the OAK-D LR cameras support hardware time synchronization / triggering? I see that many of the other Oak cameras do, but I don't see any reference to it for the LR.

[Luxonis-David]

@GoldenZephyr unfortunately OAK-D-LR was not targeted for applications where time synchronization is needed, so you are right, it does not support the hardware FrameSync signal out of the box.
Maybe you can use NTP software host time syncing protocol though, if your timings does not need to be really precise.
Thoughts?

[ghost]

Hello, are there any updates on the shipment. We order a couple of these and were just wondering when they would be sent.

[Erol444]

Hi @robsan7777 , I believe we plan to ship these in January.

[ghost]

Is that the first shipment that will be done, or old orders were shipped already? We ordered them about 3 to 4 months ago so I am just trying to get an idea of when we will get them. Thank you!!

[Erol444]

Hi @robsan7777 , is your order number 15425? If so, then we should start shipping these in a week or two, we are now working on the calibration of these devices. We apologize for the delay.

[ghost]

Yes, that is my order number! Thank you for the update.

[jimas95]

@robsan7777
running on the same issue, any solutions ?

Also the installer for windows is not working for me

[ghost]

@jimas95
I was hoping to have an SDK for the camera for easier integration but I do not think there is yet.

I tried multiple methods and ended up setting the ros_depthai_driver on ROS Iron since it is supported in just a few distributions. Was able to only see live camera images but not depth or anything else. It is a relatively new camera, not sure if they are still working on the drivers and supportive software for it.

[ghost]

@jimas95
Figured out the initial problem, when using the ethernet port from my laptop directly with the camera it does not find it sometimes. I used now a USB to Ethernet adapter and the depthai_demo works fine