1
\$\begingroup\$

I need to synchronize 4 multiple MIPI CSI Cameras. I'm using the following MCU https://www.nxp.com/products/processors-and-microcontrollers/arm-microcontrollers/i-mx-rt-crossover-mcus/i-mx-rt1050-crossover-mcu-with-arm-cortex-m7-core:i.MX-RT1050

I would like to know the proper way of connecting multiple CSI Cameras and synchronizing between them.

The idea also is to use an IMX8 processor as Main CPU and use the imxrt01050 as slaves that are connected to the cameras.

Which setup is possible with correct synchronization between all of them ? Kindly note I plan to use 1080p resolution at 30FPS.

\$\endgroup\$
8
  • \$\begingroup\$ can you elaborate on "synchronization", what does that mean to you? \$\endgroup\$ Commented Jul 11, 2021 at 10:54
  • \$\begingroup\$ @MarcusMüller For video analytics, it is often important that the camera sensors capture the frames at the same time. \$\endgroup\$
    – John
    Commented Jul 11, 2021 at 12:47
  • \$\begingroup\$ yes, I get that, but define "at the same time": is that synchronization on a 100 ms scale? 1ms ? 10 ns? \$\endgroup\$ Commented Jul 11, 2021 at 13:00
  • \$\begingroup\$ @MarcusMüller 10ns scale, I need to do stereo matching and I need a very little delay \$\endgroup\$
    – John
    Commented Jul 11, 2021 at 13:19
  • 1
    \$\begingroup\$ 10 ns: are you sure? How fast are the objects in your scenery moving? I think you just picked the lowest number I gave you without thinking about it... \$\endgroup\$ Commented Jul 11, 2021 at 13:22

1 Answer 1

2
\$\begingroup\$

Seeing you need synchronization only on a frame basis, this should be rather straightforward: nothing in your microcontroller needed to initiate the camera giving images takes in the order of 1/60 s.

So, you'd just write firmware that takes some start signal (for example, simply a GPIO input going high) and tells the camera via I²C to start sending data. If you share the same trigger with all microcontrollers, this should be rather straightforward, and the timing insecurity, especially if the triggering happens in an interrupt, should be far below 1/FPS.

Regarding how to connect 8 of these cameras to one application processor: well, tough one, because this is a lot of data.

The 100 Mb/s ethernet integrated in your MCU is not fast enough for uncompressed full HD at 30 FPS (30 F/s * 1092*1080 px * 12 bit/px is about 283 Mb/s). So, that can't work.

The only other fast interface is the USB2 High-Speed interface, which, with 480 Mb/s is nominally fast enough for 12 bit per pixel, but you'll find that it's not trivial to write USB firmware fast enough. So, you'll need to write a firmware that deals with camera data, and makes sure it is DMA'ed into the USB peripheral buffer. Of course, you'll have to implement the USB stack. NXP does have an app note on how to implement a USB Webcam style device with your MCU, but it does 640x480 at 20 fps, not six times that resolution at 1.5 times that frame rate.

Then, you will have to aggregate all that data over USB at your central processor. I doubt you've made a good choice there – you will need, bandwidth-wise, a high-speed USB controller per camera, and your central processor doesn't have 8 of these.

So, all in all: I think both your MCU and your central controller are the wrong tools here.

Realistically, you'll want an FPGA on which you can instantiate 8 MIPI CSI interfaces, and then have one link to your central controller CPU; probably via multiple PCIe lanes. That'd solve the USB bottleneck, but it does not solve the fact that this is much data, and it's very questionable your cute little quad-a72 core processor (since your workload is fully identical 8 times, the littleBig little-cores cores don't help that much) will be able to deal with 8 camera streams. Don't forget that having one or two video processing chains does not make dealing with 8 video streams simple, by any means.

\$\endgroup\$
14
  • \$\begingroup\$ There are only 4 cameras not 8, maybe that assumption make things little easier ? \$\endgroup\$
    – John
    Commented Jul 11, 2021 at 13:49
  • \$\begingroup\$ not really. The interfacing problem remains exactly the same. This is simply very much data. \$\endgroup\$ Commented Jul 11, 2021 at 13:50
  • \$\begingroup\$ Can you suggest an alternative CPU same from NXP to achieve that ? Thanks so much! a great analysis \$\endgroup\$
    – John
    Commented Jul 11, 2021 at 13:51
  • \$\begingroup\$ an FPGA solution is out of the choice, as we don't have an FPGA guy or solution that does it \$\endgroup\$
    – John
    Commented Jul 11, 2021 at 13:53
  • \$\begingroup\$ well, you don't have a choice, sorry. You simply can't do full HD at 30 fps and bit depths > 12 bit with a device that can't even use the full USB2.0 High speed bandwidth. You need a fast interface to your central CPU, no matter what you do. sometimes the architecture you've made in your head doesn't fit your system requirements - this is the case here. Have you even tested whether your stereoscopic application would run in real time on your i.MX8 when the data is already in RAM? if not, you should do that- \$\endgroup\$ Commented Jul 11, 2021 at 13:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.