I have a camera rigidly mounted together with an IMU that consists of 2-dof gyroscopes and 3-dof accelerometers. My aim is to compute the global optical flow of the image captured by the camera, in order to do global motion compensation when compressing the camera video stream. Instead of doing image processing to compute global optical flow, is it possible to obtain the optical flow based purely on the IMU readings?
closed as off-topic by Chris Stratton, Elliot Alderson, Finbarr, Edgar Brown, Sparky256 Feb 6 at 4:23
This question appears to be off-topic. The users who voted to close gave this specific reason:
- "Questions on the use of electronic devices are off-topic as this site is intended specifically for questions on electronics design." – Chris Stratton, Elliot Alderson, Finbarr, Edgar Brown, Sparky256
If by optical flow you mean the movement over time that the board does, it's possible to do that; I implemented a tracking algorithm using a Kalman filter over a STM32 microcontroller to get the orientation of an IMU,
but it's a big amount of computation and I'm not sure that it could reduce the load compared to image processing.
I actually think it could be feasible, and in fact it's what camera manufacturer do with image stabilization.
It's not that hard to do it, but I believe it's quite hard to do it well, or at least in a meaningful way;
IMUs are typically oriented to slow-ish and accurate tracking, instead you need to compensate higher frequencies of motion, like vibration and shake. You'll need to tune your tracking accordingly.
In short, you have to convert the x/y/z accelerometer axes and pitch/yaw gyroscope axes to the same set of coordinates, and apply the desired amount of filtering. Then integrate twice the accelerations to get the (linear angular) speed and the velocity of the movement, as desired. That's what you then have to compensate in the video.
You'll probably have to compensate some delays, either in the video or in the sensor readings, to time-match the compensation with the recording.