The problem

I am trying to estimate the 3D pose of a person who is observed with a single camera and 5 worn IMUs (limb extremities and upper-back). The camera frames are converted to shape-based feature vectors, and the IMUs each provide 4D quaternion representations of their orientation.

I have recovered the 3D pose using each modality by learning a mapping from the input feature space to the output pose space. Now I wish to obtain better results by combining both modalities in some way through sensor fusion.

I have tried appending the feature vectors of each modality and also using a weighted average of their outputs. These are very simple approaches, and only resulted in very small improvements on average.


What other approaches can I try to combine these two incommensurate data sources?

Is there any preprocessing on the features that should be done?

Note: My preference is to continue using a learning-based approach if possible. (i.e. I do not want to explicitly model the physics/kinematics/etc)


2 Answers 2


The Kalman filter is usually used for data fusion; you are in for a lot of work, though! Modeling the system will save a lot of time.

We used a Kalman filter where I used to work for people tracking with IR sensor array images. About two man years of work went into the project.

  • \$\begingroup\$ I was hoping to avoid using the Kalman filter. I have used it in the past for some basic tracking applications. Formulating a state transition model for human pose will lead to very a difficult problem given the complexities of human kinematics. Instead, I was thinking along the lines of projecting the features into a common meta-space and then trying to integrate them somehow; or even simpler, using a nearest-neighbours approach to find poses closest to both sensor types and interpolating or learning a mapping using this reduced (common) example set. Any other ideas? \$\endgroup\$
    – Josh
    Jun 23, 2011 at 13:01

Just some ideas, not sure how useful they are:

If you want to use a machine learning approach, the challenge is finding the right criterion.

The easiest to work with, is ground truth data, but I suppose you don't have that.

The other generic way for machine learning is using a form of smoothness criterion. One way I can think of is to predict all measured parameters (at a given time) from a smaller set (lower dimension) of variables. Both the transformation from input to low-dimensional variable (matrix or parameterized function) and the values for the smaller set of variables can be learned by, for example, a generic non-linear optimization method. You only have to define the criterion.

If that works, then at least you have some sensor fusion working. Still, you'd have to make a transformation from the learned, lower dimensional samples to the desired output variables.

Instead of, or in combination with, using fewer variables, you could also enforce smoothnes over time of the set of learned variables. It should be possible to include this into the learning method above.

If you want the learned variables to have meaning, you can include a penalty into the criterion to force them to be similar to the input samples.

About the learning methods: The easiest way to start is to define a criterion, and optimize using a standard-nonlinear optimizer (from e.g. matlab or python's scipy package)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.