Order of adding frames to the Reconstruction class with multiple sensors

RecFusion
1

I assume that with one sensor, we should pass depth/colour images to the Reconstruction::addFrame() function in the same order as we have retrieved them from this sensor. However, what happens if we use e.g. 3 sensors (marked as A, B, C)?

  • Should we use the order of the acquisition as this order has implicit time ordering? (A1, B1, C1, A2, B2, C2, …; numbers indicate the image order on a given sensor)
  • Can we mix a bit like this: A1, B1, C1, C2, B2, A2, or any other A-B-C combination for the same image number?
  • Or the best is the order of overlapping sensors (e.g. if A overlaps with C, and C overlaps with B then A1, C1, B1, A2, C2, B2, …)?
  • What happens internally, if we add all images from sensor A firstly, then B, then C: A1, A2, …, B1, B2, …, C1, C2, …?

In short, what is the recommended way, what could work, and what will not?

2

Hello Patryk,

it is important that the timeline is consistent, and that there is an overlap between a newly added frame with the rest of the reconstruction that has already been made.

So, if 1, 2, 3 indicate the timestamps (i.e. frames A1 and B1, C1 are taken appr. at the same timestamp 1, frames A2, B2, C2 at timestamp 2), then if A overlaps with C, and C overlaps with B

  • the ideal combination is:
    A1, C1, B1, A2, C2, B2
  • other combination is also possible:
    A1, B1, C1, A2, B2, C2 (even if A overlaps with C, and C overlaps with B, it’s ok because the reconstruction might skip B1, but already B2 will be added since it will have some overlap with C1)
  • you shouldn’t use the following combination:
    A1, A2, …, B1, B2, …, C1, C2, …

Best regards,
Olga

1 Like
3

Hello Olga,

Thank you! This is what we have been suspecting, but it is always worth getting the authoritative answer :slight_smile:

Best regards,
Patryk

4

Hi @gavranovic

With a parallel acquisition of depth frames, I have noticed that sensors acquire frames at various speeds. For this reasons, I might get e.g. 100 images on the first (A) and the second sensor (B), and 101 images on the third sensors (C).

For the first 100 images it is straightforward, we use the e.g. the ideal combination you have mentioned. But what about the 101st image? Should we add it or leave it? What if the third sensor acquire 2 or 3 more images than the rest? Should we still add them to the Reconstruction class?

Best regards,
Patryk

5

Hi Patryk,

you can add these additional images. It’s only important that they are close in time and space to the the recently added frames. It doesn’t matter if the number of frames is not equal for all the sensors.

Best,
Olga

1 Like