Depth Relasense Scaling for reconstruction

RecFusion
1

Hi all i work on a simple software that should use a 3d scan as base for more computations, for this “more” i use opencv. Now i have implemented a wrapper for the realsense cameras that interanly keeps the realsense images as cv::Mat.
to call the “addFrame” method successfully the depth image needs to be in the correct scale and the meassurement unit is 0.1mm - if i understand correctly.

what i currently do is

rs2::frameset frameset = m_pipe.wait_for_frames();
    if (frameset.size() == 2)
    {
        rs2::video_frame tmpcolorFrame = frameset.get_color_frame();
        rs2::depth_frame tmpdepthFrame = frameset.get_depth_frame();

        float scaleDepthFactor = m_pipe.get_active_profile().get_device().first<rs2::depth_sensor>().get_depth_scale();

        qDebug() << m_depthScale << " | " << m_depthScale*100.0;

        cv::Size size(tmpcolorFrame.get_width(), tmpcolorFrame.get_height());
        m_colorFrame = cv::Mat(size, CV_8UC3, (void*)tmpcolorFrame.get_data());
        m_depthFrame = 100.0 * cv::Mat(size, CV_16U, (void*)tmpdepthFrame.get_data());

        return true;
    }

and then


            cv::Mat &depthFrame = realsenseCamera.getDepthFrame();

            unsigned char* depthIn = (unsigned char*)depthFrame.data;
            float* depthOut = imgDepth->data();
            // Scale depth to 0.1 mm units
            for (int i = 0; i <imgDepth->width()*imgDepth->height(); ++i)
                depthOut[i] = (float)depthIn[i];
       m_rec->addFrame(0, imgDepth, imgColor, trackingStatus, imgScene, 0, 0, T);

But the reconstruction does nothting and says the tracking is lost. This code already worked with out the opencv in between.

So at the end the question could be, what OpenCV format is directly suitable for RecFusion Matricies and how i can scale the depth image correctly ?

Would be

2

Hello,

in principle there is no reason the SDK would not work with some additional processing of images with OpenCV.

There are following issues in your code:

  1. SDK expects depth image to be in mm units not in 0.1mm, therefore you need to multiply your depth matrix not by 100, but by m_depthScale *1000
  2. Also, cv::Size expects arguments to be (rows, columns). This means that you need to swap width and height arguments.

Fixing point #1 should already make the reconstruction work.

Best regards,
Olga