Annotation data added in the python plugin not shown in the GUI

Python SDK
1

Hi all,

When I add the mesh or point cloud generated in the following code to the Data List, it doesn’t get shown on the GUI. The label map I add (self.mask) works fine.

    @imfusion.Algorithm.action(display_name="To Mesh")
    def extract_meshes(self):
        if hasattr(self, 'vessel_mask_path') and self.vessel_mask_path != Path("") and os.path.exists(self.vessel_mask_path):
            self.mask = imfusion.load(str(self.vessel_mask_path))            
            self.mask[0].modality = imfusion.Data.Modality.LABEL
            print('Loaded label!')
            # self.imageset_out.add(self.mask)
        else:
            print(f'Failed to load label! {self.vessel_mask_path}')
            return imfusion.Algorithm.Status.ERROR
        mesh = imfusion.execute_algorithm('Seg.LabelToMesh', self.mask)
        pcds = imfusion.execute_algorithm('Base.MeshToPointCloud', mesh)
        print(pcds)
        pcd = imfusion.execute_algorithm('Base.CombinePointClouds', pcds)
        print(pcd)
        # imfusion.app.execute_algorithm('Mesh.MeshDistance', mesh)
        imfusion.app.data_model.add(pcd)
        imfusion.app.data_model.add(self.mask) 
        return imfusion.Algorithm.Status.SUCCESS

When I replaced imfusion.execute_algorithm with imfusion.app.execute_algorithm, I got a warning:
Could not dereference all input data while recording algorithm Base.CombinePointClouds
and still nothing was added to the data list.

Just to add a bit of context, the final goal would be to load the GT segmentation, and run mesh distance algorithm to compare it with the output of the segmentation network (which we get after calling compute).

First, can you please tell this code should add the merged point cloud to the data list, or if I should do anything differently? and second, considering the final goal, can I do it without adding the PCD or meshes to the data list?

Best,
Farid

2

Hi Farid,

thanks for reaching out. The issue here is that historically Meshes and PointClouds are not considered “Data” in the Suite. They are managed by the AnnotationModel instead of the DataModel but the python bindings currently don’t offer a way of adding something to it. The reason why the call to data_model.add does not fail outright is because we are currently transitioning them to be treated as proper data and they are already inheriting from Data now but the DataWidget cannot handle them.

I think in your case, you could use imfusion.mesh.point_distance direclty without having to add anything to the data or annotation models.
If you do need them to be displayed in the Suite you could use the workaround of writing the mesh / pc to disk first and then using imfusion.app.open afterward.

I hope this helps but let me know if you need anything else.

Cheers,
Ilja

3

Thanks Ilja for looking into it. I will try your suggestion.

Best,
Farid

4

Hi Ilja,

Your solution helped a lot, but now I have another question in continuation. I am hoping that I could get this representation in the attached image below using the python code. My workaround right now is to get the meshes, save them and load them again, all in python, but the final stage would be to select the two meshes in the GUI and call visualize the distance. Is there a better way to do this through code? If it is possible using a cpp plugin, please also let me know.

image
image1920×992 126 KB

Best,
Farid

5

Hi Farid,

I am afraid that I don’t see a better you at the moment due to the limited interactions with the annotation model through Python. As for the C++ side I am unfortunately not very familiar with that part of our codebase. I will ask around in the Vision team.

Cheers,
Ilja