Annotation data added in the python plugin not shown in the GUI

Python SDK
1

Hi all,

When I add the mesh or point cloud generated in the following code to the Data List, it doesn’t get shown on the GUI. The label map I add (self.mask) works fine.

    @imfusion.Algorithm.action(display_name="To Mesh")
    def extract_meshes(self):
        if hasattr(self, 'vessel_mask_path') and self.vessel_mask_path != Path("") and os.path.exists(self.vessel_mask_path):
            self.mask = imfusion.load(str(self.vessel_mask_path))            
            self.mask[0].modality = imfusion.Data.Modality.LABEL
            print('Loaded label!')
            # self.imageset_out.add(self.mask)
        else:
            print(f'Failed to load label! {self.vessel_mask_path}')
            return imfusion.Algorithm.Status.ERROR
        mesh = imfusion.execute_algorithm('Seg.LabelToMesh', self.mask)
        pcds = imfusion.execute_algorithm('Base.MeshToPointCloud', mesh)
        print(pcds)
        pcd = imfusion.execute_algorithm('Base.CombinePointClouds', pcds)
        print(pcd)
        # imfusion.app.execute_algorithm('Mesh.MeshDistance', mesh)
        imfusion.app.data_model.add(pcd)
        imfusion.app.data_model.add(self.mask) 
        return imfusion.Algorithm.Status.SUCCESS

When I replaced imfusion.execute_algorithm with imfusion.app.execute_algorithm, I got a warning:
Could not dereference all input data while recording algorithm Base.CombinePointClouds
and still nothing was added to the data list.

Just to add a bit of context, the final goal would be to load the GT segmentation, and run mesh distance algorithm to compare it with the output of the segmentation network (which we get after calling compute).

First, can you please tell this code should add the merged point cloud to the data list, or if I should do anything differently? and second, considering the final goal, can I do it without adding the PCD or meshes to the data list?

Best,
Farid

2

Hi Farid,

thanks for reaching out. The issue here is that historically Meshes and PointClouds are not considered “Data” in the Suite. They are managed by the AnnotationModel instead of the DataModel but the python bindings currently don’t offer a way of adding something to it. The reason why the call to data_model.add does not fail outright is because we are currently transitioning them to be treated as proper data and they are already inheriting from Data now but the DataWidget cannot handle them.

I think in your case, you could use imfusion.mesh.point_distance direclty without having to add anything to the data or annotation models.
If you do need them to be displayed in the Suite you could use the workaround of writing the mesh / pc to disk first and then using imfusion.app.open afterward.

I hope this helps but let me know if you need anything else.

Cheers,
Ilja

3

Thanks Ilja for looking into it. I will try your suggestion.

Best,
Farid

4

Hi Ilja,

Your solution helped a lot, but now I have another question in continuation. I am hoping that I could get this representation in the attached image below using the python code. My workaround right now is to get the meshes, save them and load them again, all in python, but the final stage would be to select the two meshes in the GUI and call visualize the distance. Is there a better way to do this through code? If it is possible using a cpp plugin, please also let me know.

image
image1920×992 126 KB

Best,
Farid

5

Hi Farid,

I am afraid that I don’t see a better solution at the moment due to the limited interactions with the annotation model through Python. As for the C++ side I am unfortunately not very familiar with that part of our codebase. I will ask around in the Vision team.

Cheers,
Ilja

6

Hello Farid,

I am not sure if I fully understand your wish for a custom implementation, since that there allready is an algorithm to visualize distances between meshes and point clouds in the Suite (which you also seem to have found seeing you other post here).
Should this not suffice for your needs you can try to implement your own custom Plugin which should give you access to the whole C++ API. This example should get you started. In your case you would need to modify DemoAlgorithm::createCompatible to accept two Meshes (data.getSurfaces(), or data.getPointClouds if you want to compare point clouds), make sure that those are passed to the algorithm on/after construction (line 30 in the example) and implement your desired processing in compute. Your algorithm should then show up on right-click on two selected meshes under the “Demo” category (you can modify this in DemoAlgorithmFactory).

Regards,
Thomas

7

Thanks Thomas,

I wanted to automatically visualize the distance without the need for the user to interact with the GUI. I found a workaround as Ilja suggested to extract and load the meshes via the python plugin, and to visualize the distance, in the python code I will load a workspace file that runs the visualization algorithm.
I think this might do the trick.

Best,
Farid