Image Registration(US-CT) with Imfusion SDK

Noah · April 10, 2025, 7:35am

Dear Imfusion Support Team,

I’m encountering difficulties while using the Imfusion SDK. My task involves aligning a set of ultrasound DICOM files with a set of CT DICOM files. Registration tests performed in the Imfusion Suite yielded expected results, but I’m facing issues when using the API:

Runtime Crash: The ImageRegistration object fails to initialize during runtime.
Compute() Failure: The UltrasoundRegistrationAlgorithm crashes when calling compute().
Suboptimal Results: While FeatureMapsRegistrationAlgorithm runs without errors, the transformation effect on the CT images is unsatisfactory.
config failed:The UltrasoundDISARegistrationAlgorithm and DescriptorsRegistrationAlgorithm error in extracting descriptors

Could you please assist in diagnosing these issues? Thanks a lot for the support!

Best regards,

YuanW

attachments

//The following is the outline of the code
std::vector<std::unique_ptr<SharedImageSet>> images = dicomLoader.loadImages();
SharedImageSet sweep = *images[0].get();
sweep->setModality(Data::ULTRASOUND);
SharedImageSet sisct = *images[1].get();
UltrasoundRegistrationAlgorithm ur(sweep, sisct);
FeatureMapsRegistrationAlgorithm fr(sweep, sisct);
UltrasoundDISARegistrationAlgorithm udr(sweep, sisct);
DescriptorsRegistrationAlgorithm dr(sweep, sisct);
//ImageRegistration I(&sweep, &sisct);
//ur.compute();
fr.compute();
//udr.compute();
//dr.compute();

//*****  The following is the FeatureMapsRegistrationAlgorithm terminal output
Intensity range [0..2435], scaling with 26
Intensity range [0..2666], scaling with 24
Couldn't read OpenCL buffer (-5)
Couldn't write to OpenCL buffer (-5)
Couldn't read OpenCL buffer (-5)
Couldn't write to OpenCL buffer (-5)
Couldn't read OpenCL buffer (-5)
Couldn't write to OpenCL buffer (-5)
Couldn't read OpenCL buffer (-5)
Couldn't write to OpenCL buffer (-5)
Couldn't read OpenCL buffer (-5)
5 evaluations in 0.0798547 seconds, success
-80 -80 -80 -40 -40 -40 -> 0 (-640.279)

Optimization terminated at boundary conditions for parameters 0 1 2 3 4 5

//***** The following is the UltrasoundDISARegistrationAlgorithm terminal output
Intensity range [0..2435], scaling with 26
Intensity range [0..2666], scaling with 24
[DescriptorsRegistrationAlgorithm] Fixed volume preprocessing: 16.607 ms
[DISAFeaturesAlgorithm] Preset model 'DISA-LC2' cannot be used since neither ONNX or Torch engine is available
[DescriptorsRegistrationAlgorithm] DISA Algorithm failed.
[DescriptorsRegistrationAlgorithm] Error in extracting descriptors

//***** The following is the DescriptorsRegistrationAlgorithm terminal output
Intensity range [0..63310]
Intensity range [0..63984]
[DescriptorsRegistrationAlgorithm] Fixed volume preprocessing: 70.434 ms
[ML.ModelConfiguration] Loading config file Models/DISA/Features/volume-Torch.yaml
[ML.Engine] torch is not a registered engine - registered engines are: []. If you are targeting a C++ engine, make sure the corresponding plugin has been loaded. If you are targeting a Python engine, make sure that your Python environment has the relevant framework (pytorch, onnxruntime, openvino, etc.) installed.
[ML.MachineLearningModel] Could not create engine 'torch'.
[DISAFeaturesAlgorithm] Error in running model 'Models/DISA/Features/volume-Torch.yaml'. Could not create engine 'torch'.
[DescriptorsRegistrationAlgorithm] DISA Algorithm failed.
[DescriptorsRegistrationAlgorithm] Error in extracting descriptors

schultezb · April 10, 2025, 2:44pm

Hi Noah,

given your error description, my initial guess is that most likely the used dedicated plugins like ImFusionReg, ImFusionTorch, etc. were not initialized correctly. You must make sure to not only link against them but also initialize them via one of the Framework::loadPlugin() functions.

After successful plugin initialization the log output should print sth like [Base.Framework] Available Plugins: ImFusionAS, ImFusionDicom, ImFusionDicomGui, ImFusionML, ImFusionPython, ImFusionReg, ImFusionSeg, TorchPlugin.

Could you please share the complete log of your application from start to crash? That would allow me to confirm my theory or see if it is sth else.

Cheers,
Christian

matteo_ronchetti · April 10, 2025, 3:33pm

Hi Noah,

Can you explain me which steps you followed to do the registration in the ImFusion Suite?
Does the Ultrasound Sweep come already with tracking from the DICOM file?

A few observations on your code:
SharedImageSet sweep = *images[0].get(); results in a copy and if the dicomLoader is already loading an US::UltrasoundSweep (which is a derived class of SharedImageSet) this copying process will result in the loss of the specific information associated with the US::UltrasoundSweep object. It is better to use the pointer.

UltrasoundRegistrationAlgorithm expects either a compounded ultrasound volume or an US::UltrasoundSweep, if you use it without any of these two I would not expect it to work.

FeatureMapsRegistrationAlgorithm expects multichannel feature volumes (like the ones produced by DISA), you cannot use it directly on medical images. The OpenCL error -5 it produces means that there are insufficient resources for running the registration. This is probably due to the wrong input.

UltrasoundDISARegistrationAlgorithm fails because you lack the Torch and the ONNX plugins (only one of the two would be enough). This could either be because your installer doesn’t contain them or, as Christian said, your installer has them but you need to initialize the ImFusion framework. If you successfully used “Ultrasound DISA Registration” from the Suite, then your installer has these plugins and only the initialization is missing.

DescriptorsRegistrationAlgorithm also has the problem of missing Torch and the ONNX plugins.

I hope this is helpful.

Have a nice day,
Matteo

Noah · April 11, 2025, 10:34am

Dear Imfusion Support Team,

Thank you for your response. I’ve made further attempts, but unfortunately, I still haven’t succeeded.

Here’s what I did in ImfusionSuite.exe:

Imported two DICOM folders.
Modified the Modality of one Data to “UltraSound” (or used Data already labeled as “US”).
Selected both Data and tried the following algorithms from the Algorithms menu:
- ImageRegistration
- UltrasoundRegistrationAlgorithm
- UltrasoundDISARegistrationAlgorithm
- DescriptorsRegistrationAlgorithm
  (Used default settings for “Compute” or “Run”).

Regarding initialization and linking issues, I’ve confirmed there are no problems. I used loadPlugins and observed the correct output for [Base.Framework]. I also tried the release version to ensure our environment matches ImfusionSuite.exe as closely as possible.

Why does it say the torch is not a registered engine ? Why does it work in ImfusionSuite but not in my own implementation? Here are my current suspicions:

Are there any prerequisites for using Torch? How should the engine initialization code be written?
Could it be a Torch version mismatch? How can I verify this? Do I need to manually import Torch?

I look forward to your insights.

Best regards,
[Your Name]

//***** The following log is used DescriptorsRegistrationAlgorithm and UltraSound in Imfusionsuite.exe
[Base.Framework] Available Plugins: ImFusionAS, ImFusionDicom, ImFusionDicomGui, ImFusionLiveUS, ImFusionML, ImFusionPython, ImFusionROS, ImFusionReg, ImFusionRobotics, ImFusionSeg, ImFusionStream, ImFusionUS, OpenGLDemo, TorchPlugin.
[GUI.MainWindowBase] OpenGL: 4.5.0 NVIDIA 566.14
[GUI.MainWindowBase] Vendor: NVIDIA Corporation
[GUI.MainWindowBase] GPU: NVIDIA GeForce RTX 3060/PCIe/SSE2
[GUI.MainWindowBase] Memory: 10.61 GB of 12 GB available
Keeping zero values on GPU resampling is currently unsupported, switching to CPU mode
[DescriptorsRegistrationAlgorithm] Fixed volume preprocessing: 1.015 s
[ML.ModelConfiguration] Loading config file DISA/MICCAI-2023/model-Torch.yaml
[TorchEngine] Torch loading: DISA/MICCAI-2023/model.pt, cuda: Available, mps: Not Available
[TorchEngine] Created Torch model from file DISA/MICCAI-2023/model.pt
[DescriptorsRegistrationAlgorithm] DISA Descriptor computation: 1.663 s
Keeping zero values on GPU resampling is currently unsupported, switching to CPU mode
[DescriptorsRegistrationAlgorithm] Preprocessing moving volume: 186.282 ms
[ML.ModelConfiguration] Loading config file DISA/MICCAI-2023/model-Torch.yaml
[TorchEngine] Torch loading: DISA/MICCAI-2023/model.pt, cuda: Available, mps: Not Available
[TorchEngine] Created Torch model from file DISA/MICCAI-2023/model.pt
[DescriptorsRegistrationAlgorithm] DISA Descriptor computation: 601.403 ms
[FeatureMapsRegistrationAlgorithm] Preparation time: 280.748 ms
[FeatureMapsRegistrationAlgorithm] Total evaluations: 14080 (11267.2eval/sec)
[FeatureMapsRegistrationAlgorithm] Search time: 1.250 s
[FeatureMapsRegistrationAlgorithm]  0.0774383    -4.2241   -7.46743 -0.0178556 -0.0106232 0.00463599 --> 0.715801
[FeatureMapsRegistrationAlgorithm]  -0.112738   -4.30352   -10.2066  0.0363719   0.174332 -0.0285251 --> 0.699772
[FeatureMapsRegistrationAlgorithm]      -0.17   -4.47075   -15.0312   0.303984  -0.234577 -0.0679306 --> 0.634907
[FeatureMapsRegistrationAlgorithm]   0.013052   -2.85628   -11.5988    1.44356  -0.360773 -0.0825905 --> 0.595186
[FeatureMapsRegistrationAlgorithm]  0.106553  -4.19966   13.7745  -0.01288  0.389441 0.0178691 --> 0.459164
Keeping zero values on GPU resampling is currently unsupported, switching to CPU mode
Computing pass 2
Adding Zero-Mask to Abdomen  1.0  I26f  3
LC2 PreProcessing on Abdomen  7.0  I30f  3
53 evaluations in 5.8479 seconds, success
0.0264368 -0.0243417 1.42445 0.00577342 -0.026945 0.0152414 -> 0.806759 (0.780537)

//***** The following log is displayed in my software terminal
[Core.Log] Warning: Logging not initialized, messages are printed to console only.
[SystemFingerprint] Unable to retreive hdd
[Core.GlobalSettings] Failed to synchronize settings: Timeout
[GL.ContextManager] OpenGL: 4.5.0 NVIDIA 566.14
[GL.ContextManager] Vendor: NVIDIA Corporation
[GL.ContextManager] GPU:    NVIDIA GeForce RTX 3060/PCIe/SSE2
[GL.ContextManager] Memory: 10.64 GB of 12 GB available
ImFusionLib Version 3.12.25 built on 2024-12-5.
[Base.Framework] Available Plugins: ImFusionAS, ImFusionDicom, ImFusionDicomGui, ImFusionLiveUS, ImFusionML, ImFusionPython, ImFusionROS, ImFusionReg, ImFusionRobotics, ImFusionSeg, ImFusionStream, ImFusionUS, TorchPlugin.
[GUI.Animations] No global backend set: installing a dummy one.
Directory: "D:/Program Files/ImFusion/ImFusion Suite/SuiteDev_YW/CT"
Directory: "D:/Program Files/ImFusion/ImFusion Suite/SuiteDev_YW/image"
Intensity range [24..239]
Intensity range [24..239]
Intensity range [0..2435], scaling with 26
[DescriptorsRegistrationAlgorithm] Fixed volume preprocessing: 23.411 ms
[ML.ModelConfiguration] Loading config file Models/DISA/Features/volume-Torch.yaml
[ML.Engine] torch is not a registered engine - registered engines are: []. If you are targeting a C++ engine, make sure the corresponding plugin has been loaded. If you are targeting a Python engine, make sure that your Python environment has the relevant framework (pytorch, onnxruntime, openvino, etc.) installed.
[ML.MachineLearningModel] Could not create engine 'torch'.
[DISAFeaturesAlgorithm] Error in running model 'Models/DISA/Features/volume-Torch.yaml'. Could not create engine 'torch'.
[DescriptorsRegistrationAlgorithm] DISA Algorithm failed.
[DescriptorsRegistrationAlgorithm] Error in extracting descriptors

schultezb · April 11, 2025, 11:05am

Hi Noah,

Thank you for the detailed log.
I see that you are working on Windows. It might be that your application consumes different copies of the plugin DLLs: How do you deploy and run your application? Are you using the delayload feature of the linker?

I assume that your application is linking directly against some of the Plugins such as ImFusionReg.lib, ImFusionUS.lib, etc. and therefore the app is asking for the plugin DLLs directly on startup. There are different approaches how to achieve that Windows finds those DLLs.
However, it is critical that during Framework::loadPlugin() you load the exact same DLL and not a separate copy located elsewhere!

Cheers,
Christian

Noah · April 15, 2025, 1:34am

Dear Imfusion Support Team,
Thank you for your prompt and helpful guidance! Following your advice, I resolved the issue—Windows did not find the same dll as the loaded one.
I truly appreciate your expertise and support.Looking forward to future collaboration!
Best regards,
YuanW

schultezb · April 15, 2025, 6:31am

Thank you for your update. I’m happy to hear that you were able to solve the issue.

We encountered a similar issue recently on a different project. It is already on our roadmap to improve the plugin interface so that such situations are avoided in the future or at least we detect them and can notify the user with an error message.

Cheers,
Christian