Overview of Software Creation - Image Fusion¶

The software team is charged with bringing to OnkoDICOM the functionality of co-registration or Image Fusion .

Image Fusion is the process of orienting two image sets with different timestamps so that the anatomical features present in both are entirely or partially overlaid. It is typical that one scan is fixed (say CT1), and the second (CT2) translocated until the images match. Most current software will allow a manual or an algorithmic match to occur. (DICE coefficients can also be measured.)

The process of image fusion is usually done with a pair of scans, although there may be many scans fused to a 'parent' scan or an 'ancestral tree' of scans. By this I mean that if there are 5 CT scans, they can be fused:

CT1+CT2, CT1+CT3, CT1+CT4, CT1+CT5 ('parental' process)

or in the following sequence:

CT1+CT2, CT2+CT3, CT3+CT4, CT4+CT5 ('ancestral' process)

Neither method is preferred, and neither should be prevented. It is not conceived at this point that more than 2 scans will be fused at one time (exception: see PET/CT below).

A fusion can be undertaken between CT, MRI or PET*. XR to XR could be fused.

I believe that when a fusion of CT2 to CT1 has occurred and been accepted, a DICOM transformation object is defined, which defines for the moved scan distances, and saved into the CT2 directory. **Warning - you must check that this is the case.

It is also usual that when the scans are placed in the same viewing space, they are coloured so that a correct overlay renders the colour into grey scale, and they are roughly centred over each other. Most software allows manual rotation of the mobile image around 3 axes (yaw, pitch, or roll). Once satisfactory alignment is achieved, the positioning is confirmed.

The reason for providing co-registration inside OnkoDICOM is so that Regions of Interest (ROI) drawn on any particular scan can be copied onto any other fused scan.

For example, in Lung Cancer, I might want to answer the question - Can I detect a cancer signal in the lung before I am able to see a lump of cancer? I would take scans from years before and copy today's tumour ROI across so that my search for a signal can be confined to where the cancer started.

For example, in Lung Cancer, radiotherapy scars some lungs but not others. I can make a ROI for scarred lung on a follow up CT, and then copy that back to a fused radiation plan so that I can do analysis of the contained lung and the radiation dose deposition.

My understanding leads me to suggest that you will need to:

• Research open source libraries to deal with co-registration, and manual & algorithmic image matching. Please research PyPlati [[https://github.com/pyplati/platipy]].
• Code for a Co-registration window which will: * load up two scans, the first is the base, the second is mobile * position the scans over the top of each other (?just use the 512x512 grid) * colour & alpha the scans so that the selected colours merge to grey scale when matching values are overlaid (e.g., bone on bone, fat on fat, lung on lung, etc) * allow for the overlay scan to be resized * allow for the overlay scan to be yawed, pitched and/or rolled manually * allow for an algorithm to do the matching (without or after manual correction) * allow the fusion to be accepted, and the transformation object written.

I am thinking that I should provide you with sets of scans that need to be fused - CT/CT pairs, CT/MRI pairs, CT/PET-CT pairs and MRI/PET-CT pairs. Any others?

PET is a special case. Currently a "PET scan" as produced is actually a fused PET+CT image pair. The slice numbers may not match. Although these scans are fused, it is a special case as the scans are produced from physical scanners where the physical distance between the PET and CT detector rings is known and fixed. There is no other movement permitted in the scan (if the patient wriggles up the bed, the result is a mis-registered scan pair. HOWEVER, when it is desired to fuse a PET scan to another (non-PET) CT, the fusion is done to the PET-CT and the same shift applied to the PET:

if you want to co-register MRI and PET, you have to co-register [MRI+(CT]-PET), not [MRI+(PET]-CT)