[Bug] how to load dicom-segment data

[l5769389]

Describe the Bug

the img is from OHIF Viewer。 how can i use cornerstone3D to load the dicom-segment。 or like the export demo can i load the exported file?

Steps to Reproduce

like above

The current behavior

like above

The expected behavior

like above

OS

window11

Node version

18

Browser

chrome

[jlopes90]

Segmentation Volume #664
Segmentation Stack #1059

[l5769389]

Segmentation Volume #664 Segmentation Stack #1059
The link you provided doesn't seem to fully solve the issue. I couldn't understand the code in OHIF-VIEWER, it's too complex. Are there any simpler examples?

[jlopes90]

How import Segmentation Volume #664 (comment)

Edited (27/03/2024)

[l5769389]

Import Segmentation Volume

const { viewport } = libCornerstone.getEnabledElement(element);

await libCornerstone.imageLoader.loadAndCacheImages(imageIds);

const volume = await libCornerstone.volumeLoader.createAndCacheVolume(volumeId, {
    imageIds: imageIds
});

volume.load();

await viewport.setVolumes([{ volumeId }]);

viewport.render();

// "arrayBuffer" is file segmentation dicom
const result = libCornerstone.adapters.segmentation.generateToolState(viewport.imageIds, arrayBuffer);

const derivedVolume = await libCornerstone.volumeLoader.createAndCacheDerivedVolume(volume.volumeId, {
    volumeId: segmentationId
});
const derivedVolumeScalarData = derivedVolume.getScalarData();
derivedVolumeScalarData.set(new Uint8Array(result.labelmapBufferArray[0]));

libCornerstoneTools.segmentation.addSegmentation(segmentationId);
await libCornerstoneTools.segmentation.addSegmentationRepresentation(toolGroupId, segmentationId);

how to get segmentationId?

[jlopes90]

how to get segmentationId?

// It is to create or generate id in segmentation, for example:
const uid = cornerstone.utilities.uuidv4();
const segmentationId = 'segmentation-image-volume:' + uid;

[l5769389]

let me answer the question。demo is base vue 、 offical demo and the data source is i got from ohif-view demo

<script setup lang="ts">
import { getDicomSource } from '@/config/config'
import {
  Enums,
  RenderingEngine,
  Types,
  volumeLoader,
  utilities, setVolumesForViewports
} from '@cornerstonejs/core'
import * as cornerstone from '@cornerstonejs/core'
import { adaptersSEG } from '@cornerstonejs/adapters'
import { metaData } from '@cornerstonejs/core'
import { api } from 'dicomweb-client'
import { onMounted, ref } from 'vue'
import * as cornerstoneTools from '@cornerstonejs/tools'
import { initDemo, createImageIdsAndCacheMetaData } from '@/helpers/index.js'
import { StackScrollMouseWheelTool, ToolGroupManager, SegmentationDisplayTool } from '@cornerstonejs/tools'

const { ViewportType } = Enums
let renderingEngineId = '123'
let volumeId = 'volumeId'
const viewportId = 'viewportId'
let renderingEngine
let imageIds
let volume
let toolGroup
const renderContainerRef = ref(null)
const toolGroupId = 'toolGroupId'
const {
  segmentation,
  Enums: csToolsEnums
} = cornerstoneTools
const uuid = utilities.uuidv4()
const segmentationId = 'segmentation-image-volume:' + uuid

async function getImageIds() {
  const params = getDicomSource()
  // params = {
  //   StudyInstanceUID: "1.3.12.2.1107.5.2.32.35162.30000015050317233592200000046",
  //     SeriesInstanceUID: "1.3.12.2.1107.5.2.32.35162.1999123112191238897317963.0.0.0",
  //   wadoRsRoot: `/dicom-web`
  // };
  imageIds = await createImageIdsAndCacheMetaData(params)
}

async function addSegmentationsToState() {
  const segParams = {
    studyInstanceUID: '1.3.12.2.1107.5.2.32.35162.30000015050317233592200000046',
    seriesInstanceUID: '1.2.276.0.7230010.3.1.3.296485376.8.1542816659.201008',
    sopInstanceUID: '1.2.276.0.7230010.3.1.4.296485376.8.1542816659.201009',
    wadoRsRoot: 'https://d33do7qe4w26qo.cloudfront.net/dicomweb'
  }
  const arrayBuffer = await retrieveDicomData(segParams)
  const result = await adaptersSEG.Cornerstone3D.Segmentation.generateToolState(imageIds, arrayBuffer, metaData)
  const segmentationVolume = await volumeLoader.createAndCacheDerivedSegmentationVolume(volumeId, {
    volumeId: segmentationId
  })
  segmentation.addSegmentations([
    {
      segmentationId: segmentationId,
      representation: {
        // The type of segmentation
        type: csToolsEnums.SegmentationRepresentations.Labelmap,
        // The actual segmentation data, in the case of labelmap this is a
        // reference to the source volume of the segmentation.
        data: {
          volumeId: segmentationId
        }
      }
    }
  ])
  const segmentationVolume1 = cornerstone.cache.getVolume(segmentationVolume.volumeId)
  const scalarData = segmentationVolume1.getScalarData()
  scalarData.set(new Uint8Array(result.labelmapBufferArray[0]))
}

function retrieveDicomData({ studyInstanceUID, seriesInstanceUID, sopInstanceUID, wadoRsRoot }) {
  const client = new api.DICOMwebClient({
    url: wadoRsRoot
  })
  const options = {
    studyInstanceUID,
    seriesInstanceUID,
    sopInstanceUID
  }
  return client.retrieveInstance(options)
}

function render() {
  renderingEngine.renderViewport(viewportId)
}

const initTools = () => {
  cornerstoneTools.addTool(SegmentationDisplayTool)
  toolGroup = ToolGroupManager.createToolGroup(toolGroupId)
  toolGroup.addTool(StackScrollMouseWheelTool.toolName)
  toolGroup.addTool(SegmentationDisplayTool.toolName)

  toolGroup.setToolEnabled(SegmentationDisplayTool.toolName)
  toolGroup.setToolActive(StackScrollMouseWheelTool.toolName)
}

const initRender = async () => {
  renderingEngine = new RenderingEngine(renderingEngineId)
  const viewportInputArray = [{
    viewportId,
    type: ViewportType.ORTHOGRAPHIC,
    element: renderContainerRef.value,
    defaultOptions: {
      orientation: Enums.OrientationAxis.SAGITTAL,
      background: <Types.Point3>[0.2, 0, 0.2]
    }
  }]
  renderingEngine.setViewports(viewportInputArray)
  toolGroup.addViewport(viewportId, renderingEngineId)
  volume.load()
  await setVolumesForViewports(
    renderingEngine,
    [{ volumeId }],
    [viewportId]
  )
  await segmentation.addSegmentationRepresentations(toolGroupId, [
    {
      segmentationId,
      type: csToolsEnums.SegmentationRepresentations.Labelmap
    }
  ])
}

async function createVolume() {
  volume = await volumeLoader.createAndCacheVolume(volumeId, {
    imageIds
  })
}

onMounted(async () => {
  await initDemo()
  initTools()
  await getImageIds()
  await createVolume()
  await addSegmentationsToState()
  await initRender()
  render()
})


</script>

<template>
  <div class="w-full h-[500px] bg-transparent flex justify-center">
    <div class="w-[600px] h-full bg-transparent  relative" ref="renderContainerRef">
    </div>
  </div>
</template>

<style scoped>

</style>

the addSegmentationsToState function is get the segment data and set to display。
and i got metadata get undefined。 i add a provider （you should register it to cornerstone） and save sopIntanceUid to metadata like this：

[jlopes90]

@l5769389

take a look #664 (comment)

[l5769389]

When I use the official data of ohif-view and run the code listed above, I encounter some mismatched images. I think there might be an issue with the implementation in addSegmentationsToState(). Can anyone help me resolve this?

[l5769389]

@jlopes90 i want to modify the segment render color。 but i dont know how to do. can you show me ? it looks like use the api: i got the segment data like this:

const arrayBuffer = await retrieveDicomData(segParams)
  const result = await adaptersSEG.Cornerstone3D.Segmentation.generateToolState(imageIds, arrayBuffer, metaData)
  const segmentationVolume = await volumeLoader.createAndCacheDerivedSegmentationVolume(volumeId, {
    volumeId: segmentationId
  })

[jlopes90]

@l5769389 https://www.cornerstonejs.org/live-examples/labelmapsegmentcolorchange

This is not in the examples list, I just made a PR #1192

[l5769389]

**@jlopes90 here is the way i get segment arraybuffer from PACS System ,am i right? the segment data need be accurate to sopInstanceUID ？

async function retrieveSegData({ studyInstanceUID, seriesInstanceUID, sopInstanceUID, wadoRsRoot }) {
  const client = new api.DICOMwebClient({
    url: wadoRsRoot as string,
  })
  const options = {
    studyInstanceUID,
    seriesInstanceUID,
    sopInstanceUID
  }
  return await client.retrieveInstance(options)
}
```**

[jlopes90]

@l5769389 I don't know, I've never tried PACS

[l5769389]

@jlopes90， Sorry to bother you again, but I've been using your segmentation loading demo. I've found a bug when retrieving DICOM data from a remote PACS (Orthanc) via API (similar to the demoDicom() function) and then reading segmentation data locally. In Figure 1, DICOM data is fetched from Orthanc via an API, while segmentation data is read locally using an tag. In Figure 2, both DICOM and segmentation data are read locally using tags. It's evident that there's a bug in Figure 1. Could you please help me resolve this? all use your demo。 i only change the StudyInstanceUID and SeriesInstanceUID from your demo . I believe there might be some flaws in your example.

[l5769389]

https://deploy-preview-1177--cornerstone-3d-docs.netlify.app/live-examples/segmentationvolume the demo link

[jlopes90]

@l5769389 #1201

Example:
https://deploy-preview-1201--cornerstone-3d-docs.netlify.app/live-examples/segmentationVolume.html

Try editing exConfig in the console or you can make PACS.

exConfig.fetchDicom = {
  "StudyInstanceUID": "1.3.6.1.4.1.14519.5.2.1.256467663913010332776401703474716742458",
  "SeriesInstanceUID": "1.3.6.1.4.1.14519.5.2.1.40445112212390159711541259681923198035",
  "wadoRsRoot": "https://d33do7qe4w26qo.cloudfront.net/dicomweb"
};

exConfig.fetchSegmentation = {
  "StudyInstanceUID": "1.3.6.1.4.1.14519.5.2.1.256467663913010332776401703474716742458",
  "SeriesInstanceUID": "1.2.276.0.7230010.3.1.3.481034752.2667.1663086918.611582",
  "SOPInstanceUID": "1.2.276.0.7230010.3.1.4.481034752.2667.1663086918.611583",
  "wadoRsRoot": "https://d33do7qe4w26qo.cloudfront.net/dicomweb"
};

Run Load DICOM and Load SEG.

Edited (11/04/2024)

[l5769389]

1. i tried your demo, Still no effect, just causing the display to flip.
2. in console set window.exConfig not effect in the link.
   
   this is ohif-viewer's config:

const exConfig = {
  fetchDicom: {
    StudyInstanceUID: '1.3.12.2.1107.5.2.32.35162.30000015050317233592200000046',
    SeriesInstanceUID: '1.3.12.2.1107.5.2.32.35162.1999123112191238897317963.0.0.0',
    wadoRsRoot: 'https://d33do7qe4w26qo.cloudfront.net/dicomweb'
  },
  fetchSegmentation: {
    StudyInstanceUID: '1.3.12.2.1107.5.2.32.35162.30000015050317233592200000046',
    SeriesInstanceUID: '1.2.276.0.7230010.3.1.3.296485376.8.1542816659.201008',
    SOPInstanceUID: '1.2.276.0.7230010.3.1.4.296485376.8.1542816659.201009',
    wadoRsRoot: 'https://d33do7qe4w26qo.cloudfront.net/dicomweb'
  }
}

[jlopes90]

Ops sorry. This is the code I have already tested and it is exactly the same as the one above. #1170 (comment)

exConfig.fetchDicom = {
    StudyInstanceUID: '1.3.12.2.1107.5.2.32.35162.30000015050317233592200000046',
    SeriesInstanceUID: '1.3.12.2.1107.5.2.32.35162.1999123112191238897317963.0.0.0',
    wadoRsRoot: 'https://d33do7qe4w26qo.cloudfront.net/dicomweb'
};

exConfig.fetchSegmentation = {
    StudyInstanceUID: '1.3.12.2.1107.5.2.32.35162.30000015050317233592200000046',
    SeriesInstanceUID: '1.2.276.0.7230010.3.1.3.296485376.8.1542816659.201008',
    SOPInstanceUID: '1.2.276.0.7230010.3.1.4.296485376.8.1542816659.201009',
    wadoRsRoot: 'https://d33do7qe4w26qo.cloudfront.net/dicomweb'
};

[l5769389]

@jlopes90 i copy your demo and tried it again . result seems still error . use the url like above。 you can scroll sag viewer to see the error, it may appear in about index 10/128 or index 118/128 。

[l5769389]

In the volume view, you can see the errors more clearly. Figure 1 mpr is correct, while Figure 2 shows the error that occurred using Cornerstone. they are same source。

[steph603]

@l5769389 did you ever resolve your issue with segmentations displaying misaligned? I am having something similar.

@jlopes90 I have implemented your volume segmentation example (thank you - it's an excellent demo!) - The issue is occurring when I swap from loading in a dicom file from my machine to fetching from Orthanc.

When loading the base images from a local dicom, and request segments from Orthanc (or a local file), everything works as expected.
When I swap to obtaining base images from Orthanc, whether segments are local or from Orthanc, the segments are misaligned as below.

I believe something is wrong or lacking when using createImageIdsAndCacheMetaData for the base images.
Is there another way I can get those base images loaded in that would more closely replicate using a file, like retrieveInstance which works well for segmentations?

[l5769389]

@steph603 I didn't find a solution。 Wait for the author to solve this problem

[l5769389]

my co-worker now find the solution 。you can test: @steph603 ,but it seems still error in 3d。

[steph603]

Update: We found a solution!

I'm aware of this PR that reversed the image IDs and that did seem to partially help, however, there was still something causing our segmentations to display incorrectly when base images were coming from Orthanc - like so:

We discovered that the order of the image ids was being scrambled when retrieving from Orthanc, and while the base images were displaying correctly, segmentations were not aligning.
The solution is to sort the image ids based on the Instance Number (0020,0013)

const sortedInstances = Object.values(instances).sort((a,b) => a['00200013'].Value[0] - b['00200013'].Value[0])

in createImageIdsAndCacheMetaData like this:

Now our segments are displaying as expected:

[sedghi]

I guess i'm gonna close this now that the issue is resolved