Useful Scripts

Vtk to obj converter for converting slicer atlases to blender / osg loadble objs.   vtk2obj.py

Needs a Python and vtk install.

Targets:

 

-Short term:

• Rendering at interactive frame rates (10+ fps)

• Labelling

• Manual dissection to show structures (transfer functions, organ extraction etc...)

• Shading

• Ability to load/stream large datasets(x times vram?, just go into sys ram?)

• Basic interface

 

-Long term:

• automatic processing

• ease of use

• measurement tools

• export interoperability

• plugins

• etc...

 

Aims(quoted from the “contract” I got):

• To develop new 3D Anatomy system.

• To use existing data to produce some 3D images that could be used for anatomy teaching.

• prepare a pilot of these materials for medical students.

 

 

Aspect to Learning anatomy:

 

• Areas: the body has regions and spaces which communicate and with each other, their boundries, relations and contents need to be known.

• Organs: The locations, shapes, surfaces, attachments, blood supply, lympatics etc....

• Vessels

• Nerves

• Joints: types, range of movements etc.

• Muscles: Actions, origins and insertions and innervation.

• Surface Anatomy: surface landmarks and relationships of inner structures relative to these.

• Bony landmarks.

• Vertebral Levels

• Corelation with radiological anatomy: what structures look like on scans (we need to have a switch between rendering the raw scan and the lit labelled transfer functioned volume) to teach this.

 

Integrating these in the models via labels and overlaid polygonal models would be comprehensive enough.

Grouping the labels under the above categories will make them more manageable (disable/enabling their rendering would make things less cluttered and fine tuned to intended area of interest.)

 

limitations: I doubt we will be able to visualise nerves or lymphatics (MRI and really large nerves maybe yes)

 

A good website of some anatomy concepts that are usually not covered well.

 

http://www.instantanatomy.net/

Deadline for above aims: 26^th August, basically we need a model or too that are clear and easily understood before the open day where the prototype will be shown. Also there will be an educational meeting at an unspecified time(i'll find out) where this will be shown.

 

I included possible todo's with their priorities after each section, the priorities are in realtion to the dead line and not necessary with the lifetime of the project->

 

Implementation Details

edit: i had a look on the specs of the Mac pro, i dont think we'd need the level of optimisation i mention here :-)

Rendering of data

Input data conversion to renderable format:

 

Possible sources of data include CT, MRI and cadaver sections, we have to decide about the internal format we use for rendering.

Format:

 

Input-8/16 bit scalar for CT and MRI, RGB8 for Cadaver sections.

 

Internal format- LUMINANCE8?

 

Pow2 clamping is currently in OSGVolume. Why? We arent gonna do trilinear filtering. Removing empty space around the main bulk of the data can make each slice non pow2 but will save allot of VRAM. Was this implemented for the DXT compression?

 

Cadaver sections dont have a wide variety of colours if they are photographed under diffuse lighting conditions (like visible human), we can get away with using an 8 bit texture and a LUT (in shader lookup or some sort of palette texture) for storing the colour information. I experimented with this in GIMP for palllete generation and made a difference map between a true colour and indexed slice, the result was very weak noise; no structures were lost.

 

Possible todos:

 

-Decide on the optimal internal format:High

 

Paged Data

Even volumes of 512*512 per slice don't show important anatomical structures the since about 1/25 to 1/125 of the volumes data is wasted in air around the subject. Paged LOD of textures is very important to have full potential in education. The Nasa Blue Marble example in OSG seems to have done this Texture LOD? Streamed from the hardisk, which is exactly what is needed. I`ll need to find out how it works.

 

Possible todos:

 

-get paged textures working with volumes:Medium ( we can cut chunks from high res data to see the smaller structures) for the time being.

 

 

Lighting:

 

lighting is very very very important in the perception of surfaces, without lighting the data can be very confusing.

precalculated lit abdomen, note how the lighting brings out the surfaces even when the volume's diffuse is just white.

 

 

No surface lighting:

The cheapest option but organs of similar density will seem to merge together since there are no depth or surface cues. Maybe if we have this lighting mode add fogging/distance attenuation.

 

Precalculated lighting:

 

This is the 2^nd fastest lighting option, that can be as complex as needed except with no specular highlights. If no coloured lights are used the lighting can be stored in an 8 bit LUMINANCE.

 

NB: the lighting volume and the scalar volume dont have to be the same resolution, might save allot of memory here.

 

Surface based lighting:

 

Here we load the normal in someway per voxel and read it in the fragment program.

 

-3 component normal

largest and most taxing on memory and fillrate, can be DXT compressed.

 

-2 component normal

normals should be of length 1, this can be exploited to store only 2 component of the normal, generating the normal again in the fragment program is easy.

 

Frag code

 

vec3 nr = nrv.xyz;

// unpacking and make them signed

nr.x = (nr.x - 0.5) * 2;

nr.y = (nr.y - 0.5) * 2;

float xy = dot(nr, nr);

float z = 1 - xy;

nr.z = sqrt(z);

//needed?

nr = normalize(nr);

 

-1 component normal

using an 8 bit index into a 1D normal texture of size 1*256 can be used to store normals in just 8 bits. The 1Dtexture only needs two components ( x and y). the shader code is the same as above except replace y with a if using a LUMINANCE ALPHA texture. Lowest bandwidth and memory cost.

 

C code

 

inline unsigned char resolveNormalIndex(float x,float y)

{

float xNr=(0.5f * x) + 0.5f;

float yNr=(0.5f * y) + 0.5f;

 

return ((yNr*32)+(xNr*16));

}

 

this code takes precalculated x and y normals and generates an 8 bit index value for the lookup in the 1D texture.

 

Here is what the 1D texture looks like(I painted it in 2D because its easier):

 

 

Generating the normals:

 

Currently OSG has a normal generator. I just get a black volume with some white speckles from running it with normal generation on. Maybe its the data im using? From what I see its central difference so its the same one I used to generate the image a few paragraphs above 0_0;

 

Other lighting considerations:

 

-Subsurface scattering

 

Even though this is hard in polygonal graphics, its easy in volume rendering; just make all the transfer function pixels 80% transparent, and at the borders between materials make that 50% transparent. The lighting contribution from obscured data will shine through.

 

-Shadowing

 

Yet another hard in polygons easy in volume one. Its quite easy if one limitation is placed on the light: must be at eye position. This requires MRT FBOS, and one of textures attached to the FBO must be bound to our volulme all the time while we are rendering to the FBO. Render each slice to the FBO as normal for gl_fragcolor[0],gl_fragcolor[1] will contain the opacity of the rendered voxels, this gl_fragcolor[1] is going to the bound texture mentioned previously. This allows each rendered quad(slice) to read the opacity of the previous slices. Since rendering is back to front for transparent objects max opacity will be at the nearest slices; the least shadowed ones. Reverse the value and modulate with colours on gl_fragcolor[0] to make shadows. Got this from a paper.

 

 

Possible todos:

-Normal generation: High

-Decide on normal storage: High

-Shading:High

-Subsurface scattering: High because its so easy.

-Shadowing:Low

 

Rendering Modes:

 

-CT: no transfer function applied just the CT scalars as the blend factor.

-Alpha blend: simple enough, allows structures to obstruct each other and be removed by transfer functions.

-MIP: takes the highest intensity fragment at a pixel and displays that(this is done per component), useful since it mimics the behaviour of x-rays, part of Aim "Corelation with rediological anatomy".

-Depth MIP?: I made this mode up, its basically MIP but only for a limited distance into the volume, this would be very useful in making surface anatomy models without fiddling too much in transfer functions. this inst a mode in opengl nor OSG.

 

 

Possible todos:

 

-CT:High

-Alpha blend:High

-Activate MIP projection option via state setting in OSG: High

-Investigate depth MIP, possible way is to render the volume with mip and a distance from camera falloff on each pixel:Medium

 

 

Clarification of data

 

Transfer functions:

 

Transfer functions will allow the user to select certain structure to show, also it save VRAM since 1 scalar can lookup a 4 component vector. 1D transfer functions are enough as a start. From my tests it is better to have a transfer function with more pixels than the volume has values since texture filtering will generate intermediate values.

 

-automatic

Automatic transfer functions might be good for quick generation of models. Here is one possible implementation, generate a histogram of all the values. Log10 all the histogram values. Each of the highest peaks represents a material. The low plateau shows a noisy material, usually air/connective tissues on CT's, these palteus areas can be made 100% transparent in the transfer function to show more important stuff.

elimnating diffrent tissues by clmaping above the diffrent peaks on the log10 histogram. (grey matter, white matter,subcutaneous fat)

 

-semi automatic

This involves just reading the CT, noting down the intensity of the organs of interest and painting a transfer function, simple and easy to implement and allows per pixel control over the transfer functions.

 

Problems

The biggest problem with the above two methods is that some organs (esp in abdomen) have all the same intensity.

 

-manual

This requires a more manual approach. I experimented a bit and came up with this method of colouring similar organs differently.

 

Segment required organ->compare segmented volume with full volume->common values are shifted to unused intensities.

 

For example:

 

kidney, spleen and pancreas have a density of around 55-65, I segment the kidney and have it in a separate volume. I compare the kidney only volume with the volume with other organs, the common voxels are kidney voxels, shift their intensity by +140. This gives us a volume where kidneys get their colour from transfer function pixel 200 instead of 60. The kidneys now have a separate colour from similar density organs.

 

BUT!

 

We have moved the kidneys out of their natural intensity!,around the kidney is fat of intensity 30-45

 

Since we have bi linear filtering on this will cause the area between the fat and the kidney to be an average of both intensities (120) pixel 120 of the transfer function may not have the right colour, this will coat the kidney in a wrong colour! Hence when painting transfer functions always remember these “intermediate areas”.

my manually semgmented volume, i had to turn off texture filtering to prevent the wrong values from appearing.

 

 

Possible todos:

-get osgVolume to load 1D transfunctions from image files: High

-implement manual segmentation in OSGVolume: Medium

-automatic transfer function generation: Low

 

Clipping

clipping is important since it takes away unwanted information quickly and is used often in traditional anatomy teaching(saggital ,coronal and transverse slices of cadavers).

 

-OSG clipping

This is simple planar clipping, plane are infinite though and may take off too much info. Saves fillrate.

 

-Bounding box in shader, this is easy to move and handle, does not save fillrate. Min and max are vec3 uniforms.

 

if (gl_TexCoord[1].x < max.x)

if (gl_TexCoord[1].y < max.y)

if (gl_TexCoord[1].z < max.z)

if (gl_TexCoord[1].x > min.x)

if (gl_TexCoord[1].y > min.y)

if (gl_TexCoord[1].z > min.z)

//texkill

 

Labelling

 

-rendering

the labels should be text with autotransform applied in OSG, line primitives connect them to points in the volume.

-interactvity, labels should be clickable and jump to other slides/ bring popup windows of information.

 

 

Superimpose/mix polygonal Graphics

 

In learning anatomy it is sometimes helpful to simplify spaces / surfaces of organs into cubes and other simple geometric shapes (axilla pyramid,middle ear cuboid,bladder domed pyramid etc....) superimposing the volume with hand modelled polygonal graphics would help. This wont be hard from a programming point of view since hopefully OSG will render the non transparent polygons first then the volume(no z test trouble since volume has no z test).

 

Something along the lines of this picture.

Labels coming off the faces of the polygonal model would be the best.

 

Sphere segments in OSG could be used to demonstrate the range of joint movements! This would seriously be the best type of label I have seen( as in compared to other anatomy resources) for displaying joint movements.

open scene graph sphere segment can show limits of angular movement.

Superimpose/mix another volume

 

PET scans are usually superimposed onto CT/MRI, this may require the slices of the two volumes to be interleaved properly.

 

Hollow organ extraction

 

I have made code for this, here is the result. This helps organs like bowels stand out. Quite good for diagnosis too, I can see two problems with that bowel with my untrained eye.

only hollow organs are visible here.

 

User Interface

 

Slice scrolling:

Implement slice scrolling in the volume by shifting Tcoords in the vertex program.

when the data gets shifted to the edge of the rendering brick, you get free clipping.

 

 

Slice highlight:

 

this feature makes all slices very transparent except one slice of interest, can be implemented by using a uniform float variable and falloff from the slice of interest. (the slices should have different gl_Vertex.z coords so its easy to calculate transparencies of the slices on the gpu )

 

Multiaxis slice generator:

 

CT scans generate transverse slices, we can generate slices in all 3 directions by having 3 single quads sampling the volume texture.

 

Camera manipulator:

Current OSG trackball manipulator seems fine

Head tracking? wiimote joystick (pitch and roll)?

 

Possible todos:

 

-all of the above:High

 

User tools

 

 

 

Other possible uses

 

Interventional radiology trainer.

 

 

Misc Rendering notes:

 

Opacity modification

Im guessing this can remove allot of the aliasing

 

Ray tracing and polygons:

If the ray tracing is purely making opaque iso surfaces gl_FragmentDepth can make the ray tracing fragments pass the depth test againt the polygon fragments. Make gl_FragmentDepth = to first opqaue voxel depth in world space.

 

Jitering eye position for rays or sampling TCoords for plane style rendering reduces the number of needed samples.

 

Papers and ppts

 

  kniss_tvcg02-small.pdf

 

http://www.vis.uni-stuttgart.de/vis03_tutorial/

 

dicom offis

This define makes the compiler use the right config file for the system #define HAVE_CONFIG_H 1