Vision Systems Design 2(12), December 1997, pp 30-38.
Reproduced with permission. (c) 1997 PennWell Publishing.
By Andrew Wilson, Editor
Sophisticated off-the-shelf visualization tools are allowing researchers and physicians to develop three-dimensional medical visualization systems
In the past, physicians depended as much on physical evidence as laboratory data to diagnose patients. Now, however, medical imaging systems such as CAT, MRI and PET scanners have evolved to the point where it is possible to see certain anatomical structures with very fine detail. High-resolution CT systems, for example, now allow organs to be visualized at resolutions approaching the sub millimeter level. Better still, fusion of data from different imaging modalities can yield images of the structural and functional relationships of such organs.
As such systems become more sophisticated, the data they produce increases dramatically. To understand this data, physicians and researchers need to visualize images of the anatomy in a lifelike way. To do so, they are turning to 3D visualization software to examine, explore and precisely pinpoint problem areas before determining final treatment strategies.
For those bringing such visualization applications to the medical field, the challenge is to create easy to use graphical rendering and analysis tools. To produce fast powerful visualization applications, medical application developers are turning to software development tools that convert data from a variety of medical instruments to interactive visual environments.
Providing such capability does however come at a price. Because the computational requirements and data sets are very large, most third-party packages, such as IDL from Research Systems (Boulder, CO), IMOD from the University of Colorado (Boulder, CO) and the Visualization Data Explorer from IBM (White Plains, NY) require graphics workstation or UNIX-based hosts.
In choosing a visualization package to perform a specific imaging function, developers must carefully evaluate a number of factors. These include hardware requirements, data types and file formats supported, visualization and viewing techniques used, image analysis capability, data fusion and virtual reality. Developers choosing the VoxelView from Vital Images (Minneapolis, MN), for example, must run the 3D volume rendering software on either the R5000SC or R10000SC O2 graphics workstations from Silicon Graphics (Mountain View, CA). At a price of $35,000 including the graphics workstation, VoxelView allows developers to visualize, analyze and manipulate 3D image data, highlight regions of interest, control transparency, color, lighting and create digital animations.
Like VoxelView, SciAn, developed at the Supercomputer Computations Research Institute at Florida State University (Tallahassee, Florida) is also a UNIX-based scientific visualization and animation package that runs on both Silicon Graphics and IBM RS/6000 workstations. Version 1.1 of the software, currently in beta test, supports a number of different data types. These include scalar and vector fields, structured and non-structured grids and datasets that vary over time. In operation, the software can show any number of visualizations on screen at any one time. Viewpoints of different visualizations can be linked, enabling comparison of data from observation and simulation.
Perhaps the most important consideration in choosing a visualization package are the types of file formats supported. Because data formats from MRI, PET, and CT equipment differ, software packages such as MEDx from Sensor Systems (Sterling, VA) support a number of different file formats including data types from Picker MRI, Siemens PET and GE Signa MRI scanners. Also running on a variety of UNIX workstations including the 300 Series from HAL Computer Systems (Campbell, CA), MEDx software is designed to display both 2D and 3D data under X Windows. Release 2.0 of the software includes a functional image analysis module to normalize detect and correct for head motion and an automated 3D registration algorithm developed by Roger Woods at UCLA (Los Angeles, CA) that can register non-coplanar data sets.
Once loaded into memory, a number of image analysis techniques can enhance or measure image data. Typical image processing routines include 3D matrix operations, linear and adaptive histogram functions, 2D and 3D FFT and deconvolution routines and fusion of multimodal images. These are used to both enhance and extract data from the images. To make quantitative measurements on image data, many off-the-shelf packages also include image analysis functions. These include plotting of line and trace profiles, selection and automatic plotting or regions of interest and surface and volume area measurement.
As part of its Analyze AVW package, the Biomedical Imaging Resource at the Mayo Clinic (Rochester, MN) includes a number of techniques to analyze, measure and display 3D image data. Running under UNIX, Analyze allows multiple volume images to be simultaneously accessed and processed by multiple programs in a multi-windows interface (Fig. 1). AVW functions are available to developers in two forms: either as a C-callable library of functions or as extensions to a Tcl/Tk software toolkit. The C-callable library permits the AVW functions to be complied and linked to software systems both statistically and dynamically. The Analyze package is currently available from CNSoftware (Stewartville, MN).
To display two and 3D images, visualization techniques allow data to be viewed as orthogonal or oblique sections, as volume or surface rendered objects or as a movie of volume or surface rendered images. While orthogonal sections can provide a general overview of image data sets and allow the operator to visually compare images, oblique sections can extract slices at any arbitrary orientation within a volumetric data set. Better still, new data sets with slices parallel to a selected oblique angle can be generated after resampling and interpolation.
As part of a drug design program at Bristol-Meyers Squibbs Pharmaceutical Research Institute (Princeton, NJ), Dr. Ron Behling is using MRI data sections to study problems in drug discovery and design. To do so, Behling is using PV-Wave from Visual Numerics (Houston, TX) running on a Silicon Graphics workstation.
Although there are no standard data formats in MRI, says Behling, the command language of PV-Wave allows virtually any data types to be read. Using various PV-Wave functions, such as FFTs and curve fits, Behling is developing custom procedures to study diffusion rates in tissues.
In many cases, it is useful to observe the two-dimensional structure of organs in the body. To do so, many visualization packages incorporate volume rendering, a technique used for 3D visualization of structures such as the heart and lungs. Produced by projecting rays from multiple angles of view through the volumetric data set, rendered objects can then be scaled and rotated about x, y and z axes to produce animated 3D displays.
Tumors are often very irregular and positioned close to other structures. 3D is therefore very important when you need to visualize the relative positions of these structures, says Zachary Leber, a radiotherapy development manager with Radionics Software Applications (Burlington, MA). Because of this, Radionics developed Xknife using the Application Visualization System (AVS) from Advanced Visual Systems (Waltham, MA). In operation, Xknife converts data from CT and MRI scans that shows the location of a lesion into three dimensional images that show its shape and position relative to optic nerves and other important structures (Fig. 2). Physicians use Xknife to plan noninvasive treatment of intercranial abnormalities, sparing patients the expense, pain and risk associated with open-brain surgery.
In many medical applications, however, it is not necessary to understand data within 2D data sets or to visualize data in 3D. In the manufacture of prosthesis, for example, physicians only need to understand how the surface of structures are composed. Because of this, most software packages incorporate algorithms to allow structures such as bones to be displayed as 3D shaded surfaces. To create 3D shaded surface displays of structures, surface rendering techniques often allows setting multiple threshold ranges to identify multiple objects within a volume. In this way, when two or more objects are to be displayed together, each can be assigned unique colors.
The construction and depiction of surfaces from volume data is important in the manufacture of titanium prosthesis for cranioplasty, says Dr. Jonathan Carr at the University of Cambridge (Cambridge, England). Previously with Christchurch Hospital (Chistchurch, New Zealand), Carr developed a method for designing cranial implants to repair defects in the skull. To do so, skull surfaces were extracted from CT data by ray-tracing iso-values surfaces. By using a tensor product b-spline interpolation in the ray-caster, ripples in the surface data due to large spacing between CT slices were reduced. Associated depth maps characterize the defect regions that need to be repaired. By using radial basis functions, the surface of the skull was then interpolated and the fitted surface used to produce CNS milling instructions to machine a mould in the shape of the surface from a block of hard plastic resin. A cranial implant is then formed by pressing flat titanium plate into the mould using a hydraulic press (Fig. 3).
In many imaging applications, physicians need to see the geometric relationships between anatomical structures. To do so, software packages may include techniques that allow multiple surface or volume rendered images to be rapidly displayed, thus producing the illusion of motion. This is especially useful in studying the heart, for example, where sequences of moving images can indicate loss of muscle function.
VIDA from the Department of Radiology at the University of Iowa College of Medicine (Iowa City, IO) allow developers to display images in a different number of these ways. Written in C and running under UNIX, VIDA has been developed for the manipulation, display and analysis of multidimensional data sets. Available programs in VIDA include orthogonal and oblique sectioning, volume and surface rendering ROI analysis and interactive image segmentation (Fig. 4). Once composed, a module called MovieViewer allows animation techniques to create the illusion of motion. And, because the software is modular, developers can rapidly integrate new programs.
MRI images can show the extent of a tumor in relation to other soft tissues but not necessarily whether the tumor has invaded bone structures. CT images, on the other hand, can show bone structures but may give a relatively poor image of the soft tissue. Side by side images from the same patient can be difficult to interpret as the exact correspondence of the images is not known. Because of this researchers are looking to image fusion techniques to combine both data sets in single, composite images, removing the problems associated with side by side evaluation of images from different medical imaging modalities.
At Foster Findlay Associates (FFA; Newcastle upon Tyne, England), researchers are developing CT/MRI image fusion techniques using C Images 3D from. Running on UNIX and PC-based platforms, Cimages3D is the result of a three year development between FFA and Shell Research (The Netherlands). As a development toolkit, the packages image processing functions include image enhancement, segmentation, texture analysis, ROIs within images. The package supports multiple image formats including CT, MRI and PET and is supplied with an API that allows developers to build applications by combining image sequences.
Simulation and planning of craniofacial surgery is also the aim of the Virtual Environment for Reconstructive Surgery (VERS) project. As a collaboration between the Biocomputation Center at NASA Ames Research Center (Moffett Field, CA) and the Department of Reconstruction Surgery at Stanford University (Stanford, CA), the project aims to create a virtual environment where a surgeon can plan and simulate an operation (Fig. 5). Using the Immersive Workbench from FakeSpace (Mountain View, CA), the project was demonstrated at last years SIGGRAPH by Silicon Graphics and Fakespace.
Designers looking to incorporate 3D visualization packages into their next generation of imaging equipment now have many off-the-shelf image processing packages to choose from. In specifying such packages, developers should be careful to choose modular packages based on open standards that run on a number of hardware platforms.
Although most currently available packages do not run under Windows NT, it is highly likely that this will be the OS of choice for many systems in future. Better yet, because most packages available today are written in C and run under UNIX, it will be a fairly simple task for software developers to create Internet-based, Java-enabled software that will run under multi-tasking, multi-user operating systems. When this is accomplished, sophisticated image processing routines such as 3D volumetric display, data fusion and virtual reality techniques will become as common as the hospital bedpan.
FIGURE CAPTIONS
Figure 1: Analyze AVW software from the Biomedical Imaging resource at the Mayo Foundation (Rochester, MN) permits simultaneuos use of multiple imaging tools assigned to any particular volume image or groups of volume images and other data elements.
Figure 2: Xknife software from Radionics Software Applications was developed using the Application Visualization System from Advanced Visual Systems. The Xknife user interface was built using AVS widgets and viewers and is shown with an MRI scan texture-mapped in the 3D display.
Figure 3: To develop titanium cranial implants, ray tracing is used to depict bone structures within a stack of CT data slices(a). Radial basis functions then fit a surface to the depth-map corresponding to the rendered view of the defect(b). A computer milling machine then produces a mold of the shape of the defect (c) that is then used to construct a flat titanium plate that is screwed into the skull (d and e).
Figure 4: After gathering image data on an Imatron C-150 electron beam CT scanner, color coded image of the lungs were visualized by VIDA software from the Division of Physiologic Imaging at the Department of Radiology at University of Iowa College of Medicine. The lung was segmented using VIDAs segmentation module and displayed using the VIDA surface display module.
Figure 5: Using the Immersive Workbench from Fakespace, physicians can interactively view and operate on three-dimensional images of patient data.
CNSoftware The Professional Building 100 SE Second Street Stewartville, MN 55976 Tel: (507) 533-6602 Fax: (507) 533-6642 E-mail: analyze@cnsoft.com http://www.cnsoft.com
Fakespace 241 Polaris Ave Mountain View, CA 94043 Tel: (415) 688-1940 http://www.fakespace.com
NASA Ames Research Center Moffett Field, CA Tel: (650) 604-9000 http://www.arc.nasa.gov/
Stanford University Stanford, CA 94305 Tel: (650) 723-2300 HYPERLINK http://www.stanford.edu
Radionics Software Applications (RSA) 22 Terry Avenue Burlington, MA 01803 Tel: (617) 238-0600 Fax: (617) 238-0606 E-mail: mailto:sales@radionics.com http://www.radionics.com
Bristol-Meyers Squibbs Pharmaceutical Research Institute P. O. Box 4000 Princeton, NJ Tel: (609) 252 4000
UCLA Box 951361 Los Angeles, CA 90095-1361 Tel: (310) 825-4321 http://www.info.ucla.edu/
Silicon Graphics Computer Systems 2171 Landings Drive Mountain View, CA 94043 Tel: (415) 933-3900 Fax: (415) 960-0197 http://www.sgi.com
HAL Computer Systems 1315 Dell Avenue Campbell, CA 95008 Tel: (408) 379-7000 Email: mailto:webmaster@hal.com http://www.hal.com
NASA Ames Biocomputation Center MS 239-11 Moffett Field, CA 94035-1000 Tel: (650) 604-4804 http://biocomp.arc.nasa.gov/tech2006/.index.html
Visual Numerics Corporate Headquarters 9990 Richmond Avenue, Suite 400 Houston, Texas 77042-4548 USA Tel: (800) 222-4675 Tel: (713) 784-3131 FAX: 713-781-9260 e-mail: mailto:info@houston.vni.com http://www.vni.com/index_hi.html
Advanced Visual Systems 300 Fifth Avenue Waltham, MA 02154 Tel: (781) 890-4300 Fax: (781) 890-8287 E-mail: info@avs.com http://www.avs.com
Mayo Clinic Rochester 200 First St. S.W. Rochester, MN 55905 Tel: (507) 284-2511 Fax: (507) 284-0161 http://www.mayo.edu/bir/home.html
Foster Findlay Associates Newcastle Technopole, King's Manor, Newcastle upon Tyne, England, NE1 6PA. Tel: (01144) 191 201 2180 Fax: (01144) 191 201 2190 E-Mail: mailto:info@www.ffa.co.uk http://www.ffa.co.uk
IBM North America 1133 Westchester Avenue White Plains NY 10604 United States Tel: (800) IBM 3333 E-mail: mailto:ibm_direct@vnet.ibm.com http://www-i.almaden.ibm.com/dx/
University of Colorado Boulder CO 80309 Tel: (303)492-4350 http://bio3d.colorado.edu/imod/
Research Systems 2995 Wilderness Place Boulder, CO 80301 Tel: (303) 786-9900 Fax: (303) 786-9909 email: mailto:info@rsinc.com http://www.rsinc.com/index.html
Supercomputer Computations Research Institute Florida State University, Tallahassee, Florida E-mail: mailto:lyons@scri.fsu.ed http://www.scri.fsu.edu/~lyons/scian/
Sensor Systems 103A Carpenter Drive Sterling, VA 20164 Tel: (703) 437-7651 http://www.sensor.com http://www.sensor.com
Dr. Jonathan C. Carr Fallside Laboratory Information Engineering Division Engineering Department Trumpington Street Cambridge, CB2 1PZ England Tel: (01144) 1223 332754; Fax: (01144) 1223 332662; Email: mailto:jccc@eng.cam.ac.uk http://svr-www.eng.cam.ac.uk/~jccc/Personal.html#research
Vital Images 3100 W. Lake Street Suite 100 Minneapolis, MN 55416-4510 612-915-8000 612-915-8010 (fax) http://www.vitalimages.com/
Division of Physiologic Imaging Department of Radiology, University of Iowa. Iowa City, IO http://everest.radiology.uiowa.edu/vida/vidahome.html