High Performance Remote Sensing – Beowulf in the house
Monday, October 5th, 2009With the advent of easy 64 bit computing and multi-core CPU’s time has come for every company to have its own super-computer, not just research groups in Universities. The nature of data processing in Remote sensing lends itself easily to parallelization. Most of the imagery data is multi-band 2-dimensional rasters, 3-dimensional matrices from a mathematicians viewpoint. From a computational task the major processes of geometry correction, spectral correction, collation of frames and compression for transmission can be done on a pixel by pixel or block by block basis allowing segmentation of the tasks to multiple processors.
With the uptake of more projects single machine based processing became an issue at Apogee and solutions were sought for continuously running general processing on large high resolution datasets. The processing chains have been automated and set up on a beowulf server farm with quad-core CPU’s and identical diskless systems to run in parallel using a Message Passing Interface (MPI) or Parallel Python. This solution enables us to quickly finish larger projects, serve more clients and develop more elaborate processing, since computational complexity is not a barrier any more. The processes are also more fault tolerant due to the use of a more stable and syncronized operating environment with regular checkpointing for major outage recovery.
Beowulf Cluster System Diagram
