Processing Large Rasters Using Tile Iterators
Some computers and applications may not effectively process large raster datasets at once because of limited system memory. To process datasets that are tens or even hundreds of gigabytes in size, the ENVI API provides an option to process chunks, or tiles, of data using a tile iterator.
The ENVIRaster::GetData method in conjunction with the SUB_RECT keyword to ENVISubsetRaster can also process subsets of data, one at a time. However, this process can be complicated so the ENVIRasterIterator object is provided as a more convenient option to process tiles of data. Each tile already contains data, so there is no need to call the GetData method separately. Here is the typical workflow when using tile iterators:
- Create an ENVIRaster object from the source image.
- Create an empty ENVIRaster object with the same number of rows and columns as the source raster.
- Use ENVIRaster::CreateTileIterator to create a tile iterator object.
- Use the tile iterator to get tiles of data from the source raster.
- Perform image-processing tasks on the data.
- Use the ENVIRaster::SetData method to populate the empty raster with the processed tiles of data.
- Use the ENVIRaster::Save method to close the raster for writing and to convert it to read-only mode.
Spatial and Spectral Modes
When creating a tile iterator object, you have the choice to retrieve data spatially or spectrally. Set the MODE keyword to spatial
or spectral
in ENVIRaster::CreateTileIterator.
In spatial mode (the default mode), the tile iterator retrieves all the tiles in one band, then all the tiles in the next band, etc. The iterator returns 2-D tiles of the form:
[range(columns dimension), range(rows dimension)]
The same spatial range is returned n times for n bands as specified by the BANDS keyword.
In spectral mode, the tile iterator retrieves a tile with one column of pixels in all bands. Then it retrieves the next column of pixels in all bands, etc. The iterator returns 2-D tiles of the form:
[range(columns dimension), range(bands dimension)]
Example of Spatial Mode
This example uses a QuickBird panchromatic image from the ENVI installation path to demonstrate retrieving tiles in spatial mode while applying an edge-enhancement algorithm to the data. The PRINT statements within the FOREACH loop are not required; they only demonstrate that the tiles are being processed.
Copy and paste the following code into the IDL Editor, then save the file as TileIteratorSpatialExample.pro
. Compile and run the program.
PRO TileIteratorSpatialExample
COMPILE_OPT IDL2
; Start the application
e = ENVI()
; Select input data
file = FILEPATH('qb_boulder_pan', ROOT_DIR=e.ROOT_DIR, $
SUBDIRECTORY = ['data'])
; Open the original raster
OrigRaster = e.OpenRaster(file)
; Create a new raster
newFile = e.GetTemporaryFilename()
EdgeDogRaster = ENVIRaster(URI=newFile, $
NROWS=OrigRaster.NROWS, $
NCOLUMNS=OrigRaster.NCOLUMNS, $
NBANDS=OrigRaster.NBANDS, $
DATA_TYPE=OrigRaster.DATA_TYPE)
; Iterate through the tiles of the original data
tileIterator = OrigRaster.CreateTileIterator()
count = 0
FOREACH tile, tileIterator DO BEGIN
count++
PRINT,''
PRINT, 'Tile Number:'
PRINT, count
; Process the data
processedTile = EDGE_DOG(tile)
currentSubRect = tileIterator.CURRENT_SUBRECT
EdgeDogRaster.SetData, processedTile, SUB_RECT=currentSubRect
ENDFOREACH
; Finalize data
EdgeDogRaster.Save
; Display new raster
View = e.GetView()
Layer = View.CreateLayer(EdgeDogRaster)
END
Example of Spectral Mode
This example uses a QuickBird multispectral image from the ENVI installation path to demonstrate retrieving tiles in spectral mode while subtracting data from two bands. Copy and paste the following code into the IDL Editor, then save the file as TileIteratorSpectralExample.pro
. Compile and run the program.
PRO TileIteratorSpectralExample
COMPILE_OPT IDL2
; Start the application
e = ENVI()
; Select input data
file = FILEPATH('qb_boulder_msi', ROOT_DIR=e.ROOT_DIR, $
SUBDIRECTORY = ['data'])
; Open original
OrigRaster = e.OpenRaster(file)
; Create a new raster
newFile = e.GetTemporaryFilename()
outRaster = ENVIRaster(URI=newFile, $
NROWS=OrigRaster.NROWS, $
NCOLUMNS=OrigRaster.NCOLUMNS, $
NBANDS=1, $
DATA_TYPE='double')
; Iterate through the tiles of the original data
tileIterator = OrigRaster.CreateTileIterator($
MODE='spectral', BANDS=[0,1])
count = 0
FOREACH tile, tileIterator DO BEGIN
count++
PRINT,''
PRINT, 'Tile Number:'
PRINT, count
; Subtract band 2 from band 1
processedTile = DOUBLE(tile[*,0]) - DOUBLE(tile[*,1])
currentSubRect = tileIterator.CURRENT_SUBRECT
outRaster.SetData, processedTile, SUB_RECT=currentSubRect
ENDFOREACH
; Finalize the data
outRaster.Save
; Display the new raster
View = e.GetView()
Layer = View.CreateLayer(outRaster)
END