Processing Large Rasters Using Tile Iterators

Some computers and applications may not effectively process large raster datasets at once because of limited system memory. To process datasets that are tens or even hundreds of gigabytes in size, the ENVI API provides an option to process chunks, or tiles, of data using a tile iterator.

The ENVIRaster::GetData method in conjunction with the SUB_RECT keyword to ENVISubsetRaster can also process subsets of data, one at a time. However, this process can be complicated so the ENVIRasterIterator object is provided as a more convenient option to process tiles of data. Each tile already contains data, so there is no need to call the GetData method separately. Here is the typical workflow when using tile iterators:

  1. Create an ENVIRaster object from the source image.
  2. Create an empty ENVIRaster object with the same number of rows and columns as the source raster.
  3. Use ENVIRaster::CreateTileIterator to create a tile iterator object.
  4. Use the tile iterator to get tiles of data from the source raster.
  5. Perform image-processing tasks on the data.
  6. Use the ENVIRaster::SetData method to populate the empty raster with the processed tiles of data.
  7. Use the ENVIRaster::Save method to close the raster for writing and to convert it to read-only mode.

Spatial and Spectral Modes

When creating a tile iterator object, you have the choice to retrieve data spatially or spectrally. Set the MODE keyword to spatial or spectral in ENVIRaster::CreateTileIterator.

In spatial mode (the default mode), the tile iterator retrieves all the tiles in one band, then all the tiles in the next band, etc. The iterator returns 2-D tiles of the form:

[range(columns dimension), range(rows dimension)]

The same spatial range is returned n times for n bands as specified by the BANDS keyword.

In spectral mode, the tile iterator retrieves a tile with one column of pixels in all bands. Then it retrieves the next column of pixels in all bands, etc. The iterator returns 2-D tiles of the form:

[range(columns dimension), range(bands dimension)]

Example of Spatial Mode

This example uses a QuickBird panchromatic image from the ENVI installation path to demonstrate retrieving tiles in spatial mode while applying an edge-enhancement algorithm to the data. The PRINT statements within the FOREACH loop are not required; they only demonstrate that the tiles are being processed.

Copy and paste the following code into the IDL Editor, then save the file as TileIteratorSpatialExample.pro. Compile and run the program.

PRO TileIteratorSpatialExample

COMPILE_OPT IDL2

 

; Start the application

e = ENVI()

 

; Select input data

file = FILEPATH('qb_boulder_pan', ROOT_DIR=e.ROOT_DIR, $

  SUBDIRECTORY = ['data'])

 

; Open the original raster

OrigRaster = e.OpenRaster(file)

 

; Create a new raster

newFile = e.GetTemporaryFilename()

EdgeDogRaster = ENVIRaster(URI=newFile, $

  NROWS=OrigRaster.NROWS, $

  NCOLUMNS=OrigRaster.NCOLUMNS, $

  NBANDS=OrigRaster.NBANDS, $

  DATA_TYPE=OrigRaster.DATA_TYPE)

 

; Iterate through the tiles of the original data

tileIterator = OrigRaster.CreateTileIterator()

 

count = 0

FOREACH tile, tileIterator DO BEGIN

  count++

  PRINT,''

  PRINT, 'Tile Number:'

  PRINT, count

 

  ; Process the data

  processedTile = EDGE_DOG(tile)

  currentSubRect = tileIterator.CURRENT_SUBRECT

 

  EdgeDogRaster.SetData, processedTile, SUB_RECT=currentSubRect

ENDFOREACH

 

; Finalize data

EdgeDogRaster.Save

 

; Display new raster

View = e.GetView()

 

Layer = View.CreateLayer(EdgeDogRaster)

END

Example of Spectral Mode

This example uses a QuickBird multispectral image from the ENVI installation path to demonstrate retrieving tiles in spectral mode while subtracting data from two bands. Copy and paste the following code into the IDL Editor, then save the file as TileIteratorSpectralExample.pro. Compile and run the program.

PRO TileIteratorSpectralExample

COMPILE_OPT IDL2

 

; Start the application

e = ENVI()

 

; Select input data

file = FILEPATH('qb_boulder_msi', ROOT_DIR=e.ROOT_DIR, $

  SUBDIRECTORY = ['data'])

 

; Open original

OrigRaster = e.OpenRaster(file)

 

; Create a new raster

newFile = e.GetTemporaryFilename()

 

outRaster = ENVIRaster(URI=newFile, $

  NROWS=OrigRaster.NROWS, $

  NCOLUMNS=OrigRaster.NCOLUMNS, $

  NBANDS=1, $

  DATA_TYPE='double')

 

; Iterate through the tiles of the original data

tileIterator = OrigRaster.CreateTileIterator($

  MODE='spectral', BANDS=[0,1])

 

count = 0

FOREACH tile, tileIterator DO BEGIN

  count++

  PRINT,''

  PRINT, 'Tile Number:'

  PRINT, count

 

  ; Subtract band 2 from band 1

  processedTile = DOUBLE(tile[*,0]) - DOUBLE(tile[*,1])

  currentSubRect = tileIterator.CURRENT_SUBRECT

  outRaster.SetData, processedTile, SUB_RECT=currentSubRect

ENDFOREACH

 

; Finalize the data

outRaster.Save

 

; Display the new raster

View = e.GetView()

Layer = View.CreateLayer(outRaster)

END