1

I need to stich a lot of geotiffs, located on an S3 storage into a big geotif.

Currently I using the following commands

gdalbuildvrt -input_file_list filelist.txt -overwrite /tmp/output.vrt
gdalwarp -co COMPRESS=LZW /tmp/output.vrt -overwrite /tmp/result.tif

works but painfully slow. we are dealing with around 5-11k geotiffs on S3. How can i perform this in parallel. CPU and network is neither near a limit so I assume it is running in a kind of "one thread mode". I saw there is a multi option for gdalwarp but nothing for gdalbuildvrt.

thx for your help!

Stefan Starke
  • 31
  • 3

2 Answers2

1

GDAL does not naively support concurrency for computing - however you can use multiple threads for compression in some cases. Please refer to this nice answer on gis.stackexchange.com and the relevant documentation.

So you will need to implement the concurrent processing yourself, depending on the framework / software / language / etc. you want to use. Hence, this can't be answered in this context, you'd need to ask a more concise/specific question.

One note: when reading 11k GeoTIFFs from S3, the appropriate format (e.g. Cloud Optimized Geotiff) and limiting the number of API calls (e.g. through re-using credentials for threads and processes) can have a huge performance and cost-saving impact.

Val
  • 6,585
  • 5
  • 22
  • 52
0

For speeding up the process ue gdal_translate rather gdalwarp. Besides, you should implement a tiling system, create a vrt for each tile , followed by gdal_translate. Then in the end you merge the tile again with vrt + gdal_translate.

el_selvaje
  • 158
  • 1
  • 7