75 lines
4.9 KiB
Markdown
75 lines
4.9 KiB
Markdown
---
|
|
title: 'Import Global SRTM Elevation Data Into Postgres Database Using PostGIS'
|
|
date: 2024-08-06T12:15:44+01:00
|
|
author:
|
|
name: "Sam Chance"
|
|
header_image: '/pics/blog/batch-import-postgis-rasters/batch-import-postgis-rasters.webp'
|
|
summary: "This article explains how to batch import rasters into a PostGIS table"
|
|
toc: true
|
|
---
|
|
|
|
{{< figure src="/pics/blog/batch-import-postgis-rasters/batch-import-postgis-rasters.webp" title="DEM (Digital Elevation Model) of Singapore. NASA Shuttle Radar Topography Mission (2013)." width="600">}}
|
|
|
|
## Introduction
|
|
It's possible to download the entire SRTM (Shuttle Radar Topography Mission) satellite imagery dataset and insert it into a Postgres database for personal use. This can be helpful for if you need global elevation data for your analysis and don't want to be limited by third-parties APIs.
|
|
|
|
The SRTM is a near-global dataset of elevation data with a resolution of 1-arc-second (30m). More information is available from [USGS](https://www.usgs.gov/centers/eros/science/usgs-eros-archive-digital-elevation-shuttle-radar-topography-mission-srtm?qt-science_center_objects=0#qt-science_center_objects).
|
|
In this guide we will go through downloading the data, inserting it into a Postgres database with PostGIS, then querying the final result to create a DEM for any country or region.
|
|
|
|
This guide assumes you are using Linux (this may also apply to other Unix like systems such as MacOS) and have a Postgres database with the PostGIS extension installed. More information [here](https://postgis.net/documentation/getting_started/).
|
|
|
|
## Download data
|
|
|
|
We'll download the data from the [OpenTopography](https://portal.opentopography.org/raster?opentopoID=OTSRTM.082015.4326.1) AWS S3 bucket. You should create a free account with them before downloading. Also, remember to provide citations where necessary.
|
|
|
|
To download the data, first we need to install the [AWS](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) CLI utility:
|
|
|
|
{{< highlight shell >}}
|
|
# install for linux
|
|
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
|
|
unzip awscliv2.zip
|
|
sudo ./aws/install
|
|
|
|
# MacOS
|
|
curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
|
|
sudo installer -pkg AWSCLIV2.pkg -target /
|
|
{{</ highlight >}}
|
|
|
|
Next we can run the following command to download the rasters. This will take some time depending on your internet speed. The final size is about 127GB and took me about 1 hour to download on a 600MB connection.
|
|
|
|
{{< highlight shell >}}
|
|
aws s3 cp s3://raster/SRTM_GL1/ . --recursive --endpoint-url https://opentopography.s3.sdsc.edu --no-sign-request
|
|
{{</ highlight >}}
|
|
|
|
## Using raster2pgsql to import raster tiles into PostGIS
|
|
Now we have the data downloaded on our system, we can import it into our database.
|
|
If we look inside the SRTM_GL1_srtm directory, we can see all of the 14280 raster files:
|
|
|
|
{{< figure src="/pics/blog/batch-import-postgis-rasters/raster-tiffs.webp" title=" " width="600">}}
|
|
|
|
Next we will use the `raster2pgsql` command to import these rasters into our database.
|
|
|
|
{{< highlight shell >}}
|
|
raster2pgsql -d -M -C -I -s 4326 -t 128x128 -F *.tif dem.srtm | psql --dbname=osm --username=postgres -h 127.0.0.1
|
|
{{</ highlight >}}
|
|
|
|
Here's a breakdown of the above command:
|
|
- -d This drops an existing table if it already exists. It's useful to include this in case we need to run the command again.
|
|
- -M This runs `VACUUM ANALYZE` on the table after the raster tiles have been inserted into the database. This optimizes the table by reclaiming storage space and then collecting table statistics.
|
|
- -C This applies raster constraints to the table by using the `AddRasterConstraints` function in PostGIS. This adds several types of constraints to the table which ensures data integrity and enforces rules for consistency.
|
|
- -I Creates a spatial index (GiST) on the table.
|
|
- -s Assign SRID (Spatial Reference System Identifier) to the raster.
|
|
- -t Tile size. This splits the raster into tiles of the specified size. Each tile is a row in the final table.
|
|
- -F Creates a filename column. As the raster is split into many smaller tiles, this is useful to help us identify the original file the tile belonged to.
|
|
- *.tif Selects all of the .tif files in the directory.
|
|
- dem.srtm Defines the schema and name of the table. Remember to create the `dem` schema in the database before running the command.
|
|
|
|
The `raster2pgsql` command will output a sql query which we can then pipe into the `psql` command.
|
|
|
|
For more information on the `raster2pgsql` command, please visit the PostGIS [website](https://postgis.net/docs/manual-2.1/using_raster_dataman.html).
|
|
|
|
This process will take a couple of hours. It could take longer if running over a network, so it'd be best to run this command on the same machine as the database.
|
|
|
|
#### Citations
|
|
NASA Shuttle Radar Topography Mission (SRTM)(2013). Shuttle Radar Topography Mission (SRTM) Global. Distributed by OpenTopography. https://doi.org/10.5069/G9445JDF. Accessed: 2024-08-06
|