Update content

content/_index.md

@@ -3,21 +3,8 @@ toc: False
 ---
 
 # Grounded Insights from Open Data
-## Deriving meaningful insights from data enables us to make better decisions. 
 
-Data is often chaotic and dispersed. This requires us to build solid data pipelines to efficiently transform data into a unified and useful format. We can think of this as a path with the following steps:
-
-#### Data
-Data begins as a collection of information, facts, and statistics. Often, the data we need is unstructured, comes in various formats, and originates from different sources.
-
-#### Integration
-Integration involves combining data from various sources and formats using techniques such as ETL (Extract, Transform, Load) and data modelling. This process creates a unified view of the data which we can then store in a Data Warehouse ready for analysis.
-
-#### Analysis
-Analysis involves examining and interpreting data to uncover patterns, trends, and correlations. This process often utilizes programming, statistical methods, and data visualization tools.
-
-#### Insights
-Insights are the deep understanding we've derived from our data. Specifically, this refers to patterns and relationships that have been uncovered from our data that didn't previously exist. We can use insights to make important business decisions or inform policy.
+Data is often chaotic and dispersed. This requires us to build solid data pipelines to efficiently transform data into a unified and useful format.
 
 ### Explore Based Data
 

content/blog/batch-import-postgis-rasters.md

@@ -3,12 +3,22 @@ title: 'Import Global SRTM Elevation Data Into Postgres Database Using PostGIS'
 date: 2024-08-06T12:15:44+01:00
 author: 
     name: "Sam Chance"
-header_image: '/pics/blog/batch-import-postgis-rasters/batch-import-postgis-rasters.webp'
-summary: "This article explains how to batch import rasters into a PostGIS table"
+header_image: '/pics/blog/batch-import-postgis-rasters/singapore-final.webp'
+summary: "This article explains how to batch import rasters into a Postgres database. It also covers the use of PostGIS functions to query these rasters and create new ones. These techniques are particularly useful for applications such as QGIS."
 toc: true
+tags: ["QGIS", "PostGIS", "Database", "Raster"]
 ---
 
-{{< figure src="/pics/blog/batch-import-postgis-rasters/batch-import-postgis-rasters.webp" title="DEM (Digital Elevation Model) of Singapore. NASA Shuttle Radar Topography Mission (2013)." width="600">}}
+## TLDR
+In this article, we walk through the process of obtaining and downloading rasters of global digital elevation data from the SRTM satellite.
+
+We split these rasters into tiles and import them into a local Postgres database using PostGIS.
+
+We then use this database and construct a query using PostGIS functions to generate a single DEM raster of Singapore and import this into QGIS.
+
+The final image looks like this:
+
+{{< figure src="/pics/blog/batch-import-postgis-rasters/singapore-final.webp" width="600">}}
 
 ## Introduction
 It's possible to download the entire SRTM (Shuttle Radar Topography Mission) satellite imagery dataset and insert it into a Postgres database for personal use. This can be helpful for if you need global elevation data for your analysis and don't want to be limited by third-parties APIs.
@@ -68,7 +78,145 @@ The `raster2pgsql` command will output a sql query which we can then pipe into t
 
 For more information on the `raster2pgsql` command, please visit the PostGIS [website](https://postgis.net/docs/manual-2.1/using_raster_dataman.html).
 
-This process will take a couple of hours. It could take longer if running over a network, so it'd be best to run this command on the same machine as the database.
+This process will take a couple of hours. It could take longer if running over a network, so it'd be best to run this command on the same machine as the Postgres instance.
+
+Once this process has completed, we can query the table to see how many tiles we have:
+
+{{< highlight sql >}}
+select count(*) from dem.srtm
+{{</ highlight >}}
+
+{{< highlight plain-text >}}
+  count  
+---------
+ 12009480
+{{</ highlight >}}
+
+So we have just over 12 million 128x128 tiles in the database.
+
+## Raster Clipping
+
+Now we have global SRTM data loaded in our local database, we can extract any of the tiles for further analysis and access this using any application. 
+
+Here I'll demonstrate using the following PostGIS commands: `st_clip`, `st_intersects` and `st_union` to create a digital elevation table for singapore and access this from from within QGIS.
+
+I already have countries vector data in my Postgres database that I extracted from OpenStreetMaps. You can download a pg_dump of this [here](/data/countries.sql) to insert into your database. Alternatively, I have a post [here](/blogs/import-osm-countries-data) that explains the process to do this manually.
+
+To create our table, we'll use the following SQL. I'll break down each section step-by-step.
+
+First we'll use the `st_intersects` command to select all of the raster tiles that intersect with the Singapore. Note, we'll need to create a table, spatial index then apply constraints for this to work in QGIS.
+
+{{< highlight sql >}}
+-- drop existing table and create a new table
+drop table if exists dem.singapore_srtm;
+create table dem.singapore_srtm as
+
+-- cte to create polygon of singapore
+with country as (select geom as geom from dev.countries where country = 'Singapore')
+
+-- select tiles that intersect with the Singapore polygon using a left join and st_intersects function
+select srtm.rast, srtm.rid
+from dem.srtm srtm
+join country ply on st_intersects(ply.geom, st_convexhull(srtm.rast));
+
+-- create index and apply constraints
+create index on "dem"."singapore_srtm" using gist (st_convexhull("rast"));
+analyze "dem"."singapore_srtm" ;
+select addrasterconstraints( 'dem', 'singapore_srtm', 'rast', true, true, true, true, true, true, false, true, true, true, true, true) ;
+{{</ highlight >}}
+
+Now we can query the table:
+{{< highlight sql >}}
+-- check number of tiles in the table
+select count(*) from dem.singapore_srtm
+{{</ highlight >}}
+
+{{< highlight plain-text >}}
+ count 
+-------
+   153
+{{</ highlight >}}
+
+Here we have 153 of our 128x128 tiles.
+
+We visualise these tiles in QGIS:
+{{< figure src="/pics/blog/batch-import-postgis-rasters/singapore-intersects.webp" width="600">}}
+
+Note, you can use the `st_convexhull` command to generate an outline of the raster tiles as seen in the previous figure:
+
+{{< highlight sql >}}
+select row_number() over () as uid, st_convexhull(rast) as geom
+from dem.singapore_srtm
+{{</ highlight >}}
+
+It would be nice to remove the pixels outside of the Singapore polygon. For this we can use the `st_clip` function along with `st_intersect`:
+
+{{< highlight sql >}}
+-- drop existing table and create a new table
+drop table if exists dem.singapore_srtm;
+create table dem.singapore_srtm as
+
+-- cte to create polygon of singapore
+with country as (select geom as geom from dev.countries where country = 'Singapore'),
+
+-- cte to clip tiles to the singapore polygon
+clipped_tiles as (
+select st_clip(srtm.rast, ply.geom, true) as rast, srtm.rid
+from dem.srtm srtm
+join country ply on st_intersects(ply.geom, st_convexhull(srtm.rast))
+)
+
+select rast as rast
+from clipped_tiles;
+
+-- create index and apply constraints
+create index on "dem"."singapore_srtm" using gist (st_convexhull("rast"));
+analyze "dem"."singapore_srtm" ;
+select addrasterconstraints( 'dem', 'singapore_srtm', 'rast', true, true, true, true, true, true, false, true, true, true, true, true) ;
+vacuum analyze "dem"."singapore_srtm";
+{{</ highlight >}}
+
+Give us this:
+
+{{< figure src="/pics/blog/batch-import-postgis-rasters/weird-clipping.webp" width="600">}}
+
+Hmm, this doesn't look quite right! As you can see, the clipped tiles only seem to apply within the tile boundaries outside of the clip region. These areas have been set to `nodata` (which is expected). However, outside of that, QGIS is rendering those pixels as something else (it's rendering them as having a value of 0). 
+
+The reason for this is because we are not strictly importing a single raster into QGIS, rather we are still importing all of the indivitual tiles. Since GeoTIFFs are rasters that can only exist as a grid of pixels (i.e. a rectangle), QGIS must therefore render the entire raster extent, which leads to the unusual rendering issue above.
+
+We can solve this by using the `st_union` function in our query. This takes all of the individual raster tiles and unions them into a single raster. All of the `nodata` values for the extent of the clipped region will also be unioned, thus creating a single rectangular raster that we can import into QGIS.
+
+Our final query will look like this:
+
+{{< highlight sql >}}
+-- drop existing table and create a new table
+drop table if exists dem.singapore_srtm;
+create table dem.singapore_srtm as
+
+-- cte to create polygon of singapore
+with country as (select geom as geom from dev.countries where country = 'Singapore'),
+
+-- cte to clip tiles to the singapore polygon
+clipped_tiles as (
+select st_clip(srtm.rast, ply.geom, true) as rast, srtm.rid
+from dem.srtm srtm
+join country ply on st_intersects(ply.geom, st_convexhull(srtm.rast))
+)
+
+-- union tiles into a single raster
+select st_union(rast) as rast
+from clipped_tiles;
+
+-- create index and apply constraints
+create index on "dem"."singapore_srtm" using gist (st_convexhull("rast"));
+analyze "dem"."singapore_srtm" ;
+select addrasterconstraints( 'dem', 'singapore_srtm', 'rast', true, true, true, true, true, true, false, true, true, true, true, true) ;
+vacuum analyze "dem"."singapore_srtm";
+{{</ highlight >}}
+
+{{< figure src="/pics/blog/batch-import-postgis-rasters/singapore-final.webp" width="600">}}
+
+Looks much better!
 
 #### Citations
 NASA Shuttle Radar Topography Mission (SRTM)(2013). Shuttle Radar Topography Mission (SRTM) Global. Distributed by OpenTopography. https://doi.org/10.5069/G9445JDF. Accessed: 2024-08-06

content/blog/install-artix.md

@@ -34,7 +34,7 @@ guide I'll be using an encrypted partition on an UEFI system. If if you want a
 different configuration, please consult the [Arch
 wiki](https://wiki.archlinux.org/title/Partitioning#Example_layouts).
 
-![artix-keyboard-select](/pics/blog/install-artix/artix-keyboard-select.webp)
+{{< figure src="/pics/blog/install-artix/artix-keyboard-select.webp" width="400">}}
 
 ## Partition layout
 
@@ -68,7 +68,7 @@ List all drives attached to system:
 lsblk
 {{</ highlight >}}
 
-![artix-lsblk](/pics/blog/install-artix/artix-lsblk.webp)
+{{< figure src="/pics/blog/install-artix/artix-lsblk.webp" width="400">}}
 
 Locate the target drive (in this case `/dev/sda`) where we will install Artix.
 
@@ -89,7 +89,7 @@ Run through the options to partition the disk:
  
 You should now have two partitions under `/dev/sda`:
 
-![artix-lsblk1](/pics/blog/install-artix/artix-lsblk1.webp)
+{{< figure src="/pics/blog/install-artix/artix-lsblk1.webp" width="400">}}
 
 `/dev/sda1` is the unencrypted boot partition, and `/dev/sda2` will be where we store our encrypted volume.
 
@@ -133,7 +133,7 @@ lsblk -f
 
 It should look something like this:
 
-![artix-lsblk2](/pics/blog/install-artix/artix-lsblk2.webp)
+{{< figure src="/pics/blog/install-artix/artix-lsblk2.webp" width="400">}}
 
 Note the UUIDs - they will be needed later for setting up decryption during boot.
 
@@ -285,7 +285,7 @@ grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=grub
 grub-mkconfig -o /boot/grub/grub.cfg
 {{</ highlight >}}
 
-![artix-grub-install](/pics/blog/install-artix/artix-grub-install.webp)
+{{< figure src="/pics/blog/install-artix/artix-grub-install.webp" width="400">}}
 
 ## Add Users
 
@@ -305,7 +305,7 @@ passwd user
 Edit the sudoers file to allow sudo root commands for user.
 
 {{< highlight shell >}}
-EDITOR=vim visudo`
+EDITOR=vim visudo
 {{</ highlight >}}
 
 Then uncomment the following line: