By |

GDAL has support for GDAL Virtual File System which allows GDAL library and command-line tools to work with files on network storage. This is critical in the era of Cloud-Native Geospatial where it is becoming a standard practice to access and share geospatial data via cloud storage services. In this post, we will see how to use GDAL command-line tools to read and write data to Google Cloud Storage (GCS) using the /vsigs file system handler. We will focus exclusively on Google Cloud Storage for this post -but the same concepts apply when using other cloud services such as AWS S3 or Azure Blobstore. Similarly, other GDAL-based tools – such as rasterio – will be able to access the data from GCS using the same configuration options shown here.

The post covers the following topics

  • Reading Files from Public GCS Buckets.
  • Creating Private GCS Buckets and Uploading Data
  • Configuring Authentication and Reading Data from Private GCS Buckets
  • Writing Data to Private GCS Buckets
  • Using Environment Variables

Note 1: This post assumes familiarity with the GDAL command-line tools and assumes you have installed GDAL on your machine. You will find detailed instructions for installation in our course material for Mastering GDAL Tools.

Note 2: The code snippets are split over multiple lines for redability using the Windows Line Continuation character ^. If you are running these on Mac/Linux, you can replace them with \ instead.

Continue reading →

By |

I was recently asked to deliver a session on Earth Science for kids. My daughter goes to an after-school science program at Max Science where they teach science with unique and fun hands-on experiments. They wanted to do an interactive session to introduce the kids to Earth Science and asked if I could deliver a guest talk for their kids in Grades 1 to 4. I loved the idea and developed a module titled The Science of Satellites to introduce the magic of remote sensing to primary school kids. The session ended up being a lot of fun, for the kids and me. In this post, I want to go through the materials and my experience teaching this session.

All the content developed for this session – including high-resolution graphics – is available freely for download. Scroll to the bottom to find the download link.

The 1.5 hour session was split into 3 parts:

  • Part 1 Guess the Place A game to guess the place from satellite images
  • Part 2: The Science of Satellites Learning what satellites do and how can you build and launch a satellite
  • Part 3: Your Name from Space An activity where kids create their from letters seen from satellite images.
Continue reading →

By |

In this post, we will learn how to build a regression model in Google Earth Engine and use it to estimate total above ground biomass using openly available Earth Observation datasets.

NASA’s Global Ecosystem Dynamics Investigation (GEDI) mission collects LIDAR measurements along ground transects at 30m spatial resolution at 60m intervals. The GEDI waveform captures the vertical distribution of vegetation structure and is used to derive estimates of Aboveground Biomass Density (AGBD) at each sample. These sample estimates of AGBD are useful but since they are point measurements – you cannot directly use them to calculate total aboveground biomass for a region. We can use other satellite datasets to build a regression model using the GEDI samples to map and quantify the biomass in a region.

Regression Workflow

This article shows how we can build and run the entire workflow in Earth Engine – from pre-processing to building a regression model to running the predictions. You will also learn some advanced techniques and best practices such as:

  • How to fuse datasets of different resolutions by using setDefaultProjection() and reduceResolution() to align them to a common grid.
  • How to sample pixels from rasters with sparse data efficiently and precisely by leveraging the image mask using stratifiedSample().
  • How to split your workflow into separate steps and use Exports to avoid user memory limit exceeded or computation timed-out errors.
Continue reading →

By |

ISRO recently released the full archive of medium and low-resolution Earth Observation dataset to the public. This includes the imagery from LISS-IV camera aboard ResourceSat-2 and ResourceSat-2A satellites. This is currently the highest spatial resolution imagery available in the public domain for India. In this post, I want to cover the steps required to download the imagery and apply the pre-processing steps required to make this data ready for analysis – specifically how to programmatically convert the DN values to TOA Reflectance. We will use modern Python libraries such as XArray, rioxarray, and dask – which allow use to seamlessly work with large datasets and use all the available compute power on your machine.

Continue reading →

By |

Dynamic World is a new landcover product developed by Google and World Resources Institute (WRI). It is a unique dataset that is designed to make it easy for users to develop locally relevant landcover classification easily. Contrary to other landcover products which try to classify the pixels into a single class – the Dynamic World (DW) model gives you the probability of the pixel belonging to each of the 9 different landcover classes. The full dataset contains the DW class probabilities for every Sentinel-2 scene since 2015 having <35% cloud-cover. It is also updated continuously with detections from new Sentinel-2 scenes as soon as they are available. This makes DW ideal for change detection and monitoring applications.

A key fact about this dataset is that Dynamic World is not a ready-to-use landcover product. Users are expected to fine-tune the output of DW with local knowledge into a final landcover product. Since DW provides per-pixel probabilities generated by a Fully Convolutional Neural Network (FCNN) model, a lot of difficult problems encountered in classifying remotely sensed imagery are addressed already and allows users to refine it with a relatively simple model (such as Random Forest) with small amount of local training data.

A good mental model to use for Dynamic World is to not think of it as landcover product but as a dataset that provides 9 additional bands of landcover related information for each Sentinel-2 image that can be refined to build a locally relevant classification or change detection model.

As seen in the mangrove classification example, using the Dynamic World probability bands as input to a supervised classification model can help you generate a more accurate landcover map in less amount of time. It also eliminates the need for post-processing the results.

To test this concept and explore the potential of this new dataset in developing locally relevant landcover maps – I partnered with Google and WRI to develop a training workshop and host a 5-day “Mapathon” with participants of diverse backgrounds. The event was a mix of hands-on workshop along with hackathon-style group projects to use Dynamic World for a real-world application.

The workshop was hosted by Regional Centre for Mapping of Resources for Development (RCMRD) in Nairobi, Kenya. You can read more about the event in this article. I and Elise Mazur from WRI also gave a talk about our experience at Geo for Good 2023.

In this post, I want to share more technical details about the workshop materials and code for projects for those who may want to use Dynamic World for their own applications.

Continue reading →

By |

Extracting building footprints from high-resolution imagery is a challenging task. Fortunately we now have access to ready-to-use building footprints dataset extracted using state-of-the-art ML techniques. Google’s Open Buildings project has mapped and extracted 1.8 billion buildings in Africa, South Asia, South-East Asia, Latin America and the Caribbean. Microsoft’s Global ML Building Footprints project has made over 1.24 billion building footprints available from most regions of the world.

Update: VIDA has released the most comprehensive buildings dataset by combining both Google and Microsoft building footprint dataset. This dataset is available in Google Earth Engine via the GEE Community Catalog.

Update: Google has released another dataset for building heights. See our post on Extracting Building Heights from Open Buildings 2.5D Temporal Dataset

Given the availability of these datasets, we can now analyze them to create derivative products. In this post, we will learn how to access these datasets and compute the aggregate count of buildings within a regular grid using Google Earth Engine. We will then export the grid as a shapefile and create a building density map in QGIS.

Continue reading →

By |

QGIS User Conference 2023 took place in Netherlands on 18-19 April, 2023. This conference was organized by the Dutch QGIS User Group in the small town of ‘s -Hertogenbosch.

A long awaited community event that happened after a break of 3 years. The conference was attended by over 200 people from across the world. I got a chance to participate and meet many of QGIS community members in-person.

Conference Group Photo

In this post, I hope to share some of my insights and resources from the conference.

Continue reading →

By |

In this article, I will outline a method for extracting shoreline from satellite images in Google Earth Engine. This method is scalable and automatically extracts the coastline as a vector polyline. The full code link is available at the end of the post.

UPDATE: The post now includes tidal-phase filtering using HYCOM Data.

The method involves the following steps

  1. Create a Cloud-free Composite Image from images collected during the same tidal phase
  2. Extract All Waterbodies
  3. Remove Inland Water and Small Islands
  4. Convert Raster to Vector
  5. Simplify and Extract Coastline
Video Demonstration of the Script

We will go through the details of each step and review the Google Earth Engine API code required to achieve the results.

Continue reading →

By |

Modern versions of QGIS comes with a handy command-line utility called qgis_process. This allows you to access and run any Processing Tool, Script or Model from the Processing Toolbox on a terminal. This is very useful for automation since it doesn’t require you to open QGIS or manually click buttons. You can run the algorithms in a headless-mode and even schedule them to run them at specific times.

This post covers the following topics

  • How to launch qgis_process command on Windows, Mac and Linux.
  • How to find the parameters and values for each algorithm and build your command
  • Example showing how to do a spatial join on the command-line using the Join Attributes by Location algorithm
  • Example showing how to run a model on the command line to automate a complex workflow

Want to follow along? You can download the data package containing all the datasets used in this post. Before running each command, make sure to replace the paths in the commands with the paths on your computer.

Continue reading →