Smathermather's Weblog

Remote Sensing, GIS, Ecology, and Oddball Techniques

Quantitative analysis of gorilla movement and R-Stat (part 2)

Posted by smathermather on November 23, 2016

In my previous post, I did a bit of setup on the who and what for analyzing gorilla data. Now let’s move into R-Stat a little bit, specifically installation and configuration.

For R, we’ll install whatever package is the correct one for your machine, and then also install R-Studio. This gives us a nice development environment for R.

Screen shot of R-Studio

We’re going to leverage a few packages in our work. The first is the “sp” package. This gives us “CLasses and Methods for Spatial Data” in R:

install.packages("sp")

We’ll also “rgdal”. This package gives us “Bindings for the Geospatial Data Abstraction Library”. For our use case here, this allows us to read shapefiles, geotiffs, and other spatial data.

install.packages("rgdal")

We’ll use “rmarkdown” also. This allows us to build a markdown file (think HTML but much simpler) that includes our code in code blocks, and can be compiled to a document whether a web page, pdf, etc.. This encourages us to write code and document it in one place, and ultimately will allow us to build some really great reporting functionality while getting our analysis work done.

install.packages("rmarkdown")

Finally, we’ll need the adehabitatLT package for aiding in our analysis of animal movement and habitat use.

install.packages("adehabitatLT")

We can, of course install all this in one swell foop as follows:

install.packages("sp")
install.packages("rgdal")
install.packages("rmarkdown")
install.packages("adehabitatLT")

Now we have all the building blocks we need to start analyzing our data. Stick around for part 3.

Posted in Gorillas, Karisoke, R | Tagged: , , , , | Leave a Comment »

Quantitative analysis of gorilla movement and R-Stat

Posted by smathermather on November 21, 2016

At this point in my career, it is a rare request that intimidates me. Most requests are either solved, largely solved (and I know how to find the answer), or broadly improbably / excessively expensive.

XKCD cartoon demonstrating the difficulty of distinguishing difficult problems in computer science

This is a nice place to be. I have a bit of calmness with every request, a vision for how to execute, or the escape valve of saying, “That’s harder / more expensive than you think. Perhaps we should look at these alternatives.”

Recently, I got a request that was tougher — a theoretically possible project at the edge of my knowledge domain, a desire to deliver at any reasonable cost, and no clear sense of who to ask for help.

Specifically, how does one take volumes of gorilla tracking data, and make sense of the behaviors and intentions implicit in their movement? Many know the story of Dian Fossey, the primatologist who lived with the mountain gorillas in Rwanda. What many may not know, is her research continues in Rwanda at Karisoke Research Center (and has continued for 50 years), following the troops of gorillas in Rwanda, and the project has expanded quite a bit under auspices of the Dian Fossey Gorilla fund:

Map of Africa

Map of East Africa centered on Rwanda

Map of Rwanda

Zoomed in map of Rwanda centered near Karisoke

Image of Karisoke then and now

I have the privilege of being sponsored by the Cleveland Zoological Society to got to Karisoke, observe the research, and help where needed with GIS infrastructure development and geospatial analyses. I am excited beyond belief.

So, what’s the test here? I know how to do geospatial infrastructure. Ask me to improve infrastructure and I have a decade of experience chasing down these problems, trying out different technologies, developing new ones. I also know how to perform advanced analyses. I pride myself on my analytic skills, even as I’ve moved from analyst to programmer to manager, etc.. But, I’ve never done the more advanced quantitative analyses of movement necessary to deepen understanding beyond work already done at Karisoke. Lots of great minds have already been looking at the data for years and decades. The basic question in my mind — how can I meaningfully help?

Thanks to my colleague Dr. Patrick Lorch, I have some sources. Pat did his thesis work studying the movements and behavior of Mormon crickets in the western US, and his experience with quantitative analysis of movement is directly applicable.

First we looked to Peter Turchin’s Quantitative Analysis of Movement. Then we found the adehabitat R-Stat module.

In short, the next few blog posts will be about the apparently incomparably beautiful nation of Rwanda, mountain gorillas, R-Stat, and adehabitat. There will be PostGIS mixed in for good measure. Stay tuned.

Posted in Gorillas, Karisoke, R | Tagged: , , , , | Leave a Comment »

Filtering buildings and vegetation from drone-derived surface models

Posted by smathermather on October 21, 2016

I’ve been playing for the last few days with PDAL and it’s ability to filter point clouds. More code and details later, but here’s a teaser of my results to date:

Image of unfiltered surface model.

Image of unfiltered surface model.

Image of unfiltered surface model

Image of unfiltered surface model.

Image of surface model derived from points filtered using approximate progressive morphological filter.

Image of surface model derived from points filtered using approximate progressive morphological filter.

Image of surface model derived from points filtered using approximate progressive morphological filter.

Image of surface model derived from points filtered using progressive morphological filter.

Image of surface model derived from points filtered using progressive morphological filter.

Image of surface model derived from points filtered using progressive morphological filter.

Image of surface model derived from points filtered using progressive morphological filter.

Image of surface model derived from points filtered using progressive morphological filter plus building footprint and outlier removal.

Image of surface model derived from points filtered using progressive morphological filter plus building footprint and outlier removal.

Image of surface model derived from points filtered using progressive morphological filter plus building footprint and outlier removal.

Image of surface model derived from points filtered using progressive morphological filter plus building footprint and outlier removal.

Posted in Other | Leave a Comment »

Finding peace, finding ground: Drone flights for hydrologic modeling

Posted by smathermather on October 15, 2016

Š-L-M

Lately I’ve been helping with project to predict flooding in Dar es Salaam (“Home of Peace”), the capital former capital of Tanzania in East Africa. This is a really fun project for me, in part because most of the hard work is already done, but there still remain some tricky problems to solve.

This project is a part of Dar Ramani Huria, a “community-based mapping project… training university students and local community members to create highly accurate maps of the most flood-prone areas of the city using OpenStreetMap.”

Dar es Salaam (or Dar for short), is the fastest growing city in Africa, making the identification of flood prone zones during rainy seasons absolutely critical. The Ramani Huria crew has mapped the streams, ditches, and other waterways of Dar, as well as flown imagery (via drone) over much of the city critical parts of the city.

screen-shot-2016-10-15-at-3-01-04-pm

Problem Space:

The results are stunning, but using drone imagery and photogrammetric point clouds (instead of LiDAR) has it’s limitations.

One problem, that I’m not going to get into in any depth today, is the difficulty of conflating (vertically and horizontally) different datasets flown at different times with commodity GPS.

Another problem is the difficulty of turning photogrammetrically derived point clouds into Digital Terrain Models. There is proprietary software that does this well (e.g. LasTools and others), but we seek a free and open source alternative. Let’s visualize the problem. See below a photogrammetrically-derived point cloud (processed in Pix4D, visualized at https://plas.io):

screen-shot-2016-10-15-at-3-15-44-pm

We can directly turn this into a digital surface model (DSM):

screen-shot-2016-10-15-at-3-24-23-pm

5

We can see the underlying topography, but buildings and trees are much of the signal in the surface model. If we, for example, calculate height above the nearest stream as a proxy for flooding, we get a noisy result.

OSM to the Rescue

If you recall at the beginning of this post, I said this is “a really fun project for me” because “most of the hard work is already done”. Dar is a city with pretty much every building digitized. We can take advantage of this work to remove buildings from our point cloud. We’ll use the utilities associated with the Point Data Abstraction Layer (PDAL) library. Specifically, we’ll use PDAL’s ability to clip point cloud data with 2D OGR compatible geometries (a great tutorial on this here).

First thing we need to do is extract the buildings from OSM into shapefiles. For this, and easy way is to use Mapzen’s Metro Extracts:

screen-shot-2016-10-15-at-3-58-35-pm

I chose Datasets grouped into individual layers by OpenStreetMap tags (IMPOSM) as this gives me the capacity to just choose “Buildings”. I loaded those buildings into a PostGIS database for the next steps. What I seek is a shapefile of all the areas which are not buildings, essentially from this:

building

To this:

no_building

For this we’ll need to use ST_Difference + ST_ConvexHull. ST_ConvexHull will create a shape of the extent of our data, and we’ll use ST_Difference to cookie-cutter out all the building footprints:

DROP TABLE IF EXISTS "salaam-buff-diff-split1";
CREATE TABLE "salaam-buff-diff-split1" AS
WITH subset AS (
	SELECT ST_Transform(geom, 32737) AS geom FROM "salaam-buildings" LIMIT 20000
	),
extent AS (
	SELECT ST_ConvexHull(ST_Union(geom)) AS geom FROM subset
	),
unioned AS (
	SELECT ST_Union(geom) AS geom FROM subset
	),
differenced AS (
	SELECT ST_Difference(a.geom, b.geom) AS geom FROM
	extent a, unioned b
	)
SELECT 1 AS id, geom FROM differenced;

This gives us a reasonable result. But, when we use this in PDAL, (I assume) we want a subdivided shape to ensure we don’t have to access the entire point cloud at any given time to do our clipping. We’ll add to this process ST_SubDivide, which will subdivide our shape into parts not to exceed a certain number of nodes. In this case we’ll choose 500 nodes per shape:

DROP TABLE IF EXISTS "salaam-buff-diff-split1";
CREATE TABLE "salaam-buff-diff-split1" AS
WITH subset AS (
	SELECT ST_Transform(geom, 32737) AS geom FROM "salaam-buildings" LIMIT 20000
	),
extent AS (
	SELECT ST_ConvexHull(ST_Union(geom)) AS geom FROM subset
	),
unioned AS (
	SELECT ST_Union(geom) AS geom FROM subset
	),
differenced AS (
	SELECT ST_Difference(a.geom, b.geom) AS geom FROM
	extent a, unioned b
	)
SELECT 1 AS id, ST_Subdivide(geom, 500) AS geom FROM
differenced;

subdivided

Finally, if we want to be sure to remove the points from the edges of buildings (we can assume the digitized buildings won’t perfectly match our point clouds), then we should buffer our shapes:

DROP TABLE IF EXISTS "salaam-buff-diff-split1";
CREATE TABLE "salaam-buff-diff-split1" AS
WITH subset AS (
	SELECT ST_Transform(geom, 32737) AS geom FROM "salaam-buildings" LIMIT 20000
	),
buffered AS (
	SELECT ST_Buffer(geom, 2, 'join=mitre mitre_limit=5.0') AS geom FROM subset
	),
extent AS (
	SELECT ST_ConvexHull(ST_Union(geom)) AS geom FROM subset
	),
unioned AS (
	SELECT ST_Union(geom) AS geom FROM Buffered
	),
differenced AS (
	SELECT ST_Difference(a.geom, b.geom) AS geom FROM
	extent a, unioned b
	)
SELECT 1 AS id, ST_Subdivide(geom, 500) AS geom FROM
differenced;

If we recall our before point cloud:

screen-shot-2016-10-15-at-3-15-44-pm

screen-shot-2016-10-15-at-4-38-53-pm

Now we have filtered out most buildings:

screen-shot-2016-10-15-at-4-36-15-pm

screen-shot-2016-10-15-at-4-36-31-pm

In order to do this filtering in PDAL, we do two things. First we create a json file that defines the filter:

{
  "pipeline":[
    "/data/2015-05-20_tandale_merged_densified_point_cloud_part_1.las",
    {
      "type":"filters.attribute",
      "dimension":"Classification",
      "datasource":"/data/salaam-buff-diff-split.shp",
      "layer":"salaam-buff-diff-split",
      "column":"id"
    },
    {
      "type":"filters.range",
      "limits":"Classification[1:1]"
    },
    "/data/2015-05-20_tandale_merged_densified_point_cloud_part_1_nobuild_buff.las"
  ]
}

Then we use these json definitions to apply the filter:

nohup sudo docker run -v /home/gisuser/docker/:/data pdal/pdal:1.3 pdal pipeline /data/shape-clip.json

Problems continue

This looks pretty good, but as we interrogate the data, we can see the artifacts of trees and some buildings still linger in the dataset.

screen-shot-2016-10-15-at-4-37-38-pm

screen-shot-2016-10-15-at-4-38-03-pm

 

Enter the Progressive Morphological Filter

It is possible to use the shape of the surface model to filter out buildings and trees. To do so, we start with the assumption that the buildings and vegetation shapes distinctive from the shape of the underlying ground. We have already used the hard work of the OpenStreetMap community to filter most of the buildings, but we still have some buildings and plenty of trees. PDAL has another great tutorial for applying this filter which we’ll leverage.

Again we need a JSON file to define our filter:

{
  "pipeline": {
    "name": "Progressive Morphological Filter with Outlier Removal",
    "version": 1.0,
    "filters": [{
        "name": "StatisticalOutlierRemoval",
        "setMeanK": 8,
        "setStddevMulThresh": 3.0
      }, {
        "name": "ProgressiveMorphologicalFilter",
        "setCellSize": 1.5
    }]
  }
}

And then use that filter to remove all the morphology and statistical outliers we don’t want:

nohup sudo docker run -v /home/gisuser/docker/:/data pdal/pdal:1.3 pdal pcl -i /data/2015-05-20_tandale_merged_densified_point_cloud_part_1_nobuild_buff.las -o /data/nobuild_filtered2.las -p /data/sor-pmf.json

This command will remove our colorization for the points, so we’ll see the colorization according to height only:

screen-shot-2016-10-15-at-4-53-55-pm

screen-shot-2016-10-15-at-4-54-49-pm

 

What remains is to interpolate this into digital terrain model. That is a project for another day.

Posted in 3D, Drone, Other, PDAL, Photogrammetry, UAS | Tagged: , , , | Leave a Comment »

UAS Mapping: Positional Accuracy Assessment via GeoKota

Posted by smathermather on August 10, 2016

Introduction Recently we partnered up with folks from the University of Akron to help determine how accurate UAS are compared to traditional mapping methods. Given the current difficulty to fly commercially in the National Airspace, this partnership gave us a unique opportunity to fly inside their Field House. This controlled space had a lot of […]

via UAS Mapping: Positional Accuracy Assessment — GeoKota

Posted in Other | Leave a Comment »

The Worlds of #Mapzen Sphere Maps

Posted by smathermather on July 21, 2016

Guest blog post today from Brandon Garman (@brandon_garman):

After seeing Mapzen’s blog post on Sphere Maps, I wanted to try my hand at it. I downloaded the Github repository, set up a python simple server and ‘BOOM’ had my own instance running (Mapzen makes this process very easy). So next I pulled out my Wacom tablet, opened up Photoshop and began painting in circles with different colors and brushes. After I had a few results I was happy with, I loaded each one to see how I did. Not so well it turns out! I need to spend a bit more time practicing and fine-tuning before I can get any results half as nice as Mapzen’s examples.

Instead of hunkering down and refining my painted circles, I took a look at some of the non-hand drawn examples that came in the repo (glich, yinyang, checkerboard). Since all that needs uploaded is an image file, I decided to find some images of my own. PLANETS!! What better image to use to style terrain on earth than other planets! So, I picked a few images of our planetary neighbors and I have to say I’m very happy with some of the results.

MARS
This is probably my favorite result (the moon is a close second though, see below).

mars_pic

mars.PNG

NEPTUNE
As a cartographer, seeing an all blue hillshade is jolting, but oddly amusing. Still an excellent result though.

Neptune_pic

neptune.PNG

MOON
There are some great colors and contrast here, however the result was a little blah.
moon_pic.jpg

moon

Until I realized the majority of the dark colors were in the top left, which meant the lighting was backwards from what is typical. So I rotated the sphere image 180 degrees and the result was much better, close to that of Mars.
moon_rot.jpg

moon_rotate.PNG

EUROPA – one of Jupiter’s moons
Not too bad, but has the same inverted look as the moon did.
Europa_pic

europa.PNG

Here it is rotated 180 degrees.
Europa_rot.jpg

europa_rotated.PNG

EARTH
Obviously I was going to use Earth. This color scheme here is really great, but the level of detail in the data at this scale makes it look pixellated and hard to read.
earth_pic.jpg

earth.PNG

I tried blurring the image a bit to smooth out some of the choppiness of the colors and it helped.
earth_blur.jpg

earth_blur.PNG

JUPITER
I love this color scheme as well, but has the same problems as the Earth image did.
jupiter_pic

jupiter.PNG
I blurred this image as well. It helped a bit. Still a great color scheme though.
jupiter_blur

jupiter_blur.PNG

The best part about sphere map is that the only limitations are your creativity. The ability to create great looking maps is very easy. Also, the ability to create the most useless but entertaining maps is just as easy and is sometimes more fun…

mona_lisa_round

mona_lisa.PNG

starry_night_round.jpg

starry_night.PNG

 

Deathstar_blur

deathstar_blur.PNG

Posted in Other | Leave a Comment »

Viewing Sparse Point Clouds from OpenDroneMap — GeoKota

Posted by smathermather on June 30, 2016

This is a post about OpenDroneMap, an opensource project I am a maintainer for. ODM is a toolchain for post-processing drone imagery to create 3D and mapping products. It’s currently in beta and under pretty heavy development. If you’re interested in contributing to the project head over here. The Problem So for most of the […]

via Viewing Sparse Point Clouds from OpenDroneMap — GeoKota

Posted in 3D, Image Processing, OpenDroneMap, OpenDroneMap, Optics, Other, Photogrammetry | Tagged: , , , | Leave a Comment »

Using PostGIS for Hydrologic Modeling (reblog)

Posted by smathermather on June 17, 2016

The Problem We have to filter out the roads and ditches without removing streams that cross roads or follow them closely. I’m going to use PostGIS to find the intersection of the streams lines data with a buffered roads polygon. If the intersected line is less than 50% of the length of the stream line, […]

via Filtering Roads from Extracted Streams Data — GeoKota

Posted in Database, PostGIS, PostgreSQL, SQL | Tagged: , , , , , | Leave a Comment »

Using foreign data wrapper to use PostGIS with SQLServer

Posted by smathermather on May 29, 2016

Here was the problem that needed solved last week (we have a few similar problems in upcoming projects, so this was an exciting thing to try): we needed to use PostGIS to access data in a SQLServer database. The SQLServer database backs the web site in question, the underlying content management system, etc., so no– removing SQLServer isn’t really an option at this stage. Obviously PostGIS is a requirement too… .

Before I go further, I used tds_fdw as the foreign data wrapper. The limitations here are as follows: it is a read-only connection, and only works with Sybase and SQLServer, as it uses tabular data stream protocol for communicating between client and server. This is not as generic a solution as we can use. Next time I’ll try ogr_fdw which is more generic as it can connect with other databases and other data types. Another advantage to ogr_fdw is we can use IMPORT FOREIGN SCHEMA. Regina Obe and Leo Hsu warn though to limit this to 150 tables or so for performance reasons.

With the limitations listed above, this is how we built the thang:

DROP SERVER beach_fdw CASCADE;

-- Create the server connection with FOREIGN DATA WRAPPER
CREATE SERVER beach_fdw
FOREIGN DATA WRAPPER tds_fdw
OPTIONS (servername 'name_or_ip', port '1433', database 'non-spatial-database', tds_version '7.1', msg_handler 'notice');

-- We map the postgres user to the user that can read the table we're interested in
CREATE USER MAPPING FOR postgres
SERVER beach_fdw
OPTIONS (username 'user', password 'password');

-- Create the actual foreign table connection
CREATE FOREIGN TABLE beach_closures (
AutoNumber int NOT NULL,
Title varchar NOT NULL,
StartDate timestamp without time zone NOT NULL,
WaterQuality varchar NOT NULL,
Latitude varchar NOT NULL,
Longitude varchar NOT NULL,
BeachStatus varchar NOT NULL,
ClosureColor varchar NOT NULL)
SERVER beach_fdw
OPTIONS (schema_name 'schema_name', table_name 'vw_CMPBeachClosures');

-- Now we create a spatial view using our longitude and latitude
CREATE VIEW v_beach_closures AS
SELECT
AutoNumber, Title, StartDate, WaterQuality, Latitude,
Longitude, BeachStatus, ClosureColor, ST_SetSRID(ST_MakePoint(Longitude::numeric, Latitude::numeric), 4326)	AS geom
FROM beach_closures;

Voila! A nice little PostGIS enabled view of a SQLServer view or table!

Posted in Database, PostGIS, PostgreSQL, SQL | Tagged: , , , , | Leave a Comment »

Efficient delivery of raster data part 4

Posted by smathermather on May 1, 2016

Guest post from my colleague Patrick Lorch who wrote up what we did the other day in order to view a whole bunch of tiled images in a directory in QGIS (I did some mild editing to his posts. Mistakes are mine). The great thing about the approach is that is generalizeable to most tools that use GDAL for their raster API. This is part of a series. You can view this series in reverse with this post:

Building virtual datasets from a bunch of tiffs
What do you do when someone gives you aerial images stored as tiles in different directories representing different zoom levels? The goal is to make them easy to use as baselayers in QGIS. The answer is to reference them in a virtual data set (VRT).

gdalbuildvrt is the ticket
First make lists of tiffs
If the directory structure is something like this:

total 8047
drwxr-xr-x 7 pdl    4096 Apr 29 14:22 ./
drwxr-xr-x 5 pdl    4096 Apr 27 22:10 ../
drwxr-xr-x 2 pdl 1310720 Apr 22 21:37 0/
drwxr-xr-x 2 pdl  393216 Apr 22 22:54 1/
drwxr-xr-x 2 pdl   98304 Apr 28 14:44 2/
drwxr-xr-x 2 pdl   32768 Apr 28 14:44 3/
drwxr-xr-x 2 pdl    8192 Apr 28 14:44 4/

Then you first need a set of list files listing tiffs in each directory.

ls 0\*.tif > list0.txt
ls 1\*.tif > list1.txt
ls 2\*.tif > list2.txt
ls 3\*.tif > list3.txt
ls 4\*.tif > list4.txt

Now make the vrts

gdalbuildvrt -input_file_list list0.txt aerial_2015_0.vrt
gdalbuildvrt -input_file_list list1.txt aerial_2015_1.vrt
gdalbuildvrt -input_file_list list2.txt aerial_2015_2.vrt
gdalbuildvrt -input_file_list list3.txt aerial_2015_3.vrt
gdalbuildvrt -input_file_list list4.txt aerial_2015_4.vrt

Now you can open these in QGIS depending on what zoom level you need.

These VRTs may now be loaded as ordinary rasters in QGIS or whatever you please. In this case, we retiled with multiple resample levels (see this post for more info), so we’ll have to define max/min ranges at which the different image collections are visible.

Thanks for the write up Pat!

Posted in GDAL | Tagged: , , , | Leave a Comment »