Smathermather's Weblog

Remote Sensing, GIS, Ecology, and Oddball Techniques

Archive for April, 2016

(Whichever tiler you use) and efficient delivery of raster data (image pyramid layer) (update2)

Posted by smathermather on April 15, 2016

Subdivision of geographic data is a panacea to problems you didn’t know you had.

Maybe you deal with vector data, so you pre-tile your vector data to ship to the browser to render– you’re makin’ smaller data. Maybe you use cutting edge PostGIS so you apply ST_Subdivide to keep your data smaller than the database page size like Paul Ramsey describes here. Smaller’s better… . Or perhaps you are forever reprojecting your data in strange ways, across problematic boundaries or need to buffer in an optimum coordinate system to avoid distortion. Regardless of the reason, smaller is better.

Maybe you aren’t doing vector work, but this time raster. What’s the equivalent tiling process?  I wrote about this for GeoServer almost 5 (eep!) years ago now (with a slightly more recent follow up) and much of what I wrote still applies:

  • Pre-tile your raw data in modest chunks
  • Use geotiff so you can use internal data structures to have even smaller tiles inside your tiles
  • Create pyramids / pre-summarized data as tiles too.

Fortunately, while these posts were written for GeoServer, they apply to any tiler. Pre-process with gdal_retile.

gdal_retile.py -v -r bilinear -levels 4 -ps 6144 6144 -co "TILED=YES" -co "BLOCKXSIZE=256" -co "BLOCKYSIZE=256" -s_srs EPSG:3734 -targetDir aerial_2011 --optfile list.txt

Let’s break this down a little:

First we choose our resampling method for our pyramids (bilinear). Lanzcos would also be fine here.

-r bilinear

Next we set the number of resampling levels. This will depend on the size of the dataset.

-levels 4

Next we specify the pixel and line size of the output geotiff. This can be pretty large. We probably want to avoid a size that forces the use of bigtiff (i.e. 4GB).

-ps 6144 6144

Now we get into the geotiff data structure — we internally tile the tifs, and make them 256×256 pixels. We could also choose 512. We’re just aiming to have our tile size near to the size that we are going to send to the browser.

-co "TILED=YES" -co "BLOCKXSIZE=256" -co "BLOCKYSIZE=256"

Finally, we specify our coordinate system (this is state plane Ohio), our output directory (needs created ahead of time) and our input file list.

-s_srs EPSG:3734 -targetDir aerial_2011 --optfile list.txt

That’s it. Now you have a highly optimized raster dataset that can:

  • get the level of detail necessary for a given request,
  • and can extract only the data necessary for a given request.
  • Pretty much any geospatial solution which uses GDAL can leverage this work to make for very fast rendering of raster data to a tile cache. If space is an issue, apply compression options that match your use case.

    Posted in GDAL | Tagged: , , , | 2 Comments »

    ~~North Carolina GIS~~ rhymes with Lasagna

    Posted by smathermather on April 12, 2016

    FOSS4GNA 2016 Logo

    Picture of Garfield the cat diving into a dish of lasagna

    Last year I really enjoyed attending and presenting at North Carolina GIS in Raleigh. As many of you know, Free and Open Source Software for Geospatial North America (FOSS4GNA, alleged by some to rhyme with “lasagna”) will be in Raleigh this year, in a short few weeks.

    I highly encourage you to go. First of all, it’s FOSS4GNA, so lots of free and open source geospatial goodness. But, Raleigh and NCGIS in general are a hot spot of open source geospatial stuff. Last years’ North Carolina GIS was a mini FOSS4G, mixed with the standard uhhhh, not so open source crowd. I came expecting (most respectfully) form of (name your favorite state GIS conference). What I saw was that but also more than 20 talks on FOSS Geo stuff. Don’t believe me? Here’s the list:

    mobile_global_open

  • Open-ing the Future of NOAA GIS | Speaker Tony LaVoi
  • The Rise of 3D GIS on the Web | Speaker Patrick Cozzi
  • plas.io and Greyhound: Point Clouds in Your Browser | Speaker Howard Butler
  • Open Source, Open Discussion | Speakers: Ralph Dell GISP, Randal Hale, Jason Hibbets, Dr. Helena Mitasova GISP
  • How to Build Fat Polygons | Speaker Skip Daniels
  • How to Use GitHub to Hire Your Next Analyst | Speakers: Dave Michelson, Cameron Carlyle
  • QGIS for the Desktop | Speaker Randal Hale
  • GRASS7: New Features and Tools for Spatio-Temporal Analytics and Visualization | Speaker Dr. Helena Mitasova
  • MapLoom: A New Web-client With Versioned Editing (GeoGit) Integration | Speakers: Syrus Mesdaghi, Tyler Garner
  • Quality of Life Dashboard | Speaker Tobin Bradley
  • Defaulting to Open (at least trying to…) | Speaker Justin Greco
  • Using Geospatial Applications to Build ForWarn | Speaker Bill Hargrove
  • National Map Corps: Crowdsourcing Map Features for the USGS | Speaker Silvia Terziotti
  • Wake County Open Data: Where Will It Take You? | Speakers: Carter Vickery, Bill Scanlon
  • FOSS and Web Mapping | Speaker Ashley Hanes, A-B Tech CC
  • National Park Service GIS Data + OpenStreetMap = Places of Interest
  • Spatial Analysis of Wildfire Occurrences in North Carolina Using the R Project for Statistical Computing | Speaker David Jones
  • Point Cloud in Your Pocket | Speaker Stephen Mather
  • The Unknowns: An IT Professional’s Guide to Open Source | Speaker Paul Ramsey
  • Open Data? Show Me the Money! | Speaker Blake Esselstyn
  • Exploring Spatial Data Infrastructure in an Open Source World | Speaker Jacqueline Lowe, UNC-A
  • Open Source Geospatial Foundation (OSGeo) | Speaker Doug Newcomb
  • Open Data Kit (ODK) An Exciting, Free, and Open-Source Field Data Collection Alternative | Speaker Eric Wilson
  • But, lest you think this is a new thing, now that FOSS is up and coming in the geo world, in 2013, NCGIS played host to a dozen open source geospatial presentations, and has been on this trend since at least 2001.

    I recommend checking out some of the amazing FOSS Geo work endemic to NC State, while you are there. If you don’t know Helena Mitasova from the Open Source Geospatial Research and Education Laboratory, you should. I’m hoping she and her students have on display their Tangible Landscape which ties a sandbox in to real-time GRASS DEM processing (flow accumulation, viewsheds, fire modeling, etc.):

    http://geospatial.ncsu.edu/osgeorel/tangible-landscape.html

    Screen capture of tangible landscape video.

    Finally, I’d like to really briefly address what is in the minds of many as they think on North Carolina these days — North Carolina’s HB2 legislation. The FOSS4GNA organizers have addressed this bill, and communicated their position and their accommodations. The response, quoted in part below speaks for itself:

    As we shared shortly after HB2 was passed, it was too late to relocate and/or cancel the conference. We are very grateful to the good people who recognized that boycotting FOSS4G NA hurts a very inclusive conference and community. The fact you are coming means a great deal, and we do not take it for granted. Thank you!

    After talking with so many people in the last few weeks, it is very clear that our LGBT attendees, including myself, do not stand alone.

    Here are some of the things we have put in place to help ensure all of our attendees are safe & welcome:

  • There will be 4 gender neutral restrooms. And the venue is updating their signs to clearly state they are gender neutral restrooms.
  • For those interested, we will be encouraging people to make donations to the ACLU, who is suing the State of North Carolina because of HB2.
  • Our code of conduct is in place, and will be enforced by staff.
  • There is a map that lists trans friendly restrooms in the area. Huge thanks to Emily Waggoner for creating it!
  • Our sponsors have taken a stance against HB2.
  • I love this community, and I am so happy to see these responses. Please come and join us. It will be an amazing conference.
    Post script: Thanks to Doug Newcomb for the history lesson on NCGIS. I hope to share more of his info in future posts.

    Posted in FOSS4G-NA | Leave a Comment »