Smathermather's Weblog

Remote Sensing, GIS, Ecology, and Oddball Techniques

Posts Tagged ‘FOSS’

Viewing Sparse Point Clouds from OpenDroneMap — GeoKota

Posted by smathermather on June 30, 2016

This is a post about OpenDroneMap, an opensource project I am a maintainer for. ODM is a toolchain for post-processing drone imagery to create 3D and mapping products. It’s currently in beta and under pretty heavy development. If you’re interested in contributing to the project head over here. The Problem So for most of the […]

via Viewing Sparse Point Clouds from OpenDroneMap — GeoKota

Posted in 3D, Image Processing, OpenDroneMap, OpenDroneMap, Optics, Other, Photogrammetry | Tagged: , , , | Leave a Comment »

Using PostGIS for Hydrologic Modeling (reblog)

Posted by smathermather on June 17, 2016

The Problem We have to filter out the roads and ditches without removing streams that cross roads or follow them closely. I’m going to use PostGIS to find the intersection of the streams lines data with a buffered roads polygon. If the intersected line is less than 50% of the length of the stream line, […]

via Filtering Roads from Extracted Streams Data — GeoKota

Posted in Database, PostGIS, PostgreSQL, SQL | Tagged: , , , , , | Leave a Comment »

Efficient delivery of raster data part 4

Posted by smathermather on May 1, 2016

Guest post from my colleague Patrick Lorch who wrote up what we did the other day in order to view a whole bunch of tiled images in a directory in QGIS (I did some mild editing to his posts. Mistakes are mine). The great thing about the approach is that is generalizeable to most tools that use GDAL for their raster API. This is part of a series. You can view this series in reverse with this post:

Building virtual datasets from a bunch of tiffs
What do you do when someone gives you aerial images stored as tiles in different directories representing different zoom levels? The goal is to make them easy to use as baselayers in QGIS. The answer is to reference them in a virtual data set (VRT).

gdalbuildvrt is the ticket
First make lists of tiffs
If the directory structure is something like this:

total 8047
drwxr-xr-x 7 pdl    4096 Apr 29 14:22 ./
drwxr-xr-x 5 pdl    4096 Apr 27 22:10 ../
drwxr-xr-x 2 pdl 1310720 Apr 22 21:37 0/
drwxr-xr-x 2 pdl  393216 Apr 22 22:54 1/
drwxr-xr-x 2 pdl   98304 Apr 28 14:44 2/
drwxr-xr-x 2 pdl   32768 Apr 28 14:44 3/
drwxr-xr-x 2 pdl    8192 Apr 28 14:44 4/

Then you first need a set of list files listing tiffs in each directory.

ls 0\*.tif > list0.txt
ls 1\*.tif > list1.txt
ls 2\*.tif > list2.txt
ls 3\*.tif > list3.txt
ls 4\*.tif > list4.txt

Now make the vrts

gdalbuildvrt -input_file_list list0.txt aerial_2015_0.vrt
gdalbuildvrt -input_file_list list1.txt aerial_2015_1.vrt
gdalbuildvrt -input_file_list list2.txt aerial_2015_2.vrt
gdalbuildvrt -input_file_list list3.txt aerial_2015_3.vrt
gdalbuildvrt -input_file_list list4.txt aerial_2015_4.vrt

Now you can open these in QGIS depending on what zoom level you need.

These VRTs may now be loaded as ordinary rasters in QGIS or whatever you please. In this case, we retiled with multiple resample levels (see this post for more info), so we’ll have to define max/min ranges at which the different image collections are visible.

Thanks for the write up Pat!

Posted in GDAL | Tagged: , , , | Leave a Comment »

(Whichever tiler you use) and efficient delivery of raster data (image pyramid layer) (update2)

Posted by smathermather on April 15, 2016

Subdivision of geographic data is a panacea to problems you didn’t know you had.

Maybe you deal with vector data, so you pre-tile your vector data to ship to the browser to render– you’re makin’ smaller data. Maybe you use cutting edge PostGIS so you apply ST_Subdivide to keep your data smaller than the database page size like Paul Ramsey describes here. Smaller’s better… . Or perhaps you are forever reprojecting your data in strange ways, across problematic boundaries or need to buffer in an optimum coordinate system to avoid distortion. Regardless of the reason, smaller is better.

Maybe you aren’t doing vector work, but this time raster. What’s the equivalent tiling process?  I wrote about this for GeoServer almost 5 (eep!) years ago now (with a slightly more recent follow up) and much of what I wrote still applies:

  • Pre-tile your raw data in modest chunks
  • Use geotiff so you can use internal data structures to have even smaller tiles inside your tiles
  • Create pyramids / pre-summarized data as tiles too.

Fortunately, while these posts were written for GeoServer, they apply to any tiler. Pre-process with gdal_retile.

gdal_retile.py -v -r bilinear -levels 4 -ps 6144 6144 -co "TILED=YES" -co "BLOCKXSIZE=256" -co "BLOCKYSIZE=256" -s_srs EPSG:3734 -targetDir aerial_2011 --optfile list.txt

Let’s break this down a little:

First we choose our resampling method for our pyramids (bilinear). Lanzcos would also be fine here.

-r bilinear

Next we set the number of resampling levels. This will depend on the size of the dataset.

-levels 4

Next we specify the pixel and line size of the output geotiff. This can be pretty large. We probably want to avoid a size that forces the use of bigtiff (i.e. 4GB).

-ps 6144 6144

Now we get into the geotiff data structure — we internally tile the tifs, and make them 256×256 pixels. We could also choose 512. We’re just aiming to have our tile size near to the size that we are going to send to the browser.

-co "TILED=YES" -co "BLOCKXSIZE=256" -co "BLOCKYSIZE=256"

Finally, we specify our coordinate system (this is state plane Ohio), our output directory (needs created ahead of time) and our input file list.

-s_srs EPSG:3734 -targetDir aerial_2011 --optfile list.txt

That’s it. Now you have a highly optimized raster dataset that can:

  • get the level of detail necessary for a given request,
  • and can extract only the data necessary for a given request.
  • Pretty much any geospatial solution which uses GDAL can leverage this work to make for very fast rendering of raster data to a tile cache. If space is an issue, apply compression options that match your use case.

    Posted in GDAL | Tagged: , , , | 2 Comments »

    Leaflet Crosshairs

    Posted by smathermather on February 18, 2016

    Ok, I’m not the first to come up with that name, but I like it so it’s staying for now.

    The problem space is this: Your web development team is good with forms. They build forms like diurnal animals wake — daily: day in, day out, day in, day out. They want to build a form for a mobile web app and it just happens to use HTML5 to grab the geolocation from field deployed phones, tablets, etc.. Great! Geo problem solved — when we collect our form data, we’ll get geo-data for free.

    Not so fast, form-building Valentine — what’s the quality of those data? How accurate is that phone GPS? Is it good enough? The answer is probably yes, most of the time.

    But, good enough most of the time isn’t good enough, Faye. What I want is an embedded map where I can move the crosshairs to the actual location inside the form. It’s like all the corrections you do back at the office, but you can do them in the field while the site is still fresh in your mind. Simple. Useful. (Also, infrastructure like Fulcrum App does it because it’s so simple and useful). Hence, questions like this: http://gis.stackexchange.com/questions/90225/how-to-add-a-floating-crosshairs-icon-above-leaflet-map and pages like this: https://www.mapbox.com/blog/help-search-MH370/

    I couldn’t get the solution on Stack Exchange to work for me. Besides, I think its the wrong solution. I don’t want to move the icon back to center on the map moving, I want the map to move and the icon to stay stationary. It’s a fine (and probably irrelevant) distinction, but it feels important to me.

    So, we build a small page that has the following features:

    • A crosshairs that is stationary
    • A map that moves
    • When the map moves, our lat/lon values update in our form

    Main code as follows (careful — careless use of jquery follows):

    <!DOCTYPE html>
    <html>
    <head>
    	<title>Leaflet Crosshairs</title>
    
    	<!-- Include meta tag to ensure proper rendering and touch zooming -->
    	<!--<meta name="viewport" content="width=device-width, initial-scale=1" />-->
    	<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
    
    	<!-- Include the jQuery library -->
    	<script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
    	<!-- Include jQuery Mobile stylesheets -->
    	<link rel="stylesheet" href="//ajax.googleapis.com/ajax/libs/jquerymobile/1.4.5/jquery.mobile.min.css">
    	<!-- Include the jQuery Mobile library -->
    	<script src="//ajax.googleapis.com/ajax/libs/jquerymobile/1.4.5/jquery.mobile.min.js"></script>
    
    	<!-- Include leaflet css and js -->
    	<link rel="stylesheet" href="//cdn.leafletjs.com/leaflet/v0.7.7/leaflet.css" />
    	<script src="//cdn.leafletjs.com/leaflet/v0.7.7/leaflet.js"></script>
    
    	<style>
    	body {
    		padding: 0;
    		margin: 0;
    	}
            html, body, #map {
                height: 100%;
                width:100%;
            }
    
            #metamap {
                width: 100%;
                height: 300px;
            }
            #crosshair {
                position: relative;
                z-index: 10;
                height: 200px;
                vertical-align: middle;
            }
    	#crosshair img {
    		position: absolute;
    		margin: 0;
    		top: 50%;
    		left: 50%;
    		margin-right: -50%;
    		transform: translate(-50%, -50%);
    	}
    	</style>
    </head>
    <body>
    
        <div id="metamap">
            <div id="map">
                <div id="crosshair"><img class="crosshair" src=crosshair.png /></div>
            </div>
        </div>
        <br />
        <hr />
        Latitude: <input type="text" id="txtLatitude" />
        <br /><br />
        Longitude: <input type="text" id="txtLongitude" />
        
            <script>
                // Initiate map
                var map = L.map('map');
    
                // load map
                L.tileLayer('//api.tiles.mapbox.com/v4/{id}/{z}/{x}/{y}.png?access_token=pk.eyJ1IjoiY2xldmVsYW5kLW1ldHJvcGFya3MiLCJhIjoiWHRKaDhuRSJ9.FGqNSOHwiCr2dmTH2JTMAA', {
                    maxZoom: 20,
                    id: 'mapbox.satellite'
                }).addTo(map);
    
                // Now a function to populate our form with latitude and longitude values
                function onMapMove(e) {
                    // txtLatitude.val(map.getCenter());
                    var locale = map.getCenter();
                    $('#txtLatitude').val(locale.lat);
                    $('#txtLongitude').val(locale.lng);
                }
                
                // Boilerplate...
                function onLocationError(e) {
                    alert(e.message);
                }
    
                // When the map moves we run our function up above
                map.on('move', onMapMove);
    
                // Boilerplate
                map.on('locationerror', onLocationError);
                
                // When we load the map, we should zoom to our current position using device geolocation
                map.locate({ setView: true, maxZoom: 20 });
            </script>
    </body>
    </html>
    
    <body>
    
    
    <div id="metamap">
    
    <div id="map">
    
    <div id="crosshair"><img class="crosshair" src=crosshair.png /></div>
    
            </div>
    
        </div>
    
        
    
    <hr />
    
        Latitude: <input type="text" id="txtLatitude" />
        
    
        Longitude: <input type="text" id="txtLongitude" />
        
            <script>
                // Initiate map
                var map = L.map('map');
    
                // load map
                L.tileLayer('//api.tiles.mapbox.com/v4/{id}/{z}/{x}/{y}.png?access_token=pk.eyJ1IjoiY2xldmVsYW5kLW1ldHJvcGFya3MiLCJhIjoiWHRKaDhuRSJ9.FGqNSOHwiCr2dmTH2JTMAA', {
                    maxZoom: 20,
                    id: 'mapbox.satellite'
                }).addTo(map);
    
                // Now a function to populate our form with latitude and longitude values
                function onMapMove(e) {
                    // txtLatitude.val(map.getCenter());
                    var locale = map.getCenter();
                    $('#txtLatitude').val(locale.lat);
                    $('#txtLongitude').val(locale.lng);
                }
                
                // Boilerplate...
                function onLocationError(e) {
                    alert(e.message);
                }
    
                // When the map moves we run our function up above
                map.on('move', onMapMove);
    
                // Boilerplate
                map.on('locationerror', onLocationError);
                
                // When we load the map, we should zoom to our current position using device geolocation
                map.locate({ setView: true, maxZoom: 20 });
            </script>
    </body>
    </html>
    
    

    Screen shot of app in action

    Things to fix:

    • Alignment of crosshairs so they are properly centered
    • Better looking crosshairs
    • Rounding for those coordinate values
    • Do we need jQuery? Pro’ly not

    That was my fun for the day. Shout-out to Tanetta Jordan my brilliant paired programmer for the day. Without her, this would have taken a week… .

    Oh ya, and git-repo here: https://github.com/cleveland-metroparks/leaflet-crosshairs

    Look there for updates that should include the improvements above.

    Posted in Javascript, Leaflet | Tagged: , , , | Leave a Comment »

    GDAL tools of the trade: GDAL, MrSid, and nearblack revisited

    Posted by smathermather on February 11, 2016

    internet_archiveBack in 2011 I wrote a post about GDAL, translating MrSID files and trimming off the artifacts so mosaics work. I’m revisiting some similar work this week and realized I had lost my copy of FWTools that can deal with MrSIDs. The Internet Archive came to the rescue.

    At some point LizardTech licensed a DLL that allowed for compilation compatible with FWTools licensing so we could do things like use gdal_translate to covert a MrSID file to a tiff. Later on, the licensing changed, but binaries created prior to the change hold that original license. The WayBack Machine archived one such copy here: https://web.archive.org/web/20130127013649/http://home.gdal.org/fwtools/FWTools202.exe .

    Download, install, and now for some Batch scripting fun… .

    W:\Aircraft\naip_ortho>for %I in (*.sid) do gdal_translate %I %~nI.tif

    W:\Aircraft\naip_ortho>gdal_translate ortho_1-1_1n_s_oh035_2015_1.sid ortho_1-1_
    1n_s_oh035_2015_1.tif
    Input file size is 58222, 49140
    0…10…20.

    Posted in GDAL, Image Processing | Tagged: , , , | 2 Comments »

    MSF Canada Drone Day follow-up

    Posted by smathermather on July 13, 2015

    Dirk’s MSF Canada Drone Day is officially the first blog post I have “re-blogged”. Please read: https://smathermather.wordpress.com/2015/07/13/msf-canada-drone-day/

    or better yet here: http://dirkgorissen.com/2015/07/14/msf-canada-drone-day/

    I had the pleasure of co-presenting with Dirk and Ivan, and the rest is well covered in Dirk’s post. It came together as an excellent day and I think you would be hard pressed to have had a better introduction to drones.

    The day was valuable to me as an emerging practitioner. I learned more about the state of the art in hardware, software, regulations, philosophy, and RC control from this day, and it was inspiring to inhabit the same space with such dedicated practitioners for a short time.

    Beyond the value of the workshop to the participants, the outcomes were the following, this quoted from Dirk’s post:

    As a first milestone we are looking to pull together a proposal to the Humanitarian Innovation Fund in collaboration with OpenDroneMap and supported by the Missing Maps consortium.

    I love the extension of ODM into this space. This is the real value of open source, the opportunity to collaborate across the world, across industries and use cases, and across organizations. Expect to see improvements to ODM in usability, performance, and output qualities from this initiative. More on this later.

    Another outcome / learning for me was observing Ivan’s OpenUAV. From his repo:

    This is intended to be a repository for design files, instructions, photos, documentation, and everything else needed for people wishing to build a and operate UAV (drone) in a low-income, resource-poor environment. This is not about cutting-edge UAVs, it’s about democratizing the technology and getting it into the hands of more people, particularly in poorer countries and humanitarian settings.

    Photo of OpenUAV example

    Ivan undersells it. This is a pro quality quad copter on a very nice price diet — a brilliant piece of pragmatic engineering.

    This little quad copter will make its way into drone building workshops I’ll be offering in Cleveland and Columbus Ohio and Seoul, South Korea in August and September. More details forthcoming.

    If you are in Cleveland, plan to be at FOSS4G Seoul, or Ohio GIS, come build Ivan’s capable quad.

     

    (BTW, Ivan says with a couple of 4C 8000mAh batteries, this sucker flies for 50 minutes… .)

    Posted in 3D, Bundler, Camera Calibration, FOSS4G, Image Processing, OpenDroneMap, OpenDroneMap, Optics, Photogrammetry, PMVS | Tagged: , , , , , , , , , | Leave a Comment »

    OpenAerialMap, OpenImageryNetwork, MapKnitter, OpenTerrain, and OpenDroneMap (cont. 1)

    Posted by smathermather on June 7, 2015

    Citing my previous post, let’s move on to more specifics on my thoughts regarding the integration of OpenAerialMap, OpenDroneMap, and MapKnitter as projects.

    Image from kite over Seneca Golf Course

    OpenDroneMap ❤ OpenAerialMap.

    OpenAerialMap will become a platform by which drone users can share their imagery under an open license.

    So, as the metadata spec for OpenAerialMap and OpenImageryNetwork matures, and as soon as a publicly available place for drone users to push their data comes online, ODM will write appropriate metadata and geotiffs to go into OIN to be indexed by OAM. Probably as an added bonus, ODM should be able to optionally auto-upload outputs from to the appropriate node on the OpenImageryNetwork.

    Lincoln Peak Vinyard

    OpenDroneMap ❤ MapKnitter.

    MapKnitter / ODM integration is pretty straight forward in my mind too. There are ways that MapKnitter complements ODM, and vice versa. ODM does not have a graphical user interface at this time. MapKnitter promises to fill that role in a future OpenDroneMap implementation. MapKnitter has no image blending or auto-matching tools. OpenDroneMap will soon have both.

    • Ways MapKnitter may help OpenDroneMap:
      • MapKnitter’s clever use of Leaflet to handle affine transformation of images is really exciting, and may help with improving final georeferencing for ODM datasets.
      • Regarding the above, one really useful thing for fliers launching balloons, drones, and kites without GPS would be the ability to quickly and easily perform really approximate georeferencing. I would envision a workflow where a user moves an image to its approximate position and size relative to a background aerial. ODM would be able to take advantage of this approximate georeferencing to optimize matching.
    • Ways OpenDroneMap could benefit MapKnitter
      • For large image datasets, matching images can be very tedious. Automatic feature extraction and matching can help. OpenDroneMap could be adapted to serve back match information to Mapknitter to ease this process. This will become increasingly important as MapKnitter raises the ~60 image limit on images that it can process.
      • A near future version of ODM will have image blending / smoothing / radiometric matching. For the server portion of the MapKnitter infrastructure, this feature could be a really useful addition for production of final mosaics.

    These projects (plus OpenTerrain…) are really exciting in their own right. Together, they represent amazing opportunities to foster, cultivate, process, and serve a large community of imagery providers, from individuals and small entities capturing specific datasets using kites, drones, and balloons, to satellite imagery providers hosting their own “image buckets” of open imagery data. Exciting times.

    Image over Groth Memorial from kite

    Posted in 3D, Bundler, Camera Calibration, FOSS4G, Image Processing, OpenDroneMap, Optics, Photogrammetry, PMVS | Tagged: , , , , , , , , , , | Leave a Comment »

    OpenAerialMap, OpenImageryNetwork, MapKnitter, OpenTerrain, and OpenDroneMap

    Posted by smathermather on May 29, 2015

    This tweet:

    is the beginning of some fruitful discussion, I suspect. There are some really awesome projects gaining momentum. I’ll give an overview of them as best I am able.

    Kite aerial photography image over bridle riding ring.

    Kite aerial photography image over bridle riding ring.

    Let’s start with the one nearest and dearest to my heart (if you’ve been reading my blog, you can skip this part): OpenDroneMap. OpenDroneMap is an open source toolkit for processing drone, balloon, kite imagery into geographic data. It does this by using fully automated feature-matching between images, which create a 3D point cloud. From that, we can create a 3D surface (mesh), textured mesh, and orthophoto. This guy says it better:

    But, it’s just a stand alone, Linux (Ubuntu)-based tool. It requires some geekiness to run, and it does not (at least not yet) act as a platform.  By that I mean, generically, you can’t just upload images to it and get the wonderful output from a service, and we don’t have a place to store and share all this wonderful data that comes from and will be coming from drones and other aerial platforms. This is where (from my selfish perspective) the other projects are so well timed… .

    Screenshot of DevelopmentSeed's introductory post for OpenAerialMap

    Let’s start with OpenAerialMap. From the Development Seed blog post on it (yes, you should follow the link. Don’t worry, I’ll wait until you return):

    OpenAerialMap is a set of tools for searching, sharing, and using open satellite and drone imagery. This initial release includes the core infrastructure to catalog petabytes of open imagery. It also includes an extremely usable API and an elegant web interface to submit, search and download available imagery.

    This is a reboot of a couple of previous attempts at solving this problem space, and it’s really exciting to watch passionate and brilliant work take place to make this happen. Also, this is not an easy problem space, and is being really thoughtfully simplified and implemented.

    (As a side note, I’m not going to get into the distinction between OpenAerialMap and OpenImageryNetwork — not today anyway)

    Preview of Open Terrain tumblr page

    Preview of Open Terrain tumblr page

     

    Open Terrain is a project for which a portion of its scope is to do for terrain models what OpenAerialMap and OpenImageryNetwork will do for open aerial datasets. The projects are informing each other and growing together, which is awesome collaboration to observe.

    Finally, Mapknitter has recently been rebooted too, and it’s now a really elegant tool for taking a few aerial images and knitting them into a usable map (ok, it always was p. cool — now it’s even more elegant). What’s great about MapKnitter is it specifically addresses the problem of georeferencing balloon, kite, or drone images in a simple-to-use interface in the browser.

    Snapshot of MapKnitter landing page

    So, back to the question:

    Bravo, yes. Lets. I have been thinking about, talking to people, discerning the strengths, overlaps, and complementary fittings of these projects as they have emerged. We are headed toward some really great things… . More specific thoughts to come.

    Posted in 3D, Bundler, Camera Calibration, FOSS4G, Image Processing, OpenDroneMap, Optics, Photogrammetry, PMVS | Tagged: , , , , , , , , , , | 1 Comment »

    OpenDroneMap Video for FOSS4GNA

    Posted by smathermather on March 6, 2015

    Thanks to some brilliant work by our Interpretive Technology Technician, we now have a little video for FOSS4GNA on OpenDroneMap. Come see my presentation next week in San Francisco, Weds. at 13:30 (1:30PM):

    odm_screenshot

    Posted in 3D, Bundler, Camera Calibration, FOSS4G, Image Processing, OpenDroneMap, Optics, Photogrammetry, PMVS | Tagged: , , , , , , , | 6 Comments »