Architecture of an Open Data Initiative, Fed Style

Michael Byrne ‏(@byrne_tweets), the Geospatial Information Officer (GIO) for the Federal Communications Commission (FCC) has been tweeting out queries like this:

“@tmcw @vtcraghead @SeanGorman so if you could design an open data initiative in govt, what would you do? “

to which he has gotten everything from <140 character responses from e.g. Bill Morris of:

@byrne_tweets Nationalize the high-res imaging satellites! Oh sorry – You got me all fired up there.

to presentations on data sharing collaboration:

@byrne_tweets @tmcw @vtcraghead API + bulk download + federated discovery then promote remixing and analysis with samples and partners

followed by geocommons principles for data sharing/collaboration:

as well as gist on github of “open data chatter”

Check out the last couple of days of @byrne_tweets.  This is a thread worth following.


My thoughts.  First, in principle, I should be adding this to gist above, but this won’t be structured so cleanly as that, so we’ll start with a narrative.

FedIT, Open Data Initiatives, etc., need to not only be about

  1. sharing of data now held too close to the federal bosom for general use
  2. nationalization of high-res satellites

Nor only about sharing structures, or the best modes of sharing.  Fundamentally, FedIT should leverage its primary assets:

  1. Size
  2. Hierarchy and discoverability
  3. Capacity to Guide IP choices
  4. Vision new opportunities to facilitate public good

First: Size.

A good model for cost sharing/open data is the Ohio Geographically Referenced Information Program (OGRIP) Ohio Statewide Imagery Program (OSIP).  From this document:

Ohio Statewide Imagery Program (OSIP) is a $5.6M program to acquire 1-foot pixel resolution color imagery with a 2-foot Digital Elevation Model (DEM) for the entire state to support spatial analysis. This program also provides local government the opportunity to leverage the state’s contract to enhance the state’s product and obtain an economy of scale.

This is deeper than just “Open Data”, but a program targeted at leveraging the State’s asset, relative to local governance– leverage in contract negotiations.  Thus, local municipalities have the option to “buy-up” and get better products for less capital.  It is a win for tax payers, but also facilitates fiscal opportunities for the contractors that otherwise they would not or could not create on their own.

Second: hierarchy and discoverability.

One thing that struck me in Michael Byrne’s presentation at FOSS4G North America 2012 was his insistence on the value of the humble URL as an API for data and reporting.  The disconsonant thing for me, however, was the expectation that the URL, arranged in a hierarchy, such as http://baseurl/state/county/municipality might be widely applicable.  Thinking of the datasets I create and distribute, this seemed unlikely.

But I was wrong.  Of course, if an FedIT open data initiative included providing structure and discoverability for local data, this would be and ideal geographic bin for local data, whatever the hierarchy be within that bin.

Third: Capacity to guide IP choices

FedIT/Open Data should, as a vehicle of public good, be driven by policy which encourages procurement of commercial open source software, open standards, etc., but which also strongly encourages the procurement of software and methodologies free of patents.

Fourth: Vision new opportunities to facilitate public good– data processing infrastructure

FedIT/Open Data should not only be about release and availability of data, but also the development of technical infrastructure to facilitate a new public good. *

FedIT needs to envision what the next infrastructure development piece is to facilitate public good and innovation, and needs to develop it in advance of industry such that the “personal” and competitive good of corporations is supplemented by and sublimated to the public good.

Here, it is not just about storage, capacity and APIs, but about technical infrastructure for data processing.

Simple example: photo archives

Where is the value in photo archives in the future?  The greater value is in reconstruction of unstructured data, e.g.


ultimately to fully structured point clouds in space and time:

Colosseum point cloud

FedIT should be exploring building infrastructure to consume, develop, and propagate unstructured data into structured data.  Those data, much like census data, could then be value added, packaged, and resold, but also available directly as a public good.

This example model would encourage the sharing of photos outside the walled gardens of Facebook, Google+, Apple, etc., as it forges a vision of mapping and data construction that focuses the infrastructure on the development of a common good– a fully referenced 3D point cloud and mesh in the public domain.

* footnote– originally in main text:


The web is the early version of this technical infrastructure development.  Search engines would have been another good example, but there we now depend on the benevolence of Google (and Bing and Siri and duckduckgo).  OpenStreetMap is a global phenomenon touched off by government data restrictions, and derives its own momentum from the walled gardens of data in industry and governance around the world.  It is thus an example of an “end around” to failure in governance.  I would argue that at an organic level, it is thus that OpenStreetMap is it’s own form of governance (ssh, don’t tell them I said that…).

The underlying problem with FedIT staying ahead on technical infrastructure development is hinted at in Bill Morris’s tweet:

@byrne_tweets @smathermather It’s all that’s been on my mind recently, but yes, Lidar would be nice too.

LiDAR is an example where proprietary standards are the defacto standard, but more importantly a case where the data collection happened so far ahead of a FedIT vision, that some data has forever been lost from the public domain, where local municipalities had no capacity to store the valuable but not value-added raw products.

One thought on “Architecture of an Open Data Initiative, Fed Style

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.