Smathermather's Weblog

Remote Sensing, GIS, Ecology, and Oddball Techniques

Normalizing tables in PostgreSQL: experiments with Fulcrum

Posted by smathermather on February 12, 2015

Fulcrum by Spatial Networks is an interesting and very useful hosted service for collecting geo data in the field with whatever smartphone or tablet you already own.  It works well, so long as you just want to collect points, and the field building capabilities are quite extraordinary. Also, for many applications, it’s cheaper than custom solutions, so it hits that sweet spot well. If you’ve been paying attention, I prefer my hosted solutions to be (nearly) purely open source, so as to avoid vendor lock-in. Fulcrum is sufficiently compelling, we are using it carefully and with a bit of joy, in spite of not meeting my normal preferences.

Screen shot of Fulcrum's form builder

So, I thought I’d share some table normalization we are doing on our wetland inventory data that we’ll be collecting with Fulcrum. The State of Ohio specifies a rapid assessment method for classifying wetlands called Ohio Rapid Assessment Method for Wetlands (ORAM) (warning– pdf). This assessment method has force of law, and on paper is a one-page, two-sided form with a range of physical, water regime, land cover and other characterizations, and to date we’ve always filled it out on paper, transcribed that at the office, and then populated spreadsheets, or more recently a PostGIS/PostgreSQL database.

Where we run into some difficulty is that some of our form values can be multiple values. Fulcrum returns a single field for these, with comma delimited contents:

comma_delimited

Since we need to do lookups and calculations on these fields, we really need some normalization. Enter regexp_split_to_table.

CREATE TABLE nr.oram_metrics AS
  SELECT oram_id, 'm1_wetland_area'::text AS metric, regexp_split_to_table(m1_wetland_area, ',')::text AS selection
    FROM nr.cm_oram_data;

Now in our case, we have one lookup table for ORAM. Perhaps it should be multiple tables, but it’s just one discrete (ht. Brian Timoney for grammar fix) thing, this ORAM form, so we keep it simple. This means for each of our columns that should be normalized, we can throw them all in one table. We use UNION ALL to accomplish this:

CREATE TABLE nr.oram_metrics AS
  SELECT oram_id, 'm1_wetland_area'::text AS metric, regexp_split_to_table(m1_wetland_area, ',')::text AS selection
    FROM nr.cm_oram_data

UNION ALL

  SELECT oram_id, 'm2a_upland_buffer_width'::text AS metric, regexp_split_to_table(m2a_upland_buffer_width, ',')::text AS selection
    FROM nr.cm_oram_data;

Finally, we can use a FOREIGN KEY constraint with a table that has all our metrics and selections plus our lookup value to complete the exercise:

CREATE TABLE nr.oram_metrics AS
  SELECT oram_id, 'm1_wetland_area'::text AS metric, regexp_split_to_table(m1_wetland_area, ',')::text AS selection
    FROM nr.cm_oram_data

UNION ALL

  SELECT oram_id, 'm2a_upland_buffer_width'::text AS metric, regexp_split_to_table(m2a_upland_buffer_width, ',')::text AS selection
    FROM nr.cm_oram_data;

ALTER TABLE nr.oram_metrics ADD CONSTRAINT "metric_constraint"
  FOREIGN KEY (metric, selection) REFERENCES nr.oram_score_lookup_all (metric, selection);​ 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: