Just-in-time spatial

Acquisition, storage, & lightweight presentation of dynamic data

John J Czaplewski | UW - Madison | @JJCzaplewski
Maria Hart | CFIRE | @freightplanner

Outline

• Scraping
• Presentation
• Separating geometry and attributes

Part I: Scraping

What it is:

• Programmatic data acquisition
• Data transformation and extraction from known sources
• Accessing previously inaccessible data

Who (might) care:

• Researchers
• Journalists
• Hobbyists

Why can't I just download a spreadsheet of the data?

• Data provider doesn't realize the value
• Publishing is an afterthought
• Competitive advantage

Picking the right source

• Keep it targeted
• What's it worth?

NO

Yes

Yes

Yes

Yes

Yes

topojson

An extension of GeoJSON that encodes topology

topojson + CSV > GeoJSON

Natural Earth 10m Sovereign Entities

• Shapefile: ~ 11mb
• GeoJSON: 24.7mb
• Topojson + CSV: 2mb

Know your limits (or proxies)

• Test locally
• Process locally
• Download slowly

Why all open-source?

• Tool options
• Control
• Share / Repeat

Example: City of Madison Data

Truck Driver Shortage

http://midamericafreight.org/rfs_maps/drivershortage

Energy Information Administration (EIA)

http://midamericafreight.org/rfs_maps/template/eia_example

Thank you to CFIRE/MAFC!

Find the slides at http://johnjcz.com/presentations/foss4g2013
the code at https://github.com/jczaplew/foss4gna_examples
and me @JJCzaplewski

Questions?