QGIS 2.0 and the Open Source Learning Curve

As likely everyone in the geo world knows by now, the widely awaited release of QGIS 2.0 was announced at FOSS4G this week. Having closely watched the development of QGIS 2.0 I was eager to get it installed and take a look at the new features. I am for the most part a Windows user, but I like to take the opportunity to work with the open source geo stack in a Linux environment. My Linux skills are better than a beginner, but because I don’t work with it on a daily basis, those skills are rusty. So I decided this is a good excuse to build out a new Ubuntu virtual machine, as I haven’t upgraded since 10.04.

Build Out The Virtual Machine

Let’s just say I was quickly faced with the challenges of getting Ubuntu 12.04 installed in VMware Workstation 7.1. Without getting into details, the new notions of “Easy Install” and its effect on installing VMware Tools took a while to figure out. In case anyone has the Easy Install problem, this YouTube video has the answer, also posted in code below:

sudo mv /etc/rc.local.backup /etc/rc.local
sudo mv /opt/vmware-tools-installer/lightdm.conf /etc/init
reboot the virtual machine

The second problem, about configuring VMware Tools with the correct headers, is tougher. I’m still not sure I fixed it, but given its rating on Ask Ubuntu, it’s obviously a problem people struggle with. Suffice it to say there is no way that I could have fixed this on my own, nor do I really have a good understanding of what was wrong, or what was done to fix it.

The Open Source Learning Curve

This leads me to the point with this post, open source is still really tricky to work with. The challenges in getting over the learning curve leads to two simultaneous feelings: one, absolute confusion at the problems that occur, and two, absolute amazement at the number of people who are smart enough to have figured them out and posted about it on forums to help the rest of us. While its clear the usability of open source is getting markedly better (even from my novice perspective), it still feels as if there is a “hazing” process that I hope can be overcome.

QGIS 2.0 Installation

With that in mind, here is some of what I discovered in my attempt to get QGIS 2.0 installed on the new Ubuntu 12.04 virtual machine. First, I followed the directions from the QGIS installation directions, but unfortunately I was getting errors in the Terminal related to the QGIS Public Key, and a second set on load. Then I discovered the Digital Geography blog post that had the same directions, plus a little magic command at the end (see below) that got rid of the on load errors.

sudo chown -R "your username" ~/.qgis2

After viewing a shapefile and briefly reviewing the new cartography tools, I wanted to look at the new analytical tools. But there was a problem, and they didn’t load correctly. So back to the Google, and after searching for a few minutes I found the Linfinty Blog on making Ubuntu 12.04 work with QGIS Sextante. After a few minutes downloading tools I check again, this time the image processing tools are working, but I keep getting a SAGA configuration error. After figuring out what SAGA GIS is, and reading the SAGA help page from the error dialog box, it was back to searching. The key this time was a GIS Stack Exchange post on configuring SAGA and GRASS in Sextante, this time the code snippet of glory is:

sudo ln -s /usr/lib64/saga /usr/lib/saga

Next, back to QGIS to fire up a SAGA powered processing routine, and the same error strikes again. More searching and reading, now its about differences in SAGA versions and what’s in Ubuntu repos. With no clear answer, I check the Processing options in which there is an “Options and Configuration”, which has a “Providers” section, which as an “Enable SAGA 2.0.8 Compatibility” option.  Upon selecting the option, boom, the all the Processing functions open. Mind you I haven’t checked whether they actually work, after several hours they now at least open.

"Processing" configuration options in QGIS 2.0
“Processing” configuration options in QGIS 2.0


It is clear that the evolution of open source geo took several steps forward in this last week, and the pieces are converging into what will be a true end-to-end, enterprise-grade, geographic toolkit. And while usability is improving, in my opinion, it is still the biggest weakness. Given the high switching costs for organizations and individuals to adopt and learn open source geo, the community must continue to drive down those costs. It is unfair to ask the individual developers who commit to these projects to fix the usability issues, as they are smart enough to work around them. Seems to me it is now up to the commercial users to dedicate resources to making the stack more usable.

Key Resources:

World Country Polygon Datasets

The Humanitarian Information Unit (HIU) has released several new datasets that leverage the Office of the Geographer‘s work on mapping International Boundaries. The Large Scale International Boundaries (LSIB) dataset, maintained by the Geographic Information Unit (GIU), is a vector line file that is believed to be the most accurate worldwide (non-Europe, non-US) international boundary vector line file available. The lines reflect U.S. government (USG) policy and thus not necessarily de facto control (cited from metadata attached to files). In September 2011, the HIU first released the boundaries publicly for download. Working with colleagues at DevelopmentSeed after that release, they made some substantial improvements to the underlying data structure that helped lead to this work.

The LSIB dataset is designed for cartographic representation and map production. However, this poses a problem for GIS analysis, because the dataset is only composed of vector lines of terrestrial boundaries between countries. This means they do not contain coastlines, and could not be converted into polygons for GIS analysis. To address this issue, the HIU combined the LSIB dataset with the World Vector Shorelines (1:250,000) dataset. The combination of these two datasets is one of the highest resolution country polygon datasets available. Additionally, the LSIB-WVS polygon file is believed to be the most accurate available dataset for determining island sovereignty. It corrects the numerous island sovereignty mistakes in the original WVS data (cited from metadata attached to files).

Two other modifications were made to the datasets. First, the large cartographic scale of the data also introduces a problem in that the data are too detailed for global scale mapping. Therefore, the HIU also created “generalized” versions of the original LSIB-WVS polygons that are suitable for smaller scale mapping. Second, in order to facilitate the ability to “join” data to the polygons in a GIS, several attributes were added to the database, including Country Name and several ISO 3166-1 Country Codes (ISO Alpha 2, ISO Alpha 3, and ISO Number). After a year of work, the data have been released into the public domain.

All datasets can be downloaded from the HIU Data page or the links below:

LSIB – WVS Country Polygons

High Resolution LSIB-WVS Country Polygons (Americas) :: https://hiu.state.gov/data/Americas_LSIBPolygons_2013March08_HIU_USDoS.zip

High Resolution LSIB-WVS Country Polygons (Africa/Eurasia) :: https://hiu.state.gov/data/EurasiaAfrica_LSIBPolygons_2013March08_HIU_USDoS.zip

Simplified Versions

Simplified Global World Vector Shorelines :: https://hiu.state.gov/data/Global_SimplifiedShoreline_2013March08_HIU_USDoS.zip

Simplified Global Country Polygons :: https://hiu.state.gov/data/Global_LSIBSimplifiedPolygons_2013March08_HIU_USDoS.zip

LSIB Lines

Large Scale International Boundaries (LSIB), AFRICA and the AMERICAS :: https://hiu.state.gov/data/AFRICAandAMERICAS_LSIB4b_2012Sep04_USDoS_HIU.zip

Large Scale International Boundaries (LSIB), EURASIA :: https://hiu.state.gov/data/EURASIA_LSIB4b_2012Sep04_USDoS_HIU.zip

Cartographic Guidance

Note, both the polygon and line datasets are useful for cartographic representation. This is due to the variety of different boundary classifications that are in the LSIB. Below is a subset from the metadata attached to the datasets that describes USG cartographic representation of the boundary lines.

From the LSIB lines metadata:
The “Label” attribute field provides a name for any line requiring non-standard depiction, such as “1949 Armistice Line” or “DMZ”

The “Rank” attribute categorizes lines into one of three categories:
a) A rank of “1” (includes most of the 320 international boundaries) for those which the USG considers “full international boundaries.”
b) A rank of “3” for other lines of international separation. Most are considered by the US government to be in dispute.
c) A rank of “7” for other lines of separation such as DMZ’s, No-Mans Land (Israel), UNDOF zone lines (Golan Hts.), Sudan’s Abyei, and for the US Naval Base Guantanamo Bay on Cuba.

Any line with a rank of “3” or “7” is to be dotted or dashed differently and in a manner visually subordinate to the normal rank “1” lines.

Additional information about how the LSIB dataset is produced, and the processes that went into the production of the new datasets are included in the metadata.

And for more information about the Office of the Geographer, see the article from State Magazine below:

State Magazine (March 2009) Office of the Geographer
Article about the Office of the Geographer from State Magazine in March 2009

Geography jobs at the U.S. Department of State

Update: Both jobs have been filled (21 Sept 2013)

The Jobs

Available immediately, the U.S Department of State is looking to fill two positions related to geographic analysis and geographic programming. The Office of eDiplomacy, the State Department’s knowledge management gurus, want to build a first-rate geographic application development team. The two-person team will work with DoS bureaus to understand their workflows, leverage the geographic components of their data, and build custom geographic applications to help them. The team will be composed of one GIS applications developer, and one GIS analyst. Each position will require a substantial overlap in skills, meaning the developer must understand GIS analysis and the analyst will have to have some programming experience.

The Backstory

Over the past two years I have been working as the GIS Architect at the Humanitarian Information Unit, a division of the Office of the Geographer and Global Issues, Bureau of Intelligence and Research at the U.S. Department of State.

In that capacity, I started a project to build a completely open source geographic computing infrastructure focused on humanitarian applications. Called the CyberGIS, this project is built exclusively from open source geographic technology including, PostGIS/PostgreSQL, GeoServer, TileCache, OpenLayers, and TileMill, along with the standard Ubuntu, Apache, Tomcat, jQuery components, and we host our production environment in Amazon Web Services. Using the term CyberGIS was intentional and intended to place the project inline with on-going efforts in the academic community to unite the worlds of geographic information science and cyberinfrastructure.

We have used this infrastructure to build several HIU geographic web applications, including the Imagery to the Crowd projects. These award winning projects are just the beginning for the CyberGIS at the HIU, we have several applications under development that we hope to unveil publicly in the coming months. The success of the HIU CyberGIS has raised the attention of geography in the Department, and the fact that eDiplomacy is building this development team is a huge step in expanding the power of Geography to the entire State Department. Two years ago I could not have expected that we could move this far this fast, and now we have an opportunity to fundamentally influence how the Department operates.

The Ask

If you have serious GIS analysis and open source geographic developer skills and want to be part of a geographic revolution, then I encourage you to apply. We need forward leaning, capable folks who fundamentally understand spatial analysis and geographic technololgy. You must be willing to work hard and be leaders in showing how geographic data and analysis can improve American diplomacy. This is a unique moment and we need the right team.

The main Careers page on the ActioNet website is here

The GIS Business Analyst position here

The GIS Developer position here

Export as KML for Google Earth/Google MapsOpen standalone map in fullscreen modeExport as GeoJSONExport as GeoRSS
U.S. Department of State

loading map - please wait...

U.S. Department of State 38.894607, -77.048439 Geography Jobs Here