GovHack 2014 post-mortem

UPDATE 20 October 2014: After a bungle on my Amazon EC2 instance, the demo URL on our hackerspace project page is no longer active. I've resurrected this site on my demo server on Rackspace here. Ignore the link on the hackerspace page until that page gets updated (if it will get updated, because I can't do it)

Earlier this month, I attended the GovHack 2014 hackathon, along with thousands of other fellow hackers all across the country. This was my first GovHack, but not my first hackathon. My previous hackathon was RHoK and having no idea how GovHack would turn out, I entered the GovHack event with a RHoK-based mindset of how I would expect this hackathon to turn out.

Bad idea.

I learned very quickly there was a major difference between RHoK and GovHack. Going into RHoK, you have an idea about what solutions you will get to hack on over the weekend as problem owners are present to pitch their ideas to the audience of prospective hackers. With GovHack, you need an idea about what solution you want to hack on over the weekend, all they were going to provide was the various open data and APIs. What on earth are we going to build?

So after losing nearly half the weekend to analysis paralysis, our team (named CreativeDrought, wonder why?) agreed with my suggestion of just building a MapGuide-based mashup of various open datasets, most notably, the VicRoads Crash Stats dataset and related transportation data. I obviously knew MapGuide inside-and-out and its capabilities to have a level of confidence that with the remaining weekend we should still be able to crank out some sort of workable solution. At the very least, we'd have a functional interactive map with some open data on it.

And that's the story of our CrashTest solution in a nutshell. It's a Fusion application, packed to the gills with out-of-the-box functionality from its rich array of widgets (including Google StreetView integration). The main objective of this solution was to allow users to view and analyse crash data, sliced and diced along various age, gender, vehicle type and various socio-economic parameters.

MapGuide's rich out-of-the-box capabilities, Maestro's rapid authoring functionality and GDAL/OGR's ubiquitous data support greatly helped us. I knew with this trio of tools, that we could assemble an application together in the remaining day and a bit left that we had to actually "hack" on something.

Sadly, we only got as far as putting the data on the map for the most part. Our team spent more time frantically trying to massage various datasets via ogr2ogr/Excel/GoogleDocs into something more usable than actually writing lines of code! Seriously VicRoads? Pseudo-AMG? Thank goodness I found the necessary proj4 string for this cryptic coordinate system so that we could re-project a fair chunk of the VicRoads spatial data into a coordinate system that better reflects the world we want to mash this data up with!

Still, our "solution" should hopefully still open up a lot of "what if" scenarios. Imagine looking at a cluster of accident events, not being able to ascertain any real patterns or correlation and so you then fire up the StreetView widget and lo-and-behold, Google StreetView providing additional insights that a birds-eye view could not. Also imagine the various reporting and number crunching possibilities that are available by tapping into the MapGuide API. Imagine what other useful information you could derive if we had more time to put up additional useful datasets. We didn't get very far on any of the above ideas, so just imagine such possibilities if you will :)

So here's our entry page if you want to have a look. It includes a working demo URL to a Amazon EC2 hosted instance of MapGuide. Getting acquainted with Amazon Web Services and putting MapGuide up there was an interesting exercise and much easier than I thought it would be, though I didn't have enough time to use the AWS credits I redeemed over the weekend to momentarily lift this demo site out of the free usage tier range performance-wise. Still, the site seems to perform respectably well on the free usage tier.

Also on that page is a link to a short video where we talk about the hack. Please excuse the sloppy editing, it was obviously recorded in haste in a race against time. Like the solution and/or the possibilities it can offer? Be sure to vote on our entry page.

Despite the initial setbacks, I was happy with what we produced given the severely depleted time constraints imposed on us. I think we got some nice feedback demo-ing CrashTest in person at the post-mortem event several days later, which is always good to hear. Good job team!

So what do I think could be improved with GovHack?

• Have a list of hack ideas (by participants who actually have some ideas) up some time before the hackathon starts. This would facilitate team building, letting participants with the skills, but without ideas easily gravitate towards people/teams with the ideas.
• The mandatory video requirement for each hack entry just doesn't work in its current form. Asking teams to produce their own videos puts lots of unnecessary stress on teams, who not only have to come up with the content for their video, but have to also deal with the logistics of producing said video. I would strongly prefer that teams who can/want to make their own video do so, while other teams can just do a <= 3 minute presentation and have that be recorded by the GovHack organisers. Presentations also lets teams find out how other teams fared over the weekend. While everyone else in the ThoughtWorks Melbourne office was counting down to the end of the hackathon, I was still frantically trying to record my lines and trying not to flub them! I raided the office fridge for whatever free booze that remained just to calm myself down afterwards. I don't want to be in that situation ever again!
• Finally, the data itself. So many "spatial" datasets as CSV files! So many datasets with no coordinates, but have addresses, horribly formatted addresses, adding even more hoops to geocode them. KML/KMZ may be a decent consumer format, but it is a terrible data source format. If ogr2ogr can't convert your dataset, and requires a manual intervention of QGIS to fix it, then perhaps it's better to use a different spatial data format. Despite my loathing of its limitations, SHP files would've been heavily preferred for all of the above cases. I've made my thoughts known on the GovHack DataRater about the quality of some of these datasets we had to deal with and got plenty of imaginary ponies in the process.

Despite the above points, the event as a whole was a lot of fun. Thanks to the team (Jackie and Felicity) for your data wrangling and video production efforts.

Also thanks to Jordan Wilson-Otto and his flickr photostream where I was able to get some of these photos for this particular post.

Would I be interested in attending the 2015 edition of GovHack? Given I am now armed with 20/20 hindsight, yes I would!

Other new things (or things I forgot to talk about) in Fusion for MapGuide 2.6

This post has been stuck in my draft queue for several months now. I guess I forgot about this post so some of the content in this post may have already been talked about. Still, there some content here that I'm certain hasn't been talked about, and since I currently don't have idea at the moment what has and hasn't been talked about, I've decided to just publish the whole post.

So here's some new features (or features I've probably already talked about) in Fusion for MapGuide Open Source 2.6

OpenLayers updated to 2.13.1

OpenLayers in Fusion has been updated to the latest stable release, which is 2.13.1. This gives us important fixes for tiled layers in IE10, among other fixes and improvements from the 2.12 release.

Short of a few specific customizations to the Google layer (to disable tilt at lower levels), you'll be glad to hear that the OpenLayers included with Fusion is now mostly vanilla with very few modifications. Making your own custom OpenLayers build should be a painless affair.

Built with Closure

fusionSF-compressed.js is now built with the Google Closure Compiler.

Besides giving us a 50kb reduction in file size over YUI compressor with default settings, building through Closure gives us Source Map support for modern browsers.

What this means is you can now debug and step through fusionSF-compressed.js in its uncompressed form with full comments without having to modify the template files to switch to a debug version of Fusion. Current versions of Chrome and Firefox both support source maps in their respective developer tools.

With advanced optimizations, fusionSF-compressed.js file size can be reduced by up to 500kb! Unfortunately, building with closure's advanced optimizations may actually alter application behaviour and requires a thorough audit of the code in Fusion first before we can determine if it is safe for us to flick the switch. So for now, we build with default settings.

Delayed refresh on layer/group toggle in legend

When you toggle a layer or group on/off in the legend, it triggers a map refresh. If you do this on multiple layers/groups in quick succession, it's going to fire off that many refresh requests. Not very efficient.

For this release, we've added support for a new DrawDelay extension property in the Map element of the MapGroup. Setting this property will cause any toggle in the legend to delay the refresh for that amount of time (in ms).

Having this property set will allow you to toggle multiple layers/groups in the legend in quick succession, but only trigger a single refresh request at the end as long as each individual toggle falls within the delay period. This feature is opt-in. By default, the delay is 0 which means legend widget will behave like before where each toggle triggers a refresh immediately.

More QuickPlot tweaks

The legal disclaimer is now a configurable element

Of course, you may want this element to be mandatory, so we've added a new AllowDisclaimerToggle extension property that determines whether this checkbox will be shown.

In addition, you can configure the initial checked state of these elements through the following new extension properties:

• ShowLegend
• ShowNorthArrow
• ShowCoordinates
• ShowScaleBar
• ShowLegalDisclaimer (does nothing if there is no legal disclaimer attached or AllowDisclaimerToggle = false)

Making MBTiles databases with a little help from mapguide-rest

So allow me to explain what this screenshot represents

What you are looking at is tiles stored in a MBTiles database. The Android app in question is Locus Map, which supports MBTiles databases. MBTiles is an ideal format for delivering tiled maps to mobile devices. It's just a SQLite database that uses the same intuitive XYZ tiling scheme used by OpenStreetMap and friends.

This post will show you how to make such MBTiles databases with a little help from the mapguide-rest extension.

[NOTE: These instructions were used against the current master branch of mapguide-rest which supercedes the 0.8 release, and has several fixes for tile caching. These instructions may or may not work on the 0.8 release]

First thing you need is to download and install Portable Basemap Server.

In the directory where you installed Portable Basemap Server, edit the CustomOnlineMaps.xml and add a new map source to any XYZ tileset accessible from mapguide-rest

  <onlineMapSource>
    <name>mapguide-rest Sheboygan</name>
    <url><![CDATA[http://localhost/mapguide/rest/library/Samples/Sheboygan/MapsTiled/SheboyganNoWatermark.MapDefinition/xyz/Base Layer Group/{$z}/{$x}/{$y}/tile.png]]></url>
    <tileFormat>png</tileFormat>
    <servers></servers>
  </onlineMapSource>

Note the {$x}, {$y} and {$z} placeholder tokens in the URL. These placeholders are required and Portable Basemap Server will be replacing them with actual X, Y and Z values when fetching tiles.

Once you've edited and saved CustomOnlineMaps.xml, run the PortableBasemapServer.exe as an administrator.

What is unique about Portable Basemap Server (and something I could not find in any other free/OSS packages), is that Portable Basemap Server includes a convenient function to create an MBTiles database from any online tiled map (that presumably follows a XYZ tiling scheme). This function is accessible from the Format Convert menu.

In the Format Convert window, change the Data Source Type to the map source you registered in CustomOnlineMaps.xml. If your tiled map does not span the entire world, you may want to hone in to the specific region your tiled map is in with one of the existing map sources before switching over to your mapguide-rest map source.

Now do the following:

• Specify the path of the MBTiles database you want to create
• Trace a box around the area you want to fetch the tiles for
• Tick the Download Levels (aka. Zoom levels) you want to fetch tiles for. The map preview shows the current zoom level, so generally you want to tick that level and every level above that.
• Tick Compact? to optimize storage of redundant tiles

Once you are satisfied with the settings, click Start to initiate the tile seeding process. Portable Basemap Server will spit out lots of useful progress information while building the MBTiles database.

Once the tile seeding process is complete, you can now transfer this MBTiles database file to your mobile device and load it into your MBTiles supported app.

As a bonus, the actual process of building an MBTiles database also seeds the XYZ tile cache on the server-side. So this process can also serve as a XYZ tile cache seeder if you have no use for MBTiles.

If you're interested in what platforms support MBTiles, check out the MBTiles wiki.

Announcing: MapGuide Open Source 2.6 RC2

The number of critical issues revolving around v2.6 QUERYMAPFEATURES that have cropped up since the MapGuide 2.6 RC1 release have necessitated a second release candidate.

Here's a list of changes since RC1:

• Fix: Bad sample code and data in light of the Sheboygan dataset update.
• Fix: v2.6 QUERYMAPFEATURES cannot append/update viewer selections via shift-click in AJAX and Fusion viewers.
• Fix: v2.6 QUERYMAPFEATURES cannot handle dates before Jan 1, 1970.
• Fix: GEOMETRY parameter in QUERYMAPFEATURES operation should be optional. QUERYMAPFEATURES should be able to accept a query geometry (via GEOMETRY) or an XML selection string (via FEATUREFILTER) when updating viewer selections.
• Remove setselection AJAX viewer helper script which is made redundant with the above fix.
• Fusion: Add int16 identity property support for search widget.
• Fusion: Allow blank titles in print widget.
• Fusion: Preserve inner stack trace (if possible) when reporting back un-handled AJAX errors.

Download

Oh hey ... it works!

How we make our builds of MapGuide

For this post, I'm going to shed a little insight (for those interested) into how we produce our builds of MapGuide Open Source for Windows and Linux.

Windows

We use Jenkins to coordinate our Windows builds of MapGuide. Jenkins is a Java-based continuous integration tool, though in our case we generally emphasise the continuous aspect of CI more than the integration aspect due to the nature of C++ code. Jenkins was chosen because of its ease of use and its massive array of available plugins that enhance the functionality of Jenkins. In addition to MapGuide and FDO, Maestro is also built with Jenkins.

MapGuide and FDO are built with completely free tools. For 2.5 and older releases, we used Visual C++ 2010 Express edition and some installation gymnastics with the Windows 7.1 SDK to get access to the 64-bit Visual C++ compiler. Thanks to the migration to Visual Studio 2012 as the de-facto windows compiler for MapGuide and FDO, building MapGuide/FDO with free tools is now a totally painless experience as the Windows Desktop SKU of VS2012 Express satisfies all our requirements, including both x86 and x64 C++ compilers needed for us to produce 32-bit and 64-bit builds.

Sadly, since the majority of the code we're building is C++, that does not lend itself to fast iteration cycles where build times are generally expected to be several minutes. Our build times for MapGuide and FDO are several hours which generally hampers the quick feedback cycles that are expected of normal CI processes.

So our MapGuide and FDO build jobs are manually triggered instead of triggered off of polling a remote SCM repository. Maestro however, builds a lot faster (being .net code) so it is configured to trigger off of new commits meaning we can get fresh builds of Maestro several minutes after a svn commit.

So when it comes to put out a new build of MapGuide, we simply fire up Jenkins, launch the FDO build jobs (that will trigger the downstream MapGuide build job when completed), and check back in several hours later. If a build fails, we pore through the various log files to find out why and see how it can be rectified. This process is not perfect, but it's good enough given what we have to work with.

Linux

For 2.5 and older releases, producing Linux builds of MapGuide was a somewhat manual process of making a CentOS or Ubuntu VM in VirtualBox, setting up any required tools and libraries and running a set of build scripts to compile MapGuide and FDO for that particular distro. This is why the Linux builds always lagged several days behind the Windows builds.

For 2.6, I wanted to vastly streamline this process. Like the Windows build process, the Linux build process should be something we can trigger with a press of a button or running a script and we can check back several hours later to collect the various build artifacts for upload or additional testing.

To this end, I caught wind of this wonderful tool called vagrant, which made the process of setting up and configuring VMs (a process they call provisioning) much simpler. Some leg work was required to set up the CentOS and Ubuntu base boxes for vagrant but once that was done, producing a CentOS or Ubuntu build of MapGuide was a simple "vagrant up" away.

The use of vagrant means VMs are now truly disposable entities. Our Linux VMs are created as part of the build process and are disposed of after building MapGuide to ensure a clean slate VM is spun up every time for subsequent builds. This means MapGuide/FDO will always be built from a clean environment and means we won't encounter problems due to tainted build environments.

Like the Windows build process, this one is not perfect. Some of the MapGuide unit tests involving threads have a very high chance of locking up the shell provisioner (I don't know why this happens), so these tests have to be disabled and run elsewhere in a manually supervised VM.

Also editing Vagrantfiles requires careful review and scrutiny as a typo or syntax error in the shell script code in a Vagrantfile means several hours lost as you wonder and investigate why "vagrant up" didn't produce your set of build artifacts after leaving it for several hours.

Still, the process as a whole is a lot better than what we had before which is what ultimately counts.

In closing

If you're interested in setting up your own CI environment for MapGuide, you can find the configurations I've used:

• For Jenkins here
• For Vagrant here

Now when I'm ready to put out a build of MapGuide, it's just a case of:

• Triggering the FDO build jobs in Jenkins
• Running "vagrant up" on my CentOS and Ubuntu Vagrantfiles

I don't think it can get much simpler than that!