Saturday, 30 August 2014

Bootstrap map viewer templates

So this past week, I attended another hackathon and our hack was yet-another-twitter-bootstrap-with-openlayers concoction. The problem it seems, every time I go down this path, I lose several critical hours re-creating my desired responsive layout of a map viewer with sidebar with bootstrap CSS every time. You'd think that having done this many times now, I'd have memorised the necessary HTML markup and CSS to do this by now.

But that is just not the case, so after the event (and a good long post-hackathon sleep), I fired up Sublime Text and set out to solve this problem once and for all: A set of starter bootstrap templates that should serve a useful foundation to build a bootstrap-based map viewer web application on top of. Several hours later, there's two templates available.

A 2-column template. A full-screen map viewer with a sidebar to the left:

And a 3-column template, which has sidebars to the left and right which is reminiscent of the classic MapGuide AJAX viewer.

A key feature of both templates is that the floating sidebars are both collapsible.

On small displays, sidebars start off initially collapsed, but can be brought out through their respective toggle buttons.

You can check out the templates below:

Do you have ideas for improving my initial design and have some web design skills? Send me some pull requests.

Tuesday, 26 August 2014

I have a dream

I have a dream

Where MapGuide and FDO source code are hosted and/or mirrored on GitHub.

Where by the virtue of being hosted on GitHub, these repositories are set up to take advantage of every free service available to improve our code quality and developer workflow:
  • TravisCI for Continuous Integration of Linux builds
  • AppVeyor for Continuous Integration of Windows builds
  • CoverityScan for static code analysis
  • Coveralls for code coverage analysis
  • What other awesome services can we hook on here? Please enlighten me. I'd really want to know.
Where a single commit (from svn or git) can start an avalanche of cloud-based services that will immediately tell me in several hours time (because C++ code builds so fast doesn't it?):
  1. If the build is OK (thanks to Travis and AppVeyor)
  2. Where we should look to improve our test coverage (thanks to coveralls)
  3. Areas in our codebase where we should look to change/tweak/refactor (thanks to coverity scan)
  4. Other useful reports and side-effects.
Now the difference between dream and reality is that there are clear obstacles preventing our dream from being realised. Here's some that I've identified.

1. Git presents a radically different workflow than Subversion

Yes, we're still using subversion for MapGuide and FDO (har! har! Welcome to two-thousand-and-late!). Moving to GitHub means not only a change of tools, but a change of developer workflows and mindset.

So in this respect, rather than a full migration, an automated process of mirroring svn trunk/branches (and any commits made to them) to GitHub would be a more practical solution. Any pointers on how to make this an idiot-proof process?

2. Coverage/support is not universal

MapGuide/FDO are multi-platform codebases. Although TravisCI can cover the Ubuntu side and AppVeyor can cover the Windows side, it does leave out CentOS. I've known enough from experience (or plain ignorance) that CentOS and Ubuntu builds need their own separate build/test/validate cycles.

And actually, Travis VMs being 64-bit Ubuntu Linux doesn't help us either. Our ability to leverage Travis would hinge on whether we can either get 64-bit builds working on Linux or am able to cross-compile and run 32-bit MapGuide on 64-bit Ubuntu, something that has not been tried before.

Also most service hooks (like coveralls and CoverityScan) target Travis and not AppVeyor, meaning whatever reports we get back about code quality and test coverage may have a Linux-biased point of view attached to them.

3. The MapGuide and FDO repositories are HUGE!

The repositories of MapGuide and FDO not only contain the source code of MapGuide and FDO respectively, but the source code of every external thirdparty library and component that MapGuide/FDO depends on, and there's a lot of third-party libraries we depend on.

If we transfer/mirror the current svn repositories to GitHub as-is, I'm guessing we'd probably be getting some nice friendly emails from GitHub about why our repos are so big in no time.

Also would Travis and AppVeyor let us get away with such giant clones/checkouts happening every time a build is triggered in response to a commit? I probably don't think so. Then again, I do live in a country where bandwidth doesn't grow on trees and our current government has destroyed our dreams of faster internet. What do I know?

So what do you think? Is this dream something worth pursuing?

Tuesday, 19 August 2014

Announcing: CentOS build of MapGuide Open Source 2.6

As mentioned in my previous post, the CentOS blocker is now resolved.

This means we now once again have a functional CentOS build of MapGuide Open Source 2.6 which is now available for download on the 2.6 release notes page.

I'm still scratching my head as to how aclocal/libtoolize/automake/autoconf on a thirdparty MapGuide component somehow makes iconv_open() fail!

Monday, 18 August 2014

Binary Search Algorithm: Applied in real life

When I announced the availability of MapGuide Open Source 2.6, we did not have a CentOS build available due to a serious blocking issue where iconv APIs would mysteriously fail within MapGuide.

The main symptoms of this iconv failure are:
  • Under default build settings, mgserver will fail on start up with a "could not load a transcoding service" error from the xerces library. The default build settings use iconv APIs to transcode strings within the xerces library.
  • The SHP FDO provider will throw a "memory allocation failed" error when attempting to connect to any SHP feature sources. The SHP provider also uses iconv APIs for narrow <-> wide string conversions.
While the first problem was worked around (by building xerces to use a different transcoder), the second one was truly back breaking. I didn't want to release 2.6 for CentOS where support for the most ubiquitous spatial data format was broken out of the box.

As a first priority after the 2.6 release, I went to see when this issue first cropped up and what was the offending component.

The 2.5.2 release for CentOS did not have this issue, so to eliminate FDO as the offending component I built the 2.5.2 release against the FDO 3.9 branch. Fortunately, the iconv failure did not show up, so we now knew that FDO was not the culprit. It was going to either be MapGuide or one of its Oem components that has caused this breakage.

Knowing that FDO was not the culprit, it was time to start identifying the svn revision in MapGuide that brought us this mess. Unfortunately, there's been quite a lot of revisions between 2.5.2 and 2.6 and knowing how long it takes to build and verify a single revision of MapGuide (because ... C++ code), it would be painstaking to build and verify every single revision.

So to take a logical shot in the dark as a means of reducing the set of svn revisions to identify, I picked the very first working revision of the 2.6 branch to see if this issue exists. It didn't (yay!), meaning our problem space is now reduced to 60 commits in the 2.6 branch.

One of these 60 revisions broke the CentOS build. Rather than wasting time building and verifying 60 individual revisions to identify the breaking revision, I took a more systematic approach and picked the closest "mid-point" revision that affected files in the Server/Oem/Common/Web directories.

If the problem showed up there, we can reduce the problem space to revisions older than that one being tested (ie. some revision older than the tested one introduced the problem), otherwise we can reduce it to revisions newer than that one being tested (ie. some revision newer than the tested one introduced the problem) and repeat the process, until our problem space becomes a single revision that fails. That revision is the revision that broke our CentOS build.

As the tale of my Trello card to track this problem can attest to, finding the offending revision was pretty quick.

And if the title of this post didn't give it away, this systematic process has a name: It's called a Binary Search Algorithm

Though in our case it's not a true binary search, more like a "biased" binary search in that although our problem space was 60 revisions, some of these revisions did not touch any part of the MapGuide Server/Oem/Web/Common code base, so such svn revisions can be excluded from our problem set. Also when the candidate revision to test landed beside a "big merge" revision, we tested that "big merge" revision as well just so we can immediately rule it out.

So there you have it. Knowing Binary Search is not just for passing your Computer Science exam or that Software Developer Job Interview or to implement various data structures, it has real life applications too like hunting down what commit broke your build.

Some might say, wouldn't a Continuous Integration system have caught this? Indeed it would've, but as I've talked about previously about how we make our builds of MapGuide, the Windows builds of MapGuide are built under Jenkins (which can detect and flag broken/unstable builds thanks to its rich ecosystem of plugins), but the Linux builds are not. Linux builds although they are mostly automated now, still require manual invocation (to start the vagrant provisioning process) and manual review of the various log files produced to see if anything broke. The offending revision obviously slipped through the radar.

The Linux build system obviously has much more room for improvement, something that we can now have the opportunity to explore now that 2.6 is out the door.

But before we go about that, let's put out that overdue 2.6 build for CentOS.

Thursday, 14 August 2014

Using PHP composer with MapGuide's bundled PHP

If you're building PHP applications for MapGuide (like I am currently doing with mapguide-rest) you should strongly consider using PHP composer for installing and managing all your external PHP libraries.

PHP composer is basically the PHP version of NuGET (for .net) and simplifies and accelerates development of PHP applications by allowing you to install and manage third-party PHP libraries and frameworks with ease.

No dependency hell or wondering what PHP files to include/require. PHP composer will sort all of that out, leaving you to simply "require 'vendor/autoload.php'" to start using your libraries and like, the Packagist website will help you find the PHP library you need to solve your particular problem.

So how do you make sure when installing PHP composer that it will work with MapGuide's bundled PHP? In the windows installer, simply set the PHP path to where MapGuide's PHP is installed.

For Linux or if you didn't use the Windows installer, you should make sure the php that is running when you invoke composer is the one from your MapGuide installation and not a system-based PHP if you so happen to have one installed.

Once installed, you can start to enjoy a (less painful and more exciting) PHP development experience.

MapGuide tidbits: RenderTile (or: Build Your Own Tile Management system)

Here's a little tidbit about tiles in MapGuide.

Are you not satisfied with MapGuide's tile management? The fact that saving a Map Definition blows away your precious tiles? The fact that you have no control where rendered tiles are stored?

Well it just so happens that the functional primitive for rendering tiles is also available in the MapGuide API: The RenderTile() method of MgRenderingService. This API only renders the tile, it does not store it in MapGuide's pre-defined tile location. The tile storage and management is completely up to you to implement. You can think of the GetTile() method of MgTileService as simply being RenderTile() with tile access/management built in. If you don't like how MapGuide does its tile management, you can implement this part yourself, and call RenderTile() yourself for the actual tiles.

Although we don't use this API in mapguide-rest for XYZ tiles, we follow the same concept of doing the tile access and management ourselves at the PHP level, but still tapping into the actual MapGuide API itself to render the actual XYZ "tiles".

This approach does have its own downsides. The automatic tile cache invalidation that MapGuide does when dependent upstream resources are saved and some users find annoying won't be at play here. You'll have to manually purge these cached tiles yourself, otherwise you may find that your cached tiles are horribly visually out-of-sync with respect to the current Map Definition and its layer structure and styles. This is what the automatic tile cache invalidation is trying to prevent, only it does it with the surgical precision of a nuclear warhead.

Tuesday, 12 August 2014

GovHack 2014 post-post-mortem

This past Sunday was the awards ceremony for GovHack 2014.

After a month of Judging and voting on hacks from over 150 different teams across Australia, it was time to find out who would take the prizes and the brownie points.

In the tense moments of anticipation, our team "CreativeDrought" was mentioned.

Not once.

Not twice.

But three times!

Expecting this haul was way beyond what I was expecting. Just winning one award would've been a nice feather on my cap, but winning 3 awards ... my cap has become a peacock!

This was not the first time I won a prize at a hackathon. My previous win was at RHoK 2012 once again with MapGuide playing a major part in the solution. That's 2 out of 2 MapGuide-assisted hackathon victories. Not bad if I say so myself!

Now that GovHack and the judging is over, I'll guess I'll leave our demo site up for another few weeks before I decide what to do with it.