Thursday, 18 December 2014

A note about .net extension methods

If you've ever programmed in C#, you've probably encountered this one awesome feature called extension methods that allows you to add new methods to existing types without modifying the original type (in reality, this feature is nothing more than syntactic sugar for invoking utility methods in a natural object-oriented manner. Still, being able to call utility methods in such a manner is much better and is partly the reason why the various LINQ APIs are so elegant to use)

One thing that people may overlook when writing their extension methods is the namespace in which the extension method class is defined in.

Simply put, if you are going to write extensions methods for a class/interface/enum, you should define the extension method class in the same namespace as the class/interface/enum that you are "extending"

The reason for this is simple: It improves discover-ability of these extension methods. Any time you bring in a namespace for a given class/interface/enum, the extension methods (being in the same namespace) will also be brought in automatically.

If these extension methods were in a different namespace, you'd have to manually seek out the namespace where these extension methods live in, which ranges from inconvenient to annoying. I've had to ILSpy many assemblies in my time just to find the proper namespace of this one extension method that the documentation keeps referring to, all because the extension methods lived in a separate namespace from the "extendee" class/interface/enum.

Something to think about before you write your next extension method.


Thursday, 20 November 2014

MapGuide tidbits: Log monitoring with log.io

If the previous MapGuide log monitoring solution don't cut it for you (especially on Linux), here's another one you can try.

log.io is a real-time log monitoring solution that runs in your web browser. It is powered by node.js and socket.io.

Just a note before we dive in, this post is geared towards Linux-specific installations of MapGuide. This may or may not work for Windows. The log.io install instructions assume Linux, so we're rolling with that.

To install log.io you will first have to install node.js, there's a bajillion different links out there on how to install node.js, so here's one for:


Once node is installed, you can install the log.io package via npm (on Ubuntu you'll have to elevate that command with sudo):

npm install -g log.io --user "[username that will run log.io]"

Then start the log.io server.

log.io-server

Then create a harvester.conf in your ~/.log.io/ directory that defines what log files to monitor. Here's a basic example configured for MapGuide and Apache log monitoring.

exports.config = {
    nodeName: "mapguide_server",
    logStreams: {
      apache: [
        "/usr/local/mapguideopensource-2.6.0/webserverextensions/apache2/logs/access_log",
        "/usr/local/mapguideopensource-2.6.0/webserverextensions/apache2/logs/error_log"
      ],
      mapguide_access: [
        "/usr/local/mapguideopensource-2.6.0/server/Logs/Access.log"
      ],
      mapguide_error: [
        "/usr/local/mapguideopensource-2.6.0/server/Logs/Error.log"
      ]
    },
    server: {
      host: '0.0.0.0',
      port: 28777
    }
  } 

Now start the log harvester.

log.io-harvester

Now browse to http://localhost:28778 and watch your MapGuide log files in real time.


NOTE: The above install instructions were pilfered from the log.io website itself, however I found the global install option to be problematic on my Ubuntu test VM. For some reason it always insists on building the native modules as the "root" user and not the user I designated from the npm install command. So I went for a local install instead, which means the following commands have been changed to the following:

  • Installing log.io: npm install log.io
  • Running log.io server: ~/node_modules/log.io/bin/log.io-server
  • Running log.io harvester: ~/node_modules/log.io/bin/log.io-harvester

Friday, 7 November 2014

Announcing: mapguide-rest 0.11.2

Here's another release of mapguide-rest. There was some issues that cropped up after pushing the original 0.11 release that necessitated 2 emergency point releases to fix. As a result, 0.11.2 is the official release.

Here's what's new in this release.

KML Service Support

mapguide-rest now provides RESTful routes that wrap the existing KML Service APIs in MapGuide:

  • /library/{resourcePath}.MapDefinition/kml
  • /library/{resourcePath}.LayerDefinition/kml
  • /library/{resourcePath}.LayerDefinition/kmlfeatures
These routes respectively wrap or replicate:
  • GetMapKml (re-implemented in PHP to write the correct RESTful URLs for NetworkLink) elements)
  • GetLayerKml (re-implemented in PHP to write the correct RESTful URLs for NetworkLink) 
  • GetFeaturesKml (wraps existing API)

Pagination support for feature data routes

These following routes that return feature data now support pagination:

  • /library/{resourcePath}.FeatureSource/features/{schema}/{class}
  • /library/{resourcePath}.FeatureSource/features.{type}/{schema}/{class}
  • /library/{resourcePath}.LayerDefinition/features
  • /library/{resourcePath}.LayerDefinition/features.{type}
  • /session/{sessionID}/{mapName}.Selection/features/{layerName}
  • /session/{sessionID}/{mapName}.Selection/features.{type}/{layerName}

You can paginate the results by passing two additional parameters:

  • pagesize - The number of features to return per "page"
  • page - The page "number" to return
Pagination will work for any supported representation except for CZML. The implementation of CZML output is not compatible with our pagination and errors will be thrown if attempting to pass in pagination parameters for CZML output.

The primary beneficiary of pagination support is ...


HTML representation for feature data

We've added HTML representations for feature data routes:
  • /library/{resourcePath}.FeatureSource/features.{type}/{schema}/{class}
  • /library/{resourcePath}.LayerDefinition/features.{type}
  • /session/{sessionID}/{mapName}.Selection/features.{type}/{layerName}
Where {type} can now be html in addition to the existing supported representations.

For example, a html representation of a Layer Definition would look like this



Apply pagination, and you now have a page-able view of your feature data


The real payoff for implementing this feature is that you also get this support for selected features. For selected features, we support changing the display orientation in the HTML representation as well.


Tweak the page size and orientation parameters for this route and you have what is now effectively a selected feature property palette for your custom map viewer for free, as demonstrated in a new sample that is bundled with this release.


The palette in this sample is just an iframe to this new HTML representation URL.

Improved HTML repository view

The HTML repository representation has been cleaned up

You'll notice each resource no longer show all actionable links. That's because these have been moved to a separate page that is loaded when you click on the resource link.

Clicking on any actionable link will show the result in the space below


We've also added more actionable links for the following resource types:

  • Feature Sources: View features as HTML
  • Map Definitions: Export as KML

Various other HTML representations have also cleaned up css.

Localization-ready

mapguide-rest is now localization ready. If you want to translate mapguide-rest to your language of choice, you just have to do the following:

  • Translate and copy the strings in app\res\lang\en.php to app\res\lang\{locale}.php. A small caveat with translating these strings is that you will have to translate the content in a way that respects the ordering of placeholder tokens as we use sprintf to fetch and format localizable strings.
  • Translate and copy the system templates under app\res\templates\en to app\res\templates\{locale}
  • Translate and copy the XSLT templates under app\res\xsl\en to app\res\xsl\{locale}

Once you have translated these files, simply change the Locale property to {locale} in app\config.php and mapguide-rest will use the translated strings and templates

Other changes/fixes

  • Client agent and IP addresses are now recorded when creating a site connection, ensuring all API access from mapguide-rest will be properly logged in the server's access.log
  • Fix layered PDF plotting for maps with base layer groups
  • Added various bits of missing REST API documentation
  • Error messages will now be returned in the appropriate requested format where applicable (eg. If an error occurs when making a request to a URL expecting an XML response, the error will be returned as XML). If no applicable request format can be determined, then the default error response will be in HTML.

Download

Monday, 20 October 2014

MapGuide tidbits: MapGuide Server daemon doesn't start after reboot

This one will be short and sweet.

If you have rebooted your Linux server, and for some reason you can no longer start the MapGuide Server as a daemon. You should check that the /var/lock/mgserver directory exists and create it if it doesn't.

The mgserver process will try to create and lock a file in this directory and will bail out if it can't. This directory is cleared when the Linux server is restarted (at least in my observations). None of the wrapper scripts (mgserver.sh or mgserverd.sh) actually check if the directory exists, so they blindly proceed as though this directory existed.

We'll patch the mgserverd.sh script to create this directory if it doesn't exist before running the mgserver daemon. In the meantime, you can edit the mgserverd.sh file in the MapGuide Linux installation yourself to create the /var/lock/mgserver directory before running the mgserver process.

Thursday, 25 September 2014

Docker-izing MapGuide

There was a motivation behind this public service announcement.

It was part of my investigations into being able to deploy MapGuide as a Docker container.

What is Docker you may ask? Have a read of their introduction page.

TL;DR? Docker is a tool that allows you to run and deploy software inside virtualized software containers. This is not a heavyweight VM like VirtualBox and their ilk. This is something that leverages existing features in the Linux Kernel to provide "lightweight" VMs with very low overhead.

A docker-ized application gives us various benefits:
  • Your application and its dependencies are all self-contained and will not interfere with anything outside of its container and vice versa.
  • Deployment is dead simple. No figuring out what pre-requisites that have to be installed. They'll all be part of the docker image that you can pull down with a single command.
  • Your application can run in host environments that the application was not originally compiled/tested for. Build once, run anywhere (where docker is installed :))
There's many more benefits than what's listed here. So you can probably see where a docker-ized MapGuide would really be useful from a developer and administrator perspective.

But there is a catch to all this goodness. Firstly you need a Linux distribution that uses Linux Kernel v3.8 or newer as that contains the required OS virtualization features needed for Docker to work.

Secondly, you need to run a 64-bit Linux distribution as that is what Docker only supports. In order to run MapGuide within a Docker container, we need a way to run a 32-bit MapGuide within a 64-bit Linux environment as Docker can only be run from within a 64-bit Linux host and as previously mentioned, we still don't have a functional 64-bit Linux build of MapGuide yet :( So the workaround is to install the required 32-bit packages, which you can find in my previous post.

So after applying all of this newly acquired knowledge, allow me to introduce my first Docker image for MapGuide.

This is a CentOS 6 base image that has the following software and packages pre-installed:
From this base image, you can build your own docker image that installs your own MapGuide data, applications and configurations and deploy/run that image as a docker container.

Here's a basic example of getting a docker-ized MapGuide up and running with mapguide-rest and some sample data packages using 64-bit Ubuntu 14.04 using the above docker base image:

Firstly, we install the docker package like so

sudo apt-get install docker.io

Then make a new directory and create a file named Dockerfile


Put the following text into the Dockerfile. The comments should be self-explanatory. 


# This image is based from the MapGuide docker base image
FROM jumpinjackie/mapguide-base
# Install additional packages, load your MapGuide applications and data, etc.
# Download Sheboygan dataset
RUN wget -P /usr/local/mapguideopensource-2.6.0/server/Packages http://download.osgeo.org/mapguide/releases/2.6.0/Release/Sheboygan.mgp
# Download Melbourne dataset
RUN wget -P /usr/local/mapguideopensource-2.6.0/server/Packages https://github.com/jumpinjackie/mapguide-sample-melbourne/releases/download/v0.2/Melbourne.mgp
# Download mapguide-rest 0.10
RUN wget https://github.com/jumpinjackie/mapguide-rest/releases/download/v0.10-pre/mapguide-rest-0.10.zip
# Set up install location for mapguide-rest
RUN mkdir -p /usr/local/mapguideopensource-2.6.0/webserverextensions/www/rest
# Extract mapguide-rest
RUN unzip mapguide-rest-0.10.zip -d /usr/local/mapguideopensource-2.6.0/webserverextensions/www/rest
# Fix up permissions of the cache directory (for tiles and smarty templates)
RUN chown daemon:daemon /usr/local/mapguideopensource-2.6.0/webserverextensions/www/rest/cache
# Expose the web server port to the world outside the container. The default port is 8008
EXPOSE 8008
# Run supervisor, that will start the MapGuide Server and Apache httpd server
CMD ["/usr/bin/supervisord"]


Now save the Dockerfile, and run the following command to build the docker image from that file

sudo docker build -t my-mapguide-app .

This command will download the mapguide-base docker image which will take a few moments depending on your download speed. This is only downloaded once and will remain until you explicitly remove this base image.

my-mapguide-app will be the name of the docker image, which you'll need to reference when you will run a container from it, which is what we will do next. But before we do that, we should list our docker images and see if our new image is there.

sudo docker images



Now that we have confirmation that our image has been created, we can create a container from it like so.

sudo docker -d --name mapguide -t my-mapguide-app

The -d switch indicates that this container will run in the background. The --name switch assigns the name mapguide to this container so we don't have to remember a long UUID for referencing this container in future operations, which is what is outputted when the command succeeds.


Finally the -t switch indicates that we want to create a container from the my-mapguide-app image that we just created.

Now that we have started a container, we now have:

  • MapGuide Server running
  • The Sheboygan and Melbourne sample data packages downloaded to the Packages directory of the MapGuide Server installation directory
  • The mapguide-rest extension installed
  • Apache HTTP Server running on port 8008 which is exposed
We now just need to know what IP address has been assigned to this container. To do that, we can run the following command

sudo docker inspect mapguide | grep IPAddress


Now that we have an IP address, we can see if things are in order by firing up the Site Administrator


If we login and go to the Manage Packages page, we can see the packages we've downloaded as part of building our my-mapguide-app Docker image are there.


Now load these packages, and go to the mapguide-rest sample apps landing page just to see that the extension was installed.


If the samples on this landing page work, you have just verified that your docker-ized MapGuide container is now fully operational.

So that's a little run-through of how to get MapGuide running in a Docker container. Now the other thing about Docker that is really awesome is that the big players in Cloud Computing already (or starting to) support Docker containers as a PaaS deployment option.

Can you say MapGuide on the cloud?


Now just to clarify, MapGuide on the cloud via IaaS is already relatively easy. I, the AWS noob easily got a MapGuide demo site on Amazon EC2 set up over the GovHack weekend, which stayed up long enough to impress the judges to win some awards.

However the IaaS approach to deployment puts a burden on you to maintain the actual infrastructure (VMs, etc). As a case in point, our GovHack demo site has been taken down because I butchered my Amazon EC2 instance due to a combination of failure to act on an important email alert on this issue from Amazon and my general AWS noob-ery.

The PaaS approach lets you focus solely on deploying the applications, without having to worry about the underlying infrastructure. MapGuide as a Docker container enables the possibility of using this cloud deployment option.

Hopefully this post has sold you on the power and potential of Docker and how having MapGuide as a docker container may open up deployment and development scenarios that were previously not possible or took a lot of manual effort.

As for that Docker base image I linked to, that is just the tip of the iceberg. I'm just getting my feet wet with Docker and foresee many future blog posts on this topic as I learn more about what Docker can and can't do.

I can see the possibility of having different base images in the future:
  • Individual Server and Web Tier Docker images. Imagine how easy it would be to set up a load-balanced cluster via Docker containers? I'd have to grok how these containers communicate first, but the possibilities are very interesting.
  • Web Tier images tailored for the PHP or Java installation profiles.
  • Many others?
I also wouldn't comfortably say that the above docker image is currently production-ready. There's questions that need to be answered:
  • How do we handle the case of a mgserver or httpd process falling over on a background docker container?
  • How can we easily access and manage log files in a docker container?
  • How can we perform repository backup/restoration operations within a docker container?
  • How do we perform $MAPGUIDE_SERVER_ADMIN_TASK within the docker container?
If you have an idea on how these questions can be answered, I invite you to help improve the design of this docker image or to enlighten me on the comments below.

MapGuide tidbits: Strange fonts on Linux

Does your MapGuide Linux installation render labels that look like this?


Instead of this?


This is due to MapGuide not being able to correctly locate the required font in question (Verdana). To fix this, you need to install the appropriate font package that provides the missing font. In the case of verdana, that package is msttcorefonts.

On Ubuntu: Simply install the msttcorefonts package

On CentOS: Install the following packages:

  • curl
  • xorg-x11-font-utils
  • fontconfig
And then install the following RPM packages with the rpm command
  • rpm -i http://pkgs.repoforge.org/cabextract/cabextract-1.4-1.el6.rf.i686.rpm
  • rpm -i https://downloads.sourceforge.net/project/mscorefonts2/rpms/msttcore-fonts-installer-2.6-1.noarch.rpm
Now you can verify your font in question is install by running the fc-list command and looking for the name of your font in the output

fc-list | grep Verdana

Now even after this, MapGuide may still not properly locate this font even if it is installed.

If this is still the case, copy the physical font files to the directory of the mgserver executable (eg. /usr/local/mapguideopensource-2.6.0/server/bin) and restart the MapGuide Server.

In the case of msttcorefonts, the physical font files are found in:
  • On Ubuntu: /usr/share/fonts/truetype/msttcorefonts
  • On CentOS: /usr/share/fonts/msttcore

Tuesday, 23 September 2014

MapGuide tidbits: Running 32-bit MapGuide on 64-bit Linux

We still don't have our mythical 64-bit build of MapGuide on Linux yet. So in the meantime, should you want to run the 32-bit CentOS or Ubuntu builds of MapGuide in their respective 64-bit versions, here's the packages you will need to have installed beforehand.

On 64-bit Ubuntu: Just install the ia32-libs package

On 64-bit CentOS: You will need to install the following packages:
  • glibc.i686 
  • libstdc++.i686 
  • expat.i686 
  • libcurl.i686 
  • pcre.i686 
  • libxslt.i686 
  • libpng.i686
This will satisfy the dependencies required by your 32-bit MapGuide, its bundled Apache HTTPD Server and PHP.

The bundled Tomcat and Java wrapper API has not been tested under this environment at the time this post was published, so to take a logical shot in the dark, you probably need to just install the respective 32-bit JVM package for the Tomcat and Java wrapper API to work. If it turns out I'm wrong about that, please do correct me in the comments below.

Thursday, 18 September 2014

350 posts

This train ain't stopping yet!


Tuesday, 16 September 2014

Announcing: mapguide-rest 0.10

Nope, this isn't a 1.0 release. Not only are we not using decimal release numbers, but there's still plenty of things to explore and refine before we can put the 1.0 stamp on this thing. Here's what's new and changed in this release.

(Experimental) Cesium CZML support

We now have support for outputting feature data as CZML for consumption inside the Cesium 3D web viewer. Support for CZML is made available as a representation of a given Layer Definition.

For example, the following route will return data from the trees layer as CZML:

http://localhost/mapguide/rest/library/Samples/Sheboygan/Layers/Trees.LayerDefinition/features.czml

We've included a CZML example in this release that demonstrates the level of support that has been implemented for this release.


When selecting an object, its tooltip data will be shown in Cesium's information window if available.


Note the object itself has no selection indicator unless the object is a point. Still trying to figure out if we can apply a different style for selected lines and polygons.

If your Layer Definition has elevation settings applied, they be extruded if they're polygons


Now if the (experimental) tag didn't stand out, here's the current limitations with this implementation:
  • The following properties are preserved when converted to CZML, anything not listed can be assumed lost in translation:
    • Point styles: Point color is preserved. The point size in CZML is the average of the height/width of the point as defined in the Layer Definition.
    • Line styles: Line color
    • Area/Polygon styles: Fill color. Outline color
  • If a Layer Definition has multiple scale ranges defined, mapguide-rest will only consider the first scale range when outputting CZML
  • If a given point/area/line style is themed, the default rule (the one without a filter) is ignored
Data publishing improvements

The restcfg.json now lets you specify a Layer Definition as the data source instead of a Feature Source and Feature Class name.

Also, when using a Layer Definition as a data source, you'll get tooltip, hyperlink and elevation FDO expression pre-evaluated, allowing you to use such computed properties within your templates.

You can find a new example that uses the building footprints from the Melbourne dataset.

File download support

Most GET routes can now prompt for downloads by appending download=1 to the query string of the URL.

XYZ tile improvements

Vector tiles can now be for a base layer group or for a layer within the base layer group.

Just to recap, this URL route fetches a vector tile for a given base layer group in a Map Definition

http://servername/mapguide/rest/library/{resourcePath}.MapDefinition/xyz/{groupName}/{z}/{x}/{y}/tile.{format}

If you want vector tiles for a specific layer within that group, you can use this new URL route

http://servername/mapguide/rest/library/{resourcePath}.MapDefinition/xyz/{groupName}/{layerName}/{z}/{x}/{y}/tile.{format}

You can find a new Leaflet example that uses the single-layer vector tiles.


Other changes

  • Fixed bad download links in the resource data list HTML representation
  • Fixed invalid chunked file transfer behavior under certain conditions
  • Samples updated to use OpenLayers 3 final and Cesium 1.1

Saturday, 13 September 2014

300,000 views!

In the race between 300,000 page views and 350 blog posts, the page views crossed the finish line first. For the record, this is my 348th post.

Thank you all for your continued viewership!



Wednesday, 10 September 2014

Make that 3/3

Thanks to this thread, I was finally able to crack the missing piece of the CZML output puzzle for mapguide-rest, and we can now output point, line and polygon features as CZML to Cesium.


CZML is a representation of a Layer Definition in mapguide-rest, this means we will get pre-evaluated properties to play with such as:

We currently use this information as follows

Tooltips get written as the description property of each CZML packet, allowing such information to be displayed in the Cesium information window when the object is selected.


Now as the above screenshot shows, there's no real visual way to know what object you selected. I'm still figuring out if there's a way in CZML or the Cesium APIs to specify how a selected object looks so it can stand out.

If the Layer Definition has elevation/extrusion settings applied, that's when the fun stuff happens. We apply the extruded value in the CZML packet for each feature, giving us 2.5D features. This extrusion only applies for polygon features at the moment.


Oh, and did you know Cesium works pretty well on any WebGL capable mobile browser?


Pretty cool stuff. Now that we've nailed down the fundamentals, it's time to figure out how much visual fidelity we can preserve from MapGuide to CZML:

  • Can we transfer thematics?
  • Can we transfer labels?
  • Can we transfer patterns?
  • Others?
Fun times ahead!

Friday, 5 September 2014

2/3

After lots of trial and error, I am finally able to output features from MapGuide as CZML to Cesium via mapguide-rest


However, only 2 out of the 3 geometry types are working (points and polygons). I'm still trying to figure out:
  • How to properly output CZML for line features
  • How best to apply Z extrusion where it is defined in the Layer Definition
  • What parts of a Layer Definition are transferable/translatable to CZML? Doesn't have to be 1:1. KML-level visual/information fidelity would suffice here.
If only there were more comprehensive CZML examples I could refer to that would make implementing this stuff much easier! All the examples I could find try to demonstrate every bell and whistle when all I want to know is where do I stick the lat/lon/elevation coordinates from my source geometries? Their CZML documentation could do with some improvement, a single-page document is terribly hard to navigate and without reference examples it's amazing I was even able to get this far!

But in the end, all this pain and struggle will be worth it because Cesium is just plain awesome! I truly believe that Cesium will be to 3D maps what OpenLayers is to 2D maps: A powerful web-based map viewing platform that is rich in features and support for many different vector and raster data sources.

Saturday, 30 August 2014

Bootstrap map viewer templates

So this past week, I attended another hackathon and our hack was yet-another-twitter-bootstrap-with-openlayers concoction. The problem it seems, every time I go down this path, I lose several critical hours re-creating my desired responsive layout of a map viewer with sidebar with bootstrap CSS every time. You'd think that having done this many times now, I'd have memorised the necessary HTML markup and CSS to do this by now.

But that is just not the case, so after the event (and a good long post-hackathon sleep), I fired up Sublime Text and set out to solve this problem once and for all: A set of starter bootstrap templates that should serve a useful foundation to build a bootstrap-based map viewer web application on top of. Several hours later, there's two templates available.

A 2-column template. A full-screen map viewer with a sidebar to the left:


And a 3-column template, which has sidebars to the left and right which is reminiscent of the classic MapGuide AJAX viewer.


A key feature of both templates is that the floating sidebars are both collapsible.


On small displays, sidebars start off initially collapsed, but can be brought out through their respective toggle buttons.


You can check out the templates below:


Do you have ideas for improving my initial design and have some web design skills? Send me some pull requests.

Tuesday, 26 August 2014

I have a dream

I have a dream

Where MapGuide and FDO source code are hosted and/or mirrored on GitHub.

Where by the virtue of being hosted on GitHub, these repositories are set up to take advantage of every free service available to improve our code quality and developer workflow:
  • TravisCI for Continuous Integration of Linux builds
  • AppVeyor for Continuous Integration of Windows builds
  • CoverityScan for static code analysis
  • Coveralls for code coverage analysis
  • What other awesome services can we hook on here? Please enlighten me. I'd really want to know.
Where a single commit (from svn or git) can start an avalanche of cloud-based services that will immediately tell me in several hours time (because C++ code builds so fast doesn't it?):
  1. If the build is OK (thanks to Travis and AppVeyor)
  2. Where we should look to improve our test coverage (thanks to coveralls)
  3. Areas in our codebase where we should look to change/tweak/refactor (thanks to coverity scan)
  4. Other useful reports and side-effects.
Now the difference between dream and reality is that there are clear obstacles preventing our dream from being realised. Here's some that I've identified.

1. Git presents a radically different workflow than Subversion

Yes, we're still using subversion for MapGuide and FDO (har! har! Welcome to two-thousand-and-late!). Moving to GitHub means not only a change of tools, but a change of developer workflows and mindset.

So in this respect, rather than a full migration, an automated process of mirroring svn trunk/branches (and any commits made to them) to GitHub would be a more practical solution. Any pointers on how to make this an idiot-proof process?

2. Coverage/support is not universal

MapGuide/FDO are multi-platform codebases. Although TravisCI can cover the Ubuntu side and AppVeyor can cover the Windows side, it does leave out CentOS. I've known enough from experience (or plain ignorance) that CentOS and Ubuntu builds need their own separate build/test/validate cycles.

And actually, Travis VMs being 64-bit Ubuntu Linux doesn't help us either. Our ability to leverage Travis would hinge on whether we can either get 64-bit builds working on Linux or am able to cross-compile and run 32-bit MapGuide on 64-bit Ubuntu, something that has not been tried before.

Also most service hooks (like coveralls and CoverityScan) target Travis and not AppVeyor, meaning whatever reports we get back about code quality and test coverage may have a Linux-biased point of view attached to them.

3. The MapGuide and FDO repositories are HUGE!

The repositories of MapGuide and FDO not only contain the source code of MapGuide and FDO respectively, but the source code of every external thirdparty library and component that MapGuide/FDO depends on, and there's a lot of third-party libraries we depend on.

If we transfer/mirror the current svn repositories to GitHub as-is, I'm guessing we'd probably be getting some nice friendly emails from GitHub about why our repos are so big in no time.

Also would Travis and AppVeyor let us get away with such giant clones/checkouts happening every time a build is triggered in response to a commit? I probably don't think so. Then again, I do live in a country where bandwidth doesn't grow on trees and our current government has destroyed our dreams of faster internet. What do I know?



So what do you think? Is this dream something worth pursuing?

Tuesday, 19 August 2014

Announcing: CentOS build of MapGuide Open Source 2.6

As mentioned in my previous post, the CentOS blocker is now resolved.

This means we now once again have a functional CentOS build of MapGuide Open Source 2.6 which is now available for download on the 2.6 release notes page.

I'm still scratching my head as to how aclocal/libtoolize/automake/autoconf on a thirdparty MapGuide component somehow makes iconv_open() fail!

Monday, 18 August 2014

Binary Search Algorithm: Applied in real life

When I announced the availability of MapGuide Open Source 2.6, we did not have a CentOS build available due to a serious blocking issue where iconv APIs would mysteriously fail within MapGuide.

The main symptoms of this iconv failure are:
  • Under default build settings, mgserver will fail on start up with a "could not load a transcoding service" error from the xerces library. The default build settings use iconv APIs to transcode strings within the xerces library.
  • The SHP FDO provider will throw a "memory allocation failed" error when attempting to connect to any SHP feature sources. The SHP provider also uses iconv APIs for narrow <-> wide string conversions.
While the first problem was worked around (by building xerces to use a different transcoder), the second one was truly back breaking. I didn't want to release 2.6 for CentOS where support for the most ubiquitous spatial data format was broken out of the box.

As a first priority after the 2.6 release, I went to see when this issue first cropped up and what was the offending component.

The 2.5.2 release for CentOS did not have this issue, so to eliminate FDO as the offending component I built the 2.5.2 release against the FDO 3.9 branch. Fortunately, the iconv failure did not show up, so we now knew that FDO was not the culprit. It was going to either be MapGuide or one of its Oem components that has caused this breakage.

Knowing that FDO was not the culprit, it was time to start identifying the svn revision in MapGuide that brought us this mess. Unfortunately, there's been quite a lot of revisions between 2.5.2 and 2.6 and knowing how long it takes to build and verify a single revision of MapGuide (because ... C++ code), it would be painstaking to build and verify every single revision.

So to take a logical shot in the dark as a means of reducing the set of svn revisions to identify, I picked the very first working revision of the 2.6 branch to see if this issue exists. It didn't (yay!), meaning our problem space is now reduced to 60 commits in the 2.6 branch.

One of these 60 revisions broke the CentOS build. Rather than wasting time building and verifying 60 individual revisions to identify the breaking revision, I took a more systematic approach and picked the closest "mid-point" revision that affected files in the Server/Oem/Common/Web directories.

If the problem showed up there, we can reduce the problem space to revisions older than that one being tested (ie. some revision older than the tested one introduced the problem), otherwise we can reduce it to revisions newer than that one being tested (ie. some revision newer than the tested one introduced the problem) and repeat the process, until our problem space becomes a single revision that fails. That revision is the revision that broke our CentOS build.

As the tale of my Trello card to track this problem can attest to, finding the offending revision was pretty quick.


And if the title of this post didn't give it away, this systematic process has a name: It's called a Binary Search Algorithm

Though in our case it's not a true binary search, more like a "biased" binary search in that although our problem space was 60 revisions, some of these revisions did not touch any part of the MapGuide Server/Oem/Web/Common code base, so such svn revisions can be excluded from our problem set. Also when the candidate revision to test landed beside a "big merge" revision, we tested that "big merge" revision as well just so we can immediately rule it out.

So there you have it. Knowing Binary Search is not just for passing your Computer Science exam or that Software Developer Job Interview or to implement various data structures, it has real life applications too like hunting down what commit broke your build.

Some might say, wouldn't a Continuous Integration system have caught this? Indeed it would've, but as I've talked about previously about how we make our builds of MapGuide, the Windows builds of MapGuide are built under Jenkins (which can detect and flag broken/unstable builds thanks to its rich ecosystem of plugins), but the Linux builds are not. Linux builds although they are mostly automated now, still require manual invocation (to start the vagrant provisioning process) and manual review of the various log files produced to see if anything broke. The offending revision obviously slipped through the radar.

The Linux build system obviously has much more room for improvement, something that we can now have the opportunity to explore now that 2.6 is out the door.

But before we go about that, let's put out that overdue 2.6 build for CentOS.

Thursday, 14 August 2014

Using PHP composer with MapGuide's bundled PHP

If you're building PHP applications for MapGuide (like I am currently doing with mapguide-rest) you should strongly consider using PHP composer for installing and managing all your external PHP libraries.

PHP composer is basically the PHP version of NuGET (for .net) and simplifies and accelerates development of PHP applications by allowing you to install and manage third-party PHP libraries and frameworks with ease.

No dependency hell or wondering what PHP files to include/require. PHP composer will sort all of that out, leaving you to simply "require 'vendor/autoload.php'" to start using your libraries and like NuGet.org, the Packagist website will help you find the PHP library you need to solve your particular problem.

So how do you make sure when installing PHP composer that it will work with MapGuide's bundled PHP? In the windows installer, simply set the PHP path to where MapGuide's PHP is installed.


For Linux or if you didn't use the Windows installer, you should make sure the php that is running when you invoke composer is the one from your MapGuide installation and not a system-based PHP if you so happen to have one installed.

Once installed, you can start to enjoy a (less painful and more exciting) PHP development experience.

MapGuide tidbits: RenderTile (or: Build Your Own Tile Management system)

Here's a little tidbit about tiles in MapGuide.

Are you not satisfied with MapGuide's tile management? The fact that saving a Map Definition blows away your precious tiles? The fact that you have no control where rendered tiles are stored?

Well it just so happens that the functional primitive for rendering tiles is also available in the MapGuide API: The RenderTile() method of MgRenderingService. This API only renders the tile, it does not store it in MapGuide's pre-defined tile location. The tile storage and management is completely up to you to implement. You can think of the GetTile() method of MgTileService as simply being RenderTile() with tile access/management built in. If you don't like how MapGuide does its tile management, you can implement this part yourself, and call RenderTile() yourself for the actual tiles.

Although we don't use this API in mapguide-rest for XYZ tiles, we follow the same concept of doing the tile access and management ourselves at the PHP level, but still tapping into the actual MapGuide API itself to render the actual XYZ "tiles".

This approach does have its own downsides. The automatic tile cache invalidation that MapGuide does when dependent upstream resources are saved and some users find annoying won't be at play here. You'll have to manually purge these cached tiles yourself, otherwise you may find that your cached tiles are horribly visually out-of-sync with respect to the current Map Definition and its layer structure and styles. This is what the automatic tile cache invalidation is trying to prevent, only it does it with the surgical precision of a nuclear warhead.

Tuesday, 12 August 2014

GovHack 2014 post-post-mortem

This past Sunday was the awards ceremony for GovHack 2014.

After a month of Judging and voting on hacks from over 150 different teams across Australia, it was time to find out who would take the prizes and the brownie points.


In the tense moments of anticipation, our team "CreativeDrought" was mentioned.

Not once.


Not twice.

But three times!


Expecting this haul was way beyond what I was expecting. Just winning one award would've been a nice feather on my cap, but winning 3 awards ... my cap has become a peacock!

This was not the first time I won a prize at a hackathon. My previous win was at RHoK 2012 once again with MapGuide playing a major part in the solution. That's 2 out of 2 MapGuide-assisted hackathon victories. Not bad if I say so myself!

Now that GovHack and the judging is over, I'll guess I'll leave our demo site up for another few weeks before I decide what to do with it.

Tuesday, 5 August 2014

Move over Sheboygan! There's a new MapGuide sample dataset in town!

For the longest time, the Sheboygan dataset has been our reference dataset for demonstrating the data publishing capabilities of MapGuide as well as being the focal point for all our sample code demonstrating the use of the MapGuide API.

Unfortunately, the actual spatial data in the Sheboygan dataset does not lend itself for MapGuide to demonstrate its capabilities to its fullest. Short of the Parcels layer, everything else in the dataset is just "dumb" spatial data, lacking any real attributes of substance that we could use to demonstrate features of MapGuide with.

MapGuide deserves a more comprehensive and intelligent sample dataset to show off its features and capabilities, and I think I found it.

Say hello to my home city of Melbourne


This MapGuide dataset was made possible by the various bits of open spatial data made available under City of Melbourne's open data platform. I discovered this site the night before GovHack 2014 as City of Melbourne was one of the "data custodians" for the GovHack event. Though I didn't use any of their data over the hackathon weekend, I did make a mental note about the rich and varied volume of data that they had available and that there might be enough stuff in there to make a much more compelling sample dataset for MapGuide than what we currently have with Sheboygan.

So a few weeks after GovHack, I paid another visit to that site to check out their data offerings in more detail and see what kind of dataset we can make in MapGuide with their available datasets. The above screenshot is the current work in progress. The MapGuide dataset was assembled together very quickly with the help of MapGuide Maestro. I guess that's a testament to the many years of development work put into Maestro to make it the most efficient authoring tool for MapGuide (not trying to toot my own horn here honest :)).

Most of the spatial data was in SHP format making loading into MapGuide a dead simple affair. Some of the spatial data was in CSV (huh?), so these were converted to SQLite with the help of OGR and its awesome virtual format feature. One particular dataset they had which sparked my initial interest in building this MapGuide dataset was the Building Footprints, which contains actual height attributes!


This means demonstrating MapGuide's KML support is much more exciting now because we can actually tap into the elevation and extrustion support to give actual height to our KML buildings exported from MapGuide.


As mentioned before, this MapGuide dataset is a work-in-progress which you can check out on my GitHub. If you want the actual package, you can get it from the releases page. If you are interesting in seeing a more comprehensive sample dataset for MapGuide, I encourage you to contribute to this repo. If you are so inclined, I'd also encourage you to have a look at the City of Melbourne data portal and see what other interesting datasets we could integrate.

It would be nice to also have a nice set of sample code that works against this dataset as well. Aside from porting the developer's guide samples across to work against this dataset, if you have any interesting ideas we could explore with this dataset, I'm all ears.

None of this would be possible if the City of Melbourne didn't open up their various datasets. So full kudos to them for not only opening up these datasets, but also for licensing their data under Creative Commons and not some "open" license that's entangled under various legal spiderwebs.

#opendata ftw!

Saturday, 2 August 2014

Announcing: mapguide-rest 0.9

Here's a new release of mapguide-rest, packed with some new toys.

PDF Support

The existing URL routes for DWF plotting now also support PDF output.

For example, a DWF plot URL like this:

http://localhost/mapguide/rest/library/Samples/Sheboygan/Maps/Sheboygan.MapDefinition/plot.dwf?x=-87.73&y=43.74&scale=8000



Can be turned into a PDF plot like this just by replacing the .dwf in the URL to .pdf:

http://localhost/mapguide/rest/library/Samples/Sheboygan/Maps/Sheboygan.MapDefinition/plot.pdf?x=-87.73&y=43.74&scale=8000


When plotting from a session-based runtime map, you also have the option of producing a layered PDF. To produce a layered PDF, simply append layeredpdf=1 to a PDF plot URL on a session-based runtime map. This takes longer to plot than a normal PDF, but the resulting PDF will have a toggle-able layer structure



Queryable Layer Definitions

The existing URL routes to query feature data from Feature Sources can now also be applied to Layer Definitions.

For example, suppose you originally were querying Parcels from its Feature Source

GET http://localhost/mapguide/rest/library/Samples/Sheboygan/Data/Parcels.FeatureSource/features.xml/SHP_Schema/Parcels?maxfeatures=500

You can now also query the same data via a Layer Definition that points to it

GET http://localhost/mapguide/rest/library/Samples/Sheboygan/Layers/Parcels.LayerDefinition/features.xml?maxfeatures=500

You'll notice that querying via Layer Definitions means you don't have to include the fully-qualified FDO class name in the URL as that will be inferred from the Layer Definition document. Also when querying via Layer Definitions, it will also include evaluated tooltip and hyperlink expressions for each feature (as MG_TOOLTIP and MG_HYPERLINK properties)

The Layer Definition query route supports the same parameters as the Feature Source one and supports GeoJSON output as well.

New Samples

We've added a new OpenLayers 3 example, demonstrating how to consume dynamic GeoJSON vector layers from the REST API


And a new Cesium example, also consuming GeoJSON from the REST API


Other Changes

  • GeoJSON features now include id attributes containing the identity property value. Previously the identity property value was written out as a regular feature property.
  • Improved XYZ tile cache resiliency
  • Feature query routes (ie. selecting features) no longer require an authentication challenge and can be done anonymously.
  • Feature query responses are no longer internally buffered, reducing response wait times.


Download

Announcing: MapGuide Open Source 2.6 Final

Here it is! The final release of MapGuide Open Source 2.6

Here's the changes from RC2:
  • Fix: PHP deprecation warning in schema report feature/data preview
  • Fix: Cannot select search results in Fusion Search widget
It is with regret that we have to pull the CentOS 6.x build from this final release. There will not be a CentOS build of MapGuide 2.6 until we have found a solution for this issue of non-functional iconv APIs. While this issue is present, the CentOS build will not have a functional SHP FDO provider and that is too much of a deal-breaker. If you are on CentOS, you should either stay on the 2.5.2 release or help assist us in finding a solution for this issue.

A note for those who want to use custom GDAL binaries from gisinternals for ECW/MrSID support on Windows: You should probably hold off from upgrading to 2.6. MapGuide 2.6 was built with Visual C++ 2012, and so requires custom GDAL binaries that were also built with MSVC 2012. These binaries don't exist yet on gisinternals. You should keep an eye on this issue.

Since is the final release, I will also talk a bit in this post about the mg-desktop release that also coincides with this release.

mg-desktop now requires the Visual C++ 2012 runtime library as a pre-requisite for any application that you build with this release of mg-desktop. In terms of changes, there's only two real changes:
  • Configuration Documents with external path aliases are now supported
  • Fix: Layer watermarks will not be rendered for layers that are not visible

Tuesday, 29 July 2014

GovHack 2014 post-mortem

UPDATE 20 October 2014: After a bungle on my Amazon EC2 instance, the demo URL on our hackerspace project page is no longer active. I've resurrected this site on my demo server on Rackspace here. Ignore the link on the hackerspace page until that page gets updated (if it will get updated, because I can't do it)

Earlier this month, I attended the GovHack 2014 hackathon, along with thousands of other fellow hackers all across the country. This was my first GovHack, but not my first hackathon. My previous hackathon was RHoK and having no idea how GovHack would turn out, I entered the GovHack event with a RHoK-based mindset of how I would expect this hackathon to turn out.

Bad idea.

I learned very quickly there was a major difference between RHoK and GovHack. Going into RHoK, you have an idea about what solutions you will get to hack on over the weekend as problem owners are present to pitch their ideas to the audience of prospective hackers. With GovHack, you need an idea about what solution you want to hack on over the weekend, all they were going to provide was the various open data and APIs. What on earth are we going to build?



So after losing nearly half the weekend to analysis paralysis, our team (named CreativeDrought, wonder why?) agreed with my suggestion of just building a MapGuide-based mashup of various open datasets, most notably, the VicRoads Crash Stats dataset and related transportation data. I obviously knew MapGuide inside-and-out and its capabilities to have a level of confidence that with the remaining weekend we should still be able to crank out some sort of workable solution. At the very least, we'd have a functional interactive map with some open data on it.

And that's the story of our CrashTest solution in a nutshell. It's a Fusion application, packed to the gills with out-of-the-box functionality from its rich array of widgets (including Google StreetView integration). The main objective of this solution was to allow users to view and analyse crash data, sliced and diced along various age, gender, vehicle type and various socio-economic parameters.



MapGuide's rich out-of-the-box capabilities, Maestro's rapid authoring functionality and GDAL/OGR's ubiquitous data support greatly helped us. I knew with this trio of tools, that we could assemble an application together in the remaining day and a bit left that we had to actually "hack" on something.

Sadly, we only got as far as putting the data on the map for the most part. Our team spent more time frantically trying to massage various datasets via ogr2ogr/Excel/GoogleDocs into something more usable than actually writing lines of code! Seriously VicRoads? Pseudo-AMG? Thank goodness I found the necessary proj4 string for this cryptic coordinate system so that we could re-project a fair chunk of the VicRoads spatial data into a coordinate system that better reflects the world we want to mash this data up with!

Still, our "solution" should hopefully still open up a lot of "what if" scenarios. Imagine looking at a cluster of accident events, not being able to ascertain any real patterns or correlation and so you then fire up the StreetView widget and lo-and-behold, Google StreetView providing additional insights that a birds-eye view could not. Also imagine the various reporting and number crunching possibilities that are available by tapping into the MapGuide API. Imagine what other useful information you could derive if we had more time to put up additional useful datasets. We didn't get very far on any of the above ideas, so just imagine such possibilities if you will :)

So here's our entry page if you want to have a look. It includes a working demo URL to a Amazon EC2 hosted instance of MapGuide. Getting acquainted with Amazon Web Services and putting MapGuide up there was an interesting exercise and much easier than I thought it would be, though I didn't have enough time to use the AWS credits I redeemed over the weekend to momentarily lift this demo site out of the free usage tier range performance-wise. Still, the site seems to perform respectably well on the free usage tier.

Also on that page is a link to a short video where we talk about the hack. Please excuse the sloppy editing, it was obviously recorded in haste in a race against time. Like the solution and/or the possibilities it can offer? Be sure to vote on our entry page.

Despite the initial setbacks, I was happy with what we produced given the severely depleted time constraints imposed on us. I think we got some nice feedback demo-ing CrashTest in person at the post-mortem event several days later, which is always good to hear. Good job team!


So what do I think could be improved with GovHack?
  • Have a list of hack ideas (by participants who actually have some ideas) up some time before the hackathon starts. This would facilitate team building, letting participants with the skills, but without ideas easily gravitate towards people/teams with the ideas.
  • The mandatory video requirement for each hack entry just doesn't work in its current form. Asking teams to produce their own videos puts lots of unnecessary stress on teams, who not only have to come up with the content for their video, but have to also deal with the logistics of producing said video. I would strongly prefer that teams who can/want to make their own video do so, while other teams can just do a <= 3 minute presentation and have that be recorded by the GovHack organisers. Presentations also lets teams find out how other teams fared over the weekend. While everyone else in the ThoughtWorks Melbourne office was counting down to the end of the hackathon, I was still frantically trying to record my lines and trying not to flub them! I raided the office fridge for whatever free booze that remained just to calm myself down afterwards. I don't want to be in that situation ever again!
  • Finally, the data itself. So many "spatial" datasets as CSV files! So many datasets with no coordinates, but have addresses, horribly formatted addresses, adding even more hoops to geocode them. KML/KMZ may be a decent consumer format, but it is a terrible data source format. If ogr2ogr can't convert your dataset, and requires a manual intervention of QGIS to fix it, then perhaps it's better to use a different spatial data format. Despite my loathing of its limitations, SHP files would've been heavily preferred for all of the above cases. I've made my thoughts known on the GovHack DataRater about the quality of some of these datasets we had to deal with and got plenty of imaginary ponies in the process.
Despite the above points, the event as a whole was a lot of fun. Thanks to the team (Jackie and Felicity) for your data wrangling and video production efforts.


Also thanks to Jordan Wilson-Otto and his flickr photostream where I was able to get some of these photos for this particular post.

Would I be interested in attending the 2015 edition of GovHack? Given I am now armed with 20/20 hindsight, yes I would!