Monday 31 December 2012

My MapGuide/FDO Wishlist for 2013

Unlike wishlists of previous years. This one is simple, and it applies to both MapGuide and FDO (but mainly MapGuide)

More non-Autodesk developers/contributors

The poll I made several months ago alluded to this problem. There is a serious brain-drain in the MapGuide Open Source project. I bemoan the fact that I seem to be the only non-Autodesk core developer/contributor that seems to truly care about the state of the MapGuide Open Source project.

Ok, so not everyone may have deep intimate knowledge of the MapGuide codebase, but neither do I. The MapGuide content on this blog is nothing more than a chronicle of things I've learned about MapGuide over the years, and even 6 years after the very first release of MapGuide Open Source, I'm still learning new things left and right.

Everybody can contribute regardless of skill level. Depending on your skill level, you can help and support the MapGuide project in various ways:

  • Put up some code samples
  • Help maintain/curate the MapGuide wiki and website to keep content fresh and relevant
  • Tweet/Share/Blog about your MapGuide experiences/tips/tricks/applications. It doesn't have to reach my blog's level of technicality.
  • Show off your cool MapGuide application
  • Contribute fixes and enhancements
  • If you can't contribute due to lack of technical knowledge, sponsor someone who can
  • Open a new sandbox branch to hack/explore new ideas for new MapGuide feature(s)
  • Post helpful and meaningful bug reports. Keep them up to date with any additional information, patches, screenshots or reproduction scenarios. Don't post bug reports and then auto-magically expect them to be addressed. More likely they will be deleted 6 months down the road due to lack of information and activity.
  • Help document the various undocumented public APIs in the MapGuide C++ header files. The API enhancement work means any C++ documentation efforts will also be making it to their respective .net and Java bindings as well.
  • Help improve the MapGuide Developer's Guide
  • Help improve the official sample code
  • Help answer questions on the mapguide-users mailing list
Currently I do of all of the above points with various degrees of dedication. I can't do this stuff forever. Even Dirty Harry understands that.

So with that said, have a safe and happy new year. Here's to a bright 2013 for MapGuide

My MapGuide/FDO Wishlist for 2012: Revisited

Firstly, how did my wishlist from last year turn out?

FDO

1. Raster support for the PostGIS provider

Nothing has happened on this front, and frankly adding such support is a tall order. I've looked at the GenericRdbms codebase (from which the PostGIS provider is built on top of), and it requires some major foundational changes to support raster data types, and this has to be done in a way without breaking any of the other providers that are built from the same foundation. And then there's the problem of grokking the complex GenericRdbms codebase as well.

I truly believe that FDO should have a provider that can do both raster/vector data to allow for truly integrated spatial data stores. It could be PostGIS + PostGIS raster, or it could be SpatialLite + RasterLite. Whatever it is, FDO should have a provider that supports it.

2. A more FDO-capable OGR provider

Still nothing on this front sadly.

3. FDO Join support for other FDO RDBMS providers

This is an unknown quantity. I don't think the other RDBMS providers have gained the benefits of join support added for the SQL Server Provider yet. A shame.

MapGuide

1. Improve MapGuide's standing as a geospatial services platform

Nothing has changed here either. However my improved understanding of the Web Tier's role in MapGuide   has armed me with the knowledge to explore how we can improve this situation. You can see this in the AS YET UN-ANNOUNCED PROJECT, and the discovery of the previously cryptic MgHttpRequest and MgHttpResponse classes.

So while we can't say that the 2.5 milestone of MapGuide will support all these geospatial web services that we've desired for so long, it should have the solid infrastructure from which we can build support for such required services in future releases using things such as the AS YET UN-ANNOUNCED PROJECT as a testbed for implementation ideas.

2. Native support for PDF output

We have PDF output, but it's not native and is only available for Fusion that will come with the 2.5 milestone of MapGuide. Still this will hopefully be a good intermediate solution in the meantime.

Sunday 30 December 2012

Building your own mapagent http handler in .net: An example

So I quipped previously that MgHttpRequest/MgHttpResponse makes it not only possible, but dead simple to create your own mapagent http handler in .net/Java/PHP without needing Apache or IIS.

How simple is it actually? Here's a .net one I banged out in under an hour.

If you look at the VS project, it just contains one ashx handler that does all the grunt work


And that grunt work is exactly what I described in my previous post. To recap, the handler needs to:


  • Collect the required parameters from Request.Form (for POST) or Request.QueryString (for GET)
  • Check the existence of USERNAME, PASSWORD parameter (or extract from http authentication header) or the SESSION parameter. Return a http 401 if none of these parameters are found.
  • Create a MgHttpRequest, plug in these collected parameters
  • Execute the MgHttpRequest
  • Write out the necessary response headers based on information in the MgHttpResponse
  • Write out the MgHttpResponse contents
So how can we test that this works? Start the project in Visual Studio and connect to the VS generated URL of the ashx handler with Maestro of course!


And if you see a list of folders in your site explorer like you normally do with the official mapagent, you know that your handler is now properly processing your incoming http requests from Maestro



Extra pro-tip: You can append a .fiddler to the above host name and you can see the http traffic through fiddler

Note that this sample project only implements the mapagent portion of the Web Tier, so things like the standard AJAX viewer resource preview won't work because there isn't an AJAX viewer relative to that URL. But you can use the local preview feature of Maestro that goes through the mapagent http interface (if you wanted more confirmation that the custom http handler is working as expected)


So if you're wondering like I did 5 years ago, whether this stuff is possible: Yes it is indeed!

Friday 21 December 2012

About time!

Qt on paper is an awesome framework for building cross-platform applications in C++. Yes, some people might question the need to reinvent the entire standard library or the need to butcher C++ syntax with a special pre-compiler, but no other C++ library/framework gives you so much bang for (zero) buck. There's a reason projects like QGIS are using Qt. It gives you the proverbial kitchen sink for building cross-platform applications.

So what's my problem?

Well for the longest time. For reasons unknown to me anyway, Qt on windows was built with wchar_t compiler support turned off, which has put off any plans of me exploring what Qt-based applications I can build with MapGuide/FDO or how to integrate MapGuide/FDO with other Qt-based applications (like QGIS). The entire source code of MapGuide and FDO revolves around this data type being actually supported, so if Qt has no idea about the strings I'm feeding it from MapGuide/FDO then we're going nowhere real fast.

The "workaround" for these versions of Qt was to basically rebuild the entire Qt stack with the wchar_t support enabled, which is a bit of a cop-out.
  • Rebuilding Qt from scratch takes an eternity
  • It means you now have to bundle custom Qt binaries instead of just plucking the matching dlls from the stock Qt SDK.
  • It makes integration with other Qt-based applications difficult to impossible. Yes, try our cool MapGuide/FDO plugin for QGIS. All you have to do is completely replace your QGIS installation with our own custom build of QGIS, and break all your other QGIS plugins.
As already mentioned, this goes nowhere real fast. So such plans and ideas have been put on the back-burner ... until now.

So Qt recently reached 5.0. Given it's a new major release, I had to ask the wchar_t question again. I was so happy when I saw this:


That's right, a Qt application is receiving, converting and outputting wchar_t strings from FDO! Hurrah!

And if you're wondering: No this MapGuide/FDO plugin does not exist. It's just a theoretical plugin to drive through the point of how difficult it is currently to integrate MapGuide/FDO with Qt or any application built on top of Qt. I look forward to if/when QGIS does move to Qt 5.0. It'll make the possibility of integrating MapGuide/FDO with QGIS much simpler and if anything, more OSGeo projects should get along with each other more often.

And if you're also wondering: Am I resurrecting the semi-retired FDO Toolbox? Maybe :)

Wednesday 19 December 2012

How to: Building the mg-desktop viewer from source

If you have/are using mg-desktop for your own MapGuide desktop applications, you're most likely working with the map viewer component most of the time. If you want to hack your own customizations into this map viewer outside of the provided extension points or to fix any viewer bugs, you have to be able to build this viewer from source.

Unfortunately, building mg-desktop from source is a complex multi-step affair like its MapGuide Server/Web counterpart, that requires a full checkout of the entire MapGuide source tree:

  1. We need to build FDO and/or setup the binary SDK in the correct path under MgDev\Oem
  2. We need to build the MapGuide Oem stack
  3. We need to build the MapGuide and mg-desktop shared C++ components and SWIG bindings
  4. Finally, we need to build the mg-desktop .net components
Assuming you just want to hack around with the mg-desktop .net viewer component source, there is a way to skip steps 1-3 and avoid the full checkout. This post will show you how.

We'll demonstrate using the Express Edition of Visual Studio 2012 for Windows Desktop in conjunction with the .net 4.0 build of mg-desktop. For the .net 2.0 build, you will need Visual Studio 2008 instead.

Firstly, grab the latest mg-desktop binaries.

Then, do a subversion checkout of the MgDev\Desktop directory under the 2.4 branch (as this is the source branch used to produce the latest mg-desktop binaries). The SVN checkout URL for this is:

http://svn.osgeo.org/mapguide/branches/2.4/MgDev/Desktop

Note that checking out this directory will only let you build the viewer source (the objective of this post). You cannot build the full mg-desktop set of binaries from this directory. To do that would require checking out its parent (MgDev) instead and doing the full 4 builds steps, which will take some time.

In your fresh SVN checkout, create a bin directory



Inside this bin directory, create a release directory (for 32-bit). For 64-bit, use release64

In the latest binaries zip file, extract the files under the Desktop directory to your release or release64 directory, which should now look like this after extraction



Back in the bin directory, create an Assemblies directory and move the following files from release/release64 into this directory:
  • OSGeo.MapGuide.Foundation.dll
  • OSGeo.MapGuide.Geometry.dll
  • OSGeo.MapGuide.PlatformBase.dll
  • OSGeo.MapGuide.Desktop.dll
In addition, copy the related XML files as well if you want intellisense on these assemblies


You have now replicated the binary output that steps 1-3 would've produced on a full SVN checkout, so now we can go straight to step 4 and open the mg-desktop .net solution file in Visual Studio:
  • MgDesktopDotNet.sln if you're using the .net 2.0 build and VS 2008
  • MgDesktopDotNet_VS2010.sln if you're using the .net 4.0 build and VS 2010 or VS 2012
Make sure the platform configuration matches the bitness of your mg-desktop binaries, otherwise you'll get BadImageFormatException trying to debug any of the executable projects.


Once that is set, you can now build the entire .net solution. Updated viewer component assemblies will be under bin\release for 32-bit and bin\release64 for 64-bit

Now the viewer is ready to be hacked around. The projects you probably be most interested in are:
  • MapViewer - Contains the source for the mg-desktop viewer component
  • MapViewer.Desktop - Contains the source for the mg-desktop implementation of MgMapViewerProvider
  • MapViewerTest - A test WinForms application that embeds the map viewer component.
  • MgAppLayout - The App Layout Engine front-end executable
For the VS2010 solution, these projects are suffixed with .Net40

Happy hacking/debugging.

Wednesday 12 December 2012

About MapGuide's WMS/WFS support

This post is bit of a Public Service Announcement.

When someone asks "What version of WMS/WFS does MapGuide support?", that question has two different answers depending on context

1. Are you trying to consume a WMS/WFS service with MapGuide?

The answer then becomes what version of WMS/WFS is supported by their respective FDO providers that come with your MapGuide installation.

If we look at the table of FDO RFCs, we see that:


2. Or are you trying to use MapGuide to serve out WMS/WFS services?

If we look at the table of MapGuide RFCs, we see that:


Given that the current stable release of MapGuide Open Source (2.4) was made after the introduction of these RFCs, these are the respective version numbers of WMS and WFS that can be served and consumed by the current stable version of MapGuide.

For older releases of MapGuide, you can cross-reference your MapGuide's major.minor version number against the respective table of RFCs to determine the level of WMS/WFS support.

Sunday 9 December 2012

Building your own mapagent http handler in .net/PHP/Java

This post basically answers a question I posed 5 years ago. Is it possible to build a pure .net/PHP/Java mapagent handler that does not require Apache or IIS? The motivation being, we want a "portable" self-contained web tier that does not require the cumbersome administration/configuration that Apache or IIS requires.

At the time of that post, Windows XP was still the dominant client version of Windows and as we all know IIS was absolutely crippled on XP, making ASP.net development an absolute pain. My question was born out of a desire to ditch this crippled IIS and be able to use something lightweight like Cassini to host my development MapGuide web applications.

At the time, I didn't think it was possible due to the belief that key APIs were missing to support operations required by the AJAX viewer (namely the ability to manipulate the MgMap display parameters as most MgMap properties in the public API are read-only).

Fast forward 5 years, and in the process of hacking on the AS YET UN-ANNOUNCED PROJECT, I got hit with the same problem. Without a way to manipulate the MgMap over http using the existing public MapGuide APIs, we cannot have a functional self-contained AJAX viewer free of IIS/Apache.

Well fortunately as I found out, there is a way to do this. Say hello to the following classes:

  • MgHttpRequest
  • MgHttpResponse
For the longest time, these classes were sitting there in the MapGuide API with the most scant of API documentation, and nobody having any idea what these classes actually do. As it turns out, the naming of these classes is a bit deceptive. If I were to have named these classes, they would've been called:
  • MgHttpRequestHandler
  • MgHttpRequestHandlerResponse
Which would've greatly clarified what these classes are supposed to do. Which is to:
  1. Set up a collection of key-value pairs from incoming http request parameters
  2. Invoke the request handler with these parameters
  3. Receive the handler response and output the contents
If you look at the source code for the Apache/ISAPI/CGI handlers, they all follow this same process using MgHttpRequest and MgHttpResponse:
  • The key-value pairs are collected using the respective APIs in Apache/ISAPI/CGI
  • The handler response is outputted using the respective APIs in Apache/ISAPI/CGI
So back to the pressing issue, how does one manipulate MgMap display parameters if the public API can't let us do this? Simple, we use MgHttpRequest to build up the required key-value parameter pairs for a GETDYNAMICMAPOVERLAYIMAGE request. When executed, MgHttpRequest will resolve the correct HTTP operation handler, forward your specified parameters across to the handler and receive back the response from this handler. The handlers (that you'll never directly see or access) does the actual grunt work. All we have to do from our end is pass the correct operation and parameters to MgHttpRequest.

This response returned by the handler (MgHttpResponse) you then use to output the result back out to the http response stream.

So how can we have a mapagent http handler in say ... ASP.net (WebForms)?
  • Collect the required parameters from Request.Form (for POST) or Request.QueryString (for GET)
  • Check the existence of USERNAME, PASSWORD parameter (or extract from http authentication header) or the SESSION parameter. Return a http 401 if none of these parameters are found.
  • Create a MgHttpRequest, plug in these collected parameters
  • Execute the MgHttpRequest
  • Write out the necessary response headers based on information in the MgHttpResponse
  • Write out the MgHttpResponse contents
If implemented correctly, your ASP.net page will function exactly as mapagent.fcgi, taking the same parameters and returning the same responses.

For PHP/Java the process is the same with obviously different APIs to extract http request parameters, do 401 statuses and output of http responses.

This is what we do in our AS YET UN-ANNOUNCED PROJECT, and as a result we are able to fully replicate the mapagent http handler without the need to have Apache or IIS.

The AS YET UN-ANNOUNCED PROJECT will be revealed shortly.

Sunday 2 December 2012

RHoK Melbourne: A tale from the trenches

This weekend I was at RHoK Melbourne hosted at my old stomping ground of Swinburne University (I did graduate from there after all), spending 2 days hacking out a workable solution within the given time frame and parameters.

Prologue

I first got wind of RHoK from an old uni friend on Facebook. Intrigued, I attended the free information night that was being held to provide more information to interested parties.

The theme of the event was disaster risk and crisis planning (for bush fires, extreme weather events, etc). Going through the stories of all the problem stakeholders there was another common overarching theme: the relevancy and importance of location and spatial data, which incidentally happens to be problem domain I'm intimately familiar with (if the title of this blog and the 200+ posts of related content didn't give that away :))

The challenge

The problem I chose to attack was the one of "Spatial Truth" for the simple reason that of all the problems listed, this was the one that could allow me to make maximum use of my knowledge and skills in this particular area. Other problems have a spatially-relevant component to them, but this one on paper looked 100% spatial to me.

The problem in question is one of mash-ability. The vicmap API data provided by the DSE needs to be able to be overlaid or mashed up with other mapping service providers (Google/Bing/OSM) and other assorted spatial data to allow the user to compare the various spatial data sources to ascertain which particular data source is the definitive source of truth for a particular area or region.

A small hiccup

The original intended solution was to have a HTML5 single-page application powered by the trinity of:


And use the vast capabilities of OpenLayers to mash up our all desired spatial data sources together, with Twitter Bootstrap providing us with a decent looking UI (hey I'm no web designer) that can easily scale down to mobile devices and jQuery just doing what it does best.

Well, this intended solution didn't quite pan out as we wanted to because of some problems:

  • The vicmap API was built on OpenLayers (great!) but it using its own bundled copy of OpenLayers (not so great. 2.10, pre-dates mobile/touch device support)
  • This bundled copy of OpenLayers defined its own OpenLayers Map and Layer types, making it difficult to integrate with our own (newer) OpenLayers.

The vicmap API was basically a JS wrapper around a WMS server, so we decided to eschew the provided vicmap javascript libraries and go directly to the WMS source (maybe a hack, but fits the name of this event)

After deciding to go directly to the WMS source, we found the biggest problem: The vicmap raster imagery was not in the projection we require (EPSG:3111 instead of EPSG:900913). You can't line up and/or overlay multiple raster imagery sources if they aren't in the same projection. Unfortunately JS web mapping libraries (and browser technology) have not sufficiently advanced enough to allow us to re-project raster imagery client-side (vector data is not an issue, but most of the data we're working with are raster, so no go).

So faced with this problem, I looked to an old friend who knew exactly how to deal with problems like this.

MapGuide to the rescue

I originally didn't want to involve MapGuide, despite most of the problems presented having a spatially-relevant problem (that MapGuide could partially/fully solve). I really wanted a pure client-side HTML5 map viewer. Although it would've been dead simple to just serve out a fully-featured MapGuide AJAX/Fusion viewer with the re-projected vicmap WMS layers like so ...



... and leave it at that, these viewers are notoriously difficult to adapt to tablet/smartphone displays (they were built before the emergence of HTML5/iPhone/Android after all) and are very GIS-y, which might be a bit hostile to normal users.

Accessibility and simplicity was key. The solution had to be accessible from smartphones and tablets. So in that respect, MapGuide's role in our solution was to act more as spatial data "proxy server", funnelling the vicmap imagery into a form that allows us to line it up and/or overlay it against Google/Bing/OSM imagery.

And as I've blogged in the past, OpenLayers makes mobile maps dead simple. Twitter Bootstrap gives the application a shiny enough look that can gracefully scale down on smaller displays due to heavy use of responsive design. So actually building a mobile map viewer from scratch was not exactly difficult to do.

The solution

The solution came in two parts, demonstrating 2 approaches to the objective of ascertaining the "Spatial Truth" among various spatial data sets.

The first one was a 4 viewer layout showing vicmap against Google, Bing and OSM data sets, with some pre-fabricated data to show off custom markers and heat maps (which could easily represent bushfire incidents or weather patterns)


All 4 viewers are synchronized, meaning panning or zooming one of the viewers does the same for the rest. One can easily ascertain the "Spatial Truth" by being able to look at all 4 sets of data at once instead of overlaying one on top of another. Unfortunately, this layout is clearly intended for use on the desktop and cannot clearly degrade gracefully to tablet/smartphone displays.

Hence the 2nd part of the solution, a fully responsive version.


Unlike the desktop-optimized layout, we only have one map viewer, so our ability to compare various datasets is to overlay the vicmap layers on top, through our own layer switcher.



The application takes advantage of HTML5 features like geolocation, that is near-universally supported on most smartphones and tablets


And for smartphone displays, our application scales down nicely to match due to the responsive design of the Twitter Bootstrap framework our application is built on top of



If the active layer is a Google one, the application can tap into the Google Map's geocoding API, allowing for address query and address resolution from coordinates (which we can't legally do with other layers due to Google's terms of use. We must have a Google map present and displaying to legally use their other services)



Finally, the application can allow for spatially-enabled RSS feeds (GeoRSS) to be overlaid onto the map. I scoured through many Victorian Government websites looking for any usable spatially enabled RSS feeds, but the only one I could find (which also happens to be very relevant to the theme of this event), was the GeoRSS feeds from the CFA showing the fire incidents and warnings for the last 4 days.


When overlaid, each incident and warning is indicated with custom OpenLayers markers


And because we're using OpenLayers GeoRSS layers to overlay this information, each marker can be clicked to bring up its feed item as a popup


Not bad for 2 days work I must say.

The source code for this application is available on github. This solution currently requires installing MapGuide Open Source 2.4 and the Vicmap MGP package (included in the github repo) in order to serve out the re-projected vicmap data. I've been told that the Vicmap data will also be available in a Google/Bing/OSM-compatible in the near future, which would remove the need to have MapGuide in place to do the re-projection work.

Just a small word of warning. The code quality of the solution reflects the nature of how it was built (a proto-typical proof-of-concept in 24-36 hours). Of course if we had more time, we'd do things in a more cleanly structured and architected manner.

My brief presentation slides on this solution are also available

Epilogue / Postmortem

The presented solution didn't win the first place prize for the event, but did win the special mention award. Of particular note from the Judging Panel (memory is a bit fuzzy here) was:

1. The visually stunning aspect of the demo (interactive maps have that kind of quality)

2. The amount of work done in the given timeframe with the number of team members involved (2, which includes myself). I owe this to the simplicity of OpenLayers and Twitter Bootstrap. The respective taglines of these two libraries are true. OpenLayers does make it easy to put a dynamic map in any web page. Twitter Bootstrap does make it faster and easier to build a sleek, intuitive and powerful front-end. Combine this with the write less, do more library and you have the holy trinity of tools to churn out beautiful, mobile-friendly web mapping applications in no time.

What were the lessons that could be learned from this event?

  • Spatial and Location is very important. Every solution presented at the end had some form of mapping component involved. If your solution can leverage spatial data, take advantage of it! Spatial is to data what time is to x, y and z. It is an extra dimension that gives new ways to view, query and analyse your data. I already knew this going in, and being armed with the knowledge and the tools I knew how to carry out the best approach.
  • The vicmap API could definitely be improved. Shen, my team partner for the first day of the event actually built part of the vicmap API and would agree with this sentiment (dogfooding ftw!). If the API is ultimately just a way to consume WMS data (we don't know because we bypassed the current API straight to the underlying WMS source), then there isn't really a point in wrapping that up in a custom JS API. The fact that this data is exposed through a well-supported OGC standard is good enough for most, if not all geospatial software and libraries to consume this information. If you really have to extend OpenLayers to access this data, then you realistically only need to extend the OpenLayers.Layer or one of its subclasses to support the vicmap data and nothing else. Most vendor-specific extensions to OpenLayers (like MapGuide's OpenLayers support) are done at the layer level. A fully custom version of OpenLayers should not be necessary.
  • The state of freely available spatial data in Australia is still somewhat woeful. Vicmap opening up their data is a start and hopefully more government departments follow suit. No need to have special APIs to access this data, OGC standards were invented for a reason.
  • On a similar vein, more organizations need to be spatially aware of the data they make available to the public. I was sorely disappointed at how many Australian RSS feeds are not spatially-enabled GeoRSS feeds. For example, VicRoads provides RSS feeds of traffic and roadwork information, yet this data has no geographic information, significantly hampering its utility. If this was a GeoRSS feed, we could've overlaid this information on a map and be able to see where these traffic incidents and roadworks are happening. You can't do that with regular RSS feeds.
  • Oh an manually setting up a MapGuide layer in OpenLayers (in our case, the re-projected vicmap data), is still an absolutely cumbersome process. Fusion, which also uses OpenLayers doesn't have this problem because your ApplicationDefinition document auto-plumbs all the necessary configuration values. Good luck trying to remember all those values when doing it manually!

A big thanks to everyone who organized this event. It was a blast.

I'll be waiting for my prize RHoK hoodie in the mail.