Monday, 31 December 2012

My MapGuide/FDO Wishlist for 2013

Unlike wishlists of previous years. This one is simple, and it applies to both MapGuide and FDO (but mainly MapGuide)

More non-Autodesk developers/contributors

The poll I made several months ago alluded to this problem. There is a serious brain-drain in the MapGuide Open Source project. I bemoan the fact that I seem to be the only non-Autodesk core developer/contributor that seems to truly care about the state of the MapGuide Open Source project.

Ok, so not everyone may have deep intimate knowledge of the MapGuide codebase, but neither do I. The MapGuide content on this blog is nothing more than a chronicle of things I've learned about MapGuide over the years, and even 6 years after the very first release of MapGuide Open Source, I'm still learning new things left and right.

Everybody can contribute regardless of skill level. Depending on your skill level, you can help and support the MapGuide project in various ways:

  • Put up some code samples
  • Help maintain/curate the MapGuide wiki and website to keep content fresh and relevant
  • Tweet/Share/Blog about your MapGuide experiences/tips/tricks/applications. It doesn't have to reach my blog's level of technicality.
  • Show off your cool MapGuide application
  • Contribute fixes and enhancements
  • If you can't contribute due to lack of technical knowledge, sponsor someone who can
  • Open a new sandbox branch to hack/explore new ideas for new MapGuide feature(s)
  • Post helpful and meaningful bug reports. Keep them up to date with any additional information, patches, screenshots or reproduction scenarios. Don't post bug reports and then auto-magically expect them to be addressed. More likely they will be deleted 6 months down the road due to lack of information and activity.
  • Help document the various undocumented public APIs in the MapGuide C++ header files. The API enhancement work means any C++ documentation efforts will also be making it to their respective .net and Java bindings as well.
  • Help improve the MapGuide Developer's Guide
  • Help improve the official sample code
  • Help answer questions on the mapguide-users mailing list
Currently I do of all of the above points with various degrees of dedication. I can't do this stuff forever. Even Dirty Harry understands that.

So with that said, have a safe and happy new year. Here's to a bright 2013 for MapGuide

My MapGuide/FDO Wishlist for 2012: Revisited

Firstly, how did my wishlist from last year turn out?


1. Raster support for the PostGIS provider

Nothing has happened on this front, and frankly adding such support is a tall order. I've looked at the GenericRdbms codebase (from which the PostGIS provider is built on top of), and it requires some major foundational changes to support raster data types, and this has to be done in a way without breaking any of the other providers that are built from the same foundation. And then there's the problem of grokking the complex GenericRdbms codebase as well.

I truly believe that FDO should have a provider that can do both raster/vector data to allow for truly integrated spatial data stores. It could be PostGIS + PostGIS raster, or it could be SpatialLite + RasterLite. Whatever it is, FDO should have a provider that supports it.

2. A more FDO-capable OGR provider

Still nothing on this front sadly.

3. FDO Join support for other FDO RDBMS providers

This is an unknown quantity. I don't think the other RDBMS providers have gained the benefits of join support added for the SQL Server Provider yet. A shame.


1. Improve MapGuide's standing as a geospatial services platform

Nothing has changed here either. However my improved understanding of the Web Tier's role in MapGuide   has armed me with the knowledge to explore how we can improve this situation. You can see this in the AS YET UN-ANNOUNCED PROJECT, and the discovery of the previously cryptic MgHttpRequest and MgHttpResponse classes.

So while we can't say that the 2.5 milestone of MapGuide will support all these geospatial web services that we've desired for so long, it should have the solid infrastructure from which we can build support for such required services in future releases using things such as the AS YET UN-ANNOUNCED PROJECT as a testbed for implementation ideas.

2. Native support for PDF output

We have PDF output, but it's not native and is only available for Fusion that will come with the 2.5 milestone of MapGuide. Still this will hopefully be a good intermediate solution in the meantime.

Sunday, 30 December 2012

Building your own mapagent http handler in .net: An example

So I quipped previously that MgHttpRequest/MgHttpResponse makes it not only possible, but dead simple to create your own mapagent http handler in .net/Java/PHP without needing Apache or IIS.

How simple is it actually? Here's a .net one I banged out in under an hour.

If you look at the VS project, it just contains one ashx handler that does all the grunt work

And that grunt work is exactly what I described in my previous post. To recap, the handler needs to:

  • Collect the required parameters from Request.Form (for POST) or Request.QueryString (for GET)
  • Check the existence of USERNAME, PASSWORD parameter (or extract from http authentication header) or the SESSION parameter. Return a http 401 if none of these parameters are found.
  • Create a MgHttpRequest, plug in these collected parameters
  • Execute the MgHttpRequest
  • Write out the necessary response headers based on information in the MgHttpResponse
  • Write out the MgHttpResponse contents
So how can we test that this works? Start the project in Visual Studio and connect to the VS generated URL of the ashx handler with Maestro of course!

And if you see a list of folders in your site explorer like you normally do with the official mapagent, you know that your handler is now properly processing your incoming http requests from Maestro

Extra pro-tip: You can append a .fiddler to the above host name and you can see the http traffic through fiddler

Note that this sample project only implements the mapagent portion of the Web Tier, so things like the standard AJAX viewer resource preview won't work because there isn't an AJAX viewer relative to that URL. But you can use the local preview feature of Maestro that goes through the mapagent http interface (if you wanted more confirmation that the custom http handler is working as expected)

So if you're wondering like I did 5 years ago, whether this stuff is possible: Yes it is indeed!

Friday, 21 December 2012

About time!

Qt on paper is an awesome framework for building cross-platform applications in C++. Yes, some people might question the need to reinvent the entire standard library or the need to butcher C++ syntax with a special pre-compiler, but no other C++ library/framework gives you so much bang for (zero) buck. There's a reason projects like QGIS are using Qt. It gives you the proverbial kitchen sink for building cross-platform applications.

So what's my problem?

Well for the longest time. For reasons unknown to me anyway, Qt on windows was built with wchar_t compiler support turned off, which has put off any plans of me exploring what Qt-based applications I can build with MapGuide/FDO or how to integrate MapGuide/FDO with other Qt-based applications (like QGIS). The entire source code of MapGuide and FDO revolves around this data type being actually supported, so if Qt has no idea about the strings I'm feeding it from MapGuide/FDO then we're going nowhere real fast.

The "workaround" for these versions of Qt was to basically rebuild the entire Qt stack with the wchar_t support enabled, which is a bit of a cop-out.
  • Rebuilding Qt from scratch takes an eternity
  • It means you now have to bundle custom Qt binaries instead of just plucking the matching dlls from the stock Qt SDK.
  • It makes integration with other Qt-based applications difficult to impossible. Yes, try our cool MapGuide/FDO plugin for QGIS. All you have to do is completely replace your QGIS installation with our own custom build of QGIS, and break all your other QGIS plugins.
As already mentioned, this goes nowhere real fast. So such plans and ideas have been put on the back-burner ... until now.

So Qt recently reached 5.0. Given it's a new major release, I had to ask the wchar_t question again. I was so happy when I saw this:

That's right, a Qt application is receiving, converting and outputting wchar_t strings from FDO! Hurrah!

And if you're wondering: No this MapGuide/FDO plugin does not exist. It's just a theoretical plugin to drive through the point of how difficult it is currently to integrate MapGuide/FDO with Qt or any application built on top of Qt. I look forward to if/when QGIS does move to Qt 5.0. It'll make the possibility of integrating MapGuide/FDO with QGIS much simpler and if anything, more OSGeo projects should get along with each other more often.

And if you're also wondering: Am I resurrecting the semi-retired FDO Toolbox? Maybe :)

Wednesday, 19 December 2012

How to: Building the mg-desktop viewer from source

If you have/are using mg-desktop for your own MapGuide desktop applications, you're most likely working with the map viewer component most of the time. If you want to hack your own customizations into this map viewer outside of the provided extension points or to fix any viewer bugs, you have to be able to build this viewer from source.

Unfortunately, building mg-desktop from source is a complex multi-step affair like its MapGuide Server/Web counterpart, that requires a full checkout of the entire MapGuide source tree:

  1. We need to build FDO and/or setup the binary SDK in the correct path under MgDev\Oem
  2. We need to build the MapGuide Oem stack
  3. We need to build the MapGuide and mg-desktop shared C++ components and SWIG bindings
  4. Finally, we need to build the mg-desktop .net components
Assuming you just want to hack around with the mg-desktop .net viewer component source, there is a way to skip steps 1-3 and avoid the full checkout. This post will show you how.

We'll demonstrate using the Express Edition of Visual Studio 2012 for Windows Desktop in conjunction with the .net 4.0 build of mg-desktop. For the .net 2.0 build, you will need Visual Studio 2008 instead.

Firstly, grab the latest mg-desktop binaries.

Then, do a subversion checkout of the MgDev\Desktop directory under the 2.4 branch (as this is the source branch used to produce the latest mg-desktop binaries). The SVN checkout URL for this is:

Note that checking out this directory will only let you build the viewer source (the objective of this post). You cannot build the full mg-desktop set of binaries from this directory. To do that would require checking out its parent (MgDev) instead and doing the full 4 builds steps, which will take some time.

In your fresh SVN checkout, create a bin directory

Inside this bin directory, create a release directory (for 32-bit). For 64-bit, use release64

In the latest binaries zip file, extract the files under the Desktop directory to your release or release64 directory, which should now look like this after extraction

Back in the bin directory, create an Assemblies directory and move the following files from release/release64 into this directory:
  • OSGeo.MapGuide.Foundation.dll
  • OSGeo.MapGuide.Geometry.dll
  • OSGeo.MapGuide.PlatformBase.dll
  • OSGeo.MapGuide.Desktop.dll
In addition, copy the related XML files as well if you want intellisense on these assemblies

You have now replicated the binary output that steps 1-3 would've produced on a full SVN checkout, so now we can go straight to step 4 and open the mg-desktop .net solution file in Visual Studio:
  • MgDesktopDotNet.sln if you're using the .net 2.0 build and VS 2008
  • MgDesktopDotNet_VS2010.sln if you're using the .net 4.0 build and VS 2010 or VS 2012
Make sure the platform configuration matches the bitness of your mg-desktop binaries, otherwise you'll get BadImageFormatException trying to debug any of the executable projects.

Once that is set, you can now build the entire .net solution. Updated viewer component assemblies will be under bin\release for 32-bit and bin\release64 for 64-bit

Now the viewer is ready to be hacked around. The projects you probably be most interested in are:
  • MapViewer - Contains the source for the mg-desktop viewer component
  • MapViewer.Desktop - Contains the source for the mg-desktop implementation of MgMapViewerProvider
  • MapViewerTest - A test WinForms application that embeds the map viewer component.
  • MgAppLayout - The App Layout Engine front-end executable
For the VS2010 solution, these projects are suffixed with .Net40

Happy hacking/debugging.

Wednesday, 12 December 2012

About MapGuide's WMS/WFS support

This post is bit of a Public Service Announcement.

When someone asks "What version of WMS/WFS does MapGuide support?", that question has two different answers depending on context

1. Are you trying to consume a WMS/WFS service with MapGuide?

The answer then becomes what version of WMS/WFS is supported by their respective FDO providers that come with your MapGuide installation.

If we look at the table of FDO RFCs, we see that:

2. Or are you trying to use MapGuide to serve out WMS/WFS services?

If we look at the table of MapGuide RFCs, we see that:

Given that the current stable release of MapGuide Open Source (2.4) was made after the introduction of these RFCs, these are the respective version numbers of WMS and WFS that can be served and consumed by the current stable version of MapGuide.

For older releases of MapGuide, you can cross-reference your MapGuide's major.minor version number against the respective table of RFCs to determine the level of WMS/WFS support.

Sunday, 9 December 2012

Building your own mapagent http handler in .net/PHP/Java

This post basically answers a question I posed 5 years ago. Is it possible to build a pure .net/PHP/Java mapagent handler that does not require Apache or IIS? The motivation being, we want a "portable" self-contained web tier that does not require the cumbersome administration/configuration that Apache or IIS requires.

At the time of that post, Windows XP was still the dominant client version of Windows and as we all know IIS was absolutely crippled on XP, making development an absolute pain. My question was born out of a desire to ditch this crippled IIS and be able to use something lightweight like Cassini to host my development MapGuide web applications.

At the time, I didn't think it was possible due to the belief that key APIs were missing to support operations required by the AJAX viewer (namely the ability to manipulate the MgMap display parameters as most MgMap properties in the public API are read-only).

Fast forward 5 years, and in the process of hacking on the AS YET UN-ANNOUNCED PROJECT, I got hit with the same problem. Without a way to manipulate the MgMap over http using the existing public MapGuide APIs, we cannot have a functional self-contained AJAX viewer free of IIS/Apache.

Well fortunately as I found out, there is a way to do this. Say hello to the following classes:

  • MgHttpRequest
  • MgHttpResponse
For the longest time, these classes were sitting there in the MapGuide API with the most scant of API documentation, and nobody having any idea what these classes actually do. As it turns out, the naming of these classes is a bit deceptive. If I were to have named these classes, they would've been called:
  • MgHttpRequestHandler
  • MgHttpRequestHandlerResponse
Which would've greatly clarified what these classes are supposed to do. Which is to:
  1. Set up a collection of key-value pairs from incoming http request parameters
  2. Invoke the request handler with these parameters
  3. Receive the handler response and output the contents
If you look at the source code for the Apache/ISAPI/CGI handlers, they all follow this same process using MgHttpRequest and MgHttpResponse:
  • The key-value pairs are collected using the respective APIs in Apache/ISAPI/CGI
  • The handler response is outputted using the respective APIs in Apache/ISAPI/CGI
So back to the pressing issue, how does one manipulate MgMap display parameters if the public API can't let us do this? Simple, we use MgHttpRequest to build up the required key-value parameter pairs for a GETDYNAMICMAPOVERLAYIMAGE request. When executed, MgHttpRequest will resolve the correct HTTP operation handler, forward your specified parameters across to the handler and receive back the response from this handler. The handlers (that you'll never directly see or access) does the actual grunt work. All we have to do from our end is pass the correct operation and parameters to MgHttpRequest.

This response returned by the handler (MgHttpResponse) you then use to output the result back out to the http response stream.

So how can we have a mapagent http handler in say ... (WebForms)?
  • Collect the required parameters from Request.Form (for POST) or Request.QueryString (for GET)
  • Check the existence of USERNAME, PASSWORD parameter (or extract from http authentication header) or the SESSION parameter. Return a http 401 if none of these parameters are found.
  • Create a MgHttpRequest, plug in these collected parameters
  • Execute the MgHttpRequest
  • Write out the necessary response headers based on information in the MgHttpResponse
  • Write out the MgHttpResponse contents
If implemented correctly, your page will function exactly as mapagent.fcgi, taking the same parameters and returning the same responses.

For PHP/Java the process is the same with obviously different APIs to extract http request parameters, do 401 statuses and output of http responses.

This is what we do in our AS YET UN-ANNOUNCED PROJECT, and as a result we are able to fully replicate the mapagent http handler without the need to have Apache or IIS.

The AS YET UN-ANNOUNCED PROJECT will be revealed shortly.

Sunday, 2 December 2012

RHoK Melbourne: A tale from the trenches

This weekend I was at RHoK Melbourne hosted at my old stomping ground of Swinburne University (I did graduate from there after all), spending 2 days hacking out a workable solution within the given time frame and parameters.


I first got wind of RHoK from an old uni friend on Facebook. Intrigued, I attended the free information night that was being held to provide more information to interested parties.

The theme of the event was disaster risk and crisis planning (for bush fires, extreme weather events, etc). Going through the stories of all the problem stakeholders there was another common overarching theme: the relevancy and importance of location and spatial data, which incidentally happens to be problem domain I'm intimately familiar with (if the title of this blog and the 200+ posts of related content didn't give that away :))

The challenge

The problem I chose to attack was the one of "Spatial Truth" for the simple reason that of all the problems listed, this was the one that could allow me to make maximum use of my knowledge and skills in this particular area. Other problems have a spatially-relevant component to them, but this one on paper looked 100% spatial to me.

The problem in question is one of mash-ability. The vicmap API data provided by the DSE needs to be able to be overlaid or mashed up with other mapping service providers (Google/Bing/OSM) and other assorted spatial data to allow the user to compare the various spatial data sources to ascertain which particular data source is the definitive source of truth for a particular area or region.

A small hiccup

The original intended solution was to have a HTML5 single-page application powered by the trinity of:

And use the vast capabilities of OpenLayers to mash up our all desired spatial data sources together, with Twitter Bootstrap providing us with a decent looking UI (hey I'm no web designer) that can easily scale down to mobile devices and jQuery just doing what it does best.

Well, this intended solution didn't quite pan out as we wanted to because of some problems:

  • The vicmap API was built on OpenLayers (great!) but it using its own bundled copy of OpenLayers (not so great. 2.10, pre-dates mobile/touch device support)
  • This bundled copy of OpenLayers defined its own OpenLayers Map and Layer types, making it difficult to integrate with our own (newer) OpenLayers.

The vicmap API was basically a JS wrapper around a WMS server, so we decided to eschew the provided vicmap javascript libraries and go directly to the WMS source (maybe a hack, but fits the name of this event)

After deciding to go directly to the WMS source, we found the biggest problem: The vicmap raster imagery was not in the projection we require (EPSG:3111 instead of EPSG:900913). You can't line up and/or overlay multiple raster imagery sources if they aren't in the same projection. Unfortunately JS web mapping libraries (and browser technology) have not sufficiently advanced enough to allow us to re-project raster imagery client-side (vector data is not an issue, but most of the data we're working with are raster, so no go).

So faced with this problem, I looked to an old friend who knew exactly how to deal with problems like this.

MapGuide to the rescue

I originally didn't want to involve MapGuide, despite most of the problems presented having a spatially-relevant problem (that MapGuide could partially/fully solve). I really wanted a pure client-side HTML5 map viewer. Although it would've been dead simple to just serve out a fully-featured MapGuide AJAX/Fusion viewer with the re-projected vicmap WMS layers like so ...

... and leave it at that, these viewers are notoriously difficult to adapt to tablet/smartphone displays (they were built before the emergence of HTML5/iPhone/Android after all) and are very GIS-y, which might be a bit hostile to normal users.

Accessibility and simplicity was key. The solution had to be accessible from smartphones and tablets. So in that respect, MapGuide's role in our solution was to act more as spatial data "proxy server", funnelling the vicmap imagery into a form that allows us to line it up and/or overlay it against Google/Bing/OSM imagery.

And as I've blogged in the past, OpenLayers makes mobile maps dead simple. Twitter Bootstrap gives the application a shiny enough look that can gracefully scale down on smaller displays due to heavy use of responsive design. So actually building a mobile map viewer from scratch was not exactly difficult to do.

The solution

The solution came in two parts, demonstrating 2 approaches to the objective of ascertaining the "Spatial Truth" among various spatial data sets.

The first one was a 4 viewer layout showing vicmap against Google, Bing and OSM data sets, with some pre-fabricated data to show off custom markers and heat maps (which could easily represent bushfire incidents or weather patterns)

All 4 viewers are synchronized, meaning panning or zooming one of the viewers does the same for the rest. One can easily ascertain the "Spatial Truth" by being able to look at all 4 sets of data at once instead of overlaying one on top of another. Unfortunately, this layout is clearly intended for use on the desktop and cannot clearly degrade gracefully to tablet/smartphone displays.

Hence the 2nd part of the solution, a fully responsive version.

Unlike the desktop-optimized layout, we only have one map viewer, so our ability to compare various datasets is to overlay the vicmap layers on top, through our own layer switcher.

The application takes advantage of HTML5 features like geolocation, that is near-universally supported on most smartphones and tablets

And for smartphone displays, our application scales down nicely to match due to the responsive design of the Twitter Bootstrap framework our application is built on top of

If the active layer is a Google one, the application can tap into the Google Map's geocoding API, allowing for address query and address resolution from coordinates (which we can't legally do with other layers due to Google's terms of use. We must have a Google map present and displaying to legally use their other services)

Finally, the application can allow for spatially-enabled RSS feeds (GeoRSS) to be overlaid onto the map. I scoured through many Victorian Government websites looking for any usable spatially enabled RSS feeds, but the only one I could find (which also happens to be very relevant to the theme of this event), was the GeoRSS feeds from the CFA showing the fire incidents and warnings for the last 4 days.

When overlaid, each incident and warning is indicated with custom OpenLayers markers

And because we're using OpenLayers GeoRSS layers to overlay this information, each marker can be clicked to bring up its feed item as a popup

Not bad for 2 days work I must say.

The source code for this application is available on github. This solution currently requires installing MapGuide Open Source 2.4 and the Vicmap MGP package (included in the github repo) in order to serve out the re-projected vicmap data. I've been told that the Vicmap data will also be available in a Google/Bing/OSM-compatible in the near future, which would remove the need to have MapGuide in place to do the re-projection work.

Just a small word of warning. The code quality of the solution reflects the nature of how it was built (a proto-typical proof-of-concept in 24-36 hours). Of course if we had more time, we'd do things in a more cleanly structured and architected manner.

My brief presentation slides on this solution are also available

Epilogue / Postmortem

The presented solution didn't win the first place prize for the event, but did win the special mention award. Of particular note from the Judging Panel (memory is a bit fuzzy here) was:

1. The visually stunning aspect of the demo (interactive maps have that kind of quality)

2. The amount of work done in the given timeframe with the number of team members involved (2, which includes myself). I owe this to the simplicity of OpenLayers and Twitter Bootstrap. The respective taglines of these two libraries are true. OpenLayers does make it easy to put a dynamic map in any web page. Twitter Bootstrap does make it faster and easier to build a sleek, intuitive and powerful front-end. Combine this with the write less, do more library and you have the holy trinity of tools to churn out beautiful, mobile-friendly web mapping applications in no time.

What were the lessons that could be learned from this event?

  • Spatial and Location is very important. Every solution presented at the end had some form of mapping component involved. If your solution can leverage spatial data, take advantage of it! Spatial is to data what time is to x, y and z. It is an extra dimension that gives new ways to view, query and analyse your data. I already knew this going in, and being armed with the knowledge and the tools I knew how to carry out the best approach.
  • The vicmap API could definitely be improved. Shen, my team partner for the first day of the event actually built part of the vicmap API and would agree with this sentiment (dogfooding ftw!). If the API is ultimately just a way to consume WMS data (we don't know because we bypassed the current API straight to the underlying WMS source), then there isn't really a point in wrapping that up in a custom JS API. The fact that this data is exposed through a well-supported OGC standard is good enough for most, if not all geospatial software and libraries to consume this information. If you really have to extend OpenLayers to access this data, then you realistically only need to extend the OpenLayers.Layer or one of its subclasses to support the vicmap data and nothing else. Most vendor-specific extensions to OpenLayers (like MapGuide's OpenLayers support) are done at the layer level. A fully custom version of OpenLayers should not be necessary.
  • The state of freely available spatial data in Australia is still somewhat woeful. Vicmap opening up their data is a start and hopefully more government departments follow suit. No need to have special APIs to access this data, OGC standards were invented for a reason.
  • On a similar vein, more organizations need to be spatially aware of the data they make available to the public. I was sorely disappointed at how many Australian RSS feeds are not spatially-enabled GeoRSS feeds. For example, VicRoads provides RSS feeds of traffic and roadwork information, yet this data has no geographic information, significantly hampering its utility. If this was a GeoRSS feed, we could've overlaid this information on a map and be able to see where these traffic incidents and roadworks are happening. You can't do that with regular RSS feeds.
  • Oh an manually setting up a MapGuide layer in OpenLayers (in our case, the re-projected vicmap data), is still an absolutely cumbersome process. Fusion, which also uses OpenLayers doesn't have this problem because your ApplicationDefinition document auto-plumbs all the necessary configuration values. Good luck trying to remember all those values when doing it manually!

A big thanks to everyone who organized this event. It was a blast.

I'll be waiting for my prize RHoK hoodie in the mail.

Wednesday, 28 November 2012

Improving the MapGuide API Documentation (part 3)

This post has a bit of a misleading title (it's more the wrapper API we're improving than the documentation) but whatever, we'll run with it.

So just to explain the previous screenshot (and probably my java-related tirades on Twitter and Google Plus for anyone who happens to follow me there) ...

That screenshot was "visual" confirmation of some MapGuide API wrapper enhancement work I've been doing to address some of the pain points of using the MapGuide API (namely Java, but this work has some cross-pollinating benefits for .net as well)

The Problem

For the 15 people in the room who actually use the MapGuide Java wrapper API in its current form (I kid, though it could possibly be true), you've probably had to deal with these annoying issues:
  • The Java proxy of MgException being a checked exception. Thus having to pollute your java code with "throws MgException" in calling methods or having to try/catch an exception that probably isn't thrown 99% of the time and if you do catch it, you're probably just gonna log or display that exception anyway.
  • Java class method naming being a verbatim transplant of their C++ counterparts (ie. They're in UpperCamelCase instead of lowerCamelCase)
  • Java proxies of MapGuide collection classes being verbatim wrapper classes, and not behaving like a Java collection. You can't even for-each through these collections due to key java interfaces not being implemented in the proxy classes.
  • Needing to have the web-based API reference on hand as doxygen commentary is lost in translation to the target language. Auto-complete in your Java IDE of choice gives you no documentation, because none is transferred across.
  • Needing to have the web-based API reference on hand to also find out if a given class or method is deprecated, because nothing in the proxy classes will tell you that to trigger any compiler warnings
  • Hardcore memory leaks in the Java wrapper
Most of the above points, pretty much summed up in this 6-year old ticket.

Though the memory leaks were plugged for the 2.4 release, the other problems are still present. Despite my own reservations about Java (the language), Java (the platform/ecosystem) can't be ignored. Just because we're using SWIG to churn out bindings to 3 different languages doesn't mean we can't do some tweaking here and there for a given language. We already do this for .net (to support properties and serializable exceptions), so why not give the Java wrapper some love as well?

The Solution

For those who are wondering how we actually build bindings to the MapGuide API in 3 different languages, here's a high-level overview:

As already mentioned, SWIG is the tool used to generate the language bindings to the MapGuide API, but in order to enforce some level of encapsulation in the wrapper APIs (because there's C++ internals in some MapGuide API classes that shouldn't be exposed in the wrapper classes), we run a custom pre-processor (IMake.exe) through the MapGuide C++ headers to generate a "sanitized" input file for SWIG to do its thing.

If you ever look at a any MapGuide C++ header and wonder what those PUBLISHED_API, INTERNAL_API and EXTERNAL_API markers are for, they are for IMake to generate an encapsulated view of that particular class for SWIG. Similarly, the __get and __set markers you see at the end of some method declarations are hints to IMake to generate .net properties for that particular method.

The same IMake tool is also used to generate the constants.php for PHP and the necessary constants source files for .net and Java

The solution consists of 3 parts:
  • Modifying SWIG to support our java-specific requirements
  • (Ab)using some SWIG directives for purposes beyond their original scope
  • Modifying the IMake preparation tool to take advantage of these (ab)used SWIG directives
Modifying SWIG

This part was simpler than I thought. The SWIG source code was not as hostile as I originally thought and hacking the java module to support what we need was relatively straightforward. SWIG was basically modified to support 2 optional java-only flags:
  • To omit the "throws MgException" clause in any java method generated
  • To output java methods in lowerCamelCase. Since all MapGuide API C++ methods are guaranteed to be in UpperCamelCase, this is a simple case of lower-casing the first letter of each method that SWIG writes out.
These flags are optional, because if we are going to provide a new enhanced Java wrapper API, we'd still want to keep the existing one around for compatibility purposes, with the enhanced wrapper being an opt-in choice.

(Ab)using SWIG directives

The directives in question are:
  • %typemap(javaclassmodifiers) (.net equivalent is %typemap(csclassmodifiers))
  • %javamethodmodifiers (.net equivalent is %csclassmodifiers)
Normally, these 2 directives are used to alter class/method visibility of the respective generated proxy classes

For example, if we apply this SWIG directive:

 %javaclassmodifiers MgMap::MgMap() "protected"  

We get the following java proxy class method declaration:

 public class MgMap
   protected MgMap()  

Similarly if we apply this typemap:

 %typemap(javaclassmodifiers) MgMap "private"  

We get the following java proxy class declaration:

 private class MgMap  

The trick with these directives is that that SWIG does no validation of the visibility string, meaning we can put any arbitrary content in there, say things like javadoc comments and annotations (or XML comments and attributes in .net). In fact, this technique is what the authors of SWIG recommend as a way of being able to document your proxy classes

So since we all know that the blank MgMap constructor is a deprecated API that you should stay away from. We can specify SWIG directives like this:

 %javamethodmodifiers MgMap::MgMap() %{/**  
  * This constructor is deprecated. Use MgMap(MgSiteConnection) instead  
 @Deprecated public%}  
 %csmethodmodifiers MgMap::MgMap() %{///<summary>  
 /// This constructor is deprecated. Use MgMap(MgSiteConnection) instead  
 // </summary>  
 [Obsolete(\"This API is deprecated\")] public%}  

Which will then turn into the following proxy class method declarations when SWIG processes them:

  public class MgMap   
   * This constructor is deprecated. Use MgMap(MgSiteConnection) instead  
   @Deprecated public MgMap() {   
  public class MgMap   
   /// This constructor is deprecated. Use MgMap(MgSiteConnection) instead  
   [Obsolete("This API is deprecated")] public MgMap() {   

Meaning with these directives, it is now possible for us to not only pass down (converted) doxygen comments, but also pass down deprecation intent as well, because sometimes an API reference is not enough to tell you something is deprecated. You ideally want csc or javac to throw a compiler warning at your face while compiling against the wrapper API library to truly drive the point home that you should not be using this class or method.

Modifying IMake

So we have the means to transfer documentation and deprecation from the C++ classes, but surely we aren't going to manually transcribe the documentation fragments of the hundreds of classes that are in the MapGuide API? Of course not!

The IMake tool already does a pass through all the MapGuide C++ headers. And if you look at the constants.php generated by IMake, you can see that it does pick up and collect doxygen commentary along the way (albeit dumped verbatim into the target language, making it un-usable as javadoc or .net documentation). So the IMake tool was modified to do the following:
  • Translate these collected doxygen fragments to their javadoc or .net counterparts
  • Write these fragments out as the aforementioned SWIG directives for the target language
Other assorted codegen magic

Sadly, Java being the dinosaur of a programming language that it is compared to C# (in my opinion), does not support something similar to partial classes, which would've made augmentation of generated java proxy classes so much simpler, so in order to add java.util.Collection and java.util.Iterable support, we have to once again look at SWIG directives for ways we can augment the generated java classes.

Fortunately SWIG also supports a javacode typemap, that basically allows us to inject vast bodies of java code into our SWIG generated java proxy classes. This allows us to easily implement java.util.Collection support for the collection classes and java.lang.Iterable support allowing for these objects to be iterated in a for-each fashion.

In a way, these SWIG directives are kind of like pseudo-templates for Java, minus the pain of having to deal with the half-assed Java generics, as their way of implementing generics completely blows (I can't make an array of generic type T? GTFO!)

The MgException class (from which all MapGuide exceptions inherit from) extends a pre-defined java AppThrowable class, which itself extends from Exception. As a result, all proxy exceptions are checked as a result. Making these exception classes un-checked as a simple case of re-basing the AppThrowable class to extend from RuntimeException instead.

The End Result

So how well do these changes fare? We can verify by running javadoc against the generated proxy Java class source files. Have another look at that same screenshot of the javadoc from our "enhanced" Java wrapper API:

What can we see here?
  • Java method names are now properly named in lowerCamelCase
  • Doxygen \deprecated markers have been properly translated to @Deprecated annotations (and ObsoleteAttribute in .net)
And also a simple reminder from IMake that we've still got lots of room for improvement in the API documentation content

If we dive down to the method level, here's what we currently have in doxygen:

And here's what gets transplanted to javadoc

As you can see, most of the key doxygen directives have matching/compatible javadoc equivalents allowing for easy conversion by the IMake tool.

All MapGuide exceptions are now un-checked, as they extend RuntimeException

For .net, we get the same result as going through the DoxyTransform tool. In fact, this API enhancement work will render the existing .net documentation approach obsolete, as this approach does the same thing but supports multiple target languages.

So the javadoc output is proof that we have working transplanted documentation, now comes the important bit: making this documentation easily consumable from within a Java IDE, which is incidentally where I'm currently scratching my head. 

What is the official/recommended way to consume Javadoc in a Java IDE (Eclipse, Netbeans, etc)? Is there an official way? Or does each IDE have its own way? I'd appreciate some pointers from someone more knowledgeable in Java tooling than myself. Feel free to comment below.

At the moment, from what I can gather there's 2 ways to bundle up this documentation for consumption within a Java IDE:
  • Pack the java proxy class source files into a separate -sources jar file
  • Pack the javadoc documentation files into a separate -javadoc jar file 
I tried both approaches through IntelliJ IDEA and it seems to pick up both forms:

Here's the result of ctrl+Q with the -sources jar

And the result of ctrl+Q with the -javadoc jar

But I'd still prefer to only have to choose one method that will work on all the major Java IDEs. I'm just not sure which one it is. I'm currently leaning on the -source version, but I'd still like an "official" answer.

But what about PHP?

You might have noticed that PHP is a conspicuous absence from this MapGuide API enhancement work.

Well the truth is that PHP poses the following problems:
  • Besides constants.php there is no actual PHP source files to "compile" into. So there's no actual place to drop our translated doxygen comments into.
  • There is no officially defined standard for documenting PHP code. Even if there was, the real aim of this API enhancement work is to improve integrated API documentation (ie. From within an IDE). If all the PHP documentation solutions ultimately produce something similar to what we already have in our API reference (a web-based documentation source), then what is the real point? We already have that!
What's left?

Besides answering the Java IDE documentation question, this API enhancement will need a requisite test run on Linux (especially the SWIG/IMake stuff). All the viewers still work as before on Windows post-SWIG/IMake surgery, indicating the wrapper APIs haven't been negatively affected in any way.

Also I have to draft up an RFC for these enhancements, for eventual inclusion into the next major release of MapGuide Open Source (2.5). If everything pans out, this next release will have 4 Web API options:
  • PHP
  • .net (with XML documentation files for Visual Studio)
  • Java (the old crufty one, kept for compatibility purposes) with Java IDE documentation support (whatever it will be)
  • Java (the enhanced one with the changes described here) with Java IDE documentation support (whatever it will be)

Monday, 26 November 2012

Improving the MapGuide API documentation (part 2.5)

This is the latest result of some SWIG/IMake sandbox work. So far, so good!

Tuesday, 20 November 2012

Announcing: MapGuide Maestro 5.0 beta 5 and 4.0.4 maintenance

Here's another beta for Maestro 5.0 and a matching 4.0.4 maintenance release.

Since 4.0.4 is a maintenance release there isn't much to talk about other than there's bug fixes :)

So let's talk about the 5.0 beta instead. Here's what's new.

More SharpDevelop goodness

As already mentioned, we've expanded our use of SharpDevelop code and components to give us improved versions of existing functionality:

  • A smarter, more responsive IronPython REPL
  • XML folding support for the Generic XML editor
  • FDO Expression Editor syntax highlighting
Sadly, Mono did not want to cooperate with us despite our code (and the SharpDevelop version we're using) sticking to the Mono compatible subset of the Base Class Library + Windows Forms. So the zip distribution (which you would install for Mono) does not include the IronPython REPL functionality and uses the existing FDO Expression Editor implementation on Mono.

Local resource previews

As part of eating our own dogfood, previews for certain resources will now use our own built-in map viewer component. The main benefits of using a locally-based previewing mechanism that:
  • It is generally snappier as we don't have to spin up an external web browser to show the preview
  • It is smarter about the initial view if we're previewing layers where the scale range is not 0 - Infinity
  • Our preview UI has more niceties over the browser-based previewing mechanism
If this feature is not working for you or you prefer the old way of previewing things, you can turn this option off in the Application preferences.

Updated SDK

This is the first release in the Maestro 5.0 series to include a matching SDK. This includes some new samples
All these samples have been migrated to .net Framework 4.0 / VS 2010 (You can use the VS 2012 express editions as well if you so choose). You cannot use the Maestro 5.0 API in a version of the .net Framework that is older than 4.0.

As I hinted to when I started on the road to Maestro 5.0, the new SDK also contains a build tool to "roll your own" LocalNative provider, which lets you use the MaestroAPI over the official MapGuide API. This is a much more convenient way than me having to build a LocalNative provider for n specific versions of MapGuide

Simply point to where your official MapGuide .net API assemblies are and hit Build to build your LocalNative provider. Then you can use this provider by registering the dll in the ConnectionProviders.xml of your MaestroAPI application.

Please Note: Because this is still a beta, the Maestro APIs are not set in stone. APIs may still change from now until the final release.

Other cool stuff

We've tweaked the WMS Configuration UI to better handle some of the more obscure WMS servers I've encountered in my journey thus far.
  • You can now edit the FDO logical class name/description
  • You can also swap the class name/description individually or in a batch. This is a convenience function for some WMS Servers where the layer names actually end as the FDO logical class description instead of class name when processed by the WMS FDO provider.

We've also made the FDO class name encoding/decoding in Maestro to behave more like its FDO counterpart so that problems like this should no longer happen anymore. We've backed this up with the same test suite and data used to exercise the name encoding/decoding in FDO proper. Basically, you shouldn't get strange looking FDO class names anymore. They should get properly encoded and decoded.

When deleting a resource with dependent resources, we're now nice enough to tell you what they are

For folders, we just give you a catch-all question which may or may not be true, simply because we don't want to waste CPU cycles computing affected references if we happen to be trashing a big folder.

There is also a new context menu option for selected resources that allows you to compile a full dependency list for a given resource

This produces a list of resources that the selected resource either directly or indirectly depends on, which can be saved to a text file

This feature combines nicely with an improvement to the Package builder to allow packaging an explicit list of resource ids instead of a particular folder. Because sometimes, you just want to package a specific resource and everything it depends on, instead of a particular folder which may then package a whole bunch of other junk which you don't particularly want.

On the theming front, ColorBrewer color ramps can now be reversed if you so desire.

We've improved your ability to rapidly translate Web Layout and Flexible Layout documents, but previewing such documents (or any browser-based preview actually) in the AJAX/Fusion viewer will still default to using the en locale. You can now choose what locale that browser-based previews will default to in the Application preferences.

This has the effect of basically tacking a locale=your_selected_locale to whatever preview URLs you launch from Maestro

So with that all said and done ... Go forth and download!

Monday, 19 November 2012

FDO Toolbox 1.1

Here's a new release of FDO Toolbox.

The new items are:

Please ensure you have .net Framework 4.0 and the Visual C++ 2010 redistributable (32bit or 64bit) installed before installing FDO Toolbox 1.1

Note: FDO Toolbox is still a semi-retired project. Don't expect any major new features. I've put this release out because I believe every new version of FDO should at least have a matching version of FDO Toolbox that uses that particular version.

Thursday, 15 November 2012

A work in progress

I wonder if the favicon might give you a hint? :)

Monday, 5 November 2012

Eating your own dogfood

For those who haven't heard the term, eating your own dog food as defined in Wikipedia:
Eating your own dog food, also called dogfooding, is a slang term used to reference a scenario in which a company (usually, a software company) uses its own product to demonstrate the quality and capabilities of the product
For the next beta of Maestro, we're doing some dogfooding of our own. The new Map Viewer component, will now be the default method for previewing the following types of resources across all connection types
  • Layer Definitions
  • Map Definitions
  • Watermark Definitions
Where previously, a web browser would be launched (if using a http connection) or a mg-desktop based viewer would be loaded (if using a local connection), for such resources we have our own preview UI built on top of the Maestro Map Viewer component (which also works in Mono)

The main benefit of this method of previewing becomes apparent when you try to preview layers with restricted scale ranges. The default method (launching the AJAX viewer) for such layers will usually result in seeing nothing as we don't have the ability to pass on the initial opening scale to the AJAX viewer nor do we have the required information to adjust the preview bounds. That's why as a workaround, we put in a "Zoom to Scale" command under the "Tools" menu of the preview Web Layout to let you easily zoom into the layer's visible scale range.

With the new preview method, the preview map viewer will automatically adjust its initial scale to fit into the layer's scale range, eliminating the need to manually zoom into the layer's visible scale range.

This feature is optional, and if you still prefer the old way of previewing resources, you can simply uncheck the option.

Next beta is almost here. There's some Mono-related bugs/annoyances to fix up

Tuesday, 30 October 2012

In praise of Visual Studio 2012 Express

Visual Studio 2012 gets a lot of flak for its questionable color scheme, the shouty all caps menus and Microsoft's initial decision to snub desktop-centric developers like me with their initial set of Express editions.

But this one single button makes it all worth it.

I don't think I can ever go back to 2008/2010. That one single button makes sifting through a 48-project solution (which Maestro currently is) an absolute breeze! Navigating through 48 projects worth of source code in previous versions of Visual Studio was a complete chore. In VS 2012 it's dead simple.

As an aside, the Windows Desktop Express edition of VS 2012 is quite an impressive piece of free (as in beer) software. I've been hacking on Maestro trunk with VS 2012 Express ever since it was released and I've had some of the most productive coding sessions with it.

I was expecting a bare-bones front-end to csc/vbc/cl.exe like previous Express editions with all sorts of SKU-related hurdles to overcome, but this is something out of the ordinary:
  • Support for heterogeneous solutions with assorted project types
  • .net Framework targeting
  • Support for solution folders
  • Integrated NuGet package manager
  • Attach to process debugging
  • Basic static code analysis
  • Project/Solution roundtripping with VS2010*
  • 64-bit C++ compilers included
This was stuff I was expecting from a Professional level SKU in previous versions of Visual Studio! I don't really care about MSTest (I use NUnit/xUnit externally), nor do I care about TFS integration (I'm use svn/git/mercurial externally). Actually I don't really care much about whatever integration Microsoft is marketing for their higher level SKUs as I am happy to get at such functionality through external tools. Refactoring support is very basic, but still workable (ReSharper is nice, but I don't need it like a crutch in order to crank out good code). 

So what Microsoft has actually offered in their 2012 Express editions actually meets most of my needs, which is especially surprising in light of their initial snub to desktop application developers (you know ... the people who make their platform). In their efforts to appease the many pissed off developers (like me) they punched well above their weight and did much more than I would've personally expected them to do.

So kudos to Microsoft for delivering a fine set of free developer tools. 

But if you think this is an olive branch manoeuvre that will get me to code for Windows 8? Dream on! :P

* Found out a little hitch in the round-tripping  The round-tripping breaks if you add a new project from VS 2012, as this upgrades the solution file to VS 2012 format. But the actual changes are minute and can be safely reverted (just need to revert the header portion of the solution file) to allow that solution file to be opened back in VS 2010.

SharpDevelop'ing all the things!

If you look at Maestro from a high level view, it shares many qualities and similarities to an IDE (minus the bits that edit and compile source code). As a matter of fact, Maestro re-uses a lot of code from the premier open source .net IDE: SharpDevelop

In fact, I would imagine, if you stripped out all the MapGuide authoring stuff out of Maestro and replaced it with source code editing and build tools, you would end up back with SharpDevelop itself. The design/architecture is (intentionally) that similar!

Of particular focus in this post is Maestro's use of the text editor component from SharpDevelop.

Currently this text editor is only being used for our generic XML editor, giving us nice syntax highlighting and other things that you would expect from an industrial strength text editor. The problem however is that there are many other areas in Maestro that could benefit from using this text editor. In addition, there are many IDE-isms present in SharpDevelop that would be of great use in the context of Maestro.

So for the next beta we are going all the way and using this component wherever we possibly can, due to its powerful syntax highlighting, auto-completion and other assorted features you expect from an industrial-strength text editing component.

IronPython Console

We've thrown out the existing IronPython REPL console and replaced it with one grafted from SharpDevelop's python addin.

We lose some of the built-in commands from the old one, but because we're using the SharpDevelop text editor, we now have things like Python syntax highlighting and the more important one: auto-complete

And we get nice colorful feedback about any errors that IronPython gives us

Also the design of this REPL means that python snippets are executed in a background thread ensuring the main UI is not clogged up. However if you are executing python snippets that create or interact with Windows Forms controls, that code has to run on the UI thread, so we've provided a UIInvoke function in the host application to let you do this.

Personally, I'm loving this REPL that's been implemented. It's a wonderful way to not only explore the Maestro API from within Maestro itself, but it's also a nice convenient way for me to test new functionality and/or newly implemented APIs.

Generic XML Editor

The generic XML editor was retro-fitted with the SharpDevelop text editor component since beta 2, but for the next beta, it now supports code folding (grafted from SharpDevelop's XML addin). This allows you to collapse vast swaths of XML content to make editing and navigation that much simpler.

Expression Editor

The Expression Editor basically did what the SharpDevelop text component did, but much more crudely. So we've replaced the text component in the Expression Editor as well. We get proper auto-complete behavior not just for properties and functions, but also operators (LIKE, IN, etc)

And proper syntax highlighting (due to a custom FDO expression syntax definition we've added)

One small caveat with this implementation currently is that the auto-complete is case sensitive and will only trigger if we have a partial case-sensitive match on potential auto-complete result. We'll look to make this eventually case-insensitive.

The next beta that packs these new features (among other things) will hopefully be done by the end of this week (no guarantees)