Announcing: MapGuide Maestro 4.0 beta 2

I am happy to announce the 2nd beta of MapGuide Maestro 4.0

The major new feature of this release is non-transactional package loading, which greatly improves the reliability of loading very large package files, at the small cost of losing the repository ACID-ity of the official approach. Should you encounter any failed operations during the package loading, Maestro will give you the ability to retry these failed operations until all failed operations are accounted for, or you choose to abort yourself.

Also included in this release is:

• Fixes for defects brought on by the new multi-site support
• Editor support for FDO expressions in color editor properties
• Updated to the latest mg-desktop binaries for Local Connection mode

Once again, this beta is available in the fat NSIS installer or the lean zip package (which excludes local connection mode). Users of non-windows operating systems will have to download the zip package.

Download (temporarily hosted on the mg-desktop site)

Introducing non-transactional package loading

The next beta of MapGuide Maestro 4.0 will feature a new way of loading packages which I call "non-transactional" package loading.

Normally the package loading process in Maestro is as follows.

1. Select the package file
2. Maestro invokes APPLYRESOURCEPACKAGE on the MapGuide Server and uploads the package file
3. Twiddle your thumbs, because there is no way to know how long this operation will take
4. The operation being transactional means that the operation will wholly suceeed or wholly fail.

Loading a package several hundred MBs in size only to be told that the entire operation failed at the end is needless to say, a very frustrating experience.

Non-transactional package takes a different approach. It eschews using the APPLYRESOURCEPACKAGE API (the cause of the thumb twiddling and waiting because there is no way to track progress) for an approach that allows progress to be measured. At the heart of a MapGuide Package is the resource package manifest, which is simply a list of resource service operations to execute on the MapGuide Server, which are generally a mixture of:

• SETRESOURCE
• SETRESOURCEDATA
• DELETERESOURCE

The existing approach executes all these operations on the server-side under one big transaction, the new approach executes all these operations one at a time from the client side and because we know how many operations need to be executed, we can measure the entire progress of the package loading operation.

Now when you load a package, you will be asked to choose the method of uploading:

Choosing the Transactional method will load the package in the existing fashion. Choosing the other option will use the non-transactional method and because this approach is measurable, we get a nice meaningful progress dialog to relay the upload progress.

So in what cases would you use non-transactional package loading over the default transactional one? Here's a few I've thought of:

• The package file is several hundred MBs in size
• The package file contains hundreds of resources to be loaded
• The package file is not loadable via the transactional method, which is generally the result of any of the above items.

Finally, because it's non-transactional if one operation fails it will keep continuing with the upload of the others, failed operations are logged. As a result, the upload can partially succeed and fail. Should the operation partially fail, it is planned to re-use the list of failed operations to allow you to re-run the package upload, but to skip all the successful operations, thus turning a partially failed upload to a fully successful one.


I've received some test packages which completely failed to load under the official package loading methods. These packages loaded without issues in the non-transactional mode. So no matter low large or small your data packages are, Maestro can upload it for you. One way or another.

Announcing: MapGuide Maestro 4.0 beta 1

I'm happy to announce the first beta release of MapGuide Maestro 4.0

This first beta contains 2 new groundbreaking features, which have already been covered previously:

• Support for connecting to multiple sites
• Support for a new connection type backed by the new mg-desktop library. Unfortunately, this feature is windows only and thus is only included in the windows installer. This feature is omitted from the zip distribution.

The multi-site feature majorly affects most workflows in Maestro and as a result of the yet-to-be-stabilized nature of this new feature, I will only recommend you download this release for testing and feedback purposes only

You may also notice a significant increase in the windows installer size (84MB). This is due to the inclusion of the full set of CS-Map coordinate system dictionary files that is required by the new local connection mode feature. I wish CS-Map stored its thousands of coordinate systems in a more space-efficient manner! Anyway, I am pondering whether to offer the new local connection feature as a separate download or keep it bundled it with the NSIS installer as I have done for this beta release. Let me know what you think.

This beta can reside side-by-side with older versions. The new local connection mode also shouldn't conflict or clash with any existing MapGuide Open Source or AIMS installations

Download

Taking MapGuide beyond the Server and Web right into your desktop

Previously, I briefly mentioned how the new Local Connection mode in the next release of Maestro was backed by a new desktop implementation of the MapGuide API. This (lengthy) post covers this

Preface

Let's start with a little personal backstory of mine.

Remember this software?

The MapGuide ActiveX viewer (source: Directions Magazine)

Now truth be told. I was not a major fan of Autodesk MapGuide 6.5. I guess that could be attributed to the following:

• I was just getting started not only with Autodesk products and technologies, but also on the basic concepts, practices and workflows of GIS systems. So for all intents and purposes, I was a total noob at the time.
• Windows only. Ewwwwww!
• The viewer is an ActiveX control that can only be embedded in Internet Explorer. Ewwwwwww!

Now the first time I got acquainted with MapGuide was in 2005 with the 6.5 version. It was also the time at my first ADN event when I heard about Autodesk's next version of MapGuide. Seeing the demos of this next version in action was quite a mind-blowing experience, given my not-so-positive experience with MapGuide 6.x. 

The AJAX-based viewer, multi-platform support, purty looking maps, flexible data access through this mysterious (at the time, to me) FDO technology and the list goes on. It just ticked all the boxes of what a modern, open-ended web mapping system should be. But then came the biggest bomb-shell of them all: It will all be open source!* So then several months later, the very first version of MapGuide Open Source was released and the rest (as they say) was history.^


Now despite my less-than-flattering opinion about MapGuide 6.x. It did have some very unique qualities to it that has yet to be replicated in MGOS/AIMS. Its map viewer, due to being based on ActiveX meant it could be embedded into .net applications. The second quality was that MWF files can contain static layers. This combination meant that it was possible to build mobile and disconnected MapGuide applications in .net or any other windows technology that supports COM (Component Object Model)


And it is in this respect that MGOS/AIMS has nothing comparable in its technological offerings.

• Disconnected? Not a chance! Client applications must be connected at all times. No server connectivity = no maps for you! Even the latest mobile extension for AIMS still requires constant connectivity.
• Mobility? You could have a full blown MGOS/AIMS installation on localhost, which your windows application can communicate via an embedded Internet Explorer. A functional, but very inelegant configuration. Nothing approaches the simplicity of a windows application + static MWF + 3mb ActiveX control configuration that was possible with MapGuide 6.x

So while MGOS/AIMS does most things leaps and bounds better than MapGuide 6.x, there are some other things where it pales in comparison.


Enter the MapGuide Desktop API (hereby known as mg-desktop for the rest of this post). mg-desktop is a desktop implementation of the MapGuide Platform API allowing for the same feature data access, map rendering/stylization functionality as the MapGuide Server, but without needing a MapGuide Server installed on your machine!


Genesis


The seeds of the MapGuide Desktop API were planted with this groundbreaking AU 2009 presentation. This presentation, for those unable to access that link showed how MapGuide Open Source actually has lots of shared components that can actually be re-used outside of the MapGuide Server and Web tiers. 

The main example from that presentation was a simple snapshot application that can produce a rendered map image or DWF file from a list of named Layer Definition documents without ever needing to contact a MapGuide Server ever! The first thought I had after seeing that presentation, was simply: Can't we just use a Map Definition to drive all this?


The second thought was, that AutoCAD Map already proved that the MapGuide API can be taken in a different direction, away from the MapGuide Server/Web Tier to become a API inside AutoCAD Map

So couldn't we have another implementation that makes it a fully self-contained framework that is usable by any desktop application, taking the concepts proven feasible by that AU presentation to its logical conclusion? mg-desktop is my answer to that question.

mg-desktop is a desktop implementation of the Geospatial Platform API, with its own implementation of the core platform classes, as shown below:

Where the MapGuide API operates in the context of the MapGuide Server/Web Tier and the AutoCAD Map Platform API operates in the context of inside AutoCAD Map, mg-desktop operates in a standalone context, with no dependencies on any particular application. It provides a suitable foundation for creating your own desktop-based MapGuide application.

In addition to implementing the core classes and services of the Geospatial Platform API, it also includes some of the additional service APIs provided by the MapGuide API, such as:

• MgRenderingService
• MgDrawingService
• MgTileService

mg-desktop also supports loading of MapGuide Packages (*.mgp). So you can transfer data and resources from a MapGuide Server to an mg-desktop installation.

Key Differences from the MapGuide API

Because mg-desktop is a desktop implementation of the Geospatial Platform API, some concepts which were applicable in MapGuide and/or AutoCAD Map no longer apply in mg-desktop:

• There is no session expiry :D Session resources still exist as a way to have temporary resources
• There is no resource security model
• On the same subject, there is no authentication required for access to services.
• There is no need to open/save runtime map state. Everything operates off of the current state of the runtime map in memory.

The mg-desktop implementation of MgResourceService wraps a pre-defined directory on the file system. Folders and resources in the mg-desktop repository conceptually map to files and folders in the file system.


The mg-desktop implementation of MgFeatureService is a thin wrapper around FDO. Each service API maps to a corresponding FDO command. All FDO types that would be returned are wrapped up and/or converted to their corresponding Mg* counterparts.


Aside from these differences, it's the same MapGuide API that you've come to know all along.

Unique APIs


mg-desktop not only just implements the abstract classes in the Geospatial Platform API, but extends them with our own additional APIs to correct my perceived shortcomings of the current MapGuide APIs. Here's an overview of the unique APIs of mg-desktop.


MgdFeatureService


MgdFeatureService is the desktop implementation of MgFeatureService and includes the following extra methods:

• RegisterProvider - Registers a new FDO provider into the provider registry by its library path
• UnregisterProvider - Removes a registered FDO provider from the provider registry by its provider name
• InsertFeatures - Allows for individual insertion of features into a feature class, instead of having to use the monolithic UpdateFeatures
• DeleteFeatures - Allows for individual deletion of features in a feature class, instead of having to use the monolithic UpdateFeatures
• UpdateFeatures - Although it unfortunately shares the same name as the monolithic method it's trying to replace, this allows for individual updates of features in a feature class.
• SelectFeaturesExtended - Almost identical to SelectFeatures, except this returns a scrollable feature reader that behaves similarly to the FDO scrollable feature reader

MgdLayer


MgdLayer is the desktop implementation of MgLayerBase which adds extra convenience methods, allowing you to operate on the layer itself instead of needing to obtain an MgFeatureService reference and the layer's feature source id and class name:

• BeginTransaction - Convenience form of MgFeatureService.BeginTransaction
• SelectFeaturesExtended - Convenience form of MgdFeatureService.SelectFeaturesExtended
• InsertFeatures - Convenience form of MgdFeatureService.InsertFeatures
• DeleteFeatures - Convenience form of MgdFeatureService.DeleteFeatures
• UpdateFeatures - Convenience form of MgdFeatureService.UpdateFeatures

The Map Viewer Component


So having a reusable library to build your own geospatial applications is good and all, but is there something out-of-the-box like a map viewer that we can embed in a .net application? The answer is an emphatic yes. mg-desktop includes an embeddable map viewer component!

This is not some trick of embedding a web browser to use the AJAX viewer to talk to a local or remote MapGuide Server, this is all native .net WinForms controls and using the same self-contained MapGuide rendering and stylization services provided by the mg-desktop library and it does pretty much everything you can do in the AJAX/Fusion viewer.

What about more complex stuff like digitizing and redlining? No problem. It can do that too!

And all of this is functionality is contained in one single sub-100kb dll! (OSGeo.MapGuide.Viewer.dll). The next release of Maestro will be using this viewer component for previewing Feature Sources, Layer Defintions, Watermarks and Map Definitions in the new Local Connection mode

In Closing...


I have submitted a pending RFC to have this included in the official MapGuide source tree, which would greatly improve integration, code re-use and ability to receive fixes and enhancements to upstream components.


The source and binaries# for the mg-desktop library (and map viewer) are available from the Google Code project home page. Note that FDO that is included with the binaries is a partial distribution (because I lack the ability to build certain FDO providers). You will need a Mercurial client to checkout the source code.

If you want to build the source, you will need to do a full svn checkout of MapGuide Open Source and FDO trunk. Then you will need to build FDO and the MapGuide Oem components first before you can finally build mg-desktop. The hg clone needs to reside in a Desktop subdirectory under MgDev in your MapGuide svn working copy. There is currently no 64-bit or Linux targets yet, so feel free to submit any patches.


* Let's give credit where it's due here. mg-desktop, Maestro and FDO Toolbox would not exist if it weren't for Autodesk open sourcing the MapGuide product and FDO technology. Similarly, my knowledge of MapGuide and FDO wouldn't have reached encyclopedic levels that they are today if these technologies were not open sourced.

^ As most of us found out, v1.0.0 of MapGuide Open Source turned out to be pretty rough round the edges and performance of Feature Joins (which 6.x handled brilliantly) was absolutely woeful and buggy, which sadly is still applicable to this very day. But this is a known problem with a known workaround, which is to have all the data in a centralised, spatially-enabled RDBMS, which you should be already doing in the first place.


# To reduce download size, mg-desktop contains a reduced distribution of CS-Map. If there are problems due to missing coordinate systems, download the country-specific grid files here and drop them into the Dictionaries directory.