My 2010 MapGuide/FDO Wishlist: Revisited

Around this same time last year, I made a wishlist for MapGuide and FDO. As this year has passed along, that wishlist somehow slowly transformed into my personal to-do list :P So how did things turn out 365 days later?

MapGuide


1. A desktop-based variant of the Geospatial Platform API


Yep, and mg-desktop is it's name. Last year, I had a faint idea of how this would flesh out. Now it's fully realized.


2. A HTML5/Flash/Silverlight vector viewer with offline support


My desire for this type of viewer at the time was predicated on the lack of a viewer like in MapGuide 6.5 that can operate with limited to no MapGuide Server connectivity. mg-desktop has solved this particular need for me, so my desire for such a viewer is no longer there.

Still if we want to have a HTML5 viewer, this may be a good starting point. The examples work on local SHP files (Sheboygan Parcels actually renders quite snappy), but this can probably be extended to a common feature model (allowing support for other spatial data formats) and then start trickling in support for MapGuide's rendering/stylization rules.

As for Flash or Sliverlight, I'm not even gonna go there. HTML5 is capable enough IMO. A plugin-free world is a beautiful one.


3. A scalable Fusion Legend widget


Yep, Fusion no longer sucks


4. A way to preserve tile sets


Still not there, but anyone affected by this problem knows about it and has probably come to accept it as a minor inconvenience.


5. Feature Write capabilities for the mapagent


GeoREST is still the way to go here. Given this is part of AIMS 2012 as a technology preview, perhaps this  will be integrated into MapGuide proper in the future?


6. A DWG FDO Provider


Still nothing here sadly.

FDO


1. A way to use the FDO API in languages other than C++/.net in a multi-platform manner


In my current line of thinking, mg-desktop is the way to go. It already uses/wraps FDO and is already developed in a way that facilitates easy generation of SWIG bindings. I just have to overcome my fear of gcc and automake and their cryptic error messages :P


2. Champion SQLite as the goto flat-file format for FDO


An emphatic yes on this one. The SQLite FDO provider is getting all the new FDO goodies nowadays. It was the first provider to get the new FDO Join APIs, thus allowing the Feature Join optimization shortcut to become possible.


3. A FDO provider for MapGuide


Nothing's changed here.


4. Consistent Schema Naming Conventions


This leaky abstraction is a small annoyance for anyone who wants to write provider independent code. If you have to write provider-specific code paths (FDO Toolbox had to employ quite a few), it just highlights some of the failings of FDO as an abstraction layer for spatial data access. Sadly, nothing's changed here.


5. Parametric Capabilities


Still nothing's changed here. Of course if ODBC and OGR don't expose capabilities of their own that are translatable to FDO's capabilities, then this leaky abstraction is one we'll just have to live with.


6. Android/iPhone support


Given Autodesk's recent focus on having mobile versions of everything known to man, this may look like a possibility. But you know Autodesk, I'm just staring at an opaque crystal ball here :-)

FDO Toolbox 1.0 (yes that's right!)

3 years and 1251 revisions in the making. The 1.0 final version of FDO Toolbox is finally here.

This release includes some small bugfixes from 1.0 RC1

Download

Is this be the end of FDO Toolbox as I proclaimed some time ago? Probably.

I'll still probably put out occasional releases to coincide with newer versions of FDO but other than that, FDO Toolbox has fulfilled all my needs (and I'm sure many of your needs as well). So I'll guess it's time to put this project into semi-retirement.

Fusion is actually usable now

I am going to call it right now. If you've been holding back from moving to Fusion due to the assortment of bugs and performance problems, then the version of Fusion that will appear in the next version of MapGuide Open Source and Autodesk Infrastructure Map Server will be the one which will finally sway you across.

Thanks to work undertaken on behalf of Hume City Council, we have finally tackled most of the deal-breaking issues which have made Fusion unsuitable for production use for many years ... Until now.

What issues in particular? Read on below.

Legend Performance

This is the number one showstopper to wider adoption. The root problem with the legend widget performance is that each icon in the legend requires an individual http GETLEGENDIMAGE icon request to the mapagent. Now suppose you have a layer with a 900-rule theme, attempting to preview this layer will grind your browser and the MapGuide Server to 100% CPU usage, because it will fire off 900 individual icon requests (yes you heard that right!). And that's just for one single user!

We solved this problem through a multi-pronged approach:

1. To speed up the server-side initialization, we use the new GetResourceContents API introduced in MapGuide Open Source 2.2 to fetch all our required layer definitions in a single batch. Performance gains will be readily apparent for really chunky Map Definitions with lots (in the 100s) of layers

2. We exploit the data URI feature that supported by most modern browsers. We pre-cache icons by generating the legend icons server-side and embedding them as base64 encoded data URIs as part of the json content that is returned to the legend widget for initialization. Each legend icon represented as a data URI will eliminate a mapagent icon request that would have to be executed. We use a metric of pre-caching 25 layer icons per scale range. For an average Map Definition (like the Sheboygan sample data set) no icon requests are ever made through this optimization. Now for those primitive browsers from Redmond that don't support data URIs, have no fear. Because there is also the next optimization.

3. The theme compression logic that is used by the AJAX viewer now also exists in Fusion. We use that same number (25) to determine the number of style rules that must be exceeded for a theme to be compressed. Compressed themes in Fusion look like so:

And like the AJAX viewer, you have the option of expanding compressed themes. To avoid unnecessarily expanding say: a 900-rule themed layer, we prompt for confirmation. So the choice to flood a mapguide server with icon requests will be yours and yours alone to make ;-). Even if your browser doesn't support data URIs (It means you are using IE6/7. Don't deny it!), the theme compression means that icon requests will be capped at a certain level by default, just like the AJAX viewer.

So with these changes, our map with the 900-rule themed layer now loads just slightly slower than the AJAX viewer, which is a major improvment from never.

Support for Map Definitions with Tiled and Dynamic Layers

This one wasn't our doing, but Autodesk themselves have finally addressed the other long-standing complaint of not being able to view a map that contains a combination of tiled and dynamic layers.

And as an added bonus, if you are using commerical underlays your tiles can also line up with the Google/Bing/Yahoo/OSM tiles if your Map Definition uses a Google-compatible finite scale list (and is using the WGS84.PseudoMercator coordinate system of course). In anticipation of supporting this new feature, I've added support for generating this Google-compatible scale list in the final release of MapGuide Maestro 4.0 which will be coming out soon.

Redlining

Ever used the Fusion redline widget? Me neither.
So what do you do with the downloaded redline GML file? Hell if I know!
What about if I want to plot the map with the redlines? Dream on. You can't do that!

When the redline widget was first introduced with the query and theme widgets, I was personally surpised that they didn't base the redline widget from the same sample the query and theme widgets were derived from: The Generic Tasks AJAX viewer sample.

So we've thrown out this useless widget and replaced it with one based from the Generic Tasks sample, giving us a redline widget that is much more powerful and useful:



If you've used the Generic Tasks markup sample, this redline widget functions almost the same. For those unfamiliar with the Generic Tasks markup sample, here's what the new Redline widget can do:

• Configure redline layers with support for labels using the same MapGuide stylization rules
• Add, Create, Remove and Manage redline layers with respect to the current map
• Draw geometries using the same digitizing tools provided by OpenLayers
• Your redlines will be part of the whatever plotting or rendering operation you do on the current map.
• Redline feature sources can be downloaded as SDF files, allowing for overlaying in AutoCAD Map3D and other Autodesk Geospatial products.

Quick Plot

The Quick Plot widget has improved reliability. This is due to the widget now using the actual MapGuide Rendering APIs to grab the initial un-rotated image and dumping that to a temp file, where the PHP gd library is then used to rotate the image and outputted back to the user. This approach is a more memory efficient use of the PHP gd library, and as a result should result in significantly less occurrences of this useless message:

ProTip: If you get this message, it's due to a failure of some sort in GeneratePicture.php. Inspecting the GeneratePicture.php request in Firebug should give you more information. As there was no mechanism in place to propagate back error messages, GeneratePicture.php fails silently at least from a user perspective. The QuickPlot changes improve memory efficiency and should result in signifcantly reduced (can we say no?) occurrences of this message.

The widget itself can now also have customizable paper and scale lists, which will require hacking the widget's extension XML element.

One small note: If you planning to wedge fusion trunk into an existing MGOS 2.2 installation, this widget will break because it uses APIs introduced in the next version of MGOS/AIMS. Other widgets should work just fine.

General Usability

Having tackled the major showstoppers, I became comfortable enough with the fusion framework to address a whole bunch of bugs and usability issues with other widgets. The Search, Measure and Query widgets have gotten a bunch of bugfixes.

Wrapping things up

All these changes have already landed in Fusion trunk (some have for quite some time now). Given this is usually branched off at some point, where it is then used by the Open and Commerical versions of MapGuide, you can expect these drastic Fusion improvements to a MapGuide release near you.

As a small aside. Many thanks to Hume City Council for providing the funding for us to implement these long overdue improvements. If you went to AU 2011, you may have seen some of our handiwork on display (including our Fusion work) in this presentation.

As a second small aside, if you have to use Internet Explorer, use Internet Explorer 9. Fusion absolutely FLIES in IE9 and even outperforms Firefox/Chrome in some cases! (yes, I am shocked)

轻松翻译使用 MapGuide Maestro *

* I trust Google Translate translated the above blog title correctly :D

Not everyone speaks English, so neither should your MapGuide Applications.

Currently, the process of translating your MapGuide Application is to hunt down all the english strings in your Basic or Flexible Web Layout XML document and manually replace each string with your translations. This is a slow and painstaking process.

The final release of MapGuide Maestro 4.0 will include a new convenient command to assist with the translation of Basic and Flexible Web Layouts.

Clicking the Translate button will bring up a dialog containing all the localizable strings (Maestro searches for all known XML elements that can contain localizable text). From here, tick the strings that require translation and give your provided translation. I did a quick and dirty french translation via Google Translate for this example.

Once you have completed your translation, click Apply to apply these translations to the affected strings. The editor will reload itself with the translated strings. Previewing the layout will show your translated layout.

The final release of MapGuide Maestro 4.0 (which will include this feature) will be arriving shortly. Doing a lot of API cleanup and documentation.

Improving the Feature Join experience

So what was this screenshot all about?

Time for an explanation. This screenshot was the result of an experimental mg-desktop addition to make feature joins perform much faster (around 100 times faster as evidenced by the screenshot), by taking advantage of new FDO Join APIs that were recently introduced with the 3.6 release. Why does the FDO Join APIs perform so much faster? Time for a little backstory...

Why Feature Joins suck (performance wise)

Feature Joins are performed by the GWS Query Engine component of MapGuide. From a high-level overview, this component takes two input iterators (with interfaces similar to a FDO/MG Feature Reader), and depending on the capabilities of the left and right sides of the join a "merged" feature reader is returned that will loop the two iterators in the following fashion:

• Nested Loops (both sides do not support ordering)
• Nested block join and sorted block (left side supports ordering, right side doesn't)
• Sort-Merge (both sides support ordering)
• Batch Sorted Block (left side does not support ordering, right side does)

Now these join algorithms themselves are also sensitive to the data you're working with. For example, any sorted block algorithm is going to perform bad on datasets with high-spread of join key values (low-spread of join keys means lots of sorted duplicate joins values which can be skipped). Sort merge has issues regarding case-sensitivity of key values (though this may be a bug in the implementation). Nested Loops is obviously terrible if both sides are large (as is the case of a SDF to SDF join as per the above screenshot)

MapGuide in its current form is not aware of join factors which impact performance:

• The size of both sides of the join (Nested loops is obviously bad, but the inability to determine size means this can't be avoided if this is the designated algorithm)
• The spread of join key values (Sorted blocks work best on lookup-value type spreads not foreign-key type spreads)

It is all of the above factors which make having good performing feature joins a difficult proposition. ^

The FDO Join API (or: Let the data store figure it out)

So it's clear that MapGuide does a terrible and/or erratic job as a query optimizer, so assuming we want respectable performance of joined data, you would've probably avoided Feature Joins altogether and have the bits you want to join in question residing in the same centralised RDBMS, do the joins at the database level and wrap the result in a database view.

This approach is a good solution in terms of performance because figuring out the best way to join data is now a data store responsibility, and you can apply a whole assortment of indexes and other DBMS performance improvements that MapGuide is not aware of. From MapGuide's perspective all it sees is a table/view and it just queries that. But this approach presents its own problems:

• The need for metadata hacks for views to be properly recognised as a feature class (eg. SDO_GEOM_METADATA in Oracle)
• The type of view affecting selectability (eg. SQL Server, which requires schema overrides to work around)

With the introduction of FDO Join APIs, it allows us to do these type of joins in an ad-hoc fashion without needing to wrap our joined data into database views. Joining data with the FDO Join APIs is basically the equivalent of doing a SQL SELECT query with JOIN clauses. In other words, the FDO Join APIs lets you delegate join optimization to the data store and let it figure out the best performing way to do it.

Taking advantage of FDO Joins in MapGuide (with RFC123)

So as already explained, work on bringing support for this into MapGuide started with mg-desktop. Being essentially a self-contained version of MapGuide meant that mg-desktop was a nice sandbox for playing and experimenting with new features. So once support was built into mg-desktop, bringing it over to MapGuide proper was quite an easy affair.

So how does mg-desktop and MapGuide take advantage of FDO Joins?

We use the existing Extended Feature Class mechanism to construct an equivalent FDO Join select query, applying the proper property/class aliasing to ensure that the reader that is returned presents the same property list as a default feature join query.

When querying an Extended Feature Class, MapGuide does several checks to see if the FDO Join optimization path can be taken. These checks include:

• Does the underlying FDO provider for the Feature Source support FDO Joins? (obvious one!)
• Does the primary and joined feature classes of this Extended Feature Class originate from the same feature source? (this is important, because FDO Joins work within the context of the same connection, thus the classes we are joining must come from the same source)
• A whole bunch of other minor conditions which your Extended Feature Class will most likely satisfy (detailed in the RFC)

If the Extended Feature Class being queried passes the above checks, MapGuide takes the FDO Join path, which as seen from the screenshot above, can be up to 100 times faster than the regular Feature Join approach.

Additionally, Extended Feature Classes also double up as the metadata. You don't need to do SDO_GEOM_METADATA hacks or FDO Schema Overrides for these classes to be recognised. Because MapGuide already understands these elements out of the box!

Also, in case you're wondering, the performance numbers in MapGuide are even better than mg-desktop (up to 1s faster than the numbers in the above screenshot) due to connection pooling and object caching mechanisms present in the MapGuide Server code.

Conditions Apply

Now as you can see, only certain Extended Feature Classes can take advantage of this optimization. The other limitation is FDO provider support. As of writing this post, the following providers implement the FDO Join APIs:

• SQL Server
• SQLite

Given the high usage of these providers (you are using SQLite for all your spatial data aren't you?), there is a good chance of this optimization path being used.

The beauty of FDO is that as soon as an FDO provider implements the FDO Join APIs (like King.Oracle wink, wink, nudge, nudge), such configured Extended Feature Classes in MapGuide can automatically take advantage of this optimization path.

So there you have it. 100x faster feature joins if your data ticks all the boxes.

^ There is a 5th type of join algorithm that is (suprisingly) not implemented by the GWS Query Engine: Hash Joins. Though probably the most memory intensive join algorithm (due to needing to store one side of the join in a hash-table), this should always perform better than the default nested loops. Anyway, it's 2011. Memory is cheap! Having this join type implemented should improve the Feature Join user story even further. Guess that's another item on the TODO list.

Announcing: A fixed installer for MapGuide Maestro 4.0 RC1

Oops!

It turns out that as far back as the first beta release of Maestro 3.0, that the window installers I've been making did not include the XML schema files required for validating XML content.

So I've posted an updated 4.0 RC1 installer with the missing XML schema files. Note that this is only applicable if you've been downloading the windows installer package. If you have been using the zip packages, the XML schema files have always been in there, so everything's ok on that front.

To make sure that the XML schema files are properly set up, go to Tools - Options and check that the XML schema path is configured to point to the Schemas directory under your MapGuide Maestro installation directory

Or, you can click Reset and Maestro will reset all preferences to default values and auto-discover all the required paths.

Update: OSGeo is back in order. So the fixed 4.0 RC1 installer is available for download from here

Announcing: MapGuide Maestro 4.0 RC1

Here's another RC coming your way.

This release re-introduces the Package Editor feature, which has been missing since Maestro 2.1 and includes a big bucket of fixes and small improvements since beta 2.

One particular improvement which I'm proud of finally implementing can be summarised with this screenshot:

Yep, I finally figured out how do do it!

Download (back at OSGeo)