The expressive power of MapGuide Tooltips redux

Over a year ago, I wrote about how you can have really powerful tooltips in MapGuide with some knowledge of HTML and the FDO expression language, with one major caveat:

You can't have interactive content in the AJAX viewer.

Well I've finally tackled this problem and implemented "sticky" tooltips for the AJAX viewer, so now you can have your Google Charts or Youtube videos in your AJAX viewer tooltips and be able to interact with them

I'm hoping to get this in for the 2.2 release of MapGuide Open Source. If you can't wait, and know how to apply diffs, you can grab the patch here

Using FDO XML configuration to reduce the attack surface of your data store (and improve performance)

Consider the following SQL Server database:

The screenshot isn't big enough to show it, but there are 250 feature classes in this schema. Sometimes for security purposes, you don't exactly want to make all 250 feature classes visible to any FDO client application (eg. MapGuide). This is commonly known as reducing the attack surface.

With FDO XML configuration and the Unified Data Store editor, you can override the default logical schema (250 classes) with something that is more stripped down (like say: 10 classes)

To do this, we once again use the unified data store editor

Now select the feature classes you don't want visible and remove them (for ease of use, you can remove a feature class by hitting the Delete key as long as the class node is selected)

Now for the important bit: Do not apply the changes! Doing so will actually delete these feature classes from the underlying data store which we do not want! We just want to hide them.

Instead, we export this current state of the logical schema to an XML configuration file

Now if we make another connection with this XML configuration file

We see that the connection takes on the logical schema as defined in our XML configuration file.

Another benefit of a reduced attack surface is performance. Here's the schema walk of the 250 class connection (LargeDb) and the configured one (LargeDb_compacted):

Although SQL Server already implements FDO RFC23, and thus schema walking is already efficient, and the times in question (as you can see from the screenshot) are already well under 10ms, you can objectively see that having a stripped down logical schema via XML configuration does indeed improve performance, which makes sense (less classes = faster loading). Now a really good test case would've been the ArcSDE provider and the Utah SGID dataset, but unfortunately the ArcSDE provider does not support XML configuration :( so that's a case of "what if?".

Another wonderful use of FDO XML configuration brought to you by the Toolbox.

Using FDO Schema Overrides Redux

When I announced beta 2 of FDO Toolbox 1.0, I briefly mentioned that the new Unified Data Store editor doubles as a visual FDO Schema Override editor.

In this post I'll illustrate how we can create FDO XML configuration documents without needing to modify a single line of XML in a text editor, using our Unified Data Store editor.

We will use the same sample SQL Server database from our previous post

Creating a connection to this via FDO Toolbox gives us the following logical schema view:

Notice once again, how the FeatId property is not identity. Now where the previous post used a text editor to tweak the logical schema, we will do the whole process visually using the unified data store editor.

We'll right click the connection and use the new Edit Data Store option

This brings up the Unified Data Store editor.

Now what we simply do, is modify the logical schema so that it matches what we want it to be. In this case, we want FeatId to be an Identity Property of VParcels. So we select the VParcels feature class, this will bring up the class definition editor. We check the FeatId property under the Identity Properties to make FeatId an Identity Property

Now comes the important bit: Do not apply the changes!

Instead we export this current state out to an XML configuration document via Export - XML Configuration Document

Now let's create another connection to the same database, but this time we'll use the XML configuration document we have just saved.

If we look at the VParcels feature class in this configured connection, you'll notice it takes on our overridden logical schema.

From here, the process is just like the previous post. In Maestro's feature source editor, you would edit the configuration document of that feature source, paste in the contents of this XML file and save it. Layers based off of the VParcels feature class will then be selectable.

As I mentioned in the beta 2 announcement this editor is not fully complete. For one thing, there is no user interface to edit the physical configuration of any Class Definitions and Property Definitions. Whatever mappings gets exported out to XML are the default mappings generated by the provider.

But for overriding the default logical schema (which I'd guess is the common scenario), the unified data store editor makes this process much simpler than the older method of editing XML files by hand.

Announcing: FDO Toolbox 1.0 beta 2

Here it is! The 2nd beta of FDO Toolbox 1.0

This contains many new changes from the 1st beta, some of the notable features include:

Unified data store editor

All the of schema and spatial context management functionality is now merged into a single user interface

As you can see from the above screenshot, instead of trying to win the losing battle against the WinForms PropertyGrid, all schema element editors are now specialized user interfaces.

Because we have now unified everything, we have all the necessary information needed to create full XML configuration documents. Therefore, exporting to XML will now export the full XML configuration document (Spatial Contexts, Logical Schemas and Physical Mapping).

Thus the unified data store editor doubles up as a visual schema override editor. I'll elaborate on this in a future post.

Just like the original schema editor, this functions also in a disconnected mode, free of any restrictions imposed by FDO provider capabilities.

The unified data store editor is not fully implemented yet. Some features are missing or not fully tested, such as:

• Object Property support
• Association Property support
• Class inheritance support
• Logical-Physical mapping configuration

Despite these missing features, the unified data store editor should fulfill most common data modeling scenarios.

SQLite-driven join operations

The old join engine has been replaced by one backed by SQLite. This not only simplifies the whole implementation, but performs much faster since we delegate all the complex join logic to SQLite.

In addition, join operations now support filters on both the left and right sides of the join.

More express support for RDBMS providers

The beauty of most FDO RDBMS providers deriving from the same core, is that the same concept of re-use can be applied as the user interface level as well. As such supporting additional FDO providers based on the GenericRdbms core is a piece of cake.

The express module now has support for creating the following data stores

• MySQL
• PostgreSQL/PostGIS (via the new OSGeo.PostgreSQL provider)

Also included is support for connecting via the commercial Autodesk FDO providers (as these are also based on GenericRdbms):

• SQL Server (using Autodesk.SqlServer provider)
• Oracle (using Autodesk.Oracle provider, experimental)

These commands are only available if you have the aforementioned providers already registered in your providers.xml file.

Bulk Copy UI enhancements

The process of creating bulk copy tasks is now faster than ever.

Add All Connections option

I have discovered from my common usage patterns in creating bulk copies is that I want to add all connections that I currently have open in the Object Explorer. So I've added an option to Add All Connections so you no longer have to painstakingly add each connection individually.

Auto-map properties option

Another thing I discovered, especially when bulk copying to freshly created data stores is that most of the time, I will be mapping properties to identically named properties in the target feature class (and have them created if they don't exist). As such, I've implemented an Auto-map context menu option, which will map each un-mapped property to a property of the same name, with the create if not exists option set to true. This saves lots of time if your source feature class has 50+ properties.

Other changes

Rounding out the other changes:

FDO updated to 3.5.0 RC2

FDO has been updated to the latest RC2 version. Most notable thing here is the ArcSDE provider, which finally implements FDO RFC23. This implementation dramatically cuts down the time needed to perform a schema walk on an ArcSDE data store.

Here's how long it took to schema walk the Utah gov't SGID public database with a pre-RC2 provider:

Here's how long it takes to schema walk the same database with the RC2 provider:

An 18x improvement! No need to take a coffee break waiting for the schema to fully load.

Friendlier connection names

When you drag and drop a supported file into the Object Explorer, it will normally create a connection name of the form: [Provider]_[Number]

Now the naming algorithm is more refined so that for file-based providers, it will create a connection name in the form of: [Provider]_[FileName]

Miscellaneous Fixes and Changes

• Fix "create class of same name" option not working when target schema already has classes
• Disable configuration section in the Generic Connect UI for providers that don't support configuration
• Reader fixes for decimal properties
• Make the Bulk Copy UI more flexible and resizable
• Prevent connection removal on active Bulk Copy and Join UIs
• Handle connection renaming on active Bulk Copy and Join UIs
• Prevent connection removal if open tasks depend on it
• Bulk Copy process no longer tries to execute a delete on a non-existent target class

Download

Report Issues/Enhancements

How to: auto-invoke an AJAX viewer command on startup

I've recently put up a new code sample to illustrate how to automatically invoke a specific AJAX viewer command on startup.

This uses the Initial Task Pane URL property of the Web Layout to load our autostart page, which then uses the AJAX viewer JavaScript API to locate the named command and execute it if found.

This will work with any command type, you just need to know the name (not the label) of the command.

An interesting find

This is the root directory of the Autodesk MapGuide Enterprise 2011 install image



Let's take a look inside...


Either this was included to satisfy open source licensing requirements (IANAL) OR it makes building MGE-compatible extensions much easier

What do you think?

SQLite: Is there anything you CAN'T do?

One of my key areas of focus in preparation for the next beta release of FDO Toolbox 1.0, was to bring the Join implementation up to par. Since all my efforts thus bar has been improving the Bulk Copy functionality, joins have been somewhat left out in the cold in terms of functionality and my attention.

As it stands, the primary problem with the current join implementation is one of performance. Without going into detail the current implementation is one that uses nested loops. This, being a O(m*n) operation will get slower as the size of the data you're working with increases.

In thinking about the different join algorithms available to tackle this problem, I came to the conclusion that implementing what is essentially a core DBMS component is just too much effort, so being the lazy programmer that I am, I've updated the join implementation to delegating the join functionality a DBMS which day-by-day continues to impress me with its amazing performance and capabilities: SQLite

So now, the updated implementation can be described as follows:

• Create a temporary SQLite database with table structure identical to our "left" and "right" sources.
  
• Pump the data from our "left" source into this temp SQLite database
• Pump the data from our "right" source into this temp SQLite database
• Create a view in our temp SQLite that encapsulates a join between these "left" and "right" tables
• Do some metadata hacks to make this view be recognised as a FDO feature class.
  
• Pump the data from this view out to our target data store.
  

How much faster is this approach? I've done some tests by joining the Sheboygan Parcel SHP file (only the bare ID/Geometry attributes, approx. 17600 records) with the corresponding Parcel data in an Access Database (approx. 16100 records):

• Old approach: 16 minutes
• SQLite approach: 21 seconds!

That's a 45x performance improvement! Now it is true that this is only a single data point :-), but if a single data point can indicate such a monumental improvement, then I can comfortably say that you will get similar results with different data sources.

I truly <3 SQLite!