Monday, 23 August 2010

Announcing FDO Toolbox 1.0 beta 1

Unlike commercial software, the 1.0 version number in the open source world usually signifies that that software is "complete", has all major features, and is considered reliable enough for general release.

In the nearly 2 years of development, I have come to the belief that FDO Toolbox fulfills this criteria and thus is ready for the v1.0 designation. This is the first of many beta releases, which will round out some of the remaining items to be completed.

This first beta release introduces the following new features:
  • Bulk copy now supports on-the-fly schema modifications. If copying to non-existent classes and/or properties, these will be created before copying begins, but only if the provider in question supports such capabilities.
  • Express support for creating SQL Server Spatial data stores.
  • Support for executing arbitrary SQL statements for providers that support SQL commands.
  • New menu commands to open the Log and Session directories
  • FdoUtil.exe now supports a -log parameter for the RunTask and BulkCopy commands, allowing error logs to be written to a file name of your choice.


Report Issues/Enhancements

Friday, 6 August 2010

Using FDO Schema Overrides

One of the more esoteric features of the FDO API is the ability to apply schema overrides via an XML configuration file.

Unfortunately this feature is not really well documented, so this blog post will hopefully make it much easier to understand how to use this feature.

What are Schema Overrides?

Schema Overrides allow you to define or override the following aspects of a FDO data source or MapGuide Feature Source:
  • The spatial contexts for this data source
  • The logical schema (schema/class/properties) of this data source
  • The mapping of this logical schema to the underlying physical storage (class -> table, property -> column)
This information is specified in a XML-based configuration file

How are Schema Overrides used?

For FDO providers like ODBC, there is no actual concept of:
  • Geometry properties
  • Spatial Contexts
In this case, schema overrides can be used to "declare" point geometry properties (based on X/Y/Z columns) and what spatial context these points represent. The ODBC feature source table editor in MapGuide Studio/Maestro is a specialized editor of this kind of schema overrides.

Another use, which I think is not quite known is to override how FDO interprets the tables and views in your database, especially views.

A practical example

One of the common needs in any presentation of data is the ability to have one set of data linked to another set of data. In MapGuide, Feature Joins can be used to achieve this effect.

But the performance of Feature Joins can range anywhere between fair and excruciatingly slow (especially when themes are involved), and as such I generally do any of the following:
  1. Pre-join the related bits of data before loading it into MapGuide
  2. If both sets of data come from the same database, do the join at the database level and encapsulate the result in a view.
Our example covers point #2.

Consider the following SQL Server database:

Suppose we want to have a view (called VParcels) that represents the Parcel tables and any related records from the ParcelData table. You would normally do a join similar to this:

Now if we look at this database through FDO Toolbox, you'll now see that our view is represented as a feature class:

But we have a problem, if we make a layer off of this we can't select any objects on it. Look at the ID property:

It isn't an identity property. If we create a layer from this feature class we won't be able to select objects on it because identity properties are the mechanism by which MapGuide knows which particular objects have been selected. In fact, 90%-95% of most problems where layers aren't selectable can be boiled down to a violation of this one basic rule:

No identity properties = unselectable layers

So now we know the problem, how can we fix it? We could index this view, by applying a unique clustered index (the only valid index that can be applied to a SQL server view, other DBMSes may be different), FDO (for SQL Server) will interpret the index as the identity properties.

But a unique clustered index requires that the view only use inner joins which may be a problem especially if you want to show parcels that have no related records (ie. a left outer join).

Enter FDO Schema Overrides to the rescue.

FDO Toolbox 0.9.6 introduced a new command that allows you to dump the default schema mapping to an XML file. What we will do is dump a default XML configuration, and then do the following:
  • Edit this configuration file
  • Test this configuration in FDO Toolbox
  • Loading this configuration file to MapGuide
  • Testing this configuration in MapGuide
So it goes without saying, you'll need FDO Toolbox v0.9.6 or later to do this.

If you right click our SQL Server connection, you will see a new Dump Schema Mapping command on the context menu. This command will create an XML configuration file containing the following information:
  • All the spatial contexts in this data store
  • The entire FDO logical schema
  • The logical to physical schema mapping
If you open this file it will look something like this (outline view):

All XML configuration files have the same structure:
  • They all start and end with the fdo:DataStore element
  • All spatial contexts are indicated with the gml:DerivedCRS element
  • The xs:schema elements indicate the FDO logical schema
  • Finally the SchemaMapping elements indicates how the logical schema maps to the physical schema. These elements are provider specific
Look at the logical schema section and you'll notice a pattern:
  • Feature Classes are denoted by the xs:complexType elements
  • Identity Properties of the Feature Class are denoted by the xs:element element declared directly above the matching xs:complexType element.

If you look closely there is no xs:element declaration for the xs:complexType of our view:

So let's override this default logical schema by declaring an identity property for our view. The xml fragment follows this pattern:

<xs:element name="CLASS_NAME" type="CLASS_NAMEType" abstract="false" substitutiongroup="gml:_Feature">
<xs:key name="CLASS_NAMEKey">
<xs:selector xpath=".//CLASS_NAME" />
<xs:field xpath="IDENTITY_PROPERTY_NAME" />

Where CLASS_NAME is the FDO feature class name and IDENTITY_PROPERTY_NAME is the name of the identity property in the FDO feature class. We'll add one such fragment for VParcels:

Now let's save it and load up another SQL Server connection to the same database. You'll notice that the express dialog has a field allowing you to specify an XML configuration file. We'll use our saved configuration file.

Now connect and take a look at VParcels from the configured connection:

We now have identity properties for our view! Assuming we have an existing SQL Server feature source (without a configuration). Let's run a validation test on it, you'll get something like this:

Notice that MapGuide also can't get the identity properties for VParcels (because it too uses FDO). Now let's edit the configuration of this feature source. In the feature source editor for Maestro, click Edit Configuration Document and we'll get an XML editor, paste the contents of our XML configuration file in here and save it.

Now let's try validating again:

That problem is gone. We have now successfully applied a schema override! MapGuide can now read the identity properties from our view. Just to doubly make sure, let's see if we can make a selectable layer now. I'll make one themed by whether there are related records (black = has related record, red = no related record). Remember that our view is based off of a left outer join so there may be parcels that have no related records.

Now let's house it in a map (switch on selectability) and preview it.

We now have a selectable view-based layer! Now having selectable views comes with a data modeling caveat:

We are only declaring these properties/columns to be unique to MapGuide. You have to ensure at the DBMS level (via constraints or other means) that the actual columns declared will be unique, otherwise selecting a feature may not return the correct set of attributes that you were expecting!

Hopefully this better explains how you can use schema overrides to change how FDO/MapGuide looks at your data store. I'll eventually have some kind of visual editor for FDO Toolbox so you don't have to do any painstaking XML editing by hand.

On a side note, as I've demonstrated with this post FDO Toolbox is a very useful tool for diagnosing problems with MapGuide Feature Sources and other problems that may not be readily apparent through looking at error logs. Almost every time I do a (windows) deployment of MapGuide (Open or Enterprise) I'll always install a copy of FDO Toolbox (and Maestro) as well, because having something on hand to be able to look at your data is very useful.

Thursday, 5 August 2010

FDO Toolbox v0.9.6

This is a quick bugfix release with the following changes:
  • Bulk copies to SQL Server now use regular FDO insert again. For best results set the ForceWKB option to true when copying data to SQL Server.
  • Much improved performance when loading a bulk copy definition file for execution, if a bulk copy definition copies data between connections with large schemas, this performance gain should be readily apparent.
  • A new ListBcpTasks command for FdoInfo.exe which lists all copy tasks in a bulk copy definition.
  • A new Dump Schema Mapping command which will dump a default xml schema override configuration file ready for editing.

Report Bugs / Enhancements

Monday, 2 August 2010

Revising the FDO Toolbox roadmap

Hmmm, didn't I say that the FDO Toolbox project was going to die some months ago?

Well, like your sportsman who retired and then decided to come back because it was way too early to retire, I am having the same feeling with FDO Toolbox development. My renewed interest in continuing development is because:
  • Having "dogfooded" FDO Toolbox on some of my client sites has revealed that there are still many things that can be improved.
  • The FDO API will never stop evolving, so in a way there will always be something new that needs supporting.
Somewhere in the future, FDO Toolbox and Maestro will eventually become one. But in the meantime, there will be life after 1.0 for this little toolbox.