Saturday 18 December 2010

By your powers combined...

Since everyone at my company knows about it already. It's safe to tell the rest of the world.

I am leaving AEC Systems at the end of next week (and this year) after a 4.5 year run. There's been many ups and downs, but ultimately I am grateful to AEC Systems for giving me the opportunity to learn and ultimately allowing me to ascend in my career and knowledge (professionally and personally) to where it is today.

A New Year is the perfect time for a new beginning.

So starting early January next year, I will be joining forces with fellow MapGuide guru and PSC member, Zac Spitzer and his talented crew at Ennoble Consultancy for some fun and interesting times in the MapGuide/FDO/Geospatial realm.

Stay tuned. Exciting times are ahead for sure.

Friday 10 December 2010

My MapGuide Wishlist

I made one for FDO, so here's one for MapGuide...

1. A desktop-based variant of the Geospatial Platform API

Despite web applications being touted as the best thing since sliced bread. I believe there is still a sizable market and demand for a Desktop-based MapGuide product without the full-blown feature set of AutoCAD Map3D.

Autodesk used to have a product like this. It was called Envision (before that, it was called OnSite Desktop) and it worked with MapGuide 6.x (or possibly older) SDF files. There hasn't been anything similar from Autodesk since.

That's not to say we can't do it ourselves. It is conceptually possible to take MapGuide in directions that do not involve the Server and Web Tier, but the amount of implementation (or re-implementation) work required is just too much. There are huge swaths of useful utility code that is currently stuck in the Server and WebTier code-bases that should really be inside some sort of reusable utility library.

Not to mention that there are several Server-only modules that have no real server dependencies. There is utility outside of a MapGuide Server / Web Tier in having standalone MgDrawingService, MgRenderingService, MgKmlService and GwsQueryEngine modules. For most convenience, it would be nice to have them merged into the base platform libraries (MgPlatformBase.dll).

Such a current development requires working with the full MapGuide source tree, not exactly the smallest thing out there to have to download/checkout.

That way, the only real implementation effort would be:

  • MgFeatureService (can simply be a straight wrapper around FDO).
  • MgResourceService (can simply be a pre-defined directory with a SQLite-backed database to store metadata and resource dependencies).
  • MgMapBase (can simply reuse MapGuide implementation)
  • MgLayerBase (can simply reuse MapGuide implementation)
  • MgSelectionBase (can simply reuse MapGuide implementation)

2. A HTML5/Flash/Silverlight vector viewer with offline support

A rich HTML5/Flash/Silverlight viewer has the primary benefits.
  • Reduced server load and better performance. The viewer would only need to perform queries for features. Everything else is done client-side.
  • Richer user experience due to everything being done client-side
  • Client side features are actual geometries, as we a not requesting a rendered image everytime we pan/zoom/refresh/etc. So we can do things like client-side selection, object snapping, etc.
I emphasize offline support (which HTML5 and Silverlight can do, not sure about Flash) because if the previous item is too much effort, an offline-capable web map viewer is the next best thing.

The old MapGuide 6.5 ActiveX viewer could do this through "fat" MWF files (filled with static layers). I just wish there was something similar with current MapGuide technology.

3. A scalable Fusion Legend widget

Fusion has had many teething problems since its introduction several releases ago, but nowadays it is mostly functional and usable, except for the Legend widget. Given a moderately large map, the legend widget just breaks down.

It also doesn't help that fusion is the default previewing mechanism in MapGuide Studio so you can expect previews on large themes to timeout consistently (that's what you get for embedding Internet Explorer!). I've had to advise clients from using Fusion simply because I knew it couldn't handle the large maps that they have, which is a shame because it perfectly satisfied all their other requirements!

4. A way to preserve tile sets.

Ever change something innocuous like the title of your Map Definition and MapGuide suddenly takes off and nukes your tile set from orbit? Yes, it's very annoying because it isn't exactly fast to produce these tile sets!

I think the best solution is to have a tile set defined as a separate resource, any Layer Definitions (and by extension, Feature Sources) that are used as part of this tile set resource are locked from editing. My current thinking is that a tile set definition (if you will) would have the following information:
  • Path to where the generated tiles will be stored. Can be %MG_DATA_FILE_PATH% for managed, or an external path or alias. Come to think of it, %MG_DATA_FILE_PATH% is probably a bad idea as packaging will take an eternity.
  • The Layer Definitions participating in this tile set.
  • The Coordinate System (to re-project any non-matching layers)
  • The scales applicable for this tile set.

MapGuide already knows the dependency chain of any resource, so it can stop any edits made to dependent layers and feature sources of a given tile set. The tiled layer component of the Map Definition would be deprecated in favour of references to tile sets (or probably be a new resource type so as not to break the additive qualities of previous schema revisions)

This way once a tile set is defined, it is not possible to trash the generated tiles unless you say so as layer and feature source changes/updates/deletions will be denied by MapGuide due to its knowledge of dependent resources.

5. Feature Write capabilities for the mapagent

Failing that, merge GeoREST into MapGuide, you'll get pretty much the same result.

6. A DWG FDO provider

I understood in the past why it was not possible. But the release of AutoCAD for Mac has proven that Autodesk finally have multi-platform DWG access technology. So get to it! It will definitely add extra value to a licence of MapGuide Enterprise

My FDO wishlist

Since it's that time of the year, and Autodesk loves them customer surveys and wishlists...

1. A way to use the FDO API in languages other than C++/.net in a multi-platform manner

FDO is a multi-platform API. I like .net, it is portable too assuming you target a safe and mature subset of the .net framework libraries (which I already do).

Unfortunately, the .net wrapper for FDO is not multi-platform and if you're not careful the .net Garbage Collector who is normally your friend, becomes your worst enemy as it prematurely cleans up your wrapped FDO objects from under your feet and you're greeted with the all-too-common AccessViolationException being thrown at you.

SWIG can wrap C++ code in a multi-platform manner. GDAL does this. MapGuide already does this, why not FDO as well? There's already a Python wrapper generated by SWIG (albeit Windows-only last I checked), so it's not exactly impossible!

I don't mind having to use C++ if I have no other choice. But higher level languages does yield greater productivity. It's how I got FDO Toolbox to become so feature-packed in such a short period of time. The fact our Linux users cannot use FDO Toolbox because of lack of flexible development options (that is not C++) is disappointing.

2. Champion SQLite as the goto flat-file format for FDO

Now I don't mean product support. Because it's already getting love from the latest releases of AutoCAD Map3D and MapGuide Studio. I mean having SQLite become the de-facto flat file format that showcases all of the features supported by FDO.

What do I mean exactly? I mean features you normally find in a RDBMS provider like:
  • Association Properties
  • Object Properties
  • Schema Element Attributes
  • etc
3. A FDO provider for MapGuide

Sure, there's no write capability built into the mapagent, but having a read-only FDO provider for MapGuide would greatly improve interoperability between FDO client applications and MapGuide. It would basically be like WFS, but without the impedance mismatch and loss of metadata. As MapGuide is built on FDO, it knows everything about FDO and the capabilities can be parameteric with respect to the underlying Feature Source.

You could possibly even have raster support by just extending existing Rendering Service APIs with required raster functionality (querying, sub-sampling, etc) and exposing this through the mapagent.

4. Consistent Schema Naming Conventions

This is already a well-documented problem, but I'll extend it with a possible solution.

For those providers that hard-code a schema name (because they don't support the concept of a logical schema and just need a name), how about instead we define the default schema name in an external configuration file and these providers can read off of this file instead? This way, not only is it unified, but unified to your preferences.

Having to write implementation-specific code in an abstraction layer like FDO defeats the whole concept!

5. Parametric Capabilities

Capability APIs break down on providers like ODBC and OGR due to their "one size fits all" nature of these providers. It would be nice if ODBC and OGR actually have capability discovery, for these capabilities to be translated to corresponding FDO capabilities, so we don't get a whole bunch of -1 values for FDO capabilities like maximum length of a property.

6. Android/iPhone support

Hey this is a wishlist after all! Smartphones are all the rage these days. No need to port everything. Just the base SDF/SHP/SQLite providers would suffice.

Thursday 25 November 2010

Announcing: FDO Toolbox 1.0 beta 3

Here it is! The 3rd beta of FDO Toolbox 1.0

Some of the highlights include:
Along with some fixes:
  • FDO Data Stores containing Object and Association properties can now be exported
  • Improved Object and Association property support in the Data Store Editor
  • Insert/Update/Preview support for Object and Association properties. Many thanks to Crispin for providing this patch
  • Geodetic polygons with bad orientation are fixed when bulk copying to SQL Server 2008

FDO RFC48: Easing some of the pain of copying data to SQL Server

The next beta of FDO Toolbox will come bundled with a beta version of FDO 3.6.0

The reason for the move is because FDO 3.6 incorporates RFC48, which provides APIs allowing us to correct the orientation of polygons.

Where is this API most relevant? Copying data to SQL Server 2008.

For those of us who fell for the "Microsoft Technology v1.0" trap (because frankly Microsoft never gets things right on the first attempt!), SQL Server has strict rules regarding geography types (FDO will create geography data types if your FDO Geometric Property uses lat/long coordinates). If you try to insert polygons with invalid orientation into a geography column, SQL Server will automatically reject the input. A post-insert MakeValid is not possible.

This basically meant that copying polygons (with lat/long coordinates) into SQL Server had a high chance of failure, with no way to correct the problem.

Apparently the next version of SQL Server (codenamed "Denali") will be more lenient with regards to geometry correctness (among other things) rendering this point moot. But that is the future, and we are in the present! With the next beta of FDO Toolbox, RFC48 APIs will be used exclusively for copying data to SQL Server 2008, ensuring a more pleasant bulk copy experience.

These new APIs are generic, so I can look at making the geometry correction available to all providers in the future, but for now this is only for SQL Server as it is the most relevant provider requiring such services.

A PostGIS Public Service Announcement

If you use PostGIS, avoid using the OSGeo.PostGIS provider. It hasn't been maintained for quite some time now and there are many un-fixed stability and performance problems that have made this provider unsuitable for production use.

Instead, use the OSGeo.PostgreSQL provider which is included in the 2.2 release of MapGuide Open Source (currently in RC), Autodesk MapGuide Enterprise 2011, AutoCAD Map3D 2011 and recent releases of FDO Toolbox.

The OSGeo.PostgresSQL provider is built on the same robust core as the MySQL, SQL Server and other RDBMS providers, and provides better stability and reliability than the old provider.

The OSGeo.PostGIS provider is not included in the next version of FDO. The next beta of FDO Toolbox will not include this provider, as it is using the next version of FDO.

So for those still using the old provider, take this as a warning to start migrating your PostGIS connections and layers in MapGuide over to the new provider before it's too late!

Thursday 18 November 2010

Introducing: SequentialProcess

Ever tried to chain multiple calls to FdoUtil.exe using a batch file?

I did recently on a client site, and found out that for one reason or another the Task Scheduler feature of Windows Server 2008 decided that the batch file I wrote was not going to run. After trying to find out the reason why (with no success), I tried running the individual calls to FdoUtil.exe with the Task Scheduler. Lo and behold they ran!

So armed with this knowledge, I have introduced a 3rd task type into FDO Toolbox, which will be in the next beta release: The Sequential Process.

A sequential process is simply an XML document that contains a series of calls to FdoUtil.exe, with an option at each invocation to abort the process if the previous FdoUtil.exe invocation returns a non-zero result (FdoUtil.exe returns 0 on success, otherwise it returns one of the predefined values here)

So for example, you can define a sequential process like so:

1. Run a SQL command to delete some rows
2. Execute a bulk copy task file (eg. SDF to SQL Server)
3. Run a SQL command to rebuild indexes

The SequentialProcess document would look like this:



Here's what it looks like visually (yes! This will have design support):


When you run this sequential process. Each step will be run as a separate FdoUtil.exe process, with the root process waiting for a return result from this process before continuing onto the next step.

For reference, FdoUtil.exe currently supports the following commands:
  • ApplySchema
  • CreateDataStore
  • Destroy (data store)
  • DumpSchema
  • CreateFile (data store)
  • RegisterProvider
  • UnregisterProvider
  • BulkCopy
  • RunTask
  • ExecuteSql
The sequential process task has opened up the design space for more FdoUtil.exe commands. So this list will definitely grow in size in future releases.

And yes, the RunTask command now supports SequentialProcess documents, so theoretically speaking, you can also nest Sequential Processes and other supported task types.

As I said before this is task scheduler friendly (the main reason for introducing this feature), so you can hook this up via the task scheduler for painless daily data updates or other automated spatial data maintenance.

Thursday 14 October 2010

Using Maestro in native mode with MapGuide Open Source 2.2 / MapGuide Enterprise 2011 assemblies

The release of MapGuide Enterprise 2011 and the pending release of MapGuide Open Source 2.2 has slightly changed the game with regards to using Maestro with the official API

Here's how to do it with this release.

First go to the mapviewernet/bin directory of your MapGuide installation


The highlighted dlls are the .net assemblies we need to strong name. Because of RFC68, the previous MapGuideDotNetApi.dll is now split into 5 separate assemblies. The MapGuideDotNetApi.dll in this installation is now a compatibility stub that redirects to the 5 assemblies.

To be able to use these assemblies in Maestro, we need to do the following:
  • Sign all these assemblies with the Maestro public key
  • Setup binding redirection for this new version of MapGuideDotNetApi.dll in web.config/app.config
Unfortunately, the current method using Signer.exe does not work because it doesn't properly handle assemblies with TypeForwardedTo attributes (as is the case with the new MapGuideDotNetApi.dll) nor does it handle updating inter-assembly references to signed equivalents, so to actually sign these assemblies, we need to use the ildasm.exe and ilasm.exe utilities that come with the .net Framework SDK.

So first, copy the aforementioned assemblies into a temporary folder

Rename MapGuideDotNetApi.dll to MapGuideDotNetApi-2.2.dll

Now open up a SDK command prompt into this folder and issue the following commands:
  • ildasm /all /out=MapGuideDotNetApi-2.2.il MapGuideDotNetApi-2.2.dll
  • ildasm /all /out=OSGeo.MapGuide.Foundation.il OSGeo.MapGuide.Foundation.dll
  • ildasm /all /out=OSGeo.MapGuide.Geometry.il OSGeo.MapGuide.Geometry.dll
  • ildasm /all /out=OSGeo.MapGuide.MapGuideCommon.il OSGeo.MapGuide.MapGuideCommon.dll
  • ildasm /all /out=OSGeo.MapGuide.PlatformBase.il OSGeo.MapGuide.PlatformBase.dll
  • ildasm /all /out=OSGeo.MapGuide.Web.il OSGeo.MapGuide.Web.dll
Your file listing should now look like this:

Delete the existing dll files, because ilasm.exe will rebuild them.

Before we continue, let's sidestep for a moment.

Here is a screenshot of a reflector dump of the original MapGuideDotNetApi.dll

Notice the PublicKeyToken=null part of the referenced assembly. The original MapGuideDotNetApi.dll is referencing un-signed assemblies (obviously). So when we rebuild MapGuideDotNetApi.dll with ilasm.exe, we have to make sure that it is referencing the signed versions of these assemblies, otherwise the you will get errors due to failing to locate the strongly-named assembly.

Also note that some OSGeo.MapGuide assemblies reference other OSGeo.MapGuide assemblies, thus we need to ensure that all references need to be updated. For reference, the dependency chain is as follows:
  • OSGeo.MapGuide.Foundation references nothing
  • OSGeo.MapGuide.Geometry references OSGeo.MapGuide.Foundation
  • OSGeo.MapGuide.PlatformBase references OSGeo.MapGuide.Foundation and OSGeo.MapGuide.Geometry
  • OSGeo.MapGuide.MapGuideCommon references OSGeo.MapGuide.Foundation, OSGeo.MapGuide.Geometry and OSGeo.MapGuide.PlatformBase
  • OSGeo.MapGuide.Web references OSGeo.MapGuide.Foundation, OSGeo.MapGuide.Geometry and OSGeo.MapGuide.PlatformBase

Download a copy of the maestroapi.key into this directory.

Now in the same command prompt, rebuild and sign the OSGeo.MapGuide.Foundation assembly using ilasm.exe:
  • ilasm /dll /key=maestroapi.key OSGeo.MapGuide.Foundation.il

Take a look at this assembly in reflector.

Note the public key token, it will be the same for the other signed assemblies that we will sign. We need to update the IL of MapGuideDotNetApi and others to include this public key token in their assembly references.


Open the MapGuideDotNetApi-2.2.il in a text editor.


Locate the relevant section for the IL files of the other assemblies and update them with the same public key token. Once that is done, rebuild and sign them all with ilasm.exe
  • ilasm /dll /key=maestroapi.key OSGeo.MapGuide.Geometry.il
  • ilasm /dll /key=maestroapi.key OSGeo.MapGuide.MapGuideCommon.il
  • ilasm /dll /key=maestroapi.key OSGeo.MapGuide.PlatformBase.il
  • ilasm /dll /key=maestroapi.key OSGeo.MapGuide.Web.il
  • ilasm /dll /key=maestroapi.key MapGuideDotNetApi-2.2.il
Your assemblies are now properly signed and referencing the signed versions of their needed assemblies.

Now these are ready to be used in Maestro.

Copy all the files from mapviewernet/bin (except for MapGuideDotNetApi.dll) into your Maestro installation.

Now copy all the signed assemblies from our temp directory into your Maestro installation, overwrite the existing assemblies with our signed ones.

Now edit your app.config/web.config to redirect the currently referenced MapGuideDotNetApi.dll to our signed version:

And Maestro in Native mode will correctly use the assemblies we just signed.

Note that for MapGuide Enterprise 2011, the assembly version number is different but the process is the same.

Upon the proper release of MapGuide Open Source 2.2, I'll post the signed assemblies here so you won't have to follow this long process :D

Wednesday 29 September 2010

The expressive power of MapGuide Tooltips redux

Over a year ago, I wrote about how you can have really powerful tooltips in MapGuide with some knowledge of HTML and the FDO expression language, with one major caveat:

You can't have interactive content in the AJAX viewer.

Well I've finally tackled this problem and implemented "sticky" tooltips for the AJAX viewer, so now you can have your Google Charts or Youtube videos in your AJAX viewer tooltips and be able to interact with them

I'm hoping to get this in for the 2.2 release of MapGuide Open Source. If you can't wait, and know how to apply diffs, you can grab the patch here

Monday 27 September 2010

Using FDO XML configuration to reduce the attack surface of your data store (and improve performance)

Consider the following SQL Server database:



The screenshot isn't big enough to show it, but there are 250 feature classes in this schema. Sometimes for security purposes, you don't exactly want to make all 250 feature classes visible to any FDO client application (eg. MapGuide). This is commonly known as reducing the attack surface.

With FDO XML configuration and the Unified Data Store editor, you can override the default logical schema (250 classes) with something that is more stripped down (like say: 10 classes)

To do this, we once again use the unified data store editor



Now select the feature classes you don't want visible and remove them (for ease of use, you can remove a feature class by hitting the Delete key as long as the class node is selected)

Now for the important bit: Do not apply the changes! Doing so will actually delete these feature classes from the underlying data store which we do not want! We just want to hide them.

Instead, we export this current state of the logical schema to an XML configuration file


Now if we make another connection with this XML configuration file



We see that the connection takes on the logical schema as defined in our XML configuration file.

Another benefit of a reduced attack surface is performance. Here's the schema walk of the 250 class connection (LargeDb) and the configured one (LargeDb_compacted):



Although SQL Server already implements FDO RFC23, and thus schema walking is already efficient, and the times in question (as you can see from the screenshot) are already well under 10ms, you can objectively see that having a stripped down logical schema via XML configuration does indeed improve performance, which makes sense (less classes = faster loading). Now a really good test case would've been the ArcSDE provider and the Utah SGID dataset, but unfortunately the ArcSDE provider does not support XML configuration :( so that's a case of "what if?".

Another wonderful use of FDO XML configuration brought to you by the Toolbox.

Using FDO Schema Overrides Redux

When I announced beta 2 of FDO Toolbox 1.0, I briefly mentioned that the new Unified Data Store editor doubles as a visual FDO Schema Override editor.

In this post I'll illustrate how we can create FDO XML configuration documents without needing to modify a single line of XML in a text editor, using our Unified Data Store editor.

We will use the same sample SQL Server database from our previous post


Creating a connection to this via FDO Toolbox gives us the following logical schema view:


Notice once again, how the FeatId property is not identity. Now where the previous post used a text editor to tweak the logical schema, we will do the whole process visually using the unified data store editor.

We'll right click the connection and use the new Edit Data Store option


This brings up the Unified Data Store editor.


Now what we simply do, is modify the logical schema so that it matches what we want it to be. In this case, we want FeatId to be an Identity Property of VParcels. So we select the VParcels feature class, this will bring up the class definition editor. We check the FeatId property under the Identity Properties to make FeatId an Identity Property


Now comes the important bit: Do not apply the changes!

Instead we export this current state out to an XML configuration document via Export - XML Configuration Document


Now let's create another connection to the same database, but this time we'll use the XML configuration document we have just saved.


If we look at the VParcels feature class in this configured connection, you'll notice it takes on our overridden logical schema.


From here, the process is just like the previous post. In Maestro's feature source editor, you would edit the configuration document of that feature source, paste in the contents of this XML file and save it. Layers based off of the VParcels feature class will then be selectable.

As I mentioned in the beta 2 announcement this editor is not fully complete. For one thing, there is no user interface to edit the physical configuration of any Class Definitions and Property Definitions. Whatever mappings gets exported out to XML are the default mappings generated by the provider.

But for overriding the default logical schema (which I'd guess is the common scenario), the unified data store editor makes this process much simpler than the older method of editing XML files by hand.

Announcing: FDO Toolbox 1.0 beta 2

Here it is! The 2nd beta of FDO Toolbox 1.0

This contains many new changes from the 1st beta, some of the notable features include:

Unified data store editor

All the of schema and spatial context management functionality is now merged into a single user interface


As you can see from the above screenshot, instead of trying to win the losing battle against the WinForms PropertyGrid, all schema element editors are now specialized user interfaces.

Because we have now unified everything, we have all the necessary information needed to create full XML configuration documents. Therefore, exporting to XML will now export the full XML configuration document (Spatial Contexts, Logical Schemas and Physical Mapping).

Thus the unified data store editor doubles up as a visual schema override editor. I'll elaborate on this in a future post.

Just like the original schema editor, this functions also in a disconnected mode, free of any restrictions imposed by FDO provider capabilities.

The unified data store editor is not fully implemented yet. Some features are missing or not fully tested, such as:
  • Object Property support
  • Association Property support
  • Class inheritance support
  • Logical-Physical mapping configuration
Despite these missing features, the unified data store editor should fulfill most common data modeling scenarios.

SQLite-driven join operations

The old join engine has been replaced by one backed by SQLite. This not only simplifies the whole implementation, but performs much faster since we delegate all the complex join logic to SQLite.

In addition, join operations now support filters on both the left and right sides of the join.

More express support for RDBMS providers

The beauty of most FDO RDBMS providers deriving from the same core, is that the same concept of re-use can be applied as the user interface level as well. As such supporting additional FDO providers based on the GenericRdbms core is a piece of cake.

The express module now has support for creating the following data stores
  • MySQL
  • PostgreSQL/PostGIS (via the new OSGeo.PostgreSQL provider)
Also included is support for connecting via the commercial Autodesk FDO providers (as these are also based on GenericRdbms):
  • SQL Server (using Autodesk.SqlServer provider)
  • Oracle (using Autodesk.Oracle provider, experimental)
These commands are only available if you have the aforementioned providers already registered in your providers.xml file.

Bulk Copy UI enhancements

The process of creating bulk copy tasks is now faster than ever.

Add All Connections option

I have discovered from my common usage patterns in creating bulk copies is that I want to add all connections that I currently have open in the Object Explorer. So I've added an option to Add All Connections so you no longer have to painstakingly add each connection individually.



Auto-map properties option

Another thing I discovered, especially when bulk copying to freshly created data stores is that most of the time, I will be mapping properties to identically named properties in the target feature class (and have them created if they don't exist). As such, I've implemented an Auto-map context menu option, which will map each un-mapped property to a property of the same name, with the create if not exists option set to true. This saves lots of time if your source feature class has 50+ properties.



Other changes

Rounding out the other changes:

FDO updated to 3.5.0 RC2

FDO has been updated to the latest RC2 version. Most notable thing here is the ArcSDE provider, which finally implements FDO RFC23. This implementation dramatically cuts down the time needed to perform a schema walk on an ArcSDE data store.

Here's how long it took to schema walk the Utah gov't SGID public database with a pre-RC2 provider:


Here's how long it takes to schema walk the same database with the RC2 provider:


An 18x improvement! No need to take a coffee break waiting for the schema to fully load.

Friendlier connection names

When you drag and drop a supported file into the Object Explorer, it will normally create a connection name of the form: [Provider]_[Number]

Now the naming algorithm is more refined so that for file-based providers, it will create a connection name in the form of: [Provider]_[FileName]

Miscellaneous Fixes and Changes
  • Fix "create class of same name" option not working when target schema already has classes
  • Disable configuration section in the Generic Connect UI for providers that don't support configuration
  • Reader fixes for decimal properties
  • Make the Bulk Copy UI more flexible and resizable
  • Prevent connection removal on active Bulk Copy and Join UIs
  • Handle connection renaming on active Bulk Copy and Join UIs
  • Prevent connection removal if open tasks depend on it
  • Bulk Copy process no longer tries to execute a delete on a non-existent target class


Friday 17 September 2010

How to: auto-invoke an AJAX viewer command on startup

I've recently put up a new code sample to illustrate how to automatically invoke a specific AJAX viewer command on startup.

This uses the Initial Task Pane URL property of the Web Layout to load our autostart page, which then uses the AJAX viewer JavaScript API to locate the named command and execute it if found.

This will work with any command type, you just need to know the name (not the label) of the command.

Friday 3 September 2010

An interesting find

This is the root directory of the Autodesk MapGuide Enterprise 2011 install image



Let's take a look inside...


Either this was included to satisfy open source licensing requirements (IANAL) OR it makes building MGE-compatible extensions much easier

What do you think?

Thursday 2 September 2010

SQLite: Is there anything you CAN'T do?

One of my key areas of focus in preparation for the next beta release of FDO Toolbox 1.0, was to bring the Join implementation up to par. Since all my efforts thus bar has been improving the Bulk Copy functionality, joins have been somewhat left out in the cold in terms of functionality and my attention.

As it stands, the primary problem with the current join implementation is one of performance. Without going into detail the current implementation is one that uses nested loops. This, being a O(m*n) operation will get slower as the size of the data you're working with increases.

In thinking about the different join algorithms available to tackle this problem, I came to the conclusion that implementing what is essentially a core DBMS component is just too much effort, so being the lazy programmer that I am, I've updated the join implementation to delegating the join functionality a DBMS which day-by-day continues to impress me with its amazing performance and capabilities: SQLite

So now, the updated implementation can be described as follows:
  • Create a temporary SQLite database with table structure identical to our "left" and "right" sources.
  • Pump the data from our "left" source into this temp SQLite database
  • Pump the data from our "right" source into this temp SQLite database
  • Create a view in our temp SQLite that encapsulates a join between these "left" and "right" tables
  • Do some metadata hacks to make this view be recognised as a FDO feature class.
  • Pump the data from this view out to our target data store.
How much faster is this approach? I've done some tests by joining the Sheboygan Parcel SHP file (only the bare ID/Geometry attributes, approx. 17600 records) with the corresponding Parcel data in an Access Database (approx. 16100 records):
  • Old approach: 16 minutes
  • SQLite approach: 21 seconds!
That's a 45x performance improvement! Now it is true that this is only a single data point :-), but if a single data point can indicate such a monumental improvement, then I can comfortably say that you will get similar results with different data sources.

I truly <3 SQLite!

Monday 23 August 2010

Announcing FDO Toolbox 1.0 beta 1

Unlike commercial software, the 1.0 version number in the open source world usually signifies that that software is "complete", has all major features, and is considered reliable enough for general release.

In the nearly 2 years of development, I have come to the belief that FDO Toolbox fulfills this criteria and thus is ready for the v1.0 designation. This is the first of many beta releases, which will round out some of the remaining items to be completed.

This first beta release introduces the following new features:
  • Bulk copy now supports on-the-fly schema modifications. If copying to non-existent classes and/or properties, these will be created before copying begins, but only if the provider in question supports such capabilities.
  • Express support for creating SQL Server Spatial data stores.
  • Support for executing arbitrary SQL statements for providers that support SQL commands.
  • New menu commands to open the Log and Session directories
  • FdoUtil.exe now supports a -log parameter for the RunTask and BulkCopy commands, allowing error logs to be written to a file name of your choice.

Download

Report Issues/Enhancements

Friday 6 August 2010

Using FDO Schema Overrides

One of the more esoteric features of the FDO API is the ability to apply schema overrides via an XML configuration file.

Unfortunately this feature is not really well documented, so this blog post will hopefully make it much easier to understand how to use this feature.

What are Schema Overrides?

Schema Overrides allow you to define or override the following aspects of a FDO data source or MapGuide Feature Source:
  • The spatial contexts for this data source
  • The logical schema (schema/class/properties) of this data source
  • The mapping of this logical schema to the underlying physical storage (class -> table, property -> column)
This information is specified in a XML-based configuration file

How are Schema Overrides used?

For FDO providers like ODBC, there is no actual concept of:
  • Geometry properties
  • Spatial Contexts
In this case, schema overrides can be used to "declare" point geometry properties (based on X/Y/Z columns) and what spatial context these points represent. The ODBC feature source table editor in MapGuide Studio/Maestro is a specialized editor of this kind of schema overrides.

Another use, which I think is not quite known is to override how FDO interprets the tables and views in your database, especially views.

A practical example

One of the common needs in any presentation of data is the ability to have one set of data linked to another set of data. In MapGuide, Feature Joins can be used to achieve this effect.

But the performance of Feature Joins can range anywhere between fair and excruciatingly slow (especially when themes are involved), and as such I generally do any of the following:
  1. Pre-join the related bits of data before loading it into MapGuide
  2. If both sets of data come from the same database, do the join at the database level and encapsulate the result in a view.
Our example covers point #2.

Consider the following SQL Server database:

























Suppose we want to have a view (called VParcels) that represents the Parcel tables and any related records from the ParcelData table. You would normally do a join similar to this:




















Now if we look at this database through FDO Toolbox, you'll now see that our view is represented as a feature class:

















But we have a problem, if we make a layer off of this we can't select any objects on it. Look at the ID property:




















It isn't an identity property. If we create a layer from this feature class we won't be able to select objects on it because identity properties are the mechanism by which MapGuide knows which particular objects have been selected. In fact, 90%-95% of most problems where layers aren't selectable can be boiled down to a violation of this one basic rule:

No identity properties = unselectable layers

So now we know the problem, how can we fix it? We could index this view, by applying a unique clustered index (the only valid index that can be applied to a SQL server view, other DBMSes may be different), FDO (for SQL Server) will interpret the index as the identity properties.

But a unique clustered index requires that the view only use inner joins which may be a problem especially if you want to show parcels that have no related records (ie. a left outer join).

Enter FDO Schema Overrides to the rescue.

FDO Toolbox 0.9.6 introduced a new command that allows you to dump the default schema mapping to an XML file. What we will do is dump a default XML configuration, and then do the following:
  • Edit this configuration file
  • Test this configuration in FDO Toolbox
  • Loading this configuration file to MapGuide
  • Testing this configuration in MapGuide
So it goes without saying, you'll need FDO Toolbox v0.9.6 or later to do this.

If you right click our SQL Server connection, you will see a new Dump Schema Mapping command on the context menu. This command will create an XML configuration file containing the following information:
  • All the spatial contexts in this data store
  • The entire FDO logical schema
  • The logical to physical schema mapping
If you open this file it will look something like this (outline view):











All XML configuration files have the same structure:
  • They all start and end with the fdo:DataStore element
  • All spatial contexts are indicated with the gml:DerivedCRS element
  • The xs:schema elements indicate the FDO logical schema
  • Finally the SchemaMapping elements indicates how the logical schema maps to the physical schema. These elements are provider specific
Look at the logical schema section and you'll notice a pattern:
  • Feature Classes are denoted by the xs:complexType elements
  • Identity Properties of the Feature Class are denoted by the xs:element element declared directly above the matching xs:complexType element.





















If you look closely there is no xs:element declaration for the xs:complexType of our view:

So let's override this default logical schema by declaring an identity property for our view. The xml fragment follows this pattern:

<xs:element name="CLASS_NAME" type="CLASS_NAMEType" abstract="false" substitutiongroup="gml:_Feature">
<xs:key name="CLASS_NAMEKey">
<xs:selector xpath=".//CLASS_NAME" />
<xs:field xpath="IDENTITY_PROPERTY_NAME" />
</xs:key>
</xs:element>

Where CLASS_NAME is the FDO feature class name and IDENTITY_PROPERTY_NAME is the name of the identity property in the FDO feature class. We'll add one such fragment for VParcels:



















Now let's save it and load up another SQL Server connection to the same database. You'll notice that the express dialog has a field allowing you to specify an XML configuration file. We'll use our saved configuration file.














Now connect and take a look at VParcels from the configured connection:























We now have identity properties for our view! Assuming we have an existing SQL Server feature source (without a configuration). Let's run a validation test on it, you'll get something like this:


















Notice that MapGuide also can't get the identity properties for VParcels (because it too uses FDO). Now let's edit the configuration of this feature source. In the feature source editor for Maestro, click Edit Configuration Document and we'll get an XML editor, paste the contents of our XML configuration file in here and save it.




















Now let's try validating again:


















That problem is gone. We have now successfully applied a schema override! MapGuide can now read the identity properties from our view. Just to doubly make sure, let's see if we can make a selectable layer now. I'll make one themed by whether there are related records (black = has related record, red = no related record). Remember that our view is based off of a left outer join so there may be parcels that have no related records.






















Now let's house it in a map (switch on selectability) and preview it.



















We now have a selectable view-based layer! Now having selectable views comes with a data modeling caveat:

We are only declaring these properties/columns to be unique to MapGuide. You have to ensure at the DBMS level (via constraints or other means) that the actual columns declared will be unique, otherwise selecting a feature may not return the correct set of attributes that you were expecting!

Hopefully this better explains how you can use schema overrides to change how FDO/MapGuide looks at your data store. I'll eventually have some kind of visual editor for FDO Toolbox so you don't have to do any painstaking XML editing by hand.

On a side note, as I've demonstrated with this post FDO Toolbox is a very useful tool for diagnosing problems with MapGuide Feature Sources and other problems that may not be readily apparent through looking at error logs. Almost every time I do a (windows) deployment of MapGuide (Open or Enterprise) I'll always install a copy of FDO Toolbox (and Maestro) as well, because having something on hand to be able to look at your data is very useful.

Thursday 5 August 2010

FDO Toolbox v0.9.6

This is a quick bugfix release with the following changes:
  • Bulk copies to SQL Server now use regular FDO insert again. For best results set the ForceWKB option to true when copying data to SQL Server.
  • Much improved performance when loading a bulk copy definition file for execution, if a bulk copy definition copies data between connections with large schemas, this performance gain should be readily apparent.
  • A new ListBcpTasks command for FdoInfo.exe which lists all copy tasks in a bulk copy definition.
  • A new Dump Schema Mapping command which will dump a default xml schema override configuration file ready for editing.
Download

Report Bugs / Enhancements

Monday 2 August 2010

Revising the FDO Toolbox roadmap

Hmmm, didn't I say that the FDO Toolbox project was going to die some months ago?

Well, like your sportsman who retired and then decided to come back because it was way too early to retire, I am having the same feeling with FDO Toolbox development. My renewed interest in continuing development is because:
  • Having "dogfooded" FDO Toolbox on some of my client sites has revealed that there are still many things that can be improved.
  • The FDO API will never stop evolving, so in a way there will always be something new that needs supporting.
Somewhere in the future, FDO Toolbox and Maestro will eventually become one. But in the meantime, there will be life after 1.0 for this little toolbox.

Wednesday 28 July 2010

FDO Toolbox v0.9.5

Here's a new release for you all.

New Features
  • Full support for side-by-side installation of 32 and 64-bit versions of FDO Toolbox. You can install and run both x86 and x64 versions of this release on your 64-bit machine, and each version is self-contained with their own settings and preferences.
  • Added drag and drop support for Excel documents
  • The RunTask command for FdoUtil.exe has been updated to support running a specific copy task within the bulk copy definition file.
  • A new ExecuteSql command is included in FdoUtil.exe allowing for execution of non-SELECT SQL statements.
  • A new streamlined naming convention is employed for added connection.
  • Bulk copy for SQL Server now uses an experimental approach using SQL INSERT statements instead of the FDO insert. SQL Server is very fickle about its geometries and FDO's insert operation often fails due to "bad" geometries. Now with the new approach, we pump direct geometry WKT strings to SQL Server, and put the onus on SQL Server to fix the "bad" geometries. This only applies to SQL Server, other providers will still use the regular FDO insert method.
Also included are the following bug fixes:
  • Data Query will now gracefully continue if the method of counting the number of results fails.
  • Bundled a custom built SQLite provider which includes a fix for multiple spatial contexts.
  • Fix bulk copy editor not removing deleted copy tasks on save.
  • Fix refresh issue for OSGeo.PostgreSQL connections
  • Fix Object Explorer loading issue when more than one class of the same un-qualified name exists.
  • Fix a RDBMS connection issue where non-FDO datastores were not being listed.
  • 64-bit fixes for ODBC connections to Excel and Microsoft Access. You don't have to use the cumbersome workaround anymore.
  • 64-bit fixes for drag/drop of Access Databases and Excel spreadsheets.
IMPORTANT NOTE: If you plan to use the 64-bit version of FDO Toolbox, be sure to download and install the Microsoft Access Database Engine 2010 redistributable package otherwise you won't be able to work with Excel spreadsheets or Access Databases through the ODBC provider.

Download

Report bugs or feature requests

Thursday 8 July 2010

MapGuide Maestro 2.1

After 2 preview releases, I am proud to announce the general availability of MapGuide Maestro 2.1

New features include:
  • Ability to preview feature sources locally (note that this is only enabled for releases newer than MapGuide Open Source 2.2 Beta 1)
  • Support for connecting via the official MapGuide API (a.k.a. Web-less editing)
  • Basic support for SDF and SHP Load Procedures
  • Site Explorer context menu enhancements (provided by Crispin from 1spatial)
  • XML editor find and replace (provided by Crispin from 1spatial)
  • Support for applying overrides to multiple coordinate systems (provided by Crispin from 1spatial)
  • Many bug fixes for LocalNativeConnection
  • Support for adding layers to a map by dragging and dropping them from the Site Explorer.
  • Open layer in Layer Editor when double clicked from the Map Editor.
  • Support for selecting DSNs in the ODBC feature source editor.
  • Updated french translation provided by Pierre Cardinal
  • New Croatian translation provided by Ivan Miličević
A comprehensive list of changes since the 2.0 release can be found here

2.1 represents the last release in the 2.x line that will have new features. Future releases in the 2.x line will be strictly bug fixes only.

New features and/or enhancement requests will all be moved for the 3.0 release

Download (msi)


Download (zip)

Tuesday 6 July 2010

Connecting to MS Access databases with the 64-bit FDO ODBC provider

One of the (many) joys of going 64-bit is the astronomically huge amount of memory (2^64 bytes) you can throw at any application.

However, one of the pains thus far (on Windows at least) is the lack of connectivity to MS Access databases due to a lack of a fully native 64-bit ODBC driver.

Well it seems Microsoft have finally listened to its 64-bit userbase and finally released 64-bit drivers for Microsoft Access Databases last month.

Now how can we use these drivers in the just released 64-bit FDO Toolbox? If you thought the Connect to ODBC command choosing MsAccess as the data source type (like I first did), then you thought wrong.

It turns out the Office 2010 ODBC drivers have slightly tweaked the required connection string.

Here's what a current MS Access ODBC connection string would look like:

Driver={Microsoft Access Driver (*.mdb)};Dbq=C:\parcels.mdb

If you used that same connection string in a 64-bit FDO Toolbox, that will not work (you'll get an error message similar to: RDBMS: No open database). You have to tweak that string so it looks like this:

Driver={Microsoft Access Driver (*.mdb, *.accdb)};Dbq=C:\parcels.mdb

That will load the correct 64-bit driver.

Now I had just made this discovery after releasing the first ever 64-bit FDO Toolbox, so in the meantime to connect to MS Access databases using the 64-bit FDO Provider, choose Connect to ODBC command using the Generic data source type.

This will present a simple UI which will allow you to enter the raw ODBC connection string.

And remember that FDO Toolbox is just another FDO client application, so this method should work for any other application that uses 64-bit FDO, like: AutoCAD Map 2011 and MapGuide Enterprise 2011