Monday 14 December 2009

An interesting AU session on the internals of MapGuide

Though I didn't go to Autodesk University in person or virtually (maybe I will one day), I'm always interested in checking out the Geospatial-focused sessions that have been posted online. This year, there seemed to be the usual lot of sessions introducing MapGuide/FDO/Map3D, but one particular session caught my eye of interest:

http://au.autodesk.com/?nd=class&session_id=5241

This session is particularly special because at face value, it is not introductory in any way. You are assumed to know about what MapGuide and FDO are about. The session goes through the internals of MapGuide and demonstrates an example of using MapGuide in a standalone context (no server tier, no web tier).

This is a one of a kind presentation. I recommend checking out the slides/sample/podcast. It has certainly opened my eyes to the possibilities!

Wednesday 9 December 2009

FDO Toolbox v0.9.1

Here's a new release, with lots of useful fixes.

New Stuff
  • Added an express connection option for ArcSDE
  • The BulkCopy command in FdoUtil.exe now has a geometry flattening parameter (-flatten)
  • A "fix schema incompatibilities" option is now available when creating an SDF/SHP/SQLite files with a loaded feature schema definition.
  • You can now delete/update multiple selected rows in a Data Query result set. Same restrictions apply (eg. No identity properties in result set = Can't update/delete)
  • You can now have connection names with slashes in them, these are interpreted as folder names when saving to / loading from the session directory.
Fixed Stuff
  • When doing a partial schema save, additional checks are performed to prevent an invalid schema from being saved (eg. A unique constraint for a property that is removed)
  • Fixed the BulkCopy command in FdoUtil.exe to work with the new Bulk Copy API
  • Fixed the RunTask command in FdoUtil.exe where it was really possible for the wrong data source to be trashed if your copy tasks had DeleteTarget = true
  • Queries performed in the Data Query UI are now done using qualified class names. This should now work on data stores with multiple schemas each containing 1 or more identically named feature classes.
  • Fixed SQL Server saved connections not reloading on next startup
Other Stuff of note
  • Since we are now targeting .net Framework 2.0 SP1, I can now take advantage of some of the new C#3.0 features while still being able to target framework 2.0. As a result the project files have been migrated to Visual Studio 2008 format. You'll need msbuild from .net Framework 3.5 to build the source code if you do not have Visual Studio 2008 installed.
  • Data Query is currently broken in ArcSDE. I'm looking for a solution

Wednesday 18 November 2009

No more short coffee breaks in VS2010

As some of you may know, when you invoke the Add References dialog in Visual Studio, you might as well go for a coffee break, because the .NET and COM references take f o r e v e r to load. It is especially annoying when all you want to do is add a reference to another project from the same solution!

Well in Visual Studio 2010 (Beta 2), this is no longer the case. Invoking the Add References dialog is super fast! The reasons being:

1) .NET and COM references are now loaded asynchronously in the background.
2) The default tab is the Projects tab (which is the one I go for most of the time)

Truth be told, I would happily upgrade to VS2010 just on this one feature. But I'd probably lose out on the coffee breaks :-)

Thursday 12 November 2009

To those who got email spammed by me today...

Apologies.

My gmail account was compromised this morning by someone operating out of Beijing, China (at least that's what IP geolocation tells me).

My current theory is that it was due to me momentarily losing all semblance of sanity and reason, by falling for a Twitter phishing scam. Although I promptly changed my twitter login after discovering this fact, it must've been hunting for any similar logins and found my gmail account.

So now I'm off to change every login I conceivably know of. Regular programming will resume shortly.


Tuesday 10 November 2009

FDO Toolbox v0.9.0

Here's a new release of FDO Toolbox.

New Stuff:

1. Support for "flattening" geometries. Previously when attempting to preview spatial data with Z or M coordinates, it will fail. Now these extra coordinates are stripped away when previewing. Geometry flattening is also a new bulk copy option (regular and express).

2. The ability to see what your geometry WKT looks like. This is used in the Expression Editor.

3. Schemas, Classes and Properties are now alphabetically sorted in the Object Explorer. Makes FDO connections with 100+ class feature schemas easier to sift through.

4. The Save Schema as XML and Save Schema as SDF commands have been replaced with a new schema saving dialog. This dialog allows you to save the full schema or a subset of the Feature Schema by un-ticking the feature classes and properties you don't want saved.



Other Stuff:

1. FDO has been updated to 3.4.1 RC2. This fixes a bug that caused the faster Object Explorer loading to not work at all. Now if you have large feature schemas, they should load faster if the provider supports the enhanced IDescribeSchema command. All previous workarounds implemented up to this point have been removed (you won't notice it really).

2. We are once again using the custom gdal16.dll from the FDO binary distribution. The original motivation for using the offical gdal16.dll was for a more capable OGR provider, but bundling the official dlls turned out to cause conflicts with the PostGIS provider. If you want a more capable OGR provider, follow the instructions here.

3. IronPython has been updated to 2.0.3



Thursday 5 November 2009

My FOSS4G presentation, recorded and online

For those who could not attend FOSS4G, here's a recording of my FDO Toolbox presentation.



Many thanks to the folks from FOSSLC for recording this presentation (and many others) at FOSS4G 2009

Wednesday 4 November 2009

Why not ....

While I was walking home from work today I had a lightbulb moment.

Why isn't there a FDO provider for ... MapGuide!?

Conceptually, there is no impedance mismatch whatsoever, MapGuide builds on FDO so its knows about feature schemas, capabilities, and other FDO terminology. There is a near perfect 1:1 mapping of everything!

A MapGuide FDO provider would allow for roundtripping between a FDO client application and a MapGuide Server that would not have been previously possible.

A MapGuide FDO provider would allow you to easily pull snapshots of data from a MapGuide Server. You could bulk copy data from a MapGuide Feature Source and not care whether that Feature Source points to a SDF file or an Oracle instance, or a WFS server.

A MapGuide FDO provider would allow you to have MapGuide Feature Sources sourcing data from other MapGuide servers.

You could probably do all this through a WFS facade, but that requires configuration from the MapGuide side of things, and you would lose some of the richness of FDO by going through WFS (I know I've had to deal with things like zero-length properties from a WFS data source)

A MapGuide FDO provider wouldn't need such configuration, just feed it the required connection parameters:
  • MapAgent URL
  • MapGuide username
  • MapGuide password
  • MapGuide Feature Source Id
And you're off and racing!

There's many new possibilities with a MapGuide FDO provider. Unfortunately, I lack the C++-fu to fully explore this idea :-S

Friday 30 October 2009

Simple FDO development tip.

When you develop any application in .net in Visual Studio using the FDO API you would normally reference the following dlls from your SDK bin directory.

OSGeo.FDO.dll
OSGeo.FDO.Common.dll
OSGeo.FDO.Geometry.dll
OSGeo.FDO.Spatial.dll

However, when debugging/running the application, you need to have *all* the files from the SDK bin directory in the application's output directory, otherwise things will break.

You could create a post-build event to copy these required files, or alternatively you could do this:

Right click your project and choose Add - Existing Item



Browse to your directory that contains your FDO dlls, select everything and choose Add As Link

Create a "com" directory in your project, and repeat this process for the .sql files in the "com" directory.


Now these linked files are in your project. They should have a shortcut overlay to indicate they are linked files. Select all these linked files, and set the Copy to Output Directory property to Copy Always


Now everytime you build your project (for debugging or deployment), your application and everything in your FDO bin directory are copied to the output directory. No post-build hackery required!


Visualizing textual geometry

Wouldn't be nice to "see" what that gibberish you just typed actually looks like?

Coming in the next release of FDO Toolbox, yes you can!





Tuesday 27 October 2009

FDO Toolbox v0.8.8

Here's a short bugfix release. The installer has an updated .net framework detection routine, which should now properly work on Vista and Windows 7.

This release also includes official GDAL dlls missing from the previous release (meaning you now have a more capable OGR provider) and fixes some data preview issues with the Oracle Provider, bringing Oracle support up to a usable level.

Thursday 22 October 2009

Is FDO Toolbox an FME?

Here's a question that was thrown my way during FOSS4G:

"Are you trying to do an FME?"

The answer: Yes and No.

Yes, because like FME, FDO Toolbox can do transformation of data.

But also no, because FDO Toolbox is a multi-purpose tool, of which data transformation is one such purpose. The data transformation of FDO Toolbox is for simple scenarios (converting one data source to another) and I intend to keep it this way. FME does what it does well and can handle whatever complex scenario you throw at it.

If you need to do simple conversion or basic transformation of data, FDO Toolbox is suited for the job. If you need to do complex data transformation/massaging, then use FME which was made to serve this purpose.

Wednesday 21 October 2009

FDO Toolbox 0.8.7 (Fresh from FOSS4G)

Here's a new release of FDO Toolbox, fresh from the FOSS4G 2009 conference. This was the exact build I was demonstrating during the presentation, which went better than I expected.

Being my first presentation (ever!), I had been frantically practising the night before just to get the content and pacing right. I guess it all paid off at the end, as the presentation went quite smoothly, with none of the stuttering and loss of words that I had during my test runs :-)

The new main features include:

Enhanced Bulk Copy

The bulk copy mechanism has been drastically improved in this release, it has a whole new user interface and XML file definition. The changes are too much for this post, which I will leave for a future post.

IronPython Scripting

With the introduction of a IronPython scripting engine, FDO Toolbox now has a 2nd extensibility point. You can now customise and drive FDO Toolbox through python scripts. I will be posting some example scripts in the near future to demonstrate what can be done with a scripting engine.

The side-effect of introducing IronPython is that the minimum .net Framework requirement is now 2.0 SP1. The installer has been updated to check for this version. If you have .net Framework 3.5 (or 3.5 SP1) installed, you will already have this installed.


Note: My presentation slides are also available for download from that site. The presentation (as with every presentation at FOSS4G) has been recorded, so expect a video version to crop up soon.

Monday 19 October 2009

FOSS4G 2009 beckons.

I'll be flying off to Sydney tomorrow for FOSS4G 2009, but I won't be attending Day 1.

Instead, Tuesday will be spent fine-tuning my presentation material with the help of my Sydney-based colleagues for the main event on Wednesday, and I'll be mingling about on Thursday, finally departing mid-Friday afternoon.

Hope to see you there.

Wednesday 16 September 2009

Google Earth plays nice with MapGuide again.

Google recently released Google Earth 5.1 and one of the first things I wanted to find out was, if it would play nice with MapGuide again.

After some testing, my conclusion is that everything is fine again. So when you want to consume KML from MapGuide, stay away from Google Earth releases from 4.3 to 5.0

Monday 14 September 2009

FDO Toolbox v0.8.2(.1)

Time for a new release.

New Features / Changes:

Colorful Map Previews.



Monochromatic map previews are a thing of the past. Though if for some reason you still prefer black and white, there's a new option available in Preferences to enable/disable this

Spatial Indexing Support


Thanks to the C# Spatial Index library we now have basic spatial indexing for our datasets. Because this enables us to finally do intersection testing on our datasets, the Data Preview can now do cool things like:


Expression Editor value fetching


The expression editor has been enhanced with distinct value fetching support. So now not-only do you get property/function auto-complete, now you get value auto-complete as well!

Other Changes
  • Added a new option to view the capabilities of any FDO connection.
  • You can now attempt to fix and correct incompatibilities in the Schema Designer when operating in contextual mode. This is useful for applying schemas to connections with very restrictive capabilities (eg. Shape Files). Before it told you the schema to be applied was incompatible. Now, you can try to do something about it.
  • Fix some bugs related to fixing schema incompatibilities.
  • Fixed crashing on Data Preview for ODBC connections as a result of a half-baked attempt to work around an annoying bug. You should definitely get this release if you work with lots of ODBC connections.
  • Auto-pad zero-length string properties (to 255) when applying a schema that has zero-length string properties.
  • Added OGR connection builders for: ESRI Shape Files, ESRI ArcCoverage, Microstation DGN (v7 and older), GeoJSON, S-57, Atlas BNA and OGR Virtual Driver. You can now totally bypass using the Generic connection dialog for the OGR Provider (which for some reason the provider does not want to play nice)

EDIT: Release has been pulled because of a showstopper defect that slipped through the cracks. A 0.8.2.1 release will be made available soon that fixes this defect.

EDIT: The 0.8.2.1 release is up. Download away!

Download

Report bugs or feature enhancements

Tuesday 25 August 2009

FDO Toolbox v0.8.0

It's been a while since the last release of FDO Toolbox, so here's a new release.

This version finally has working ArcSDE provider support (thank you, Utah State Govt for providing a public ArcSDE instance for me to test :-)), faster Object Explorer loading and that new and enhanced Expression Editor that I've been raving about.

Download

Wednesday 5 August 2009

Viewing multiple selected features in the AJAX viewer

One of the limitations with the current AJAX viewer (and a pet peeve of mine) is when you do multiple selections, you get the most unhelpful user feedback ever.


Those of you in Fusion land however, get the proper treatment with regards to multiple selected features. It will display the attributes of all the selected features for you. As shown below.


To do the same thing in the current AJAX viewer requires a custom development effort, which shouldn't have to be, but sadly is the case if you want similar functionality in the AJAX viewer.

I believe functionality as basic as this should have always existed in the AJAX viewer in the first place, which is why as part of my work on MapGuide RFC71, I am porting this functionality to the AJAX viewer.



One thing you may notice is that there is a button beside the selected dropdown. What this does is it allows you to zoom and center on the current selected feature. Allowing you to somewhat wade through that sea of blue polygons in large selection sets.


This zoom level is currently hardcoded to 200, and will probably change as I keep hacking away.

Currently, this new feature is only available for the PHP viewer. The .net and Java viewers will follow in due course (the scope of the RFC covers all 3 viewers). I have tested this on Firefox 3/3.5 and Internet Explorer 7/8 (surpisingly even IE6, that #$^% browser that just refuses to die!), it should also work on Chrome and Safari (RFC scope also covers these web browsers)

If you want to play with this new feature (and I would like people's feedback as I implement this), you will have to do the following:

1) Build mapguide trunk from source.
2) Install to a test machine.
3) Rename the web extensions root (www) directory.
4) Using a subversion client, checkout (or export) a copy of the RFC71 sandbox into your www directory.
5) Copy all the files and folders from the renamed directory into the www directory except for:
  • mapviewerphp
  • mapviewernet
  • mapviewerjava
  • stdicons
  • viewerfiles
  • localized
6) Copy constants.php into the checked out mapviewerphp. When the .net implementation is done, you'll have to copy the mapguide dlls into the bin directory of the checked out mapviewernet.

For those of you currently testing the 2.1 beta. Sorry, this feature is scoped for a 2.2 release. But I may be able to find some time to backport this to 2.1 and provide this enhancement as a separate download.

Finally for those of you who do want to test drive this feature (following the above instructions). Just remember that this is a first cut. Expect things to break :-P

Monday 3 August 2009

Negative buffer analysis

Mike Schlosser has been thinking outside the box, and realised that buffering can go both ways. In his post, he demonstrates buffering using negative distances in AutoCAD Map3D.

Well just out of curiosity I wondered if MapGuide can also handle negative buffers...












Guess so.

Monday 27 July 2009

The Expression Editor gets smarter (updated)

Once in a blue moon, you have an idea for something that's so awesome, that you obsessively try to implement it as fast as you can.

I present to you, intellisense for the FDO Toolbox Expression Editor!














This new feature will be in the next release of FDO Toolbox. This is obviously a first cut, and I'm sure the auto-complete may pop up at inappropriate times, but it's quite a usable first cut. It'll definitely make building expressions much easier.

Update: 29/7: This has been ported to Maestro, and it works in Mono!

Sunday 26 July 2009

Coming to a mapguide open source installer near you...

... The ability to choose which FDO providers to install.




















This works for both installation, and change (post-installation).

The only thing left to do here is to figure out in WiX how to initially disable the ArcSDE, Oracle, MySQL and PostGIS providers (since these providers require external libraries present) so that a "default" configuration works 100% out of the box.

Wednesday 22 July 2009

fdotoolbox-users on Google Groups

I've finally created a google group for discussing FDO Toolbox. You need a Google Account to join.

http://groups.google.com/group/fdotoolbox-users

Wednesday 15 July 2009

Resource Templating in MapGuide

We've all heard about the benefits of templates and templating. It allows for reuse, consistency and standardisation. In this post, I'm going to illustrate how you can apply templating concepts to create reusable resource templates in MapGuide. Before we begin, I'm going to talk a bit about resources in MapGuide.

Resources in MapGuide are basically XML documents describing a whole host of assorted things. For example:
  • Feature Sources describe a FDO connection
  • Layer Definitions describe how a FDO feature class will be rendered
  • Map Definitions describe the layer structure and drawing order
  • Web Layouts describe the user interface elements of the AJAX viewer
  • Application Definitions (Flexible layouts) describe the user interface elements of the Fusion application
  • Symbol Definitions describe symbols
  • Load Procedures describe how a consuming application (ie. MapGuide Studio) would load a certain set of data.
These XML documents are sent back and forth from server to client (and back) via MapGuide's Resource Service API.

Consider a user who is editing a Layer Definition in MapGuide Studio or MapGuide Maestro. The following process occurs when the user opens the resource for editing.
  1. MapGuide Studio/Maestro issues a GETRESOURCECONTENT call to the MapGuide Server for a specified resource id
  2. Server returns the XML content for that resource
  3. MapGuide Studio/Maestro parses this XML content and presents this information in a layer editor user interface
Now consider the user has finished editing and saves the Layer Definition. The following process will take place:
  1. MapGuide Studio/Maestro rebuilds the XML document from the user interface.
  2. MapGuide Studio/Maestro issues a SETRESOURCECONTENT call to the MapGuide Server on that particular resource id. The XML content is sent across the wire and saved into the repository.
By understanding this basic resource editing concept, I can now illustrate how you can effectively create resource templates in MapGuide, using Web Layouts as an example.

Consider your standard AJAX viewer web layout.



Now perhaps you don't like the default toolbar/context menu layout, or perhaps you've decided on a standardised user interface for all your MapGuide applications. So naturally you would edit the web layout in MapGuide Studio/Maestro so that it matches what you're after.



Now imagine you are now creating another MapGuide application, but you need to maintain the same user interface as before. You could create a new Web Layout in Studio/Maestro and then edit it to match the previous one, or you could do the following:

1. Save the original web layout as an XML document.

Both MapGuide Studio and Maestro offer this function, you'll be asked to specify where you want to save the document.

2. Open the document in a text editor.

This is what the above Web Layout looks like (only part of the content shown to avoid a 20-page blog post :-) ):



3. Take out the map resource id and save it.

Or alternatively put some kind of textual placeholder in there so you know where to go. You now have a Web Layout resource template that can be reused for other maps.



So how do you use this template for other maps? Like so:

1. Get the resource id of the map in question

The resource id is similar to: Library://Path/To/My.MapDefinition

2. Open and Edit a copy of the Web Layout Template

Look for the map resource id element (or your textual placeholder, if you added one in) and put the above resource id in its place.



3. Save this document into the repository.

Here's where you need to understand how resource editing in MapGuide works under the hood (explained at the beginning of this post. Go on, I'll wait for you :-) )

So now the question becomes, how do you issue this SETRESOURCECONTENT call?

When Studio/Maestro are sending and receiving these commands and XML documents to/from the MapGuide Server, they are talking to a component of MapGuide known as the mapagent.

The mapagent is a public endpoint that exposes MapGuide's services to the outside world via HTTP. All MapGuide client components, whether it is Studio, Maestro, the AJAX viewer or the Fusion viewer are talking to this mapagent for most of their operations.

So now we know we have to talk to the mapagent to load our Web Layout document, but still how do we do it?

Well the mapagent comes with a rudimentary web-based user interface (rudimentary because it's designed for automated tests, not humans :-) ) that allows you to talk to the mapagent from your web browser. Assuming a standard Apache/PHP MapGuide installation, the mapagent web interface is located at:

http://yourmapguideserver:8008/mapguide/mapagent/index.html

For MapGuide Enterprise [year], it is:

http://yourmapguideserver:8008/mapguide[year]/mapagent/index.html



What we want to do now, is locate the SetResource form, and bring it up

Looking at the fields, most are self-explanatory. We just need to specify:

a) The resource id where we want to save this Web Layout
b) The path of the Web Layout document to upload



So fill these values in and click Submit. You'll be asked for a username and password, because any mapagent operation to be performed requires authentication. So fill in the username and password for the Administrator and click OK.

Once that is done, you'll get a nice blank screen which is the mapagent's way of saying success. If you have Studio/Maestro open, refreshing it will now show your web layout right where you specified.



Load the AJAX viewer with this web layout and VOILA! You'll be greeted with the same identical user interface but with a different map.



This technique can be applied to any MapGuide resource (they are all XML documents remember), but it may prove cumbersome or error-prone for certain types of resources (eg. You could have "templated" Layer Definitions with consistent zoom levels and style rules, but the feature source you're referencing better have the expected feature class and properties, otherwise things will break)

... and that is how you can create reusable resource templates in a nutshell.

Wednesday 8 July 2009

The expressive power of MapGuide Tooltips

One feature in MapGuide that some people may overlook (in terms of utility) is tooltips.


They look simple enough, you use them to display certain attributes when your mouse pointer is over a feature. But what some may not realise is how these tooltips are implemented in the AJAX (or Fusion) viewer.

These tooltips are implemented in the viewer as HTML div containers. What this means is that given some knowledge of basic HTML, you can create some really cool and powerful tooltips.

Case in point, I present to you exhibit A:



(from the SpokaneMaps.com MapGuide Application)

The HTML for this tooltip would be something similar to this (image only, I gave up on trying to escape markup in blogger):


In MapGuide Studio/Maestro this would translate as something like (image only for the same reasons):




It might look incredibly complex, but what that expression is basically doing is stringing together multiple fragments of HTML with specific feature class attributes. It's a bit cumbersome to have to use so many nested concat() calls, but the Expression Editor in MapGuide Studio can really help you here. It is also worth noting that there is going to be a cleaner concat() function in the near future, so until then this is the only way to do it.

Here's a practical example.

Google offers a free API that lets you generate dynamic charts and graphs all from a simple url. This graph below is one such example (right click and view the page source if you want)



This chart is basically a HTML img tag with the following url:

http://chart.apis.google.com/chart?cht=p3&chd=t:60,40&chs=250x100&chl=HelloWorld

If you study the url carefully, you'll see that there is a parameter asking for the values to be charted, and another parameter asking for the labels for these charted values. Now consider the Parcels layer of the Sheboygan sample dataset:





Suppose we want to represent the values of RYEAR, GEOEXTRA and RBLDGVC in a pie chart (I know this probably makes no sense from a reporting perspective, it's just for the sake of argument)

The tooltip expression for the Parcels layer would be something like:





When we now put our mouse over a Parcel, this is what we get:





Pretty cool eh? So now you may be thinking, how far can I take this?

If you know your HTML, you know that there is an element called IFRAME. Short for Inline Frame, this element allows you to "embed" other web content inside the iframe element. What this means is that, armed with the knowledge of iframes and MapGuide expressions, you can embed almost anything inside your tooltip!

I'll leave this to your imagination. But before you go too crazy on tooltips, just remember that the tooltip will always be following the mouse pointer, so it's kind of pointless to put HTML content that requires user input in there (like forms or buttons)

Happy tooltipping!
Update (19 August): Fixed the 2nd code image

Wednesday 1 July 2009

MapGuide Migration Tool (finally updated)

Those of you who remembered when MapGuide Enterprise 2007 first came out, there was also a migration tool that allowed you to convert an existing MapGuide 6.5 dataset over to an MGE 2007 repository.

However, as new yearly releases of MapGuide Enterprise came out, the migration tool hadn't been updated to match the new releases, until now.

Autodesk have finally updated their data migration tool. Now you can migrate a MapGuide 6.5 dataset to a MapGuide Enterprise 2008, 2009 or 2010 repository.

As for me personally, I don't really have a need to use such a tool. It's great for a quick-and-easy migration job, but over time you (or your colleagues) will have to maintain that spatial data, and you really want to do it without the 6.5 cruft that may still reside in your spatial data. (eg. Breaking away from the restrictive Key/Url/Name trinity)

But if you've been stuck in 6.5 land and have just gotten started with MapGuide Enterprise, it's a nice way to easily migrate data. It is also a nice way to visualise how your 6.5 data looks like in 2008, 2009 or 2010, since MapGuide 6.5 and MapGuide Enterprise are totally different (architecturally and conceptually)

Thursday 18 June 2009

Want to hear more about FDO Toolbox at FOSS4G 2009?

A few weeks ago, I submitted a presentation abstract for FOSS4G 2009.

My idea of this presentation is simply an introductory presentation about FDO Toolbox and a summary of some of the cool things you can do with it. Given the alloted 25 mins presentation time, that's probably all I can cover :-)

It's now up against 170 other submissions vying for a presentation slot at FOSS4G 2009.

If you are attending the event, and would like to hear more about FDO Toolbox, vote #1 for the abstract titled "Introduction to FDO Toolbox"

Wednesday 10 June 2009

Calling all 64-bit users...

I have put a new release of FDO Toolbox (0.7.8) that is at least known to run on Windows XP x64.

If you have a 64-bit windows box lying around, give this release a spin and let me know how it goes

Sunday 7 June 2009

FDO Toolbox v0.7.7

It's that time... for a new release of FDO Toolbox. Some of the new features:

Custom file drag and drop handler support

You can now drag and drop the following files into FDO Toolbox
  • SDF, SHP and SQLite files
  • Feature Schema files
  • Saved connection files
  • Bulk Copy Definitions
Missing 0.6 features re-implemented
  • When bulk copying form a source connection to a target connect and the target connect has no schemas defined, it will copy every source feature class.
  • RunTask command for FdoUtil.exe re-implemented
Other new features
  • Schema Editor now supports Association and Object Properties
  • You can now run non-SELECT SQL queries in the Data Query view
  • Features that failed ot be processed in an ETL task are now properly logged with exception details and a textual dump of the failed feature.
Also, this release includes some major fixes:
  • Fix copy spatial contexts for SQL Server 2008
  • Bulk Copy now properly handles multiple feature classes
  • Bulk Copy now properly handles computed expression properties
  • Protected properties are now obfuscated in the connection tooltip, to not leak any sensitive data.
  • Invalid spatial context assocations in a Feature Schema are now handled gracefully when loaded and applied from a saved definition.
Finally this includes an updated FDO 3.4.0 distribution. This should have the updated OGR provider with an important refcount fix.

It is highly recommeded to get this release if you use the bulk copy feature a lot, as this fixes some major defects in that department.


Tuesday 26 May 2009

MapGuide Open Source 2.1 beta released

It's been a long winding road up to this point, but it has finally arrived. MapGuide Open Source 2.1 is now in beta!

Get the MGOS 2.1 beta 1 installer here

So what's new in this release?

Brand New Installer

Here is main reason why 2.1 took so long to get out the door.

The first thing you will notice when installing MapGuide Open Source 2.1 is the new wix-based,OSGeo-branded installer, developed with the combined efforts of Jason Birch, Kenneth Skovhede and myself.

There is no more separate Server and Web Extensions installers, they are now both integrated into the same installer. It functions just like the old one, but with more extras. You should get a much better and more flexible installation experience than with previous releases.

Unfortunately for this release, a reboot is mandatory after installation completes. We are looking at ways to avoid this in the next release.

A side-benefit of this new installer is that we now have the infrastructure in place to produce more frequent releases, once that build server is up and running at full operational capacity, we can start to see nightly/weekly MapGuide builds for testers or those who like to live on the edge.

Also the installer is localization-ready. Translating the installer is a piece of cake.

Please note that the installer will require at least 1.5 GB of free space during installation.

Fusion 2.0

MapGuide also comes with the latest release of Fusion, which has undergone major architectural changes to not only simplify the codebase but to also have better performance. The 5 mapguide templates have also undergone a makeover in the visual department. IMO, Slate 2.0 looks damn hot!





CS-Map

MapGuide now uses the CS-Map library, instead of Proj.4 for handling coordinate systems. This library has previously only been available in Autodesk's geospatial product line (AutoCAD Map, MapGuide Enterprise). Incidentally, this is also the reason for the vastly inflated installer size, because we had to pack in 350MB of coordinate system dictionaries (around 4000 different coordinate systems).

AJAX Viewer Fixes/Enhancements

The AJAX viewer has also undergone some fixes and enhancements which although minor in the grand scheme of things, actually improve the end user experience immensely.


  • The task pane url now supports parameters. In previous versions, if your task pane url is like (mypage.php?foo=bar), the query string part would've been lost. Now those query string parameters get properly passed down. This was a bug that had annoyed me for quite some time now and I am glad to see it got fixed.
Site Administrator Enhancements

There are also some small enhancements to the MapGuide Site Administrator.
  • Packages now show file size, useful for determining how long a package would take to load.
  • The package list can now be refreshed. Before, you would've had to go to another page and then back to Package Management. Very cumbersome to say the least.
Others

The 2.1 release also has the improvements already found in the just released MapGuide Enterprise 2010. These include:
  • Greater legend flexiblity
  • Raster Re-Projection
  • Better performing schema description APIs
And from what our testers have been saying, raster stability in this release has been greatly improved as well.

Also a mountain of bug fixes have landed since the previous release (2.0.2)

So in all, this is a very solid release. I know personally that some of my long standing pet peeves have been resolved (being able to do something about them directly kind of helps :-)). So if you haven't already, download a copy and take it for a spin. This is a beta release, so bugs (if any) should be reported.

Tuesday 28 April 2009

Oh Qt, you almost had me sold

The past week and a bit, I've been getting to know more about the Qt cross platform toolkit. It's one of those software toolkits that looks like the total package. At first glance, it seems to have all the attributes I was after:
  • Multi-platform
  • High-performance
  • Rich set of features
  • Excellent documentation
  • Powerful IDE
It is one of those C++ libraries that would make we want to seriously relearn C++ :-) (I haven't really used C++ since my University days, ie. 5 years ago!)

I was about to possibly consider using this toolkit for one or more of my company's future commerical software projects, because it seems to have ticked all the right boxes...

... until I saw this legal disclaimer:
You must purchase a Qt Commercial License from Qt Software or from one of its authorized resellers before you start developing commercial software. The Commercial license does not allow the incorporation of code developed with the Qt GNU LGPL v. 2.1 or GNU GPL v. 3.0 license versions into a commercial product.
IANAL, but as a Qt newbie, this is an absolute bummer. It means if I used the Open Source edition for evaluation purposes, as a testbed for ideas and concepts (because we can't just dive in head first without testing the waters), they are forever bound under the Open Source licensing terms. I'm aware a 30-day commercial evaluation license exists, but in my line of work, 30 days flies by real quickly.

While I'm all for Open Source software, this is really disappointing in my company's interests. If I was to use Qt in my company, I would want to use it under the commercial license for peace of mind (even though LGPL would probably be safe enough). To me, that statement reads more like:
If you decide to purchase a Qt commercial license, you better damn well know what you're going to do with it.
Operating essentially as a "one-man band" developer in my company, it's quite hard to justify a purchase of a commerical license to "learn" this toolkit which at the end of the day, may not do what I was originally hoping it would do. It is even harder in this economic climate (It's cliché I know).

This would be like saying, if you use MapGuide Open Source to serve your maps and spatial data, you can't upscale to MapGuide Enterprise (or even Topobase) in the future. Others may not think this is the most suitable analogy :-)

I may have come off as ranting a bit, but I'm not really. I guess I'll have to shelve my plans for using Qt in my company for now (or convince them it is safe to use LGPL libraries). Maybe I'll come back in a year or two, when I'm no longer a Qt newbie and can dive into development straight away.

But for my future personal projects? I'm still waiting for Qt to finish compiling.

Monday 27 April 2009

Using the Data Query view in FDO Toolbox (part 2): CRUD operations

Read Part 1 here.

One of the new features in the recently released FDO Toolbox is the CRUD (Create, Read, Update, Delete) feature. This feature finally exposes the full read/write capabilities of the FDO API to whatever providers that support such capabilities (nearly all bundled FDO providers support the full quartet of IInsert, ISelect, IUpdate and IDelete commands).

Part 1 covered the R part of CRUD (and the many ways to do it). This part will cover the rest of the quartet.

Creating (Inserting) new features

If the underlying FDO provider supports the IInsert FDO command, then you can use FDO Toolbox to insert new features into that particular data store. There are two ways to insert a new feature, you can either right click the feature class (to insert into) from the Object Explorer and choose Insert New Feature:


Or you can go into Data Query, select the feature class (to insert into) and click New Feature.
Either command will take you to the following dialog which will allow you to enter the attributes and geometry of this new feature:

Any property/attribute with a greenish background is an optional attribute (ie. In FDO terms, it is a nullable property). To enter geometry values, specify the FGF textual form of the geometry.

Unlike SQL INSERT commands, FDO does not actually support computed expressions for insert values. For example, you can't use the CurrentDate() expression function as a value for a datetime property. This is a limitation of the FDO API, that may hopefully be addressed in the future.

If the FDO provider supports transactions, the Use Transactions check box will be available for you. Once you have entered this information, click Insert to insert the new feature.

Updating existing features

If the underlying FDO provider supports the IUpdate FDO command, then you can use FDO Toolbox to update existing features in a particular data store.

To update existing features, perform a query using the Data Query view in Standard Mode, and then right click one of the results and choose Update this Feature. It will perform some checks to determine if the resulting update will only affect the selected result. If the checks pass, you will get a dialog similar to the Insert one:

Once again, the greenish fields are optional and geometry values must be in FGF text format. Again, the Use Transaction check box is available if the FDO provider supports transactions.

Unlike SQL UPDATE commands, FDO also does not actually support computed expressions for update values. For example, you can't use the CurrentDate() expression function as a value for a datetime property. Or you cannot set a string property (eg. Name) equal to the expression Translate(Name, 'a', 'o'). This is once again, a limitation of the FDO API, that may hopefully be addressed in the future.

Make your changes and click Update to update that feature. You can verify the feature has been updated by re-running the same query, the result that was edited will have been updated.

Deleting features

If the underlying FDO provider supports the IDelete FDO command, then you can use FDO Toolbox to delete existing features in a particular data store.

To delete existing features, perform a query using the Data Query view in Standard Mode, and then right click one of the results and choose Delete this Feature. It will perform some checks to determine if the resulting delete operation will only affect the selected result. If the checks pass, you will get asked for confirmation.

Click Yes to delete the selected feature. You can verify the feature has been deleted, by re-running the same query, the selected feature will no longer be in the query result set.

Bulk Updating features


WARNING: This is a dangerous operation. If you are not careful with the update filter specified, it could cause irreversible data changes! An empty filter, will cause bulk updates to be applied to all the data!

If the underlying FDO provider supports the IUpdate FDO command, then you can use FDO Toolbox to bulk update a series of features. When used carefully and properly, this is a useful and powerful way to change or correct certain attributes in a feature class.

To perform a bulk update, right click a feature class object in the Object Explorer and choose Bulk Update. Because of the dangerous nature of this operation, you will be asked for confirmation.


Click Yes and you will get a dialog like the one below:

The beauty of this dialog, is that only the checked properties will be applied in the bulk update, this allows for a great degree of flexibility in your bulk updates.

If the FDO provider supports transactions, the Use Transactions check box will be available.

Specify the update filter for this bulk update. This is important, because an empty filter will apply your updated values to all the features in the feature class! You can test the update filter by clicking Test. This will check how many features the bulk update operation will affect.

Once you are comfortable with the test results, click Update to do the real thing. Once again, due to the dangerous nature of this operation, you will be asked for confirmation. Say Yes to perform the bulk update.

Bulk Deleting features

If bulk update is Little Boy, then bulk delete is the Tsar Bomba. With that in mind...

WARNING: This is an extremely dangerous operation! If you are not careful with the delete filter specified, it could cause irreversible data loss! An empty filter, will cause the delete operation to delete everything in the feature class! You have been warned ;-)

If the underlying FDO provider supports the IDelete FDO command, then you can use FDO Toolbox to bulk delete a series of features from a data store.

To perform a bulk delete, right click the feature class object in the Object Explorer and choose Bulk Delete. Due to the potential for butchering your data, you will be asked for confirmation.


Click Yes and you will get the bulk delete dialog:


If the FDO provider supports transactions, the Use Transaction check box is available.

Unless you want to delete everything, specify a delete filter, just like the bulk update dialog, you can test to see how many features would be deleted by clicking the Test button.


Once you are certain everything is alright, click Delete to perform the actual delete operation. Just in case you have second thoughts, you will be asked for confirmation again. Say Yes to finally start performing the bulk delete operation. The time it takes for the operation to complete depends on number of features affected (using the Test function)

And that wraps up the CRUD feature of FDO Toolbox. Watch this space for further showcases of FDO Toolbox's features.