Friday, 19 June 2015

For MapGuide, OpenLayers 3 is ready for prime-time.

As some of you may recall, nearly two years ago I landed in MapGuide support for OpenLayers 3.

However, this support was partial as it only targets un-tiled Map Definitions. None of the tiling mechanisms in OpenLayers 3 were compatible with the tiling scheme used by MapGuide meaning OpenLayers 3 cannot consume non-XYZ tiles from MapGuide ...

... Or so I thought.

It turns out that it is indeed possible to consume non-XYZ tiles from MapGuide, here's one such example using the just released v3.6.0 of OpenLayers.

What this means is that OpenLayers 3 is at feature parity with OpenLayers 2 in terms of MapGuide support, meaning it has the core functionality we need to build a MapGuide client mapping application out of.

Many thanks to Andreas Hocevar of Boundless for the code sample.

MapGuide tidbits: SQLite interoperability

Sometimes, your data has to escape the MapGuide/FDO ecosystem and be consumed from other geo-software. If the limitations of the omnipresent SHP file format leave a lot less to be desired, then the next best thing is SQLite databases.

The reason I say this is because:

  • SQLite files (in our context) are just regular SQLite database with some special tables so mapping software know how to read features from them. You don't need FDO to peek inside the structure of a SQLite database. Any other SQLite database tool or library can get at the data as well.
  • SQLite is a relational(-ish) database. So features you'd expect in an RDBMS (eg. views) are also available in SQLite. Can you say that about SHP files?
  • The biggest reason. There's an OGR driver for SQLite, and it understands FDO's flavor of SQLite databases, and if your mapping software isn't using FDO, it's using GDAL/OGR!
Sure, there's a few quirks here and there, but they can be worked around or fixed.

MapGuide tidbits: Fixing SQLite data files with no coordinate systems

From the "I lost several hours discovering this so you don't have to" department

Today, I was tasked with producing some KML files. The spatial data was in SQL Server and it needed to be married with some CSV files I had, but I wanted it to get into a more portable format so I can do the bulk of the styling work in MapGuide without having to touch the SQL Server again.

Due to some quirks in this SQL Server database, the SQLite file that were produced from an FDO Toolbox bulk copy did not have any spatial contexts in them. This turns out to be a big problem because MapGuide has to know what coordinate system your data is in for rendering and various outputs to work, especially if coordinate system transformation is required. You can fix and/or flub such a coordinate system using the Coordinate System override section of the Feature Source, but this only works if MapGuide/FDO can read a spatial context from the data store for you to stick in a replacement coordinate system in.

This actually broke the KML support in MapGuide I was intending to use because it needs to know if your data is LL84 or transformable to it and since the SQLite file had none, MapGuide didn't do anything with the data when I wanted a KML export of it. Rather than waste time re-exporting this data in another format, I wanted to see if such files can be "fixed" in-place and be made usable again.

Since SQLite files are actual relational(-ish) databases, you can open them any SQLite database viewer^ (there's a lot of them) to take a look at the contents. When you open such a file, you'll see various tables representing each of your feature classes and some special tables:

  • fdo_columns
  • geometry_columns
  • spatial_ref_sys
fdo_columns contains the FDO property metadata for each column of each table. Without this table, the SQLite FDO provider will treat all columns as strings, even if your column stores numbers, dates, etc.

geometry_columns contains the metadata for the geometry columns.

spatial_ref_sys contains coordinate system information.

In the event that MapGuide reads 0 spatial contexts from your SQLite data store, it's because:
  • There are no rows in spatial_ref_sys. Each row in this table is read as a spatial context
  • Any rows in geometry_columns either have an srid of -1 or an srid with no match in spatial_ref_sys
So with this knowledge, to "fix" such files so MapGuide knows what coordinate system they're in:
  • Insert a new row into spatial_ref_sys
  • Update the srid column of all rows in geometry_columns to the srid of the row you inserted into spatial_ref_sys
Or if none of what I just said made sense, just run the commands in this gist in your SQLite database admin tool of choice. Once done, your SQLite database will have a spatial context which you can then put an override in your Feature Source so that MapGuide knows what coordinate system to interpret your data in.

^ Though FDO Toolbox can technically be used to admin SQLite databases, I am suggesting alternative tools as I am not sure whether the SQLite FDO provider will let you "fix" something that it has an already-open connection to, nor do I know if the SQL commands the fix this will work as intended when going through the SQLite FDO provider. Better to be safe than sorry.

    Tuesday, 16 June 2015

    Announcing: MapGuide Open Source 3.0 RC1

    With no real show-stopping bugs of note, I've decided to put out the first release candidate of MapGuide Open Source 3.0

    RC1 rolls back a problematic change introduced in beta 2 that resulted in a feature reader from SelectFeatures query not returning all results in the reader. Other than that, this contains some minor fixes for Fusion and the SDF and SQLite FDO providers. All the details can be found the release notes below


    Thursday, 11 June 2015

    Announcing: mapguide-rest 1.0 RC2

    Here's the second release candidate for mapguide-rest 1.0

    The main addition in this release is support for defining a "whitelist" in config.php that will restrict feature service APIs to only the specified operations and feature sources defined in the whitelist, allowing you to reduce the attack surface of your mapguide-rest install by only controlling API access to feature sources of your choosing. You can find a commented-out whitelist example in config.php that you can adapt to your own installation.

    Other changes and fixes include:
    • Fix runtime map XML generation for versions of MapGuide that do not support CREATERUNTIMEMAP
    • Fix XYZ routes for Tile Set Definition resources
    • Support a bboxistargetcs flag that allows the bbox parameter of any feature query route to be declared in the coordinate system indicated by the transformto parameter
    • Display of error response stack traces can now be toggled in config.php
    • Fix invalid GETMAPIMAGE operation version in map image adapter
    • When doing a search result query via the data publishing framework, if the query produces a single result, the "single" view template will be shown instead of the "many" view template
    • When doing a search result query via the data publishing framework, if the query produces no results, the "none" view template will be shown instead of the "many" view template
    I'm thinking there will probably be at least one more RC after this as there is still some more attack surface reduction work to do before I can confidently slap a 1.0 final label on it.


    Tuesday, 12 May 2015

    Announcing: MapGuide Open Source 3.0 Beta 2

    Here's the 2nd beta of MapGuide Open Source 3.0

    One of the most apparent things you'll notice as you download this release is ... the size.

    As an experiment (since we're still in beta), we've excised some USA-specific coordinate system grid files and un-used Geoid height files from our installation packages. As a result our installation packages are up to 150MB lighter than they were previously.

    Our test suites indicate nothing is affected by the omission of these files except for if your spatial data uses a US-based coordinate systems based on an NSRS datum. Unless your data is based in the US and you have strict data requirements that require such a coordinate system be used for your data and maps, you will probably be unaffected by the omission of these files, so why carry an extra 150MB download burden for something you'll never use amirite?

    But don't fret, these grid files are made available as a separate package for download as a supplemental package should your installation of MapGuide require them. However, if this change does break functionality in your installation (and you're not using a US-based coordinate system based on a NSRS datum), we'd like to know about it.

    If things go smoothly with this change, we'll most likely extend this installer weight-loss program to the next (long overdue) 2.6.1 point release as well.


    Friday, 24 April 2015

    Custom GDAL binaries for MapGuide Open Source 2.6 and 3.0

    A question that gets normally asked on our mailing list is how do you get the GDAL FDO provider to work with formats like ECW or MrSID. Our normal response would be (provided you are licensed to use ECW, MrSID or any other non-standard GDAL supported format) to point you over to GIS internals and grab one of their custom windows GDAL binaries to replace the GDAL dlls in your current MapGuide installation.

    The reason we ask you to do this is because when we build GDAL for use with the FDO provider, we build GDAL using only the standard profile of supported formats. That is to say any format listed here:
    Where the Compiled by default option is unconditionally Yes. It is not possible for us logistically to build GDAL/OGR with the proverbial kitchen sink of raster/vector format support, so that's where GIS internals comes in as their builds of GDAL/OGR have greater raster/vector format support. As long as you grab the same release of GDAL and make sure to pick the build that is built with the same MSVC compiler used to build the release of MapGuide/FDO you're using, you should then have GDAL and OGR FDO providers with expanded vector and raster format support.

    This suggestion worked up until the 2.5.2 release, where the right version of GDAL built with the right version of MSVC (2010 at the time) was available for download. But for 2.6 and the (pending) 3.0 releases, this suggestion is not applicable because that site does not offer a MSVC 2012 build of GDAL 1.10, which is what MapGuide 2.6 and 3.0 both use for their GDAL FDO provider.

    So this leaves some of you in a pickle, being stuck on 2.5.2 and unable to move to 2.6 or 3.0 because you need to support one of these esoteric data formats. Well, I have partially alleviated this issue for you.

    Tamas has not only made these custom GDAL binaries for download, but also the development kits used to build these binaries as well. So in these past few days, I grabbed the MSVC 2012 dev kit, paired it with our internal GDAL 1.10 source tree in FDO and made a few tweaks to some makefiles here and there and here's the end result.

    A custom build of GDAL 1.10 with support for the following additional raster data formats:
    • ECW (rw): ERDAS Compressed Wavelets (SDK 3.x)
    • JP2ECW (rw+v): ERDAS JPEG2000 (SDK 3.x)
    • FITS (rw+): Flexible Image Transport System
    • GMT (rw): GMT NetCDF Grid Format
    • netCDF (rw+s): Network Common Data Format
    • WCS (rovs): OGC Web Coverage Service
    • WMS (rwvs): OGC Web Map Service
    • HTTP (ro): HTTP Fetching Wrapper
    • Rasterlite (rws): Rasterlite
    • PostGISRaster (rws): PostGIS Raster driver
    • MBTiles (rov): MBTiles
    And support for the following additional vector formats:
    • "PostgreSQL" (read/write)
    • "NAS" (readonly)
    • "LIBKML" (read/write)
    • "Interlis 1" (read/write)
    • "Interlis 2" (read/write)
    • "SQLite" (read/write)
    • "VFK" (readonly)
    • "OSM" (readonly)
    • "WFS" (readonly)
    • "GFT" (read/write)
    • "CouchDB" (read/write)
    • "ODS" (read/write)
    • "XLSX" (read/write)
    • "ElasticSearch" (read/write)
    • "PDF" (read/write)
    You might notice some omissions from this list. Where's MrSID? Where's Oracle? Where's $NOT_COMPILED_BY_DEFAULT_DATA_FORMAT?

    Well I did say I have partially alleviated the issue and not fully alleviated it. The issue is that due to what I gather is licensing restrictions, the development kit can't bundle the necessary headers and libraries needed to build GDAL with driver support for MrSID, OCI, etc. As such the custom build of GDAL I have made available does not include support for such formats.

    What can be done about this. For something like Oracle, we already have a dedicated FDO provider for that. For something like MrSID? I'm afraid you're out of luck. You'll either have to stick on the 2.5.2 release for that much longer, or just bite the bullet and gdal_translate those MrSID files to something more accessible. I've heard some good things about RasterLite. I've also heard that you could get some great performance out of carefully prepared GeoTiffs

    Any thing to liberate yourself from MrSID because you won't see the right GDAL binaries with this support built in for the foreseeable future.

    You can find the download links for the custom GDAL builds in our updated GDAL provider guide for MapGuide 2.6 and 3.0.

    One more thing before I finish that is worth re-iterating. Some formats like ECW, require you to be have a license to use ECW technology in a server environment. Other formats carry their own licensing shenanigans, so make sure you are properly licensed to use any of the the additional formats that are made available with this custom build of GDAL. The GISInternals build system repo on GitHub has all the applicable licenses in RTF format for your perusal.

    Also worth pointing out is that this custom build of GDAL is not supported by me or anyone on the MapGuide development team. I only make this build available to you so you have the ability to access these additional data formats should you so choose and nothing more. There is no obligation by us to provide support for any issues that may arise as a result of using this custom GDAL release (inferred or otherwise). Use this custom build of GDAL at your own discretion.

    /end lawyer-y talk. Enjoy!