The Map Guy(de): March 2015

Tuesday, 31 March 2015

MapGuide 3.0 feature showcase: Tile Service Enhancements

If I were to pick the most compelling feature of MapGuide Open Source 3.0, it would be the one I am about to show you in this post.

Do you bemoan the fact that tiled maps are tied to a given Map Definition preventing their re-use with other Map Definitions?

There are ways to work around this problem, by using the "hack" of stacking multiple Map Definitions in a Fusion Application Definition. In this case, you would have a purely tiled Map Definition and you combine this with a purely dynamic Map Definition in your Application Definition and provided both maps are in the same coordinate system, both tiled and dynamic layers will line up.

You can then reuse this tiled map definition with other Fusion applications by mixing and matching tiled and dynamic Map Definitions together. However as already mentioned, this is a hack because it exploits an esoteric feature of Fusion: The ability to have more than one Map Definition specified in a Fusion MapGroup. This technique doesn't work with the AJAX viewer.

For the 3.0 release, we have solved this problem by introducing a new resource type to define shareable tile sets: The TileSetDefintion.

For a cliff notes summary: A TileSetDefinition is a re-usable tile cache that is effectively the BaseMapDefinition element of the Map Definition Schema separated out into its own resource. A TileSetDefinition lets you:

Define one or more base layer groups, each with 0 or more layers
Define a finite set of scales
Define the coordinate system of this tile set
Define the bounds of this tile set

Basically the same tiled layer settings you would've previously defined in a Map Definition. In addition a TileSetDefinition also lets you define additional settings that are specific to that tile set:

Tile size
Tile image format
Tile storage location

No longer are your tile sets constrained by global tile settings in serverconfig.ini and no longer are your rendered tiles always stored in a global defined tile cache directory. Each tile set can have their own tile size, format, storage location.

Here's an example of such a Tile Set Definition, notice the structural similarity to the Base Map section of the Map Definition

<?xml version="1.0" encoding="utf-8"?>
<TileSetDefinition xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xsi:noNamespaceSchemaLocation="TileSetDefinition-3.0.0.xsd">
  <TileStoreParameters>
    <TileProvider>Default</TileProvider>
    <Parameter>
      <Name>TilePath</Name>
      <Value>%MG_TILE_CACHE_PATH%</Value>
    </Parameter>
    <Parameter>
      <Name>TileWidth</Name>
      <Value>256</Value>
    </Parameter>
    <Parameter>
      <Name>TileHeight</Name>
      <Value>256</Value>
    </Parameter>
    <Parameter>
      <Name>TileFormat</Name>
      <Value>PNG</Value>
    </Parameter>
    <Parameter>
      <Name>FiniteScaleList</Name>
      <Value>200000,100000,50000,25000,12500,6250,3125,1562.5,781.25,390.625</Value>
    </Parameter>
    <Parameter>
      <Name>CoordinateSystem</Name>
      <Value>GEOGCS["LL84",DATUM["WGS84",SPHEROID["WGS84",6378137.000,298.25722293]],PRIMEM["Greenwich",0],UNIT["Degree",0.01745329251994]]</Value>
    </Parameter>
  </TileStoreParameters>
  <Extents>
    <MinX>-87.764986990962839</MinX>
    <MaxX>-87.695521510899724</MaxX>
    <MinY>43.691398128787782</MinY>
    <MaxY>43.797520000480347</MaxY>
  </Extents>
  <BaseMapLayerGroup>
    <Name>Base Layer Group</Name>
    <Visible>true</Visible>
    <ShowInLegend>true</ShowInLegend>
    <ExpandInLegend>true</ExpandInLegend>
    <LegendLabel>Tiled Layers</LegendLabel>
    <BaseMapLayer>
      <Name>Roads</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Roads.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Roads</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Districts</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Districts.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Districts</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Buildings</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Buildings.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Buildings</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Parcels</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Parcels.LayerDefinition</ResourceId>
      <Selectable>true</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Parcels</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Islands</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Islands.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Islands</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Hydrography</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Hydrography.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Hydrography</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>CityLimits</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/CityLimits.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>CityLimits</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
  </BaseMapLayerGroup>
</TileSetDefinition>

Finally, there's the re-usability aspect. Once a TileSetDefinition has been created, you can link to it from any Map Definition. Tile caches are now bound to the TileSetDefinition instead of the Map Definition. When a Map Definition links to a TileSetDefinition, it will override the following settings from the Map Definition:

Coordinate System
Map Extents

The reason we do this is because dynamic map layers can be re-projected, but tiles cannot. So in the event that the Map Definition has a different coordinate system from the one in the linked tile set, the linked tile set "wins" and all the dynamic layers in the Map Definition will be re-projected to the coordinate system of the tile set, ensuring that both dynamic and tiled layers will line up properly.

And the best thing about this is that both AJAX and Fusion viewers support shareable tile sets*. The above multiple map hack is no longer needed.

* For this release, this will only work with shareable tile sets using the "Default" tile provider. Why? Read on below.

XYZ Tile Support

The TileSetDefinition schema allows tile set settings to be defined in a key/value fashion like you would in a Feature Source. This schema design is intentional as it allows us to define a FDO-style provider model behind the scenes for tile access. Through this provider model, we are able to add support for different tile access schemes, one of which is XYZ tiles. This means you no longer need mapguide-rest to serve XYZ tiles. MapGuide 3.0 will have this support natively.

Here's an example of a Tile Set Definition using the XYZ tile provider

<?xml version="1.0" encoding="utf-8"?>
<TileSetDefinition xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xsi:noNamespaceSchemaLocation="TileSetDefinition-3.0.0.xsd">
  <TileStoreParameters>
    <TileProvider>XYZ</TileProvider>
    <Parameter>
      <Name>TilePath</Name>
      <Value>%MG_TILE_CACHE_PATH%</Value>
    </Parameter>
    <Parameter>
      <Name>TileFormat</Name>
      <Value>PNG</Value>
    </Parameter>
  </TileStoreParameters>
  <Extents>
    <MinX>-9770571.93250815</MinX>
    <MaxX>-9755615.48593707</MaxX>
    <MinY>5416573.69002144</MinY>
    <MaxY>5436091.17493748</MaxY>
  </Extents>
  <BaseMapLayerGroup>
    <Name>Base Layer Group</Name>
    <Visible>true</Visible>
    <ShowInLegend>true</ShowInLegend>
    <ExpandInLegend>true</ExpandInLegend>
    <LegendLabel>Tiled Layers</LegendLabel>
    <BaseMapLayer>
      <Name>Roads</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Roads.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Roads</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Districts</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Districts.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Districts</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Buildings</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Buildings.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Buildings</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Parcels</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Parcels.LayerDefinition</ResourceId>
      <Selectable>true</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Parcels</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Islands</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Islands.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Islands</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Hydrography</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Hydrography.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Hydrography</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>CityLimits</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/CityLimits.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>CityLimits</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
  </BaseMapLayerGroup>
</TileSetDefinition>

Notice that the XYZ tile set definition does not specify a coordinate system. This is because XYZ tile sets are always in WGS84.PseudoMercator (aka. EPSG:3857), so there is no need to specify this. The extents describe the bounds of this tile set in EPSG:3857 coordinates.

You'll notice in both examples, that we have the concept of a tile provider along with a set of key/value pairs to describe settings applicable for that provider, which is similar to the how a Feature Source specifies a FDO provider and a set of key/value pairs that form the set of FDO connection properties.

And what is this %MG_TILE_CACHE_PATH%? It is a placeholder for the global server tile cache path, and just like %MG_DATA_FILE_PATH% for feature sources, you can replace this with actual physical paths or MapGuide aliases. That's right, your Tile Set Definitions can have their own defined tile storage location instead of being all under a globally defined path in serverconfig.ini

As you can see, the similarity between tile providers and FDO was quite intentional :)

This provider model is currently hard-coded for these 2 providers. There is no formal plugin model/API yet that allows for external third-party tile providers to be defined. Whether we will add such support in a future release is a discussion for another time.

The best thing about this XYZ support: No API changes required! You still use the same GETTILE requests to the mapagent, only now you pass in the actual X, Y and Z for the ROW, COL and SCALE parameters.

Now there is one limitation of the XYZ support for this release. You cannot create MgMap instances from a Map Definition that links to an XYZ tile set, and as a result you cannot load such Map Definitions in the AJAX and Fusion viewers. The main use case for XYZ tile support in this release is to enable easy consumption of XYZ tile sets in MapGuide by external clients like OpenLayers, Leaflet, etc. We'll correct this limitation for the next major release after 3.0.

With this release, there's a new sample you can check out that demonstrate how to consume XYZ tiles from OpenLayers.

When you look at the code, you'll find consuming XYZ tile sets from MapGuide is as simple as consuming OpenStreetMap and friends in your JS mapping library of choice, be it OpenLayers, Leaflet or any other library that can consume XYZ tile sets.

Now that our tile storage and access has been abstracted behind a Tile Set Definition resource, it paves the way for us to easily enhance the tiling capabilities in the future with:

Support for additional storage backends (eg. MBTiles)
Support for additional tile formats (eg. Vector tiles, UTFGrid)
Standardizing tile access (eg. WMTS)

That's not to say such items are definitely on our current roadmap, but it will make implementing such features a more realistic and feasible possibility now that the groundwork has been laid in this release.

For more information about these new Tile Service Enhancements, you can consult MapGuide RFC 140

Monday, 30 March 2015

Announcing: MapGuide Open Source 3.0 Beta 1

After a month or two lost due to mucking around with packer and vagrant to get the perfect repeatable and disposable build environment for both Windows and Linux builds of MapGuide/FDO, I'm now happy to announce the availability of MapGuide Open Source 3.0 Beta 1.

Who knew debugging build infrastructure would be such a PITA? And who also knew that vagrant and windows is like oil and water? Ugh! I gave up on that part, so as it stands our current build infrastructure for putting out releases of MapGuide is still Jenkins for windows and (slightly improved) vagrant for Linux.

Stay tuned for the (promised) series of posts talking about what's new in MapGuide Open Source 3.0

Download

Friday, 6 March 2015

Announcing: mapguide-rest 1.0 RC1

I've decided that mapguide-rest has enough features to lock in the 1.0 version number.

Here's what's new in this release.

Cleaner JSON responses

If you've used JSON output from mapguide-rest or MapGuide proper, you'll notice that the JSON is ugly and cumbersome to use:

Every JSON property is an array
All property values are strings

The main reason this JSON is so ugly is because of the way that XML is converted to JSON. The conversion process knows nothing about the content model of the XML document it is trying to convert, so as a lowest-common-denominator solution, it forces all converted XML elements to JSON array properties and all XML text elements as string values. While this allows for rudimentary usage by client applications, it's not a pleasant experience.

For this release, we've set out to solve this problem. The JSON produced which would've looked like this:

Now looks like this:

How did we achieve this? Simply put, we now maintain a hard-coded list of known XML element paths (compiled from all the MapGuide XML schemas) and make note of:

Which paths must always be processed as a JSON array
Which paths have values whose data type is not a string

With this list, we now know which XML elements need to be converted to JSON arrays and which elements need to be converted to JSON property values, and the proper data type to convert such values to.

The end result is that the new JSON responses are much more natural and intuitive to work with. Not to mention these new JSON responses are smaller than their previous counterparts. Those extra [ and " characters do add up.

This is a PHP-level solution. With this release, all JSON conversion/output is now handled by mapguide-rest. While slower than doing it natively in MapGuide, it gives us more usable JSON structures and this increased usability is worth it for the slower performance.

As for the old JSON response structure, that is going away in mapguide-rest. This new structure will be used for all APIs in mapguide-rest that return JSON. If you want the old JSON structured responses, you'll have to go back to the mapagent.

In one particular case, we eschew the convert-to-JSON-from-XML approach and define a new dedicated JSON response format.

Ever tried to describe the schema of a Feature Source? This is the XML response:

Have you seen the JSON version of this monstrosity?

How on earth would any sane person know how to use this response in their applications? Yeah me neither.

So for schemas and class definitions that are returned as JSON, we introduce a more cleaner and intuitive JSON format

Isn't that a more intuitive and self-explanatory JSON response?

This new response structure is only applicable for JSON schemas and class definitions. The XML version of these schemas and class definitions will still be its same ugly and verbose self. From my perspective, XML is the "legacy" response format in mapguide-rest. JSON is the preferred response format for any client application and so we should strive to make consuming such data in this format as simple and clean as possible. With this release, hopefully we've achieved that objective.

While we're still on the subject of JSON, for certain REST APIs that previously accepted XML request bodies, such APIs now support JSON-equivalent request bodies. For example, consider this route to set resource content:

POST /rest/library/{resourcePath}/content.xml

To set resource content, you'd make a POST request to this route and include the resource XML as part of the request body.

You can now set the same resource content via its JSON representation

POST /rest/library/{resourcePath}/content.json

By making a POST request to this route and include the equivalent JSON as part of the request body. Because our JSON formats now structurally follow their XML counterparts, knowing what JSON to construct should be easy to figure out. On the server-side such JSON bodies will be converted back to its XML counterpart before passing down to the underlying MapGuide APIs (because sadly XML is still the canonical format for storage and data transfer throughout the official MapGuide APIs)

Authoring APIs for restcfg.json files

We've added a set of new APIs in the data publishing framework to allow for restcfg.json and any supporting files to be uploaded from a mapguide-rest client without needing physical access to the directory where the restcfg.json files are stored.

These authoring APIs allow you to:

Upload a restcfg.json file to a desired subdirectory within mapguide-rest based on the given URI part
Upload any supporting files to the subdirectory mapped to the given URI part
Delete any restcfg.json configuration
Delete any supporting file of a restcfg.json configuration

Because URI parts are mapped to physical subdirectories within a mapguide-rest installation, there could be a opportunities for malicious activity without proper security precautions. To this effect, we do the following:

Such APIs require Administrator or Author-level roles to be assigned to the authenticated user. The MapGuide Anonymous users will not have such roles, and thus any attempt at accessing these APIs will be denied.
Any URI parts and filename parameters are sanitized to strip off any semblance of relative paths and any other path navigation tricks that would cause mapguide-rest to save/delete a file or directory outside of its installation directory.

You can find out how to use these new APIs in the built-in interactive API reference.

API additions and cleanups

APIs that previously returned simple primitive values now return a XML or JSON boxed version of that value. Simply put, if an API returned a boolean value like this:

true

It will now return a boxed value, which looks like this for XML:

And looks like this for JSON:

{ "PrimitiveValue": { "Type": "Boolean", "Value": true } }

While this looks more verbose, it is simply to provide a consistent developer experience. If you have been expecting XML or JSON responses for 99% of the APIs, why make an exception for this 1% of APIs that return basic primitive values? They should be XML or JSON respectively as well and in order for that to happen, we have to box such values.

In addition many new APIs have been added to cover missing pieces and loose ends in our REST API:

Creating Feature Sources
Setting and deleting resource data items
Listing Long Transactions of a supported Feature Source
Getting the coordinate system base library
Validating Coordinate System WKT
Converting Coordinate System WKT to EPSG codes
Testing the connectivity of a Feature Source
Enumerating unmanaged (aliased) data
Getting site information status/statistics
DescribeSchema with class name hint
Setting content and header of any resource in a single request
Fusion layout authoring support APIs
Interrogating and setting the edit-ability of a Feature Source with the REST API

You can find out how to use these new APIs in the built-in interactive API reference.

(Geo)JSON CRUD support in restcfg.json

The data publishing framework now supports CRUD for JSON representations. The JSON representation is much more simpler and compact in its request payloads. The requests to create, update and delete features are structurally the same as the XML one (using the same XML-to-JSON pattern for most of our JSON data structures)

The existing Editable Properties example has been modified to use the JSON representation instead of the XML one. For doing CRUD through the data publishing framework, I'm sure you'll find it much easier to work with this JSON representation than the XML one.

Customizable HTML representation

The HTML representation for features are now customizable and localizable as the previous bits of hardcoded HTML that were rendered out by the HTML renderer, have all been extracted out into Smarty templates, where you can customize the HTML to your own liking.

MapGuide 3.0 support

This release of mapguide-rest supports new features introduced in MapGuide Open Source 3.0.

Your tile service routes can also work with Tile Set Definitions
XYZ tile requests will pass through to the MapGuide API on a MGOS 3.0 install
Runtime Map responses use the v3.0.0 response schema

Other changes

A new separate log_config.php lets you control logging behaviour in mapguide-rest
XYZ tile requests now work on Map Definitions with multiple tiled groups
When plotting a layered PDF, layers are no longer locked.
mapguide-rest now gracefully operates when you don't have a URL rewriting module installed.

Where to from here?

From now to the final 1.0 release of mapguide-rest, it will be nothing but bug fixes and (really) minor enhancements.

After 1.0 as I've somewhat hinted, if mapguide-rest as a project should have a continued evolution, I strongly believe it should be on the ASP.net 5.0 platform. Given ASP.net 5.0 won't see light for many more months, I'll have plenty of time to play and experiment with many ideas that are possible with ASP.net 5.0

Download

The Map Guy(de)

Tuesday, 31 March 2015

MapGuide 3.0 feature showcase: Tile Service Enhancements

Monday, 30 March 2015

Announcing: MapGuide Open Source 3.0 Beta 1

Friday, 6 March 2015

Announcing: mapguide-rest 1.0 RC1

Blog Archive

Labels

My Blog List

The Map Guy(de)

Tuesday, 31 March 2015

MapGuide 3.0 feature showcase: Tile Service Enhancements

Monday, 30 March 2015

Announcing: MapGuide Open Source 3.0 Beta 1

Friday, 6 March 2015

Announcing: mapguide-rest 1.0 RC1

Blog Archive

Labels

Subscribe To

My Blog List