Friday, 24 April 2015

Custom GDAL binaries for MapGuide Open Source 2.6 and 3.0

A question that gets normally asked on our mailing list is how do you get the GDAL FDO provider to work with formats like ECW or MrSID. Our normal response would be (provided you are licensed to use ECW, MrSID or any other non-standard GDAL supported format) to point you over to GIS internals and grab one of their custom windows GDAL binaries to replace the GDAL dlls in your current MapGuide installation.

The reason we ask you to do this is because when we build GDAL for use with the FDO provider, we build GDAL using only the standard profile of supported formats. That is to say any format listed here:
Where the Compiled by default option is unconditionally Yes. It is not possible for us logistically to build GDAL/OGR with the proverbial kitchen sink of raster/vector format support, so that's where GIS internals comes in as their builds of GDAL/OGR have greater raster/vector format support. As long as you grab the same release of GDAL and make sure to pick the build that is built with the same MSVC compiler used to build the release of MapGuide/FDO you're using, you should then have GDAL and OGR FDO providers with expanded vector and raster format support.

This suggestion worked up until the 2.5.2 release, where the right version of GDAL built with the right version of MSVC (2010 at the time) was available for download. But for 2.6 and the (pending) 3.0 releases, this suggestion is not applicable because that site does not offer a MSVC 2012 build of GDAL 1.10, which is what MapGuide 2.6 and 3.0 both use for their GDAL FDO provider.

So this leaves some of you in a pickle, being stuck on 2.5.2 and unable to move to 2.6 or 3.0 because you need to support one of these esoteric data formats. Well, I have partially alleviated this issue for you.

Tamas has not only made these custom GDAL binaries for download, but also the development kits used to build these binaries as well. So in these past few days, I grabbed the MSVC 2012 dev kit, paired it with our internal GDAL 1.10 source tree in FDO and made a few tweaks to some makefiles here and there and here's the end result.

A custom build of GDAL 1.10 with support for the following additional raster data formats:
  • ECW (rw): ERDAS Compressed Wavelets (SDK 3.x)
  • JP2ECW (rw+v): ERDAS JPEG2000 (SDK 3.x)
  • FITS (rw+): Flexible Image Transport System
  • GMT (rw): GMT NetCDF Grid Format
  • netCDF (rw+s): Network Common Data Format
  • WCS (rovs): OGC Web Coverage Service
  • WMS (rwvs): OGC Web Map Service
  • HTTP (ro): HTTP Fetching Wrapper
  • Rasterlite (rws): Rasterlite
  • PostGISRaster (rws): PostGIS Raster driver
  • MBTiles (rov): MBTiles
And support for the following additional vector formats:
  • "PostgreSQL" (read/write)
  • "NAS" (readonly)
  • "LIBKML" (read/write)
  • "Interlis 1" (read/write)
  • "Interlis 2" (read/write)
  • "SQLite" (read/write)
  • "VFK" (readonly)
  • "OSM" (readonly)
  • "WFS" (readonly)
  • "GFT" (read/write)
  • "CouchDB" (read/write)
  • "ODS" (read/write)
  • "XLSX" (read/write)
  • "ElasticSearch" (read/write)
  • "PDF" (read/write)
You might notice some omissions from this list. Where's MrSID? Where's Oracle? Where's $NOT_COMPILED_BY_DEFAULT_DATA_FORMAT?

Well I did say I have partially alleviated the issue and not fully alleviated it. The issue is that due to what I gather is licensing restrictions, the development kit can't bundle the necessary headers and libraries needed to build GDAL with driver support for MrSID, OCI, etc. As such the custom build of GDAL I have made available does not include support for such formats.

What can be done about this. For something like Oracle, we already have a dedicated FDO provider for that. For something like MrSID? I'm afraid you're out of luck. You'll either have to stick on the 2.5.2 release for that much longer, or just bite the bullet and gdal_translate those MrSID files to something more accessible. I've heard some good things about RasterLite. I've also heard that you could get some great performance out of carefully prepared GeoTiffs

Any thing to liberate yourself from MrSID because you won't see the right GDAL binaries with this support built in for the foreseeable future.

You can find the download links for the custom GDAL builds in our updated GDAL provider guide for MapGuide 2.6 and 3.0.

One more thing before I finish that is worth re-iterating. Some formats like ECW, require you to be have a license to use ECW technology in a server environment. Other formats carry their own licensing shenanigans, so make sure you are properly licensed to use any of the the additional formats that are made available with this custom build of GDAL. The GISInternals build system repo on GitHub has all the applicable licenses in RTF format for your perusal.

Also worth pointing out is that this custom build of GDAL is not supported by me or anyone on the MapGuide development team. I only make this build available to you so you have the ability to access these additional data formats should you so choose and nothing more. There is no obligation by us to provide support for any issues that may arise as a result of using this custom GDAL release (inferred or otherwise). Use this custom build of GDAL at your own discretion.

/end lawyer-y talk. Enjoy!

Thursday, 2 April 2015

Announcing: MapGuide Maestro 6.0M4

Just so I don't scare you people off with the "alpha" label, I'll be calling these releases "milestones" instead :)

So here's the 4th milestone release of Maestro 6.0, here's the goodies you can find in this release.

MapGuide 3.0 support

Maestro now supports the new resource types and schema revisions introduced with MapGuide Open Source 3.0

The new Tile Set Definition resource has specialized editor support.


Map Definitions using the new schema, have the option of the choice of tile source


An inline tile source will be as before, where you can define base map groups and layers within the Map Definition.

An external tile source, will replace the base map UI with a new UI that simply asks for the Tile Set Definition you want to link against.


When linking against a Tile Set Definition, it is important to note that its coordinate system will take precedence

Also important to note is that you can only link to Tile Sets using the Default tile provider. Linking against XYZ tile providers is not supported in MGOS 3.0 and Maestro will prevent such an action from taking place.


MgCooker has also been updated to support tile seeding of Tile Set Definitions.



However, only Tile Set Definitions using the Default tile provider are supported. For Tile Set Definitions using the XYZ provider, consider using other tile seeding mechanisms for now.

FDO Expression and Filter support

Maestro now supports the ability to parse FDO Expression and Filter strings in pure 100% managed .net code without needing to use FDO itself. This feat was made possible through the use of the Irony language implementation library which made implementing the grammar and appropriate parsers for the FDO expression and filter syntax a dead simple affair.

With the ability to parse FDO Expressions and Filters, our Expression Editor is enhanced as follows:

We now have the ability to validate the given filter or expression for syntax validity.



With the ability to analyse the structure of FDO filters and expressions, we can now also ensure you are referencing valid identifiers in your filter or expression. Such invalid identifiers will be caught by the validation process.


Finally, a new tool is included that lets you see the parse tree of the given filter or expression.



As a developer, this functionality is accessible in its own 100% managed .net class library (OSGeo.FDO.Expressions.dll) should you have a need to parse FDO filters and expressions in your application without using FDO itself.

Reading user input from IronPython console

The IronPython script console now supports reading user input from stdin with Python's input() and raw_input() functions. Previously, you would've had to construct your own input GUIs and display them in order to capture any user input from your IronPython scripts.



With this in place, such workarounds are no longer required and allows for more flexible Maestro automation scripts as a result.

Advanced Stylization editor improvements

We've improved the Composite Symbolization editor to better work with component symbols.

Firstly, component symbols now show the applicable geometry context.



Secondly, component symbols can be re-ordered as ordering affects how the final composite style will look, so having the ability to re-order component symbols is important.



Finally, the best bit of these improvements is that now you can export a component symbol out to its own Symbol Definition.


This promotes better re-use of Symbol Definitions within layers as you can already use an external Symbol Definition as a component symbol. This new feature helps to make this process much more streamlined.

mapguide-rest support

Maestro now includes a dedicated addin to easily author up, upload and manage restcfg.json files in mapguide-rest.



This addin takes advantage of new authoring APIs introduced in mapguide-rest 1.0 RC1, and thus this is the minimum required version of mapguide-rest that this addin will work with.

This feature deserves its own blog post so stay tuned.

Other stuff

We've now moved to .net Framework 4.5. This means Maestro will no longer run on Windows XP.

From a developer perspective, we've done some major re-structuring to allow for better developer scenarios. The most pertinent example being that all the various ObjectModels assemblies have now been combined into a single assembly. Referencing this single assembly gives you the ability to work with every supported resource and schema revision from MapGuide 1.0 to MapGuide 3.0 as strongly-typed classes instead of XML.

Download

Tuesday, 31 March 2015

MapGuide 3.0 feature showcase: Tile Service Enhancements

If I were to pick the most compelling feature of MapGuide Open Source 3.0, it would be the one I am about to show you in this post.

Do you bemoan the fact that tiled maps are tied to a given Map Definition preventing their re-use with other Map Definitions?

There are ways to work around this problem, by using the "hack" of stacking multiple Map Definitions in a Fusion Application Definition. In this case, you would have a purely tiled Map Definition and you combine this with a purely dynamic Map Definition in your Application Definition and provided both maps are in the same coordinate system, both tiled and dynamic layers will line up.

You can then reuse this tiled map definition with other Fusion applications by mixing and matching tiled and dynamic Map Definitions together. However as already mentioned, this is a hack because it exploits an esoteric feature of Fusion: The ability to have more than one Map Definition specified in a Fusion MapGroup. This technique doesn't work with the AJAX viewer.

For the 3.0 release, we have solved this problem by introducing a new resource type to define shareable tile sets: The TileSetDefintion.

For a cliff notes summary: A TileSetDefinition is a re-usable tile cache that is effectively the BaseMapDefinition element of the Map Definition Schema separated out into its own resource. A TileSetDefinition lets you:
  • Define one or more base layer groups, each with 0 or more layers
  • Define a finite set of scales
  • Define the coordinate system of this tile set
  • Define the bounds of this tile set
Basically the same tiled layer settings you would've previously defined in a Map Definition. In addition a TileSetDefinition also lets you define additional settings that are specific to that tile set:
  • Tile size
  • Tile image format
  • Tile storage location
No longer are your tile sets constrained by global tile settings in serverconfig.ini and no longer are your rendered tiles always stored in a global defined tile cache directory. Each tile set can have their own tile size, format, storage location.

Here's an example of such a Tile Set Definition, notice the structural similarity to the Base Map section of the Map Definition

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
<?xml version="1.0" encoding="utf-8"?>
<TileSetDefinition xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xsi:noNamespaceSchemaLocation="TileSetDefinition-3.0.0.xsd">
  <TileStoreParameters>
    <TileProvider>Default</TileProvider>
    <Parameter>
      <Name>TilePath</Name>
      <Value>%MG_TILE_CACHE_PATH%</Value>
    </Parameter>
    <Parameter>
      <Name>TileWidth</Name>
      <Value>256</Value>
    </Parameter>
    <Parameter>
      <Name>TileHeight</Name>
      <Value>256</Value>
    </Parameter>
    <Parameter>
      <Name>TileFormat</Name>
      <Value>PNG</Value>
    </Parameter>
    <Parameter>
      <Name>FiniteScaleList</Name>
      <Value>200000,100000,50000,25000,12500,6250,3125,1562.5,781.25,390.625</Value>
    </Parameter>
    <Parameter>
      <Name>CoordinateSystem</Name>
      <Value>GEOGCS["LL84",DATUM["WGS84",SPHEROID["WGS84",6378137.000,298.25722293]],PRIMEM["Greenwich",0],UNIT["Degree",0.01745329251994]]</Value>
    </Parameter>
  </TileStoreParameters>
  <Extents>
    <MinX>-87.764986990962839</MinX>
    <MaxX>-87.695521510899724</MaxX>
    <MinY>43.691398128787782</MinY>
    <MaxY>43.797520000480347</MaxY>
  </Extents>
  <BaseMapLayerGroup>
    <Name>Base Layer Group</Name>
    <Visible>true</Visible>
    <ShowInLegend>true</ShowInLegend>
    <ExpandInLegend>true</ExpandInLegend>
    <LegendLabel>Tiled Layers</LegendLabel>
    <BaseMapLayer>
      <Name>Roads</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Roads.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Roads</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Districts</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Districts.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Districts</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Buildings</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Buildings.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Buildings</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Parcels</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Parcels.LayerDefinition</ResourceId>
      <Selectable>true</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Parcels</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Islands</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Islands.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Islands</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Hydrography</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Hydrography.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Hydrography</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>CityLimits</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/CityLimits.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>CityLimits</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
  </BaseMapLayerGroup>
</TileSetDefinition>

Finally, there's the re-usability aspect. Once a TileSetDefinition has been created, you can link to it from any Map Definition. Tile caches are now bound to the TileSetDefinition instead of the Map Definition. When a Map Definition links to a TileSetDefinition, it will override the following settings from the Map Definition:
  • Coordinate System
  • Map Extents
The reason we do this is because dynamic map layers can be re-projected, but tiles cannot. So in the event that the Map Definition has a different coordinate system from the one in the linked tile set, the linked tile set "wins" and all the dynamic layers in the Map Definition will be re-projected to the coordinate system of the tile set, ensuring that both dynamic and tiled layers will line up properly.

And the best thing about this is that both AJAX and Fusion viewers support shareable tile sets*. The above multiple map hack is no longer needed.

* For this release, this will only work with shareable tile sets using the "Default" tile provider. Why? Read on below.

XYZ Tile Support

The TileSetDefinition schema allows tile set settings to be defined in a key/value fashion like you would in a Feature Source. This schema design is intentional as it allows us to define a FDO-style provider model behind the scenes for tile access. Through this provider model, we are able to add support for different tile access schemes, one of which is XYZ tiles. This means you no longer need mapguide-rest to serve XYZ tiles. MapGuide 3.0 will have this support natively.

Here's an example of a Tile Set Definition using the XYZ tile provider

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
<?xml version="1.0" encoding="utf-8"?>
<TileSetDefinition xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xsi:noNamespaceSchemaLocation="TileSetDefinition-3.0.0.xsd">
  <TileStoreParameters>
    <TileProvider>XYZ</TileProvider>
    <Parameter>
      <Name>TilePath</Name>
      <Value>%MG_TILE_CACHE_PATH%</Value>
    </Parameter>
    <Parameter>
      <Name>TileFormat</Name>
      <Value>PNG</Value>
    </Parameter>
  </TileStoreParameters>
  <Extents>
    <MinX>-9770571.93250815</MinX>
    <MaxX>-9755615.48593707</MaxX>
    <MinY>5416573.69002144</MinY>
    <MaxY>5436091.17493748</MaxY>
  </Extents>
  <BaseMapLayerGroup>
    <Name>Base Layer Group</Name>
    <Visible>true</Visible>
    <ShowInLegend>true</ShowInLegend>
    <ExpandInLegend>true</ExpandInLegend>
    <LegendLabel>Tiled Layers</LegendLabel>
    <BaseMapLayer>
      <Name>Roads</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Roads.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Roads</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Districts</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Districts.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Districts</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Buildings</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Buildings.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Buildings</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Parcels</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Parcels.LayerDefinition</ResourceId>
      <Selectable>true</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Parcels</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Islands</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Islands.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Islands</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>Hydrography</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/Hydrography.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>Hydrography</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
    <BaseMapLayer>
      <Name>CityLimits</Name>
      <ResourceId>Library://Samples/Sheboygan/Layers/CityLimits.LayerDefinition</ResourceId>
      <Selectable>false</Selectable>
      <ShowInLegend>true</ShowInLegend>
      <LegendLabel>CityLimits</LegendLabel>
      <ExpandInLegend>true</ExpandInLegend>
    </BaseMapLayer>
  </BaseMapLayerGroup>
</TileSetDefinition>

Notice that the XYZ tile set definition does not specify a coordinate system. This is because XYZ tile sets are always in WGS84.PseudoMercator (aka. EPSG:3857), so there is no need to specify this. The extents describe the bounds of this tile set in EPSG:3857 coordinates.

You'll notice in both examples, that we have the concept of a tile provider along with a set of key/value pairs to describe settings applicable for that provider, which is similar to the how a Feature Source specifies a FDO provider and a set of key/value pairs that form the set of FDO connection properties.

And what is this %MG_TILE_CACHE_PATH%? It is a placeholder for the global server tile cache path, and just like %MG_DATA_FILE_PATH% for feature sources, you can replace this with actual physical paths or MapGuide aliases. That's right, your Tile Set Definitions can have their own defined tile storage location instead of being all under a globally defined path in serverconfig.ini

As you can see, the similarity between tile providers and FDO was quite intentional :)

This provider model is currently hard-coded for these 2 providers. There is no formal plugin model/API yet that allows for external third-party tile providers to be defined. Whether we will add such support in a future release is a discussion for another time.

The best thing about this XYZ support: No API changes required! You still use the same GETTILE requests to the mapagent, only now you pass in the actual X, Y and Z for the ROW, COL and SCALE parameters.

Now there is one limitation of the XYZ support for this release. You cannot create MgMap instances from a Map Definition that links to an XYZ tile set, and as a result you cannot load such Map Definitions in the AJAX and Fusion viewers. The main use case for XYZ tile support in this release is to enable easy consumption of XYZ tile sets in MapGuide by external clients like OpenLayers, Leaflet, etc. We'll correct this limitation for the next major release after 3.0.

With this release, there's a new sample you can check out that demonstrate how to consume XYZ tiles from OpenLayers.


When you look at the code, you'll find consuming XYZ tile sets from MapGuide is as simple as consuming OpenStreetMap and friends in your JS mapping library of choice, be it OpenLayers, Leaflet or any other library that can consume XYZ tile sets.

Now that our tile storage and access has been abstracted behind a Tile Set Definition resource, it paves the way for us to easily enhance the tiling capabilities in the future with:

  • Support for additional storage backends (eg. MBTiles)
  • Support for additional tile formats (eg. Vector tiles, UTFGrid)
  • Standardizing tile access (eg. WMTS)

That's not to say such items are definitely on our current roadmap, but it will make implementing such features a more realistic and feasible possibility now that the groundwork has been laid in this release.

For more information about these new Tile Service Enhancements, you can consult MapGuide RFC 140

Monday, 30 March 2015

Announcing: MapGuide Open Source 3.0 Beta 1

After a month or two lost due to mucking around with packer and vagrant to get the perfect repeatable and disposable build environment for both Windows and Linux builds of MapGuide/FDO, I'm now happy to announce the availability of MapGuide Open Source 3.0 Beta 1.

Who knew debugging build infrastructure would be such a PITA? And who also knew that vagrant and windows is like oil and water? Ugh! I gave up on that part, so as it stands our current build infrastructure for putting out releases of MapGuide is still Jenkins for windows and (slightly improved) vagrant for Linux.

Stay tuned for the (promised) series of posts talking about what's new in MapGuide Open Source 3.0

Download

Friday, 6 March 2015

Announcing: mapguide-rest 1.0 RC1

I've decided that mapguide-rest has enough features to lock in the 1.0 version number.

Here's what's new in this release.

Cleaner JSON responses

If you've used JSON output from mapguide-rest or MapGuide proper, you'll notice that the JSON is ugly and cumbersome to use:
  • Every JSON property is an array
  • All property values are strings
The main reason this JSON is so ugly is because of the way that XML is converted to JSON. The conversion process knows nothing about the content model of the XML document it is trying to convert, so as a lowest-common-denominator solution, it forces all converted XML elements to JSON array properties and all XML text elements as string values. While this allows for rudimentary usage by client applications, it's not a pleasant experience.

For this release, we've set out to solve this problem. The JSON produced which would've looked like this:


Now looks like this:


How did we achieve this? Simply put, we now maintain a hard-coded list of known XML element paths (compiled from all the MapGuide XML schemas) and make note of:
  • Which paths must always be processed as a JSON array
  • Which paths have values whose data type is not a string
With this list, we now know which XML elements need to be converted to JSON arrays and which elements need to be converted to JSON property values, and the proper data type to convert such values to.

The end result is that the new JSON responses are much more natural and intuitive to work with. Not to mention these new JSON responses are smaller than their previous counterparts. Those extra [ and " characters do add up.

This is a PHP-level solution. With this release, all JSON conversion/output is now handled by mapguide-rest. While slower than doing it natively in MapGuide, it gives us more usable JSON structures and this increased usability is worth it for the slower performance.

As for the old JSON response structure, that is going away in mapguide-rest. This new structure will be used for all APIs in mapguide-rest that return JSON. If you want the old JSON structured responses, you'll have to go back to the mapagent.

In one particular case, we eschew the convert-to-JSON-from-XML approach and define a new dedicated JSON response format. 

Ever tried to describe the schema of a Feature Source? This is the XML response:


Have you seen the JSON version of this monstrosity?


How on earth would any sane person know how to use this response in their applications? Yeah me neither.

So for schemas and class definitions that are returned as JSON, we introduce a more cleaner and intuitive JSON format


Isn't that a more intuitive and self-explanatory JSON response?

This new response structure is only applicable for JSON schemas and class definitions. The XML version of these schemas and class definitions will still be its same ugly and verbose self. From my perspective, XML is the "legacy" response format in mapguide-rest. JSON is the preferred response format for any client application and so we should strive to make consuming such data in this format as simple and clean as possible. With this release, hopefully we've achieved that objective.

While we're still on the subject of JSON, for certain REST APIs that previously accepted XML request bodies, such APIs now support JSON-equivalent request bodies. For example, consider this route to set resource content:

POST /rest/library/{resourcePath}/content.xml

To set resource content, you'd make a POST request to this route and include the resource XML as part of the request body.

You can now set the same resource content via its JSON representation

POST /rest/library/{resourcePath}/content.json

By making a POST request to this route and include the equivalent JSON as part of the request body. Because our JSON formats now structurally follow their XML counterparts, knowing what JSON to construct should be easy to figure out. On the server-side such JSON bodies will be converted back to its XML counterpart before passing down to the underlying MapGuide APIs (because sadly XML is still the canonical format for storage and data transfer throughout the official MapGuide APIs)

Authoring APIs for restcfg.json files

We've added a set of new APIs in the data publishing framework to allow for restcfg.json and any supporting files to be uploaded from a mapguide-rest client without needing physical access to the directory where the restcfg.json files are stored.

These authoring APIs allow you to:

  • Upload a restcfg.json file to a desired subdirectory within mapguide-rest based on the given URI part
  • Upload any supporting files to the subdirectory mapped to the given URI part
  • Delete any restcfg.json configuration
  • Delete any supporting file of a restcfg.json configuration
Because URI parts are mapped to physical subdirectories within a mapguide-rest installation, there could be a opportunities for malicious activity without proper security precautions. To this effect, we do the following:
  • Such APIs require Administrator or Author-level roles to be assigned to the authenticated user. The MapGuide Anonymous users will not have such roles, and thus any attempt at accessing these APIs will be denied.
  • Any URI parts and filename parameters are sanitized to strip off any semblance of relative paths and any other path navigation tricks that would cause mapguide-rest to save/delete a file or directory outside of its installation directory.

You can find out how to use these new APIs in the built-in interactive API reference.

API additions and cleanups

APIs that previously returned simple primitive values now return a XML or JSON boxed version of that value. Simply put, if an API returned a boolean value like this:

true

It will now return a boxed value, which looks like this for XML:


And looks like this for JSON:

{ "PrimitiveValue": { "Type": "Boolean", "Value": true } }

While this looks more verbose, it is simply to provide a consistent developer experience. If you have been expecting XML or JSON responses for 99% of the APIs, why make an exception for this 1% of APIs that return basic primitive values? They should be XML or JSON respectively as well and in order for that to happen, we have to box such values.

In addition many new APIs have been added to cover missing pieces and loose ends in our REST API:

  • Creating Feature Sources
  • Setting and deleting resource data items
  • Listing Long Transactions of a supported Feature Source
  • Getting the coordinate system base library
  • Validating Coordinate System WKT
  • Converting Coordinate System WKT to EPSG codes
  • Testing the connectivity of a Feature Source
  • Enumerating unmanaged (aliased) data
  • Getting site information status/statistics
  • DescribeSchema with class name hint
  • Setting content and header of any resource in a single request
  • Fusion layout authoring support APIs
  • Interrogating and setting the edit-ability of a Feature Source with the REST API

You can find out how to use these new APIs in the built-in interactive API reference.

(Geo)JSON CRUD support in restcfg.json

The data publishing framework now supports CRUD for JSON representations. The JSON representation is much more simpler and compact in its request payloads. The requests to create, update and delete features are structurally the same as the XML one (using the same XML-to-JSON pattern for most of our JSON data structures)

The existing Editable Properties example has been modified to use the JSON representation instead of the XML one. For doing CRUD through the data publishing framework, I'm sure you'll find it much easier to work with this JSON representation than the XML one.

Customizable HTML representation

The HTML representation for features are now customizable and localizable as the previous bits of hardcoded HTML that were rendered out by the HTML renderer, have all been extracted out into Smarty templates, where you can customize the HTML to your own liking.

MapGuide 3.0 support

This release of mapguide-rest supports new features introduced in MapGuide Open Source 3.0.
  • Your tile service routes can also work with Tile Set Definitions
  • XYZ tile requests will pass through to the MapGuide API on a MGOS 3.0 install
  • Runtime Map responses use the v3.0.0 response schema
Other changes
Where to from here?

From now to the final 1.0 release of mapguide-rest, it will be nothing but bug fixes and (really) minor enhancements.

After 1.0 as I've somewhat hinted, if mapguide-rest as a project should have a continued evolution, I strongly believe it should be on the ASP.net 5.0 platform. Given ASP.net 5.0 won't see light for many more months, I'll have plenty of time to play and experiment with many ideas that are possible with ASP.net 5.0

Thursday, 12 February 2015

A small IronPython console improvement

I wasn't going to put out a new release of MapGuide Maestro without crossing at least this item off my list. So after several weeks of coder's block about how to solve this particular problem, I finally cracked it.

So I proudly present: Support for capturing user input from stdin in the IronPython console!


And yes, you can store the result as variables


You can also use input() for capturing user input from stdin, but it has the caveat of auto-evaluating whatever you enter as a Python expression. What this means is that if the user tried the same thing as above with input(), it will fail.


This is because the input of Jackie Ng is not a string literal, which causes Python to throw the SyntaxError. With input() if you are capturing strings, you have to quote them.


The real power of input() is allowing you to input raw python expressions, like capturing an array as input.


If you just want to capture string input, follow the advice of the Python documentation and use raw_input() instead.

The code to get this working doesn't look pretty IMO, but that's better than what we had before, which was nothing! So here's to more flexible python scripts in the next release of Maestro.

As for that next release, still some more pending items to tick off first.

Monday, 26 January 2015

MapGuide on Windows 10

Since Microsoft released a new preview build of Windows 10, I'd thought I give it a spin on VirtualBox.

My first impressions are simply that I am so happy to see the return of the Start Menu!



The other pleasant surprise was to see that the command prompt was finally resizable!

But my main motivation for giving Windows 10 a spin was to see if MapGuide will run on it. So I downloaded the 3.0 preview release.

I ran the installer and got this.



Okay, this normally means I have to manually "unblock" the executable in the file properties before retrying, so I did exactly that.

Except unblock does nothing! Is this a bug?



So before I chew out Microsoft for their "UAC v2.0" over-zealousness with this SmartScreen feature, I decided to click the "More Information" link on the SmartScreen dialog to see if it does anything and lo and behold, it gives me the options to finally run the thing!


Terrible UX there Microsoft. The "Run Anyway" option should be visible IMO and not concealed in the "More Information" link.

Nevertheless, after negotiating this little obstacle the installer ran after the UAC prompt. I decided to try the IIS/.net install option since:

  • PHP and its MapGuide API bindings is installed regardless (it's why mapguide-rest will/should just work out of the box on any supported version of MapGuide), so we can also verify the PHP bits are working by loading a package through the Site Administrator
  • This is the configuration that we always want to confirm and verify when testing support for newer versions of windows. The Apache/PHP/Java configurations are mostly self contained and can be generally trusted to work on any windows platform where there is a matching Visual C++ runtime library and a working Java SDK/RE for it.
So after waiting the normal amount of time for a MapGuide install to complete.


The Site Administrator runs without issues, allowing me to load my sample data package. Firing up the AJAX viewer (that is bound to the .net implementation due to my installation choice), shows that it is indeed functional


So there you have it. MapGuide 3.0 works on Windows 10. 

Given its current Technical Preview status, we cannot realistically have Windows 10 as a supported platform for MapGuide 3.0, but should you want to try, chances are MapGuide should work as evidenced by this post.