Tuesday, 3 September 2019

MapGuide 4.0 showcase: Supercharged tile sets

As promised, this is the first (of many) blog posts showcasing the new features of the just released MapGuide Open Source 4.0 Preview 1. Before we dive in, here's a little recap for some extra context.

Way back in MapGuide Open Source 3.0, we introduced the concept of a Tile Set Definition, a new resource type that allows one to define a shareable, re-usable tile set that was free of (most) server-imposed global default settings and is decoupled from any particular map or application. Before the introduction of shareable tile sets, a tile cache could only be defined within a Map Definition and was thus coupled to any application consuming this particular map. This setup also made the respective tile cache extremely prone to invalidation upon any minute changes to the Map Definition. Tile settings like size and image format were global (in serverconfig.ini) making it impossible to have tile sets of various sizes and image formats.

With the introduction of shareable tile sets, it means that tile sets no longer need to be defined on the Map Definition. Instead, it can be defined in an external Tile Set Definition resource and linked into any Map Definition (hence the shareable aspect). Being a linked resource, also means any changes to a Map Definition will not errantly flush cached tiles for the associated tile set as we have solid guarantees that editing a Map Definition that links to a tile set would not alter or compromise the layer/group structure of a tile set being linked that would normally necessitate invalidating the tile cache by deleting all previously cached tile images.

Because tile sets are now their own separate resources, it also opens up a design space that allows us to do things with these tiles that were previously not possible. Each Tile Set Definition can:
  • Specify their own tile sizes
  • Specify their own image formats
  • Specify their own storage location (internally managed by MapGuide or an explicit directory)
  • ... and much more
Tile Set Definitions also support a mini-FDO-style provider model that allows us to support different tile access semantics. This is how we're able to support XYZ tiles, while retaining the same method of accessing such tiles without any API changes.

For MapGuide 4.0, we've expanded the "...and much more" bullet point above by building on top of the shareable tile set foundation with new capabilities.

UTFGrid Tile Support

UTFGrid is a specification for rasterized interaction data. UTFGrid allows us to pre-render tooltip interaction data into tiles of textual content, which when superimposed on top of your existing map, allows us to have rich tooltips without having to send a single mapagent request for tooltip data! 

If you installed 4.0 Preview 1 with samples, there is a new sample demonstrating UTFGrid tiles in action. Notice the distinct lack of mapagent requests (beyond the requests for viewer assets, tile images and UTFGrid tiles).

UTFGrid files follow the XYZ tile access scheme, and thus are supported in Tile Set Definitions using the XYZ Tile Provider with the new tile output format of UTFGRID


With the introduction of UTFGrid tiles, the 4.0 preview 1 release opens up a new viable and potent interactive web mapping combination that was previously not possible.
  • Base map imagery provided by XYZ tiles (whether from an XYZ tile set served from MapGuide or from external services like OpenStreetMap)
  • Overlaid with tooltip interaction data provided by UTFGrid tiles served from another MapGuide XYZ tile set.
This combination requires zero MapGuide sessions to operate. If you recall my now-somewhat-ancient thoughts on scalability in MapGuide, session repositories (and their creation/maintenance/teardown) is a main contributing factor to load in a MapGuide Server.

You can pre-cache all the required content up-front through stateless tile request APIs. You can also choose to output generated tiles to a custom directory of your choosing (by editing the TilePath setting) and front that tile cache directory as a source of static file content in a HTTP web server of your choosing for even more scalability/cacheability. If your mapping interactivity requirements are simple, this new combination of XYZ + UTFGrid tiles served from MapGuide may be a viable combination for you.

(...and it just occurred to me in the process of writing the above blocks of text that this combination opens up brand new map publishing workflows and capabilities that were previously not possible! Here's one such idea I'm brewing for a future release of MapGuide Maestro)

Beyond the provided sample, there is nothing out of the box in the Preview 1 release that currently takes advantage of UTFGrid tile support. UTFGrid support exists in OpenLayers, both in:
So it is possible for some kind of out-of-the-box viewer support for UTFGrid tiles to appear in a future 4.0 preview release (no promises right now!)

Meta-Tiling Support

Meta-tiling is the process of rendering a bigger tile and slicing it down into sub-tiles of the original requested size. Here's a visual explanation:



By rendering bigger tiles and slicing the result down to sub-tiles for tile cache storage, we improve tile generation throughput via:
  • Less queries for feature data (because we're rendering for bigger tiles)
  • Requesting 1 tile and getting 2^n cached tiles for free for other requests because we cache those other sliced sub-tiles as well.
Meta-tiling was a feature that was actually implemented many years ago in a sandbox branch, but it was never merged back into trunk to be made available in a future release. With the advent of shareable tile sets introduced in MGOS 3.0, it was somewhat impractical to merge this work back into trunk as-is, so the existing meta-tiling support was refactored by yours truly to make it easy to retrofit into the new shareable tile set infrastructure.

As a result, in the 4.0 preview 1 release, meta-tiling support is available for Tile Set Definitions in both the Default and XYZ tile providers through 2 new properties:
  • MetaTileFactor
  • MetaTileLockMethod
The MetaTileFactor is a value > 1 that indicates how big our meta-tile to render should be. The specified value will be the multiple of the size of the tile we're requesting. For example, if we're requesting a 256x256 tile from a tile set with a MetaTileFactor of 2, MapGuide will render a 512x512 tile for division into four 256x256 tiles.

The MetaTileLockMethod is an optional setting that controls the locking behavior during this whole process.

  • 1 - Uses a mutex to ensure thread-safe tile writing
  • 0 or unset - Uses a file-based lock for the same effect
Generally speaking, this is a setting you don't really need to tweak and can leave as unset.


Hi-DPI (Retina) tiles

For Tile Set Definitions using the XYZ provider, a new RetinaScale property is available that allows you to generate Retina tiles. Retina tiles are (256 * n) x (256 * n) tiles where n is the specified RetinaScale value. Such tiles provide higher visual fidelity on high-resolution devices such as iPhones, iPads and high-end Android devices.

Our current out-of-the-box viewer offerings do not take advantage of Retina tiles and there is no plans to add such support. This feature is made available primarily for custom viewer offerings (like mapguide-react-layout) to take advantage of.

Other Loose Ends

The previously global TileExtentOffset property can now be configured on a per-tileset basis through a tile set property of the same name. This value is used to internally "buffer" a tile by a certain amount to be rendered to allow label placement algorithms more "breathing space" when determining optimal label placement.

We've also removed the previous restriction that you cannot create a new MgMap from a Map Definition that links to a XYZ tile set (NOTE: Maestro still enforces this restriction in the UI, but this restriction will be removed if we're connecting to a 4.0 server). To see what removing this restriction can give you, there's a new code sample that demonstrates that we can now view any conceivable Map Definition or Tile Set Definition.


Announcing: MapGuide Open Source 4.0 Preview 1.1

When I announced the availability of the 1st Preview release of MapGuide Open Source 4.0, I discovered shortly afterwards, a nasty show-stopping bug in one of the new APIs that breaks interaction with the bleeding edge version of mapguide-react-layout (the sole consumer of this new API).

As this is somewhat of a blocker in terms of what I'm am currently working with, I've put out a new 1.1 Preview release that includes a fix for this bug.

Download/Release Notes

If you don't use mapguide-react-layout, this bug probably doesn't affect you and you can stick with the existing Preview 1 release, if you have already downloaded and installed that.

And since we're here, the Linux binaries will be delayed until Preview 2 as my revised packaging scheme still needs more incubation time.

Wednesday, 21 August 2019

Announcing: MapGuide Open Source 4.0 Preview 1

Time to get the first part of my plan in motion: Get a new release of MapGuide out there!

This 1st preview release of MapGuide Open Source 4.0 is the culmination of 2+ years (Wow! Time flies!) of trunk development since the release of MapGuide Open Source 3.1. Competing interest in other side projects like mapguide-react-layout undoubtedly contributing to the long stretch.

A cliff-notes summary of what's new in this release (along with the download links) can be found on the 4.0 Preview 1 release notes wiki page. This blog will showcase all the new features in an upcoming series of posts.

Only 64-bit windows binaries are available as of this post. Linux binaries will soon follow as I am still experimenting with a new packaging mechanism for the Linux builds.

And if the "Preview" designation does not give it away, this release represents a work-in-progress and is not indicative of what the 4.0 final release will look like. There is still plenty of work to be done and more preview releases to follow.

Friday, 26 July 2019

Plans and stuff

Since I'm back, I'll start with some rough plans for MapGuide and my related projects.

The next version of MapGuide Open Source will be: 4.0

The next version of MapGuide Open Source was originally slated to be 3.3, but due to the volume of features implemented for this milestone and with some planned future work that will have "breaking change" ramifications (namely dropping support for PHP5 and supporting PHP7) means that the next release is better represented as 4.0

I'll be putting out a 4.0 preview release shortly that showcases all of the features implemented thus far, all of which I will be blogging about here. This will still support PHP5 for the PHP flavor of the MapGuide API for now.

Supporting PHP7 is a non-negotiable blocker towards a final 4.0 release, MapGuide Open Source 4.0 will only be final once we have stable and reliable PHP7 support.

I have an experimental testbed on GitHub which has been somewhat dormant since I first announced it, but will soon be re-activated and most likely incorporated back into the MapGuide sources where I'll continue this work on an experimental sandbox branch.

That's not to say that PHP7 support is the only remaining feature between the 4.0 preview and the final release. There may still be new features implemented in the meantime.

I currently can't tell you how long this journey will take (I honestly do not know how long this work to support PHP7 will take), but at least you know what the final destination will be.

MapGuide Maestro

The main theme for the next milestone of MapGuide Maestro is to support or take advantage of the capabilities/features introduced in MapGuide Open Source 4.0. Check this issue for details.

mapguide-react-layout

I will shortly put out a new 0.12 release which will bridge the 750+ commit gap since the last 0.11.2 release (the bulk of these commits, undoubtedly greenkeeper-assisted pull requests)

After this release, this project will be on a short hiatus as I give MapGuide Open Source 4.0 development maximum priority

mapguide-rest

I will eventually sunset this project with the release of MapGuide Open Source 4.0. The final pieces (which will finally culminate in a 1.0 final release) are:

  • Knocking off the remaining items on this list
  • Blocker: Being fully operational under MapGuide Open Source 4.0
Having mapguide-rest fully operational under PHP7 (that we assume MGOS 4.0 final will be shipping with) will be my final barometer of "it is done" for this project.

Monday, 22 July 2019

Ping replied in 10095780000ms

This blog has been quiet as of late. Not because I have run out of things to blog about, merely I've been on a self-imposed blogging hiatus to get a serious chunk of MapGuide development time in.

And the time is right to start talking about some of the stuff that has landed into MapGuide in the past few months as well as a some updates on my other MapGuide/FDO-related projects.

Stay tuned!

Thursday, 28 March 2019

Announcing: MapGuide Open Source 3.1.2

I am pleased to announce the availability of MapGuide Open Source 3.1.2

The notable changes in this release are detailed below:

Updated Web Tier Components

MapGuide Open Source bundles updated web tier components:

  • Apache HTTPD 2.4.37
  • Tomcat 7.0.92
  • PHP 5.6.40
Since PHP 5.6 is now end-of-life, this is most likely the last release in the 3.1 branch as we work to bring PHP 7 support for the MapGuide API in the next major release.

Functional SHP FDO provider for 64-bit Linux

Over the new year, I had set out to solve one of the last remaining blockers that prevented us from having a functional 64-bit SHP FDO provider for 64-bit Linux.

I had speculated that the blocker is due to incorrect data size assumptions in the SHP spatial indexing code on 64-bit Linux which would throw off spatial index queries. Intensive debugging sessions proved this to be the case and I was finally able to land the long-awaited fixes which is now available with the SHP provider bundled with this release.

The proof that the provider was truly functional was being able to load a SHP-heavy dataset into a 64-bit MapGuide Server and being able to pan, zoom, select and buffer SHP features without any errors or features disappearing in and out at various zoom levels (due to previously out-of-whack SHP spatial index queries)


Experimental King.Oracle FDO provider for Linux (32 and 64-bit)

Since I got a major win with one Linux FDO blocker and was on a roll, I figured we can do a double-whammy with the other major Linux FDO blocker: The lack of a functional King.Oracle FDO provider on Linux.

For this one I had speculated that the problem was due to heavy usage of wchar_t for strings which (due to different size assumptions on Linux) meant that most strings going in and out of the provider on Linux were garbage, meaning the SQL queries we think we were sending to OCI for execution were completely scrambled.

Due to the lack of solid examples of using OCI with wchar_t strings on Linux, I figured the code as it currently stands will never work on Linux. So I took the page out of a reference implementation I knew to definitely work on Linux (the OCI driver for GDAL/OGR) and wholesale #ifdef'd the entire provider codebase to use narrow (char) strings for Linux and their expected OCI functions, while retaining wide (wchar_t) strings and current OCI function usage for Windows.

The ability to easily spin up an Oracle XE environment (well until recently at least) meant that I was able to quickly verify that switching to narrow strings on Linux was indeed the way to go. This also provided motivation for me to implement a basic (and repeatable) unit test suite for this provider, finally giving us an objective baseline level of validation and verification for this provider.

The end result of all this is that the King.Oracle provider now not only builds on Linux, but it now finally works!


However, I'm certain that there will be bugs in this provider for Linux waiting to be revealed, so the provider gets the "experimental" label. Use at your own risk/discretion. Please do report any issues you find.

OGR FDO Provider Improvements

The OGR provider has undergone many "quality of life" improvements:
  • We've cleaned up various correctness issues around provider behavior. For example, attempting to access invalid property names in an OGR feature reader will now throw instead of silently failing before.
  • The provider supports a new DefaultSchemaName property that lets you specify a default schema name other than OGRSchema
  • The provider also supports a new DataSourceEncoding property to allow you declare the encoding for an OGR data source where it cannot be inferred by default by the OGR library so that properties/names/etc with non-english characters are rendered properly.
Fusion Changes
Other Changes


Friday, 15 March 2019

Announcing: vscode-map-preview 0.4.6

A kind user alerted me to the fact that my Map Preview extension for Visual Studio Code was using APIs that have long been deprecated and could be removed any moment in a future release, breaking this extension in the process.

Although I haven't touched this code in a while (not intentional btw. One person can only tend to so many different open source projects), getting back into VSCode extension development was a very comfortable affair:

  • Update a few packages here and there. 
  • Update TypeScript. 
  • Read up on the new Webview API, which is what our extension should be using now.
  • Replace usages of vscode.previewHtml with the new Webview API.
  • Verify the extension still works
  • Get re-acquainted with how to publish VSCode extensions
And ... voila! A new version of the Map Preview extension is available for your VSCode installation to auto-update to.


In terms of changes, beyond some updates to OpenLayers and a small bug fix or two, there isn't really much to write home about, it should still be the same extension as before. This release was driven primarily by the need to move away from their old vscode.previewHtml API over to their new Webview API.

Once things have settled down on my other projects, I'd like to give this project some more attention. There's still some cool ideas I want to explore with this extension. I don't know if I still want to go down the Cesium route as previously hinted as there have been many more data viz players that have arrived on the scene since then and I also sort of do not want to detract from the original goal and purpose of this extension: An easy way to preview on a Map textual content that you are already editing or viewing in VSCode.