Saturday, 3 June 2017

Look ma, no tooltip requests!

So what does tiles of ascii-like content give us?

UTFGrid tiles allows us to pre-render tooltip interaction data. Because we're pre-rendering such data, no mapagent requests are ever made, we can fetch for such tiles just like other map image tiles.

If you remember my previous notes about MapGuide scalability, session repositories add memory and administrative burden/bookkeeping to the running MapGuide Server. If you can get away with not having to create a session repository and keeping it alive, you should. Currently, tooltip requests sadly, require a runtime map (thus, requiring a session be created first and the initial runtime map state to be saved to its session repository).

Because this is pre-rendered tiles of interaction data, paired with map image tiles, it's a very scalable map viewing solution. No CREATERUNTIMEMAP operation was needed to demonstrate this example, nor did we have to create or hit any session-based resources.

Also, because UTFGrids are accessed using the same XYZ tile access scheme, we can re-use the XYZ tile provider in the Tile Set Definition and just specify the new TileFormat of UTFGRID


Accessing UTFGrid tiles is just like accessing XYZ tiles, via the GETTILEIMAGE mapagent operation, but using the UTFGrid Tile Set Definition as the resource id.

The usual mapping suspects of OpenLayers and Leaflet already support UTFGrids, so you can easily take advantage of UTFGrids when it finally lands in MapGuide proper.

Now to make sure UTFGrid support also works on our Linux builds before I draft up that RFC.

Thursday, 25 May 2017

Announcing: mapguide-react-layout 0.9

After a small hiatus (due to hacking on MapGuide proper and my day job), here's a new release of mapguide-react-layout.

Here's the highlights of this release.

Now available as an NPM module

This is the first release available on npm. The npm module allows you to customize the viewer in the following ways:
  • Creating your own custom viewer templates
  • Creating your own script commands
  • Customizing your viewer bundle by omitting features you will never use
Want to see how that's done? Check out the new example that demonstrates all of the above.

Short of cloning my github repo and hacking/building the source yourself, the npm module is the best way to fully customize nearly all aspects of the viewer.

Now re-uses the Fusion PHP backend

In previous releases, we bundled a separate copy of the PHP backend tailored to service the following commands:
  • Buffer
  • Query
  • Theme
  • FeatureInfo
  • Redline
  • QuickPlot
  • Search Commands
Once the need to make this a npm module arose, this idea proved to be un-workable.
  • We'd have to ask the user to make some intricate post-build set up in their build configurations to make sure the PHP backend content to a location relative to the viewer bundle so that the above commands will work.
  • Putting PHP code into npm, a JS package registry? Uh ... okay back to the drawing board!
So in the process of revising this idea, the solution turned out to be quite simple:

Just assume MapGuide is installed with Fusion and re-use its PHP backend services

And it turns out that this assumption was a very safe one. Like PHP, Fusion will always be installed with MapGuide on Windows or Linux.

And with this assumption, we just demand one requirement for anyone using this viewer or building a bundle with the npm module: You need to install it into a subdirectory of MapGuide's www directory, which I would assume you are already doing because that's what our current install instructions say!

With this change, my mission statement with this viewer needs a small refinement This is no longer a modern map viewer that is a replacement for Fusion. This is now a modern map viewer that happens to re-use some Fusion backend services.

Which is fine by me because another (useful) side-effect of this exercise is that ...

Partial support for Fusion viewer APIs

... I had to polyfill various client-side Fusion viewer APIs so that the existing front-end content for the above commands can work against the mapguide-react-layout viewer without any modifications required. A subset of Fusion events are also supported in this release.

However, do not expect full 100% replication of the Fusion viewer API here. If you're going to migrate to this viewer, I recommend you go the whole nine yards and take advantage of the simpler and cleaner viewer APIs offered by mapguide-react-layout

Updated and "smarter" Templates

This release includes a new redux state branch for viewer template element visibility.



With this branch we also have new redux actions to push changes to it. Applicable templates now subscribe to the state branch and be able to automatically toggle visibility of template elements in response to dispatched actions.

What this enables us to do is to apply some extra smarts to the Fusion templates. For example, running an InvokeURL command (targeted at the Task Pane) will automatically show/open the Task Pane because the template reducer function also listens on this particular InvokeURL action to push a [Task Pane is now visible] state, as demonstrated in the gif below


Notice how I didn't have to manually expand the Task Pane first? The template reducer function already took care of it because I ran an InvokeURL command to load the Query widget.

The LimeGold and TurquoiseYellow templates have been updated to use the new Tabs2 blueprint component as our existing Tabs component is deprecated and not to mention that Tabs2 actually plays nicer with our new redux state branch.

The sidebar and ajax-viewer templates can now work with Application Definitions under the following conditions:
  • Your appdef has a widget container named Toolbar that will house the primary toolbar
  • Your appdef has a widget container named MapContextMenu that will house the context menu
  • Your appdef has a widget container named TaskMenu that will house the menu in the Task Pane bar
If you create a new Application Definition document in Maestro, all of the above conditions will already be satisfied.



Also if you take a look at the Aqua template, the 3 element togglers on the top-right are gone now.



This is because we were able to find a suitable replacement for InvokeScript commands as described below.

Script Command Support (ie. The InvokeScript replacement)

This feature is the replacement for InvokeScript commands/widgets and currently requires using the npm module. The viewer provides a registry API that allows for custom commands to be registered with the viewer.

How can you link these commands to toolbars and menu items in your WebLayout/ApplicationDefinition? With the existing InvokeScript definition.

The key difference here is that this viewer completely ignores the inline script content part and just invokes the equivalent script command registered in the viewer by the same name as defined in the InvokeScript definition in the WebLayout/ApplicationDefinition.

As a result, the new methodology for using these types of commands is to "bake them in" to your own viewer bundle, while with this approach we still retain the existing authoring experience by re-using InvokeScript definitions. You just have to give it the same name as what they're registered under in the viewer bundle.

To illustrate this, the viewer by default has registered script commands for toggling the:
  • Task Pane
  • Legend
  • Selection Panel
These script commands leverage the new redux actions to push element visibility state to the layout template components, replicating the behaviour of their old InvokeScript counterparts.

Because they are also registered under the same names as their old InvokeScript counterparts, a standard new Application Definition (created by Maestro) will no longer show [X] ERROR placeholders for these commands when fed to any of the 5 ported Fusion templates.


As we how have replicated the old behaviour of the Aqua template, the hard-coded element togglers on the top-right are no longer needed.

Bing Maps Support

The viewer now supports Bing Maps as external base layers.

As mentioned previously, you will now need to acquire an API key to use Bing Maps as the API-key-less version will shut down on June 30, 2017. Fusion proper has the same problem and will also require an API key to use Bing Maps. The next release of MapGuide Maestro will have an updated Fusion editor to configure this.

Fusion Application Definitions passed to this viewer with Bing Maps support are assumed to be ones configured up with a Bing Maps API key and Bing Maps (not VirtualEarth) base layers.

Command Parameterization

This release implements initial support for parameterization of commands. In practical terms this means that the viewer will now start recognizing and support the various Fusion widget extension properties that you may have defined in the widgets of your Application Definition.

Currently, only the (recently ported over) geolocation command is covered, but the next release will begin to see greater support for other Fusion widgets.

API Documentation

With TypeDoc, we now have API documentation for the viewer.

This only currently covers the npm module, which won't really make much sense if you're trying to use the viewer from a "browser globals" context, which would be the case if you're trying to interact with the viewer from Task Pane content.

As a workaround, I've covered the applicable parts of the "browser globals" API in the ...

New Project Home Page

And this API documentation is linked from a brand-spanking new project home page!



Surprisingly, this was much harder to set up than originally thought. Turning on GitHub pages and getting it to link to the TypeDoc-generated API documentation was made difficult due to:
  • TypeDoc-generated documentation being prefixed with underscores
  • HTML files with underscores having a special meaning in Jekyll, the default static site generator for GitHub Pages that prevents us from linking to them.
Jekyll being written in Ruby, made dev/testing my project home pages on Windows a royal pain, so I had to look at other static site generators.

After a few days of evaluating different static site generators, the closest one that matched by basic needs of: 
  • Pumping out some static HTML pages from markdown files
  • Link to TypeDoc-generated HTML files from one of them
  • Have one or more theme that exudes some polish (hey, I build web applications, not web sites and definitely not designing web sites!)
was a PHP tool called couscous. It wasn't ideal. For example, deploy totally did not work for me, but there was fortunately a workaround. Compared to the other available options, this was the best of the lot for what I needed.

Hopefully the project site is informative enough to take you to where you need to go. If not, send me some feedback.

Other Changes
  • Viewer has been updated to use the latest and greatest React, Blueprint, TypeScript and OpenLayers.
  • Viewer uses the new OpenLayers ES2015 modules which means we now only use the parts of OL that we actually use and not inflate our viewer bundle with things we aren't using.
  • Viewer now 100% uses typings provided by npm @types. This means that we no longer need to use the external typings tool to acquire d.ts files, which is relevant to the npm module as you will get automatic TypeScript type definitions for mapguide-react-layout and its dependent libraries out of the box when you install the npm module.
  • Many other updated dependencies thanks to greenkeeper.io integration.
  • The viewer now supports frame targeting (ie. Target = SpecifiedFrame and you specified a frame name) for InvokeURL and Search commands
  • The viewer now supports running InvokeURL and Search commands in a "New Window". What we will actually do is run the command in a floating modal.

Friday, 19 May 2017

MapGuide tidbits: Fusion and Bing Maps

If you use Fusion with Bing Maps, you should start applying for an API key because on June 30th, 2017, the legacy Bing Maps controls (ie. The ones that don't require an API key) will shut down.

I gather Autodesk have already taken care of this problem for their next release of AIMS, but for the rest, I'll make sure that this is backported to older Fusion branches before the June 30th deadline.

And also to make sure that Maestro has the updated Fusion Editor UI to match.

Wednesday, 10 May 2017

vcpkg is very interesting too

Part of what spurred my interest in gRPC was its ease of consumption on Windows through vcpkg allowing me to easily build and prototype some gRPC services and be able to easily interop from clients in different programming languages.

For those not in the know, vcpkg is Microsoft's attempt at a package manager for C/C++ libraries on Windows. This tool is pertinent to my interests because, look at what we currently keeping in our MapGuide/FDO source repos!



That's several GBs of source code for external thirdparty libraries we need to build MapGuide and FDO. It's a situation we can't avoid at the moment (especially on Windows) because until vcpkg, we had no choice. There was nothing akin to "nuget for C++ libraries" for Windows. On Linux we at least have the option of offloading to system installed libraries provided by the default distro package manager.

With vcpkg, this would greatly simplify the MapGuide/FDO build system on Windows. Most of the libraries listed here already exist as ports in vcpkg and most are more up-to-date as well. Imagine just doing a svn sparse checkout, skipping having to download whole parts of Oem/Thirdparty, point the MapGuide/FDO sources to our existing pre-built vcpkg packages and enjoy much faster build times, because we're not wasting a majority of it building external libraries. It may be even fast enough to continuously integrate!

It would also almost certainly spell the end of the difficult story around custom GDAL binaries as well, because I've been doing my part to add/enhance (geospatially-)relevant vcpkg ports and to light their current GDAL vcpkg port up with as many features as technically and legally possible. If we ever do get to building FDO using vcpkg-sourced external libraries, you're gonna get a GDAL/OGR provider with maximum vector/raster format support (barring the obvious omissions like Oracle, DWG, ECW, etc due to legal hurdles in obtaining and using their respective libraries and SDKs)

A vcpkg-based MapGuide/FDO build system for Windows is definitely something on my radar.

React-ing to the need for a modern MapGuide viewer (Part 16): It didn't have to be this complicated

The major theme of the next release of mapguide-react-layout is to open up the floodgates to all sorts of viewer customizations.

In particular the next release will be my first foray into npm modules as I intend to publish this next release as one. The npm module allows for the following types of customizations:

  • Custom viewer layout templates
  • Custom viewer script commands
  • Being able to selectively include/exclude viewer features to reduce bundle size
These customization options will only be available through the npm module

One major roadblock that has appeared is that we currently include a verbatim copy of our Fusion PHP backend so that certain viewer commands will work:
  • Buffer
  • Query
  • Redline
  • Theme
  • Feature Information
  • QuickPlot
  • Search
How does this roadblock us? Well with this PHP backend bundled, the npm module story is more complicated:
  • It would involve some serious webpack (or some other post-build) shenanigans to get the backend PHP content deployed in the right place, relative to your custom viewer bundle.
  • It bloats the size of the npm module (based on my simulations using npm pack).
  • I feel extremely icky publishing a JavaScript library that bundles a PHP backend as a required dependency.
In hindsight, bundling a copy of this Fusion backend is totally not necessary. It is simpler to just make our viewer commands point and talk to the existing Fusion backend. When MapGuide is installed, Fusion and PHP will always be there, so this is a safe assumption to make. It's the same environmental assumption that allows for mapguide-rest to be an easy "drop into www" and it works.

By re-using the existing Fusion backend for these tools, there also another positive side effect. It means we have to also polyfill whatever Fusion viewer APIs are needed for the frontend HTML/JS content of these Fusion widgets to work within mapguide-react-layout without modifications. What this means is that migration should be even easier as we are now polyfilling various common Fusion viewer APIs in addition to the AJAX viewer APIs.

So with this next release, the viewer will now reuse the Fusion backend of your MapGuide installation. As a result, it also means the zip packages for this release will be much smaller as well.

Tuesday, 2 May 2017

It's like the early days of the internet!

Been going back on the MapGuide dev train.

One of the features I'm experimenting with is UTFGrid rendering.

I've mentioned in previous posts, how in the process of fleshing out an idea that you don't know whether it will work or not, that there comes a flash point. A moment where your idea tips from uncertainty to "yes it can be done, so let's get it done!"

I have just reached that moment again.


It's like the early days of dial-up internet. ASCII art galore!

So off the bat, 2 things I can see:

  • The Y-axis is flipped, oops.
  • I misread the UTFGrid spec. I'm not meant to do a pixel-by-pixel replacement here.
Once these 2 things are sorted, it's just a case of mapping each character to the feature attributes (that the renderer is also tracking) and we have enough data to assemble the final UTFGrid tile

Thursday, 23 March 2017

gRPC is very interesting

MapGuide in its current form is a whole bucket of assorted libraries and technologies:
  • We use FDO for spatial data access
  • We use ACE (Adaptive Communication Environment) for:
    • Basic multi-threading primitives like mutexes, threads, etc
    • TCP/IP communication between the Web Tier and the Server Tier
    • Implementing a custom RPC layer on top of TCP/IP sockets. All of the service layer methods you use in the MapGuide API? They're all basically RPC calls sent over TCP/IP for the MapGuide Server to invoke its server-side eqvivalent. Most of the other classes that you pass into these service methods are essentially messages that are serialized/deserialized through the TCP/IP sockets. When you think about it, the MapGuide Web API is merely an RPC client for the MapGuide Server, which itself is an RPC server that does the actual work
  • We use Berkeley DBXML for the storage of all our XML-based resources
    • We have an object-oriented subset of these resource types (Feature Sources, Layer Definitions, Map Definitions, Symbol Definitions) in the MdfModel library with XML serialization/parsing code in the MdfParser library
    • Our Rendering and Stylization Engine work off of these MdfModel classes to render the maps that you see on your viewer
  • We use xerces for XML reading/writing XML in and out of DBXML
  • We use a custom modified (and somewhat ancient) version of SWIG to generate wrappers for our RPC client so that you can talk to the MapGuide Server in:
    • .net
    • Java
    • PHP
So why do I mention all of this?

I mention this, because I've recently been checking out gRPC, a cross-platform, cross-language RPC framework from Google.

And from what I've seen so far, gRPC could easily replace and simplify most of the technology stack we're currently using for MapGuide:
  • ACE? gRPC is the RPC framework! The only reason we'd keep ACE around would be for multi-threading facilities, but the C++ standard library at this point would be adequate enough to replace that as well
  • DBXML/MdfModel/xerces? gRPC is driven by Google Protocol Buffers.
    • Protobuf messages are strongly typed classes that serialize/deserialize into compact binary streams and is more efficient and faster than slinging around XML. Ever bemoan the fact you have to currently work with XML to manipulate maps/layers/etc? In .net you are reprieved if you use the Maestro API (where we provide strongly-typed classes for all the resource XML types), but for the other languages you have to figure out how to use the XML APIs/services provided by Java/PHP to work with the XML blobs that the MapGuide API gives and expects. With protobuf, you have none of these problems.
    • Protobuf messages can evolve in a backward-compatible manner
    • Because protobuf messages are already strongly-typed classes, it makes MdfModel/MdfParser redundant if you get the Rendering/Stylization engine to work against protobuf messages for maps/layers/symbols/styles/etc
    • If we ever wanted to add support for Mapbox Vector Tiles (which seems to be the de-facto vector tile format), well the spec is protobuf-based so ...
    • Protobuf would mean we no longer deal in XML, so we don't need Xerces for reading/writing XML and DBXML as the storage database (and all its cryptic error messages that can bubble up from the Resource Service APIs) can be replaced with something simpler. We may not even need a database at this point. Dumping protobuf messages to a structured file system could probably be a simpler solution
  • SWIG? gRPC and protobuf can already generate service stubs and protobuf message classes in the languages we currently target:
    • .net
    • Java
    • PHP
    • And if we wanted, we can also instantly generate a gRPC-based MapGuide API for:
      • node.js
      • Ruby
      • Python
      • C++
      • Android Java
      • Objective-C
      • Go
    • The best thing about this? All of this generated code is portable in their respective platforms and doesn't involve native code interop through "flattened" interfaces of C code wrapping the original C++ code, which is what SWIG ultimately does for any language we want to generate wrapper bindings out of. If it does involve native code interop, it's a concern that is taken care of by the respective gRPC/protobuf implementation for that language.
  • Combine a gRPC-based MapGuide Server with grpc-gateway and we'd have an instant REST API to easily build a client-side map viewer out of
  • gRPC works at a scale that is way beyond what we can currently achieve with MapGuide currently. After all, this is what Google uses themselves for building their various services
If what I said above doesn't make much sense, consider a practical example.

Say we had our Feature Service (which as a user of the MapGuide API, you should be familiar with) as a gRPC service Definition

// Message definitions for the request/response types below are omitted for brevity but basically every request and
// response type mentioned below will have eqvivalent protobuf message classes automatically generated along with
// the service

// Provides an abstraction layer for the storage and retrieval of feature data in a technology-independent way.
// The API lets you determine what storage technologies are available and what capabilities they have. Access
// to the storage technology is modeled as a connection. For example, you can connect to a file and do simple
// insertions or connect to a relational database and do transaction-based operations.
service FeatureService {
    // Creates or updates a feature schema within the specified feature source.
    // For this method to actually delete any schema elements, the matching elements
    // in the input schema must be marked for deletion
    rpc ApplySchema (ApplySchemaRequest) returns (BasicResponse);
    rpc BeginTransaction (BeginTransactionRequest) returns (BeginTransactionResponse);
    // Creates a feature source in the repository identified by the specified resource
    // identifier, using the given feature source parameters.
    rpc CreateFeatureSource (CreateFeatureSourceRequest) returns (BasicResponse);
    rpc DeleteFeatures (DeleteFeaturesRequest) returns (DeleteFeaturesResponse);
    // Gets the definitions of one or more schemas contained in the feature source for particular classes.
    // If the specified schema name or a class name does not exist, this method will throw an exception.
    rpc DescribeSchema (DescribeSchemaRequest) returns (DescribeSchemaResponse);
    // This method enumerates all the providers and if they are FDO enabled for the specified provider and partial connection string.
    rpc EnumerateDataStores (EnumerateDataStoresRequest) returns (EnumerateDataStoresResponse);
    // Executes SQL statements NOT including SELECT statements.
    rpc ExecuteSqlNonQuery (ExecuteSqlNonQueryRequest) returns (ExecuteSqlNonQueryResponse);
    // Executes the SQL SELECT statement on the specified feature source.
    rpc ExecuteSqlQuery (ExecuteSqlQueryRequest) returns (stream DataRecord);
    // Gets the capabilities of an FDO Provider
    rpc GetCapabilities (GetCapabilitiesRequest) returns (GetCapabilitiesResponse);
    // Gets the class definition for the specified class
    rpc GetClassDefinition (GetClassDefinitionRequest) returns (GetClassDefinitionResponse);
    // Gets a list of the names of all classes available within a specified schema
    rpc GetClasses (GetClassesRequest) returns (GetClassesResponse);
    // Gets a set of connection values that are used to make connections to an FDO provider that permits multiple connections.
    rpc GetConnectionPropertyValues (GetConnectionPropertyValuesRequest) returns (GetConnectionPropertyValuesResponse);
    // Gets a list of the available FDO providers together with other information such as the names of the connection properties for each provider
    rpc GetFeatureProviders (GetFeatureProvidersRequest) returns (GetFeatureProvidersResponse);
    // Gets the locked features.
    rpc GetLockedFeatures (GetLockedFeaturesRequest) returns (stream FeatureRecord);
    // Gets all available long transactions for the provider
    rpc GetLongTransactions (GetLongTransactionsRequest) returns (GetLongTransactionsResponse);
    // This method returns all of the logical to physical schema mappings for the specified provider and partial connection string
    rpc GetSchemaMapping (GetSchemaMappingRequest) returns (GetSchemaMappingResponse);
    // Gets a list of the names of all of the schemas available in the feature source
    rpc GetSchemas (GetSchemasRequest) returns (GetSchemasResponse);
    // Gets all of the spatial contexts available in the feature source
    rpc GetSpatialContexts (GetSpatialContextsRequest) returns (GetSpatialContextsResponse);
    // Inserts a new feature into the specified feature class of the specified Feature Source
    rpc InsertFeatures (InsertFeaturesRequest) returns (stream FeatureRecord);
    // Selects groups of features from a feature source and applies filters to each of the groups according to the criteria set in the aggregate query option supplied
    rpc SelectAggregate (SelectAggregateRequest) returns (stream DataRecord);
    // Selects features from a feature source according to the criteria set in the query options provided
    rpc SelectFeatures (SelectFeaturesRequest) returns (stream FeatureRecord);
    // Set the active long transaction name for a feature source
    rpc SetLongTransaction (SetLongTransactionRequest) returns (BasicResponse);
    // Connects to the Feature Provider specified in the connection string
    rpc TestConnection (TestConnectionRequest) returns (TestConnectionResponse);
    // Executes commands contained in the given command set
    rpc UpdateFeatures (UpdateFeaturesRequest) returns (UpdateFeaturesResponse);
    // Updates all features that match the given filter with the specified property values
    rpc UpdateMatchingFeatures (UpdateMatchingFeaturesRequest) returns (UpdateMatchingFeaturesResponse);
}

Running this service definition through the protoc compiler with grpc plugin gives us:
  • Auto-generated (and strongly-typed) protobuf classes for all the messages. ie: The request and response types for this service
  • An auto-generated FeatureService gRPC client ready to use in the language of our choice
  • An auto-generated gRPC server stub for FeatureService in the language of our choice ready for us to "fill in the blanks". For practical purposes, we'd generate this part in C++ and fill in the blanks by mapping the various service operations to their respective FDO APIs and its return values to our gRPC responses.
And at this point, we'd just need a simple C++ console program that bootstraps gRPC/FDO, registers our gRPC service implementation, start the gRPC server on a particular port and we'd have a functional Feature Service implementation in gRPC. Our auto-generated Feature Service client can connect to this host and port to immediately start talking to it.

The only real work is the "filling in the blanks" on the server part. Everything else is taken care of for us.

Extrapolate this to the rest of our services (Resource, Rendering, etc) and we basically have a gRPC-based MapGuide Server.

Also filling in the blanks is a conceptually simple exercise as well:
  • Feature Service - Pass down the APIs in FDO.
  • Rendering Service - Setup up FDO queries based on map/layers visible and pass query results to the Rendering/Stylization engine.
  • Resource Service - Read/write protobuf resources to some kind of persistent storage. It doesn't have to be something complex like DBXML, it can be as simple as a file system (that's what mg-desktop does for its resource service implementation btw)
  • Tile Service - It's just like the rendering service, but you're asking the Rendering/Stylization engine to render tile-sized content.
  • KML Service - Just like rendering service, but you're asking the Rendering/Stylization engine to render KML documents instead of images.
  • Drawing Service - Do we still care about DWF support? Well if we have to support this, it's just passing down to the APIs in DWF Toolkit.
  • Mapping Service - It's a mish-mash of tapping into the Rendering/Stylization engine and/or the DWF Toolkit.
  • Profiling Service - Just tap into whatever tracing/instrumentation APIs provided by gRPC.
Now because gRPC is cross-language, nothing says we have to use C++ for all the service implementations, it's just that most of the libraries and APIs we'd be mapping into are already in C++, so in practical terms we'd stick to the same language as well.

Front this with grpc-gateway, and we basically have our RESTful mapagent to build a map viewer against.

There's still a few unknowns:
  • How do we model file uploads/downloads?
  • Can server-side service implementations call other services?
Google's vast set of gPRC definitions for their various web services can answer the first part. The other part, will need more playing around.

The thought of a gRPC-based MapGuide Server is very exciting!

React-ing to the need for a modern MapGuide viewer (Part 15): Play with it on docker

Today, I found a very interesting website from the tech grapevine:

http://play-with-docker.com

What is this site? It is an interactive docker playground. If you've ever used sites like JSFiddle to try out snippets of JS/HTML/CSS, this is basically the docker equivalent to try out Docker environments.

With PWD, I now have a dead simple way for anyone who wants to try out this viewer to spin up a demo MapGuide instance on PWD for themselves to check out the viewer.

Once you've proved to the site that you're are indeed a human and not a robot, you will enter the PWD console. From here, click + ADD NEW INSTANCE to start a new shell.



Then run the following commands to build the demo docker image and spin up the container

git clone https://github.com/jumpinjackie/mapguide-react-layout
cd mapguide-react-layout
./demo.sh

After a few minutes, you should see a port number appear beside the IP address



This is a link to the default Apache httpd page that is confirmation that the demo container is serving out web content to the outside world.



Now simply append /mapguide/index.php to that URL to access the demo landing page for this viewer. Pick any template on the list to load the viewer using that template.



You now have a live demo MapGuide Server with mapguide-react-layout (and the Sheboygan dataset) preloaded for you to play with to your heart's content for the next 4 hours, after which PWD will terminate your session and all the docker images/containers/etc that you created with it.

This was just one use case that I thought up in 5 minutes after discovering this awesome site! I'm sure there's plenty of other creative uses for such a site like this.

Many thanks to brucepc for his MGOS 3.1 docker image from which the demo image is based from.

Wednesday, 8 March 2017

React-ing to the need for a modern MapGuide viewer (Part 14): The customization story so far.

I've been getting an increasing amount of questions lately about "How do you do X?" with mapguide-react-layout. So the purpose of this post is to lay out the customization story so far, so you have a good idea of whether the thing you want to do with this viewer is possible or not.

Before I start, it's best to divide this customization story into two main categories:

  1. Customizations that reside "inside" the viewer
  2. Customizations that reside "outside" the viewer
What is the distinction? Read on.

Customizations "inside" the viewer

I define customizations "inside" the viewer as customizations:
  • That require no modifications to the entry point HTML file that initializes and starts up the viewer. To use our other viewer offerings as an analogy, your customizations work with the AJAX/Fusion viewers as-is without embedding the viewer or modifying any of the template HTML.
  • That are represented as commands that reside in either a toolbar or menu/sub-menu and registered/referenced in your Web Layout or Application Definition
  • Whose main UI reside in the Task Pane or a floating or popup window and uses client-side APIs provided by the viewer for interacting with the map.
These customizations are enabled in our existing viewer offerings through:
  • InvokeURL commands/widgets
  • InvokeScript commands/widgets
  • Client-side viewer APIs that InvokeURL and InvokeScript commands can use 
  • Custom widgets

    From the perspective of mapguide-react-layout, here is what's supported

    InvokeURL commands

    InvokeURL commands are fully supported and do what you expect from our existing viewer offerings:
    • Load a URL (that normally renders some custom UI for displaying data or interacting with the map) into the Task Pane or a floating/popup window.
    • It is selection state aware if you choose to set the flag in the command definition.
    • It will include whatever parameters you have specified in the command definition into the URL that is invoked.
    If most/all of your customizations are delivered through InvokeURL commands, then mapguide-react-layout already has you covered.

    InvokeScript commands

    InvokeScript commands are not supported and I have no real plans to bring such support across. I have an alternate replacement in place, which will require you to roll your own viewer. 

    Client-side viewer APIs

    If you use AJAX viewer APIs in your Task Pane content for interacting with the map, they are supported here as well. Most of the viewer APIs are mostly implemented, short of a few esoteric APIs.

    If your client-side code is primarily interacting with APIs provided by Fusion, you're out of luck at the moment as none of the Fusion client-side APIs have been ported across. I have no plans to port these APIs across 1:1, though I do intend to bring across some kind of pub/sub event system so your client-side code has the ability to respond to events like selection changed, etc.

    Custom Widgets

    In Fusion, if InvokeURL/InvokeScript widgets are insufficient for your customization needs, this is where you would create a custom widget. Like the replacement for InvokeScript commands I intend to enable a similar system once again through custom builds of the mapguide-react-layout viewer.



    My personal barometer for how well mapguide-react-layout supports "inside" customizations is the MapGuide PHP Developer's Guide samples. 



    If you load the Web Layout for this sample in the mapguide-react-layout viewer, you will see all of the examples (and the viewer APIs they demonstrate) all work as before. If your customizations are similar in nature to what is demonstrated in the MapGuide PHP Developer's Guide samples, then things should be smooth sailing.


    Customizations "outside" the viewer

    I define customizations "outside" the viewer as primarily being one of 2 things:
    • Embedding the viewer in a frame/iframe or a DOM element that is not full width/height and providing sufficient APIs so that code in the embedding content document can interact with the viewer or for code in the embedding content document to be able to listen on certain viewer events.
    • Being able to init the viewer with all the required configuration (ie. You do not intend to pass a Web Layout or Application Definition to init this viewer)
    On this front, mapguide-react-layout doesn't offer much beyond a well-defined entry point to init and mount the viewer component.

    Watch this space for how I hope to tackle this problem.

    Rolling your own viewer

    The majority of the work done since the last release is to enable the scenario of being able to roll your own viewer. By being able to roll your own viewer, you will have full control over viewer customization for things the default viewer bundle does not support, such as:
    • Creating your own layout templates
    • Creating your own script commands
    • Creating your own components
    If you do decide to go down this path, there will be some things that you should become familiar with:
    • You are familiar with the node.js ecosystem. In particular, you know how to use npm/yarn
    • You are familiar with webpack
    • Finally, you are familiar with TypeScript and have some experience with React and Redux
    Basically, if you go down this road you should have a basic idea of how frontend web development is done in the current year of 2017, because it is no longer manually editing HTML files, script tags and sprinkles of jQuery.

    Because what I intend to do allow for this scenario is to publish the viewer as an npm module. To roll your own viewer, you would npm/yarn install the mapguide-react-layout module, write your custom layouts/commands/components in TypeScript, and then set up a webpack configuration to pull it all together into your own custom viewer bundle.

    I hope to have an example project available (probably in a different GitHub repository) when this is ready that demonstrates how to do this.

    In Closing

    When you ask the question of "How can I do X?" in mapguide-react-layout, you should reframe the question in terms of whether the thing you are trying to do is "inside" or "outside" the viewer. If it is "inside" the viewer and you were able to do this in the past with the AJAX/Fusion viewers through the extension points and APIs offered, chances are very high that similar equivalent functionality has already been ported across.

    If you are trying to do this "outside" the viewer. you'll have to wait for me to add whatever APIs and extension points are required.

    Failing that, you will have the ability to consume the viewer as an npm module and roll your own viewer with your specific customizations.

    Failing that? 

    You could always fork the GitHub repo and make whatever modifications you need. But you should not have to go that far.

    Tuesday, 21 February 2017

    React-ing to the need for a modern MapGuide viewer (Part 13): My first* pull request

    Previously, I switched our testing stack for mapguide-react-layout over to Jest, which had some positive flow-on effects, like being able to finally upgrade to Webpack 2 and being able to try out the new OpenLayers npm package, resulting in a nice reduction in production bundle size due to only pulling the bits of OpenLayers that we are actually using. Jest also has code coverage built in, and by piping its coverage output to node-coveralls, TravisCI will automatically upload said coverage reports to coveralls.io resulting in yet another shiny badge to show on our project page.

    These badges are becoming like Pokemon: I just want to catch 'em all.

    So the next badge for me to collect was greenkeeper. Greenkeeper is a free service that monitors your GitHub repository and keeps your node package dependencies up to date. So last night I enabled greenkeeper integration for mapguide-react-layout.

    Today I got a GitHub notification for a new pull request on mapguide-react-layout. Great! I love pull requests. Except, this pull request is not from a human, it's from the greenkeeper bot (*my first non-human pull request). Looking at the pull request in detail was most amusing.


    A bot (coveralls) commenting on a pull request opened by another bot (greenkeeper)!

    I wonder how many pull requests out there are nothing but full of bot-on-bot comments? How deep does this bot rabbit hole go?

    When bots can start writing their own code, I think that's when we can pack it in as the human race and submit to our bot overlords.

    Monday, 20 February 2017

    React-ing to the need for a modern MapGuide viewer (Part 12): A positive cascading effect

    The move to Jest for our testing/coverage needs has opened up some opportunities that were previously roadblocks.

    Mainly, we can finally upgrade to Webpack 2. Previously, we were roadblocked because the karma runner just wouldn't work with webpack 2 configurations. Also unlike earlier attempts with Webpack 2 beta releases, this upgrade to Webpack 2 was less painful and more importantly, the bundle size remained the same.

    Also OpenLayers recently released 4.0.0, which also includes experimental ES2015 modules, the ES2015 module facilitates a "pay for only what you use" model which is great for us as we don't necessarily want to use the kitchen sink, only the parts of the library we actually use. It turns out based on their webpack example that it requires Webpack 2 to work as Webpack 1 will include said modules verbatim causing most browsers to blow up on the various ES2015 language constructs (like imports).

    Well, how convenient that we just upgraded to Webpack 2! Switching over to the new ol package and its ES2015 modules, and making the required fixes in our codebase to use this new package, and checking the final production bundle size shows promise.



    That is 150kb smaller than our current production bundle! Once other libraries we're using adopt ES2015 modules, we can expect even more weight loss.

    Saturday, 18 February 2017

    React-ing to the need for a modern MapGuide viewer (Part 11): I don't say this in jest

    ... but seriously, Jest completes my holy trinity of web front-end development nirvana.

    • React, because of its high performance and revolutionary component-based way of building frontend UIs. I could never go back to jQuery, data-binding, string templating and those other primitive ways of building frontends.
    • TypeScript, because it is in my opinion, the only sane programming language for frontend development. Just like jQuery was the glue that held together inconsistent browser APIs for many years, TypeScript the the glue that lets us play with future JS technologies in the present. TypeScript is JavaScript with C#-quality static typing. I love that a whole class of errors are eliminated through a simple compile step. I can't fathom having to maintain large code bases in a dynamically-typed language. TypeScript brings order and sanity in that regard. And with TypeScript 2.0, I don't have to deal with billion dollar mistakes.
    • And finally, Jest which I believe to be the total package for JavaScript unit testing that is sooooooo easy to set up! Code coverage is also included.
    Before I tried Jest, the current unit test suite for mapguide-react-layout was a convoluted stack of:
    I also tried to get code coverage working. But this required a tool called istanbul and because my code was TypeScript, it needed some TypeScript source map plugin for istanbul to recognise it, which resulted in some rube-goldberg-esque contraption that formed the foundation of my unit test suite, that didn't even get the coverage right! So I scrapped the code coverage part and just coasted along with karma/mocha/chai combo until now.

    With the introduction of Jest, it does the job of karma/mocha/chai/istanbul in a single, easy to install package. Only some small changes to my unit test suite was required (porting chai assertions to their jest equivalents) and the whole test suite was passing. With a simple addition of a --coverage flag to jest, it then automagically generates code coverage results and reports.



    Since I am now getting code coverage reports, the obvious next step was to upload this to the coveralls.io service. It turns out TravisCI already supports automatic upload of code coverage reports to coveralls. It just needed installing node-coveralls and piping the jest coverage output to it.

    And with that, I get another shiny badge to brandish on the project home page


    This is almost too easy. Results like this easily incentivize you to write good quality code.

    One last thing before I close out this post. The 63% coverage is a bit of a misnomer. It turns out that it is actually the percentage of code in modules that your unit tests are currently testing, which makes sense. The moment I start bringing in other components and classes under test, I expect this percentage to plummet, which is merely incentive to write more tests to bump it back up.

    Thursday, 16 February 2017

    Announcing: mapguide-react-layout 0.8

    Here's a new release of mapguide-react-layout

    Here's what's new in this release

    Multiple Map Support

    If you load an Application Definition with multiple map groups, the viewer now properly supports them.



    Thanks to the use of redux (as my previous blog adventure post explained), map state is all nicely isolated from each other and makes it easy for components and commands to easily be aware of multiple maps, such is the case of the measure component (notice how recorded measurements switch along with the active map)


    Also for Task Pane content, we added some smarts so that you know whether current task pane content is applicable or not to the current active map.



    Other Changes

    • Update Blueprint to 1.9.0
    • Update React to 15.4.2
    • Improved performance of redux aware components to avoid unnecessary re-rendering
    • Sidebar Template: Fix a small sliver of the Task Pane content visible when collapsed
    • Legend: Fix infinite loop on maps with multiple (>2) levels of group nesting
    • Hover styles no longer render for disabled toolbar items
    • Clicking an expanded panel in an accordion no longer collapses it (an expanded panel should be collapsed by clicking another collapsed panel). This affects viewer templates that use accordions (eg. Slate)
    • Added support for InvokeURL command parameters
    • Fix default positioning of modal dialogs


    Download

    Monday, 6 February 2017

    Announcing: MapGuide Maestro 6.0m6

    Here's another long overdue new milestone release for MapGuide Maestro 6.0.

    Here's a summary of what's new.

    Friendlier connection error dialog

    Rather than say MgConnectionFailedException, perhaps this dialog can explain things better?


    Theme support

    The latest version of DockPanelSuite supports theming. Maestro now includes a bunch of VisualStudio-style skins that you can choose from the options UI. For example, here's Maestro with the Visual Studio "blue" theme.


    Non-modal map previews

    Resource previews that use the local map viewer (ie. It doesn't launch an external web browser), now display as additional document-resident view content instead of a modal dialog.


    MgCooker

    Rather than punching in coordinates for override extents manually, you can now set the extents interactively with Maestro's map viewer component.


    Also on that subject, when generating tiles with overridden extents, the total tiles calculation should be more accurate now, having verified the overridden extents against the contents of the physical tile cache, the visual verification made much easier with this new feature.

    Other Changes
    • Maestro now has new validation rules for ODBC configuration documents
    • You can now enter a Google Maps API key in the Flexible Layout editor (if you require it)
    • The mapguide-rest addin now generates configuration documents with valid representation nodes
    • Fix NullReferenceExceptions when editing Fusion Flexible Layouts with multiple map groups. A nice timely fix to have to make way in preparation for when this is ready.
    • Upgrading a WebLayout to a 2.4 or newer schema version will now include the MapTip command if it doesn't exist
    • Upgrading a Layer Definition to a 2.4 or newer schema version no longer discards any URL settings.
    • XML editor now automatically trims off null '\0' characters (the source of many bogus content validation errors)
    • FDO stylization functions are no longer falsely reported as unsupported functions by the FDO expression validator.
    • Fusion and Symbol Definition editors should now trigger dirty state more often.
    • IronPython updated to 2.7.7


    Tuesday, 31 January 2017

    A few days layer ...

    Not only does AppVeyor gives us continuous integration, it can give us continuous delivery as well.

    So after getting the newly migrated MapGuide Maestro on GitHub hooked up to AppVeyor for CI and automated publishing of code coverage results to Coveralls, I wanted to look at how AppVeyor can be used for automating the creation of GitHub releases. The ideal scenario, is that when I decide to put out a release, I hit some button or run some command and a new GitHub release will show up a few minutes later.

    It turns out, AppVeyor has a whole section on their documentation dedicated to this very subject! Some tweaks were required to the AppVeyor build system so that it also builds the API/User documentation and zip/installer packages and then by specifying these artifacts in the artifacts section of appveyor.yml, they will appear in the artifacts section of the AppVeyor build page.

    What that means is that you now have access to the latest builds of MapGuide Maestro straight after AppVeyor builds and validates them. However, this is not the way I want you to access official releases of Maestro, I still want GitHub releases created for this particular case, and ideally in an automated fashion.

    This is where we set up our AppVeyor configuration to deploy using the GitHub deployment provider, with the trigger being a push of a git tag. What this means in practice is when I decide to put out an official release, I make the release tag in Git, push it out to GitHub. This triggers AppVeyor to do its usual CI and artifact packaging. But because this is a release tag, it will also automatically create a corresponding GitHub release in a draft state and uploads the associated artifacts of this build with it.

    From there, I can visit the draft release page and tidy up the release description and add release notes, click the publish button and voila! A new release is ready for you to download. How well is this automated process? Check out this test release and find out for yourself.

    I'm really liking this well-oiled CI/CD machine, further solidifying my justification for making the move to GitHub in the first place. Oh, and as the above screenshot shows, I also got my first pull request today for Maestro too. Things are looking up!

    Friday, 27 January 2017

    A few hours later ...

    Moving MapGuide Maestro to GitHub has already paid off!

    The code is hooked up to AppVeyor for automatic continuous integration

    Our unit test suites are hooked up to OpenCover, whose reports AppVeyor will automatically upload to coveralls.io.

    And these services are all free!

    And we get some nice badges to show for it!

    So just to give some context for that coverage result: This coverage is for the MaestroAPI part of Maestro. It does not cover the WinForms application itself, which I find trying to write automated end-to-end tests (and hence coverage) to be a fruitless endeavor. It will either work or not, that's as good enough of a test as any. MaestroAPI on the other hand, is the engine that drives the application and this is something that definitely needs the test validation and coverage as it has uses outside of the Maestro application and we need objective confidence in the stability and robustness of the MaestroAPI, which these aforementioned services will give us.

    Thursday, 26 January 2017

    Announcing: MapGuide Maestro is moving to GitHub

    After some thoughts and considerations, I've decided to move MapGuide Maestro out of the MapGuide Open Source SVN repo and out into its own GitHub repository.

    The primary motivation was performance (git is way faster than svn and enables more productive development) and secondly, the "pull request" model of GitHub and other DVCSes is more conductive to external contributions than what we currently have now, which is code sitting as a sub-project in a SVN repository that requires special vetting and access for commit access to be granted.

    Whereas with GitHub and friends it's just fork the repository, make your changes and send back a pull request. At no point in the above process do I ever have to get involved beyond reviewing and accepting any pull requests that comes my way. Less administrative burden on my end, less bureaucratic road blocks for external contributions on yours. Win win for everyone!

    As of this post, I will cease to make any more Maestro-related commits to the SVN repo, I'll be migrating all Maestro-related issues and wiki content over to GitHub. Do not make anymore Maestro-related issues/tickets on the MapGuide Trac instance, make them on the GitHub project instead. Once the migration of issues and wiki content has completed, I'll remove Maestro from the SVN repo. Old releases will stay on the OSGeo download server, but new ones will be on GitHub.

    Thursday, 19 January 2017

    React-ing to the need for a modern MapGuide viewer (Part 10): The end goal of this refactoring

    This post is a little backstory as to why I needed to do some behind-the-scenes refactoring.

    It is so we can finally bring across this important feature from Fusion and cross off another TODO list item in the process: Support for multiple maps!


    To finally support multiple maps, we had to re-structure our redux state tree so that:

    1. All map-specific state (selections, layer/group toggling state, current view/scale, etc) now exist in individual branches under the mapState reducer. Each branch is keyed on the name of the respective runtime map.



    2. A new active map name property is added to the configuration state branch. We then simply change our respective mapStateToProps implementations in these components to interrogate this new property to determine which map state sub-branch to read their required map state from. The beauty of this is that the state that these redux-aware components pass down to child components is mostly un-changed. They are all insulated from having to know anything about the current map.



    3. To service the MapMenu component, we store the array of available map names along side the new active map name property. The entries in the MapMenu component are bound from this array. Any change from this component dispatches an action to update the active map name, and all map-related components will then auto-magically update and re-render themselves to reflect the state of the new active map.



    With this set up we get a nice encapsulation of map state as everything map-related now drives off of the active map name (that the MapMenu component has the sole responsibility of toggling), as best demonstrated by this animation below


    The above animation demonstrates that map-specific state like selections, layer/group visibility, current view, external base layers, etc are all perfectly isolated within their respective redux state branches and the mere act of switching maps is just deciding which particular state branch we consider to be the "active" map. It is not possible for states in other maps to "bleed over" into the current map. We never got this kind of encapsulation or safety guarantees with the current Fusion implementation and no doubt is the cause of many bugs with Fusion and AppDefs with multiple Map Groups.

    There's still some aspects of the viewer that still need verification under a multi-map configuration, but for the most part this is working pretty well.

    Wednesday, 11 January 2017

    React-ing to the need for a modern MapGuide viewer (Part 9): It's like a Christmas tree!

    After figuring out how to make an accordion and getting the 0.7 release out the door, I needed to do some necessary refactoring of how we're currently modelling application state in redux to make my intended marquee feature of the next release possible.

    After the refactoring, I saw some noticeable sluggishness in the viewer across all templates. This is where my choice to build this viewer on top of React has been validated (once again for the umpteenth time!), as it comes with top-notch developer tool support. So I flicked over to the React Developer Tools, activated update tracing and lo and behold ...


    The viewer lit up like a Christmas tree, without any user interaction! What this was showing was that some React component was constantly updating and re-rendering itself and the sluggishness I was experiencing was in part due to that.

    So it was clear that the refactoring caused some component to constantly be updating. But what?

    That's where I turned to a new feature in the latest 15.4 release of React, which is by simply appending react_perf to the query string


    It activates performance profiling in React itself. I then switched to the Timeline tab in the Chrome devtools and recorded 5 seconds of this "idle" activity


    And we see there is activity that corresponds to this constant updating/re-rendering. If we drill down to one of these hot spots.


    We finally identify the culprit:

    • It is the map viewer component
    • Something is causing the busy count to constantly go up and down. We store busy count as redux state so that the Navigator (aka. Zoom slider) component can listen on to determine whether to show/hide the "busy" animating indicator at the top of the component
    The incrementBusyWorker and decrementBusyWorker functions are registered handlers for the imageloadstart and imageloadend events provided by OpenLayers MapGuide image sources. The only way these events would constantly fire from OpenLayers would be if something was constantly refreshing the map.

    Which meant the possible suspect location is most likely in the component's componentWillReceiveProps lifecycle hook function which we use to interact with OpenLayers in response to component prop changes.


    I stuck a breakpoint here to see if it was being hit while "idle", and it did!


    While paused in this breakpoint, I evaluated the above two expressions to see what was being compared

    And there was the problem! My visibility difference check was insufficient as it only did a shallow object reference comparison. The possible cause for this behaviour? My refactoring and over-zealous use of the new object spread operator in TypeScript 2.1 for shallow state cloning in my various redux reducers exposed and brought this problem into the light.

    The fix was two parts. Firstly, to replace these (obviously incorrect) checks with helper functions that not only check the references (a null/undefined to set reference transition or vice versa is still a legit difference), but the actual object properties themselves.

    Secondly, the layerGroupVisibility prop was a nested object prop that was being created in our component's connect function. This is actually a bad thing, which upon further reading makes perfect sense as returning new objects made in connect() will break the shallow object comparison that redux will do against a previous connect-invocation to short-circuit unnecessary component re-rendering. The fix here is to flatten this particular prop by replacing the layerGroupVisibility prop with the 4 individual array props and update all prop usages/references in the component code accordingly.

    With this change in place, turning on React update tracing again shows that it no longer flashes around like a Christmas tree.


    So what were the lessons learned here?
    • Component props and state, and how you structure and compare them are important when it comes to performance. The endless articles out there on the art of properly implementing shouldComponentUpdate underscores how important this fact is.
    • On the same vein, Redux has its own set of best practices for performance. This is good reference.
    • React's developer tooling is top-notch, and allows for performance issues like this one to be easily identified and debuggable.
    Now to make sure this major refactoring didn't regress any other stuff.