Wednesday, 17 May 2023

Announcing: FDO Toolbox 1.5.3

Here's the first stop of the tour, a new release of FDO Toolbox.

I knew that a new release of FDO Toolbox would be coming when I was using the previous 1.5.2 release and it spectacularly failed in my dogfooding of trying to load some GDA2020 SHP files to SQL Server. The actual problem was actually pretty minor, but this inevitably started a chain of dealing with many other annoyances and reported issues, culminating in this release you see here.

Here's a summary of notable changes in this release.

The configuration support check has been fixed so that the configuration doc field is enabled based on actual provider capability check instead of the previously (dumb) approach of checking the provider name. This means the configuration document field is no longer disabled when connecting to PostgreSQL/PostGIS databases and one can finally supply a XML configuration document to apply schema overrides!


The data store editor in standalone mode now supports deleting schemas, to support the common FDO schema override use case of trimming out extraneous schemas and feature classes.



The annoyance of loading a saved bulk copy definition file and a whole bunch of connections with "Connection1/Connection2/Connection3" names being created is now fixed. We will now try to use the original name on the definition file if there is no open connection using the same name.


Also did you know that FDO Toolbox has a neat little feature to help you easily visualize geometry WKT?


You probably didn't know because this feature was hidden in the depths of the FDO Expression Editor when editing FDO expressions or filters and could not be accessed on its own.

That's why in this release, the Geometry Visualizer is now also accessible from the Tools menu.

Finally, to return back to the original issue motivating the production of this new release, the ExtendedCoordSys.txt support file for the SQL Server FDO provider has been updated to match the copy from current FDO trunk and has been updated with a CS alias mapping for GDA2020, allowing one to create spatial contexts on a SQL Server data store with this particular coordinate system (and many others!)

Now onto the next tour stop!

Download

Friday, 5 May 2023

The "I haven't forgotten about you $PROJECT" tour

As I said near the end of my MGOS 4.0 Beta 1 announcement, I will be momentarily stepping away from MapGuide development/maintenance to give some of my sibling projects in the MapGuide/FDO space some long-overdue attention.

So for the next few weeks, I will be embarking on the "I haven't forgotten about you $PROJECT" tour with stopovers at the following projects:

mapguide-react-layout

Next Release: 0.14.8

Objectives: This will be the last release in the 0.14.x series and the last release to officially support Internet Explorer.

MapGuide Maestro

Next Release: 6.0m12

Primary Objective: This release will have authoring parity with MapGuide Open Source 4.0 Beta 1
Secondary Objective: Assorted fixes and minor enhancements

FDO Toolbox

Next Release: 1.5.3

Objective: Some fixes for bugs and usability problems reported since 1.5.2

mapguide-rest

Next Release: 1.0 RC6

Yes! I am finally giving this project some very-overdue attention!

Primary Objective: Make sure it is compatible with PHP 8.1 that is bundled with MapGuide Open Source 4.0 Beta 1. Also doubles as a secondary sanity test of the new vanilla SWIG-generated PHP bindings, which is my main motivator for revisiting this project.

Secondary Objective: See if we can do this will still being able to support older PHP versions bundled with older MapGuide versions. If this is not tenable, we may need to branch codebases and make separate releases. I hope that this is not the case.


When I finish this tour, then I will be returning to get MapGuide Open Source 4.0 to the final release finish line



Wednesday, 5 April 2023

Announcing: MapGuide Open Source 4.0 Beta 1

Due to real life priorities and commitments, this release look longer than expected to finally come to fruition, but that's all in the past because we're finally here! The first (and probably only) beta release of MapGuide Open Source 4.0 is finally available for the following platforms:

  • Windows
  • Ubuntu Linux 22.04
  • CentOS Linux 7.0
Some notable changes / improvements over the previous Preview 4 release are detailed below.

GeoJSON output improvements

The GeoJSON output support has several improvements in this release:
  • Where possible, the GeoJSON will now always include the crs property (if the source data's coordinate system has an EPSG code representation), allowing for the GeoJSON content to self-identify its coordinate system.
  • GeoJSON from WFS GetFeature requests will now properly transform feature data if a different SRS/CRS is specified in the query parameters
  • Such requests that return GeoJSON content now properly return a application/json mime type

Additional security options for mapagent hardening

Currently, an Anonymous (guest) user or session can read any resource in a MapGuide Server's repository via the mapagent HTTP endpoint. While this capability is required for certain client applications to work properly (for example, mapguide-react-layout needs to be able to read WebLayout/ApplicationDefinition/MapDefinition resources as part of viewer initialization), it doesn't mean that *every* resource should be allowed to be read by anonymous users/sessions.

In particular, Feature Sources rarely need to be read by anonymous users/sessions and it may be considered a security risk to some that connection settings in such Feature Sources (especially ones that connect to relational databases) can be read by anonymous users/sessions, exposing names of internal db servers in the process.

The current resource permission model in MapGuide does allow for read access certain resources to be denied (in their resource headers), but this model is a sledgehammer approach. (ie. It will break rendering operations that need to query data from a feature source you just denied access to in their resource header).

We need a more fine-grained approach where we can deny direct resource API access operations to things like feature sources, but still allow resource API access operations to such resources in the context of things like map rendering.

This release introduces several new webconfig.ini properties to help reduce the attack surface of the mapagent in this regard.
  • AnonymousDenyGetResourceContent
  • AnonymousDenyGetResourceData
  • AnonymousDenyGetResourceHeader
These properties accept a comma-delimited list of resource ids or resource id prefixes and when set, if an anonymous user/session attempts a GETRESOURCECONTENT, GETRESOURCEHEADER or GETRESOURCEDATA with a resource id that matches any id or prefix specified, the mapagent will deny them access to that resource.

With this feature, you can reduce the attack surface of your mapagent by reducing anonymous resource access to only resources needed for a MapGuide client application to function.

Please note that this feature only covers the mapagent and not your custom application code.

Web Tier Component updates

This release bundles updated web tier components:
  • PHP 8.1.17
  • Apache httpd 2.4.56
  • Apache Tomcat 9.0.73

Some long-standing bugs/limitations finally fixed!

A 14-year old bug where un-formatted WebLayout XML cannot be loaded has finally been fixed.

Another 14-year old limitation where labels under basic stylization is always left-justified has been addressed with new options in the 4.0.0 Layer Definition schema allowing you to control the feature label justification.

Sorry for the long turnaround on such issues. Sadly, one person can only do so much.

What's next?

Before we begin the journey to Release Candidate (and then Final release). I will be stepping away from MapGuide development/maintenance work for a few weeks to give some of the surrounding projects like Maestro and mapguide-react-layout some long needed attention, and I expect new releases of MapGuide Maestro and mapguide-react-layout during this period as a result.

Once that is out of the way, then it is back onto the 4.0 release train, driving it to its final destination.

Friday, 4 November 2022

Announcing: MapGuide Open Source 4.0 Preview 4

After a long an arduous development journey, it is finally here! The 4th preview release of MapGuide Open Source 4.0 is now available for the following platforms:

  • Windows
  • Ubuntu Linux 22.04
  • CentOS Linux 7.0
The main feature new to the preview 4 release is the one feature that took the longest time to finish, but was something that needed to be done: We have overhauled all of our MapGuide API language bindings to now be generated with the official version of SWIG instead of our ancient and heavily modified version.

It is through this work that are finally able to drop PHP 5.6 and bundle a newer version of PHP, which for this release is PHP 8.1.11.

On the .net front, this release now offers the .net bindings as a series of netstandard2.0 nuget packages. These packages can be found in the Web/nuget directory of your preview 4 installation. Come the final release of MapGuide Open Source 4.0, we will publish these packages to the official NuGet gallery and will replace the existing MapGuide API nuget package going forward.

On the Java front, we no longer ship MapGuideJavaApi.jar (the "crufty" wrapper) and MapGuideJavaApiEx.jar (the "non-crufty" wrapper) together. We now just ship a single MapGuideJavaApi.jar, which is based on the "non-crufty" wrapper. If your Java MapGuide application has been working against MapGuideJavaApiEx.jar, then migration back to MapGuideJavaApi.jar  is 99.9% seamless.

While this release is not deemed production ready, you should use this release as a means to start getting your MapGuide applications ready for the 4.0 final release. For all 3 languages, refer to the RFC for an overview of what migration you will need to perform on your codebase.

After this release, I'm not intending to introduce or implement any new features in the 4.0 timeframe. It will be all bug fixes and library updates (where required) going forward, so I'm expecting a relatively short Beta > RC > Final cycle.

Saturday, 10 September 2022

MapGuide dev diary: It is done!

In the previous installment, we had gotten a test MapGuide installation with bundled PHP 8.1 up and running, and we were able to successfully produce a PHP error when hitting the Site Administrator.

This PHP error referenced previously was a milestone because it meant that our PHP 8.1 setup (FastCGI on Apache via mod_fcgid) was working, our PHP code was actually running and so the actual errors is the result of the vast swaths of our current PHP web tier applications needing to be migrated across to work against this new PHP 8.1 binding for the MapGuide API.

And so for the next few months I did just that, not just for PHP, but also for Java and .net. You can consider this post to be a preview of what you'll need to do yourself if you want to migrate your PHP/Java/.net MapGuide applications to MGOS 4.0.

PHP Migration Overview

The PHP binding was the one I expect to be the most work because besides finally supporting PHP 8.1, the major feature of this new PHP binding is that constants.php is no longer required! This is because with vanilla SWIG we now can bake all the constants of the MapGuide API into the PHP binding extension itself! So I expect a lot of  "include 'constants.php'" references needing to be removed.

Once all the constants.php references are removed, we found the prime issue with this new PHP binding. PHP didn't like some of our C++ classes had overloaded methods whose signatures did not exist in the parent class. This manifested in the form of fatal PHP errors like the following:

PHP Fatal error:  Declaration of MgCoordinateSystemMeasure::GetDistance(MgCoordinate|float|null $arg1, MgCoordinate|float|null $arg2, float $arg3, float $arg4): float must be compatible with MgMeasure::GetDistance(?MgCoordinate $arg1, ?MgCoordinate $arg2): float in Unknown on line 0

In this case, our MgMeasure class has a GetDistance method of the following signature:

double GetDistance(MgCoordinate* coord1, MgCoordinate* coord2)

In the derived MgCoordinateSystemMeasure, it has a new overload of GetDistance that has this signature:

double GetDistance(double x1, double y1, double x2, double y2)

However, when this converted to PHP proxy classes by SWIG, PHP doesn't like this class setup because under its inheritance model, it is expecting the GetDistance overload with 4 double parameters to also exist in the base MgMeasure class. This is not the case, and thus PHP throws the above fatal error.

To workaround this problem, we had to use the SWIG %rename directive to rename the conflicting overload signature in MgCoordinateSystemMeasure in PHP to the following:

double GetDistanceSimple(double x1, double y1, double x2, double y2)

With this rename, there is no longer a signature conflict in the generated proxy class and PHP no longer throws a fatal error. Fortunately, only 2 classes in the MapGuide API have this problem, so the amount of method renaming is minimal.

Once this issue was addressed, I tried firing up the PHP implementation of the AJAX viewer and I got an interactive map! Everything seemed to be working until I tried to generate a map plot, and found my second problem. I was getting PHP fatal errors like this:

PHP Fatal error:  Uncaught TypeError: No matching function for overloaded 'MgRenderingService_RenderMap' in quickplotgeneratepicture.php:115

Fortunately, this one was easier to explain and fix. In PHP 8.1 (maybe even earlier in the PHP 7.x series), the type checking became more stricter which meant int parameters must take integers, double parameters must take doubles, etc, etc, you couldn't pass ints as doubles or vice versa, and for a method like RenderMap of MgRenderingService, there are lots of overloads that take many different combinations of int and double parameters.

Our map plotting code was passing in strings where int/double parameters were expected. In PHP 5.6 this was allowed because the type checking was evidently more lax. Now such cases cause the above PHP fatal error. This was easy enough to fix, we just use the intval and doubleval functions to make sure ints are being passed into int parameters and doubles are being passed into double parameters.

And with that, the rest of the changes involving fixing up our exception handling code due to a major change with how MapGuide applications should be handling exceptions from the MapGuide API. As part of this SWIG binding work, we've flattened the MapGuide exception hierarchy into a single MgException class, and introduced a new exception code property to allow handling MapGuide exceptions on a case-by-case basis.

So if you were handling specific MapGuide exceptions like this:

1
2
3
4
5
6
7
8
try
{
   //Some code that could throw
} 
catch (MgUnauthorizedAccessException $e) 
{
    ...
}

You would now rewrite them like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
try
{
   //Some code that could throw
} 
catch (MgException $e) 
{
    if ($e->GetExceptionCode() == MgExceptionCodes::MgUnauthorizedAccessException) {
        ...
    }
}

The reason for flattening the exception hierarchy was:

  • To make wrapping exceptions simpler (we now only need to wrap the sole MgException class in SWIG) and not have to handle exception class inheritance chains in a consistent manner across all 3 language bindings.
  • Most of the example MapGuide API code pretty much only caught MgException anyways and rarely catches any of its derived exception classes (and I imagine that this is the case in your MapGuide applications as well). Any code that cared to handle specific exception cases, we can just include the relevant sub-classification as a property of MgException itself as the above code example shows.
Once this final change was made, the AJAX viewer was fully functional. Porting Fusion to PHP 8 was a similar process.

Java Migration Overview

This migration I expect to be a cakewalk because although this binding is now also being generated by vanilla SWIG, it is 99% identical to the existing MapGuideJavaApiEx.jar that we have been generating and shipping for many releases of MapGuide.

So all I expect is just to:
  • Fix up references to MapGuideJavaApiEx and rename them to MapGuideJavaApi
  • Update exception handling blocks to take action based on the captured exception code in the caught MgException instead of catching for specific subclasses of MgException
  • Replace MapGuideApiEx.jar with MapGuideApi.jar (we're using the old jar name for the new binding) in our MG installation (and effectively, going back full circle to the way things were for Java in MapGuide)
And ... it went just exactly what I said above! This was by far the easiest migration effort of the lot.

.net Migration Overview

This migration was the one I had been dreading the most. Not because I feared this binding was going to be fragile, because we already had an exhaustive test suite which this binding passed with flying colors.

But rather, I had been dreading this one because all of our .net code that is going to use this binding (AJAX viewer, code samples, etc) are all legacy pre-historic aspx webforms and I wasn't sure if such code would accept the brand new .net development story I had planned for it.

Consider the current .net development story.

  1. You would reference the 5 OSGeo.MapGuide.* assemblies from mapviewernet/bin in your MapGuide application.
  2. You would then have to manually copy the remaining dlls from mapviewernet/bin to your MapGuide application's output directory so that the .net MapGuide API binding doesn't fail due to missing native dll dependencies
The alternative to this is to use the NuGet package, which make this story more seamless, but the process to build this NuGet package was a bespoke affair, with hand-crafted powershell scripts that trigger on nuget package installation to set up the necessary project build events to copy the native dlls needed by the OSGeo.MapGuide.* assemblies to the right location. Such functionality is tightly-coupled to Visual Studio, so if you were installing this NuGet package and building your MapGuide application outside of Visual Studio, none of the required post-build events would fire and the result is a broken MapGuide .net application because the native dll dependencies were not being copied to your application's output directory.

For this new .net binding, we build each OSGeo.MapGuide.* assembly as a separate SDK-style csproj project files. This new csproj file format has several benefits:
  • You don't have to reference individual C# source files that need to be compiled. Any C# source fil in the same directory as the csproj file is implied to be compiled as part of the project. This is great because it means we can run SWIG to generate the .cs files straight into the project and build that project straight away afterwards.
  • This csproj file format supports NuGet packages as first-class project output
  • The NuGet packages produced have first-class support for native dependencies. This is the real killer feature because it means in terms of packaging, we just have to include these native dlls in a well known location and they will be bundled up automatically as part of NuGet package creation. Such a package when consumed will have its native dependencies automatically copied to the right place and loaded from the right spot without any custom post-build events to set this stuff up!
  • And finally, it means instead of targeting .net Framework, we can target netstandard2.0

What does netstandard2.0 support imply? It implies your MapGuide application using these packages can work on all of these platforms. Now practically speaking, despite now being netstandard2.0 packages, these packages will only work on platforms where the underlying OS is Windows and (maybe) Linux as those are the platforms where we can actually compile the underlying supporting native libraries needed by these nuget packages. So no Mac OSX, no Xamarin, etc. 

In practical terms, it means you are no longer shackled to legacy .net framework for building MapGuide .net applications. You can now use .net core from its earliest netstandard2.0-supported iterations all the way to the latest .net 6.0 (as of this post). 

That's great and all, but going back to the original issue: Can the current suite of aspx webforms code accept this new way of consuming the .net MapGuide API and come along for the ride? I hope so! Because the alternative is to rewrite all of this code with more modern .net web technologies (razor pages maybe?), and while such a rewrite has merit and probably warranted, it is not warranted right now because that would add many more months of dev work to my already time-poor schedule. We have bigger fish to fry! We just hope this current codebase will cooperate with our new .net packaging paradigm with minimal effort.

So let's start with the basic facts.

  • MapGuide's .net integration requires IIS and .net framework to already be installed
  • We can assume that for the purpose of being able to use this netstandard2.0 library, that the installed .net framework version must .net framework 4.8. Building your own MapGuide applications for .net core and .net 5.0+ is something you can opt-in to, but it is not something to be demanded by our existing .net web tier code.
With these facts established we have our first hurdle, and it is one of setup/deployment.

The AJAX viewer and sample code have no Visual Studio solution/project files! How do these things ever get built?

The AJAX viewer for .net is a series of raw .aspx files, will be "compiled" to .net assemblies on first request to IIS. As part of this compilation, it will check its respective "bin" directory for any references. That's why the mapviewernet/bin has the OSGeo.MapGuide.* assemblies in there because that is what is being referenced when the .aspx files get compiled. The .net sample code also follows the same pattern.

So we have a bunch of folders of .aspx files and we need to get the correct set of dlls from inside our brand new shiny nuget packages into these folders. How would we go about this without needing to disruptively set up solution/project files for them?

Here's my approach. We setup a stub SDK-style project that targets net4.8 and references the 5 OSGeo.MapGuide.* nuget packages produced from our new .net binding project setup.

As part of the main build, we perform a framework-dependent publish of this project. Because of the first-class native dependency support, the publish output of this project would be the project's assembly, the 5 OSGeo.MapGuide.* assemblies and (importantly) all of their native dll dependencies in one single output directory. Once we done the framework-dependent publish, we can then copy all the dll files in this publish output folder (except for the stub project) into the bin directory of our AJAX viewer and sample directories.

It turns out this approach does result in a functional .net AJAX viewer and code samples. What was needed in addition to using a stub project to setup the required dll file list, is that the .net AJAX viewer and code samples need a web.config file that references the netstandard assembly due to our OSGeo.MapGuide.* assemblies now target netstandard2.0

Is this a complete and utter hack? Totally!

But this approach gives us a functional .net AJAX viewer and code samples. Considering the alternative solutions and my current timelines, this is a workable approach and sometimes ...


So that was the ugly setup/deployment aspect, but what about the code itself? Well that was relatively simple. Like PHP and Java before it, we only needed to fix up the exception handling code to match to our new pattern of checking for specific exception codes in the caught MgException to handle for certain error cases.

Getting this to work on Linux

With the bindings now all being generated by vanilla SWIG and all working on Windows, it was a case of getting the build system fully working again on Linux with these new bindings and updated web tier components.

Fortunately, on the binding front, most of the CMake configurations added in the initial phases of this work only needed minor adjustments, so the bulk of the work was actually building PHP 8.1 and  integrating this into the Apache httpd server, which we are also building from source and updating our various httpd/php config file templates to work with this new version of PHP.

Where we stand now

We now finally have MapGuide API bindings generated with vanilla, unmodified SWIG that work on both Windows and Linux. This has been a long and arduous journey and I can finally see the light at the end of this tunnel!

A new RFC has been posted for PSC discussion/voting. I hope there isn't strong opposition/dissent on these changes, because I truly believe that this work is absolutely necessary for MapGuide going forward. Newer versions of .net/Java/PHP will always get released and inevitably we will need our MapGuide API bindings to work on these newer versions. Our current infrastructure to keep up with newer .net/Java/PHP versions is just not maintainable or tenable.

If/when this RFC is adopted, the long overdue Preview 4 release should drop not too long after!

Wednesday, 24 August 2022

MapGuide Site Administrator XSS security fix available

A security fix is now available for MapGuide Open Source.

This fix mitigates several XSS vulnerabilities reported in the MapGuide Site Administrator tool.

Download the fix here

To apply, simply extract the zip contents to the www/mapadmin folder of your MapGuide installation and overwrite all existing files.

This fix can be applied to the following versions of MapGuide Open Source:

  • 2.6.1
  • 3.0.0
  • 3.1.0
  • 3.1.1
  • 3.1.2
  • Any preview release of 4.0.0

Special thanks to Eitan Shav of mend.io who found and reported this vulnerability

Wednesday, 3 August 2022

Announcing: MapGuide Maestro 6.0m11

In the interest of getting back into the habit of releasing things again and to line up authoring expectations/experience for another upcoming MapGuide Open Source 4.0 preview release, here's a long overdue release of MapGuide Maestro. Here's a summary of what's changed since the last release (Wow! It really has been 4 years since the last one?)

MapGuide Open Source 4.0 authoring support

This release of Maestro takes advantage of features/capabilities introduced in the upcoming MapGuide Open Source 4.0. For all these demonstrated features, we assume you have the current Preview 3 release of MGOS 4.0 installed or newer.

A new Layer Definition resource template based on the v4.0.0 schema is now available.



What features/capabilities does this offer? A new toggle option to determine if QUERYMAPFEATURES requests that hit this layer should include bounding box information or not. When bounding box data is not included, client-side viewer tools like zooming to selection will not work due to the lack of this information.




The WMS metadata UI now has support for exposing or hiding the geometry field data from WMS GetFeatureInfo responses.



The basic label style editor finally has the missing support for editing advanced placement settings



MapGuide Open Source 4.0 introduced bulk coordinate transformation capabilities in the mapagent and Maestro will now take advantage of this feature for places in the UI that require transforming coordinates, such as setting the map extents for example



MapGuide Open Source 4.0 now also removes the restriction that you cannot CREATERUNTIMEMAP or MgMap.Create() a Map Definition that links to a XYZ tileset, so the Map Definition editor will no longer throw this warning and block you from linking to a XYZ tile set definition if you are connected to a MGOS 4.0 instance.


Notable UI Changes

Your published WFS and WMS layers are now visible as top-level nodes in the Site Explorer! This allows for quick visual verification that you have correctly applied the appropriate WFS/WMS publishing metadata to your Feature Source or Layer Definition.


These new nodes aren't just for show. For the published WMS layers, there are context menu actions to follow back to the source Layer Definition or for the more exciting option, the ability to preview this WMS layer through the new OpenLayers-driven content representation for WMS layers



The local map viewer component (used for local map previews) now has a panel to show selection attributes


MgCooker is no more. Replaced with MgTileSeeder

The venerable MgCooker tool for pre-seeding tilesets in MapGuide has been removed. MgTileSeeder is now the full replacement for MgCooker and is capable of more things than MgCooker (like being a generic XYZ tileset seeder). All existing Maestro UI integrations with the MgCooker tool have also been removed as a result.

Maestro API package is now SourceLink-enabled

If you use the Maestro API to build your own MapGuide client applications, the Maestro API nuget package is now built with SourceLink support meaning that when you go to a definition of any class/type of the Maestro API, you will now see the full source code of that class/type instead of the class/type outline from inferred .net metadata.

Similarly, when debugging you can now step into the source code for any method in the Maestro API!


To take advantage of SourceLink requires some small Visual Studio settings changes, which are covered in detail here.

Maestro is now a self-contained .net 6.0 windows application

Aside from being able to take advantage of the new capabilities and performance improvements of .net 6.0, the other upside of this move is that this means that you no longer have to download/install the .net Framework first before installing Maestro. Being a self-contained application means that the support files needed to run a .net 6.0 application are now included with the installer itself.