Saturday, 10 September 2022

MapGuide dev diary: It is done!

In the previous installment, we had gotten a test MapGuide installation with bundled PHP 8.1 up and running, and we were able to successfully produce a PHP error when hitting the Site Administrator.

This PHP error referenced previously was a milestone because it meant that our PHP 8.1 setup (FastCGI on Apache via mod_fcgid) was working, our PHP code was actually running and so the actual errors is the result of the vast swaths of our current PHP web tier applications needing to be migrated across to work against this new PHP 8.1 binding for the MapGuide API.

And so for the next few months I did just that, not just for PHP, but also for Java and .net. You can consider this post to be a preview of what you'll need to do yourself if you want to migrate your PHP/Java/.net MapGuide applications to MGOS 4.0.

PHP Migration Overview

The PHP binding was the one I expect to be the most work because besides finally supporting PHP 8.1, the major feature of this new PHP binding is that constants.php is no longer required! This is because with vanilla SWIG we now can bake all the constants of the MapGuide API into the PHP binding extension itself! So I expect a lot of  "include 'constants.php'" references needing to be removed.

Once all the constants.php references are removed, we found the prime issue with this new PHP binding. PHP didn't like some of our C++ classes had overloaded methods whose signatures did not exist in the parent class. This manifested in the form of fatal PHP errors like the following:

PHP Fatal error:  Declaration of MgCoordinateSystemMeasure::GetDistance(MgCoordinate|float|null $arg1, MgCoordinate|float|null $arg2, float $arg3, float $arg4): float must be compatible with MgMeasure::GetDistance(?MgCoordinate $arg1, ?MgCoordinate $arg2): float in Unknown on line 0

In this case, our MgMeasure class has a GetDistance method of the following signature:

double GetDistance(MgCoordinate* coord1, MgCoordinate* coord2)

In the derived MgCoordinateSystemMeasure, it has a new overload of GetDistance that has this signature:

double GetDistance(double x1, double y1, double x2, double y2)

However, when this converted to PHP proxy classes by SWIG, PHP doesn't like this class setup because under its inheritance model, it is expecting the GetDistance overload with 4 double parameters to also exist in the base MgMeasure class. This is not the case, and thus PHP throws the above fatal error.

To workaround this problem, we had to use the SWIG %rename directive to rename the conflicting overload signature in MgCoordinateSystemMeasure in PHP to the following:

double GetDistanceSimple(double x1, double y1, double x2, double y2)

With this rename, there is no longer a signature conflict in the generated proxy class and PHP no longer throws a fatal error. Fortunately, only 2 classes in the MapGuide API have this problem, so the amount of method renaming is minimal.

Once this issue was addressed, I tried firing up the PHP implementation of the AJAX viewer and I got an interactive map! Everything seemed to be working until I tried to generate a map plot, and found my second problem. I was getting PHP fatal errors like this:

PHP Fatal error:  Uncaught TypeError: No matching function for overloaded 'MgRenderingService_RenderMap' in quickplotgeneratepicture.php:115

Fortunately, this one was easier to explain and fix. In PHP 8.1 (maybe even earlier in the PHP 7.x series), the type checking became more stricter which meant int parameters must take integers, double parameters must take doubles, etc, etc, you couldn't pass ints as doubles or vice versa, and for a method like RenderMap of MgRenderingService, there are lots of overloads that take many different combinations of int and double parameters.

Our map plotting code was passing in strings where int/double parameters were expected. In PHP 5.6 this was allowed because the type checking was evidently more lax. Now such cases cause the above PHP fatal error. This was easy enough to fix, we just use the intval and doubleval functions to make sure ints are being passed into int parameters and doubles are being passed into double parameters.

And with that, the rest of the changes involving fixing up our exception handling code due to a major change with how MapGuide applications should be handling exceptions from the MapGuide API. As part of this SWIG binding work, we've flattened the MapGuide exception hierarchy into a single MgException class, and introduced a new exception code property to allow handling MapGuide exceptions on a case-by-case basis.

So if you were handling specific MapGuide exceptions like this:

1
2
3
4
5
6
7
8
try
{
   //Some code that could throw
} 
catch (MgUnauthorizedAccessException $e) 
{
    ...
}

You would now rewrite them like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
try
{
   //Some code that could throw
} 
catch (MgException $e) 
{
    if ($e->GetExceptionCode() == MgExceptionCodes::MgUnauthorizedAccessException) {
        ...
    }
}

The reason for flattening the exception hierarchy was:

  • To make wrapping exceptions simpler (we now only need to wrap the sole MgException class in SWIG) and not have to handle exception class inheritance chains in a consistent manner across all 3 language bindings.
  • Most of the example MapGuide API code pretty much only caught MgException anyways and rarely catches any of its derived exception classes (and I imagine that this is the case in your MapGuide applications as well). Any code that cared to handle specific exception cases, we can just include the relevant sub-classification as a property of MgException itself as the above code example shows.
Once this final change was made, the AJAX viewer was fully functional. Porting Fusion to PHP 8 was a similar process.

Java Migration Overview

This migration I expect to be a cakewalk because although this binding is now also being generated by vanilla SWIG, it is 99% identical to the existing MapGuideJavaApiEx.jar that we have been generating and shipping for many releases of MapGuide.

So all I expect is just to:
  • Fix up references to MapGuideJavaApiEx and rename them to MapGuideJavaApi
  • Update exception handling blocks to take action based on the captured exception code in the caught MgException instead of catching for specific subclasses of MgException
  • Replace MapGuideApiEx.jar with MapGuideApi.jar (we're using the old jar name for the new binding) in our MG installation (and effectively, going back full circle to the way things were for Java in MapGuide)
And ... it went just exactly what I said above! This was by far the easiest migration effort of the lot.

.net Migration Overview

This migration was the one I had been dreading the most. Not because I feared this binding was going to be fragile, because we already had an exhaustive test suite which this binding passed with flying colors.

But rather, I had been dreading this one because all of our .net code that is going to use this binding (AJAX viewer, code samples, etc) are all legacy pre-historic aspx webforms and I wasn't sure if such code would accept the brand new .net development story I had planned for it.

Consider the current .net development story.

  1. You would reference the 5 OSGeo.MapGuide.* assemblies from mapviewernet/bin in your MapGuide application.
  2. You would then have to manually copy the remaining dlls from mapviewernet/bin to your MapGuide application's output directory so that the .net MapGuide API binding doesn't fail due to missing native dll dependencies
The alternative to this is to use the NuGet package, which make this story more seamless, but the process to build this NuGet package was a bespoke affair, with hand-crafted powershell scripts that trigger on nuget package installation to set up the necessary project build events to copy the native dlls needed by the OSGeo.MapGuide.* assemblies to the right location. Such functionality is tightly-coupled to Visual Studio, so if you were installing this NuGet package and building your MapGuide application outside of Visual Studio, none of the required post-build events would fire and the result is a broken MapGuide .net application because the native dll dependencies were not being copied to your application's output directory.

For this new .net binding, we build each OSGeo.MapGuide.* assembly as a separate SDK-style csproj project files. This new csproj file format has several benefits:
  • You don't have to reference individual C# source files that need to be compiled. Any C# source fil in the same directory as the csproj file is implied to be compiled as part of the project. This is great because it means we can run SWIG to generate the .cs files straight into the project and build that project straight away afterwards.
  • This csproj file format supports NuGet packages as first-class project output
  • The NuGet packages produced have first-class support for native dependencies. This is the real killer feature because it means in terms of packaging, we just have to include these native dlls in a well known location and they will be bundled up automatically as part of NuGet package creation. Such a package when consumed will have its native dependencies automatically copied to the right place and loaded from the right spot without any custom post-build events to set this stuff up!
  • And finally, it means instead of targeting .net Framework, we can target netstandard2.0

What does netstandard2.0 support imply? It implies your MapGuide application using these packages can work on all of these platforms. Now practically speaking, despite now being netstandard2.0 packages, these packages will only work on platforms where the underlying OS is Windows and (maybe) Linux as those are the platforms where we can actually compile the underlying supporting native libraries needed by these nuget packages. So no Mac OSX, no Xamarin, etc. 

In practical terms, it means you are no longer shackled to legacy .net framework for building MapGuide .net applications. You can now use .net core from its earliest netstandard2.0-supported iterations all the way to the latest .net 6.0 (as of this post). 

That's great and all, but going back to the original issue: Can the current suite of aspx webforms code accept this new way of consuming the .net MapGuide API and come along for the ride? I hope so! Because the alternative is to rewrite all of this code with more modern .net web technologies (razor pages maybe?), and while such a rewrite has merit and probably warranted, it is not warranted right now because that would add many more months of dev work to my already time-poor schedule. We have bigger fish to fry! We just hope this current codebase will cooperate with our new .net packaging paradigm with minimal effort.

So let's start with the basic facts.

  • MapGuide's .net integration requires IIS and .net framework to already be installed
  • We can assume that for the purpose of being able to use this netstandard2.0 library, that the installed .net framework version must .net framework 4.8. Building your own MapGuide applications for .net core and .net 5.0+ is something you can opt-in to, but it is not something to be demanded by our existing .net web tier code.
With these facts established we have our first hurdle, and it is one of setup/deployment.

The AJAX viewer and sample code have no Visual Studio solution/project files! How do these things ever get built?

The AJAX viewer for .net is a series of raw .aspx files, will be "compiled" to .net assemblies on first request to IIS. As part of this compilation, it will check its respective "bin" directory for any references. That's why the mapviewernet/bin has the OSGeo.MapGuide.* assemblies in there because that is what is being referenced when the .aspx files get compiled. The .net sample code also follows the same pattern.

So we have a bunch of folders of .aspx files and we need to get the correct set of dlls from inside our brand new shiny nuget packages into these folders. How would we go about this without needing to disruptively set up solution/project files for them?

Here's my approach. We setup a stub SDK-style project that targets net4.8 and references the 5 OSGeo.MapGuide.* nuget packages produced from our new .net binding project setup.

As part of the main build, we perform a framework-dependent publish of this project. Because of the first-class native dependency support, the publish output of this project would be the project's assembly, the 5 OSGeo.MapGuide.* assemblies and (importantly) all of their native dll dependencies in one single output directory. Once we done the framework-dependent publish, we can then copy all the dll files in this publish output folder (except for the stub project) into the bin directory of our AJAX viewer and sample directories.

It turns out this approach does result in a functional .net AJAX viewer and code samples. What was needed in addition to using a stub project to setup the required dll file list, is that the .net AJAX viewer and code samples need a web.config file that references the netstandard assembly due to our OSGeo.MapGuide.* assemblies now target netstandard2.0

Is this a complete and utter hack? Totally!

But this approach gives us a functional .net AJAX viewer and code samples. Considering the alternative solutions and my current timelines, this is a workable approach and sometimes ...


So that was the ugly setup/deployment aspect, but what about the code itself? Well that was relatively simple. Like PHP and Java before it, we only needed to fix up the exception handling code to match to our new pattern of checking for specific exception codes in the caught MgException to handle for certain error cases.

Getting this to work on Linux

With the bindings now all being generated by vanilla SWIG and all working on Windows, it was a case of getting the build system fully working again on Linux with these new bindings and updated web tier components.

Fortunately, on the binding front, most of the CMake configurations added in the initial phases of this work only needed minor adjustments, so the bulk of the work was actually building PHP 8.1 and  integrating this into the Apache httpd server, which we are also building from source and updating our various httpd/php config file templates to work with this new version of PHP.

Where we stand now

We now finally have MapGuide API bindings generated with vanilla, unmodified SWIG that work on both Windows and Linux. This has been a long and arduous journey and I can finally see the light at the end of this tunnel!

A new RFC has been posted for PSC discussion/voting. I hope there isn't strong opposition/dissent on these changes, because I truly believe that this work is absolutely necessary for MapGuide going forward. Newer versions of .net/Java/PHP will always get released and inevitably we will need our MapGuide API bindings to work on these newer versions. Our current infrastructure to keep up with newer .net/Java/PHP versions is just not maintainable or tenable.

If/when this RFC is adopted, the long overdue Preview 4 release should drop not too long after!

Wednesday, 24 August 2022

MapGuide Site Administrator XSS security fix available

A security fix is now available for MapGuide Open Source.

This fix mitigates several XSS vulnerabilities reported in the MapGuide Site Administrator tool.

Download the fix here

To apply, simply extract the zip contents to the www/mapadmin folder of your MapGuide installation and overwrite all existing files.

This fix can be applied to the following versions of MapGuide Open Source:

  • 2.6.1
  • 3.0.0
  • 3.1.0
  • 3.1.1
  • 3.1.2
  • Any preview release of 4.0.0

Special thanks to Eitan Shav of mend.io who found and reported this vulnerability

Wednesday, 3 August 2022

Announcing: MapGuide Maestro 6.0m11

In the interest of getting back into the habit of releasing things again and to line up authoring expectations/experience for another upcoming MapGuide Open Source 4.0 preview release, here's a long overdue release of MapGuide Maestro. Here's a summary of what's changed since the last release (Wow! It really has been 4 years since the last one?)

MapGuide Open Source 4.0 authoring support

This release of Maestro takes advantage of features/capabilities introduced in the upcoming MapGuide Open Source 4.0. For all these demonstrated features, we assume you have the current Preview 3 release of MGOS 4.0 installed or newer.

A new Layer Definition resource template based on the v4.0.0 schema is now available.



What features/capabilities does this offer? A new toggle option to determine if QUERYMAPFEATURES requests that hit this layer should include bounding box information or not. When bounding box data is not included, client-side viewer tools like zooming to selection will not work due to the lack of this information.




The WMS metadata UI now has support for exposing or hiding the geometry field data from WMS GetFeatureInfo responses.



The basic label style editor finally has the missing support for editing advanced placement settings



MapGuide Open Source 4.0 introduced bulk coordinate transformation capabilities in the mapagent and Maestro will now take advantage of this feature for places in the UI that require transforming coordinates, such as setting the map extents for example



MapGuide Open Source 4.0 now also removes the restriction that you cannot CREATERUNTIMEMAP or MgMap.Create() a Map Definition that links to a XYZ tileset, so the Map Definition editor will no longer throw this warning and block you from linking to a XYZ tile set definition if you are connected to a MGOS 4.0 instance.


Notable UI Changes

Your published WFS and WMS layers are now visible as top-level nodes in the Site Explorer! This allows for quick visual verification that you have correctly applied the appropriate WFS/WMS publishing metadata to your Feature Source or Layer Definition.


These new nodes aren't just for show. For the published WMS layers, there are context menu actions to follow back to the source Layer Definition or for the more exciting option, the ability to preview this WMS layer through the new OpenLayers-driven content representation for WMS layers



The local map viewer component (used for local map previews) now has a panel to show selection attributes


MgCooker is no more. Replaced with MgTileSeeder

The venerable MgCooker tool for pre-seeding tilesets in MapGuide has been removed. MgTileSeeder is now the full replacement for MgCooker and is capable of more things than MgCooker (like being a generic XYZ tileset seeder). All existing Maestro UI integrations with the MgCooker tool have also been removed as a result.

Maestro API package is now SourceLink-enabled

If you use the Maestro API to build your own MapGuide client applications, the Maestro API nuget package is now built with SourceLink support meaning that when you go to a definition of any class/type of the Maestro API, you will now see the full source code of that class/type instead of the class/type outline from inferred .net metadata.

Similarly, when debugging you can now step into the source code for any method in the Maestro API!


To take advantage of SourceLink requires some small Visual Studio settings changes, which are covered in detail here.

Maestro is now a self-contained .net 6.0 windows application

Aside from being able to take advantage of the new capabilities and performance improvements of .net 6.0, the other upside of this move is that this means that you no longer have to download/install the .net Framework first before installing Maestro. Being a self-contained application means that the support files needed to run a .net 6.0 application are now included with the installer itself.


Monday, 23 May 2022

MapGuide dev diary: ... and now we begin the great migration

Previously, we made some important breakthroughs on the PHP8 binding for MapGuide. Our sanity PHP tests passed (meaning that the core essentials of the MapGuide API were working), but the main PHP test suite itself had a large amount of failures.


Thanks to dead-easy PHP debugging in VSCode, I was able to step into the PHP test code line-by-line and was able to discover that most of these failures was due to a change in how we're meant to be reading out expected test results from our test fixture SQLite databases. We were reading out test results as SQLite BLOBs which broke test result comparison in PHP8. Switching over to reading out these test results as strings instead fixed up along with updating how we check for exceptions thrown by the MapGuide API fixed up all but one test case.

That one remaining test case failure is due deleting a resource created in a preceding test that doesn't exist. While I could spend more time finding out why, the fact of the matter is that 99.9% of the current test suite is now passing on Windows, and it would be more fruitful now to sort out the revised bundling and configuration story with PHP 8 and try to get a working MapGuide installation with the PHP 8 bundled installation, fixing up whatever code and configurations along the way until we have all of our PHP-based MapGuide applications fully functional, which for reference is the following:

  • The Site Administrator
  • The Feature Source schema report/preview
  • The AJAX viewer with PHP backend
  • Fusion
In the process of getting a new MapGuide installation with bundled PHP 8 setup, I've changed how Apache httpd is set up to talk to PHP. Previously, we actually did something incorrect. We previously bundled the thread-safe build of PHP and used that for both Apache and IIS setups. It turns out we bundled the wrong kind of build for the IIS configuration, as we should have actually been using the NTS (non-thread-safe) build of PHP if setting up a MapGuide Web Tier with IIS and FastCGI.

In the interest of getting things right this time round and not have to bundle 2 copies of PHP (TS and NTS builds), I've decided to go all in on using the NTS build of PHP for both IIS and Apache. This means that setting up IIS will now use the correct binaries for FastCGI, but means that we can no longer use mod_php when setting up Apache httpd as that can only be used with the thread-safe builds of PHP. Instead, for Apache httpd we will now set up FastCGI for use with PHP as well via mod_fcgid.

So I grabbed a compatible copy of mod_fcgid.so from apachelounge and dropped it into my httpd installation, updated httpd.conf to use FastCGI for PHP, flipped display_errors = On in php.ini, spun up both the MapGuide Server and httpd, opened the MapGuide Site Administrator and was greeted with the following.


And I am so glad to see this error message :)

Because that is an error message reported by PHP, so we know PHP itself (via FastCGI) is working.

Also the error is about a missing constants.php file, which if you've been following my previous posts is a file that is no longer necessary as our new PHP 8 bindings defines the constants within the binary extension itself.

This error is one of who-knows-how-many things in our PHP applications that will need fixing up to be PHP 8 compatible.

Time to start a-fixing!

Friday, 22 April 2022

MapGuide dev diary: Another (tangential) breakthrough!

Since I got the basic sanity checking PHP script running without crashing, there was one problem remaining around MgByteReader contents being read incorrectly. This turned out to be another out-of-date SWIG typemap around byte arrays. Once that got fixed we are able to read out the contents of the MgByteReader without content corruption. This is an important thing to test because a lot of results from the MapGuide API (XML results, rendered map images, map plots, etc) come in the form of MgByteReaders. If we can't read content out of these readers without data corruption we have major problems, but that is fortunately not the case (so far!).

With that out of the way, it was onto to the current PHP test suite for the MapGuide API and getting it fixed up and passing.

But before we go there, I wanted to check if our new PHP binding was leaking memory in the most obvious cases. To that effect, I built our PHP binding with reference counting diagnostics enabled and wrote this PHP script.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
<?php

function exceptionTest() {
	/*
	Expected release book-keeping:
		MgException - 1
	 */
	try {
		$rid = new MgResourceIdentifier("iamnotvalid");
	} catch (MgException $ex) {

	}
}

function throwingFunction() {
	$agfRw = new MgAgfReaderWriter();
	$wktRw = new MgWktReaderWriter();

	$rid = new MgResourceIdentifier("iamnotvalid");
}

function exceptionTest2() {
	/*
	Expected release book-keeping:
		MgAgfReaderWriter - 1
		MgWktReaderWriter - 1
		MgException - 1
	 */
	try {
		throwingFunction();
	} catch (MgException $ex) {

	}
}

function exceptionTest3() {
	/*
	Expected release book-keeping:
		MgAgfReaderWriter - 1
		MgWktReaderWriter - 1
		MgException - 2
	 */
	try {
		$rid = new MgResourceIdentifier("iamnotvalid");
	} catch (MgException $ex) {

	}

	try {
		throwingFunction();
	} catch (MgException $ex) {
		//Previous iterations of the binding would leak the previous $ex when
		//reusing the variable in this catch block
		//If the release book-keeping for MgException is 2, then this is no
		//longer the case
	}
}

function geometryXformTest() {
	/*
	Expected release book-keeping:
		MgCoordinateSystemFactory - 1
		MgCoordinateSystem - 2
		MgTransform - 1
		MgWktReaderWriter - 1
		MgPoint - 2
		MgCoordinateXY - 2
	 */
	$csFactory = new MgCoordinateSystemFactory();
	$cs1 = $csFactory->CreateFromCode("LL84");
	$cs2 = $csFactory->CreateFromCode("WGS84.PseudoMercator");
	$xform = $csFactory->GetTransform($cs1, $cs2);
	$wktRw = new MgWktReaderWriter();
	$pt = $wktRw->Read("POINT (1 2)");
	$coord = $pt->GetCoordinate();
	echo "Point (".$coord->GetX().", ".$coord->GetY().")\n";
	$pt = $pt->Transform($xform);
	$coord = $pt->GetCoordinate();
	echo "XPoint (".$coord->GetX().", ".$coord->GetY().")\n";
}

function connectionServiceTest() {
	/*
	Expected release book-keeping:
		MgUserInformation - 1
		MgSiteConnection - 1
		MgProxyResourceService - 1
		MgProxyFeatureService - 1
		MgProxyRenderingService - 1
		MgProxyMappingService - 1
		MgProxyKmlService - 1
		MgProxyDrawingService - 1
		MgProxyTileService - 1
		MgProxyProfilingService - 1
	 */

	$userInfo = new MgUserInformation("Anonymous", "");
	$site = new MgSiteConnection();
	$site->Open($userInfo);

	$service1 = $site->CreateService(MgServiceType::ResourceService);
	$service2 = $site->CreateService(MgServiceType::FeatureService);
	$service3 = $site->CreateService(MgServiceType::RenderingService);
	$service4 = $site->CreateService(MgServiceType::MappingService);
	$service5 = $site->CreateService(MgServiceType::KmlService);
	$service6 = $site->CreateService(MgServiceType::DrawingService);
	$service7 = $site->CreateService(MgServiceType::TileService);
	$service8 = $site->CreateService(MgServiceType::ProfilingService);
}

function functionWithParam($userInfo) {
	$userInfo->SetLocale("en");
}

function parameterPassingTest() {
	// We want to check parameters of Mg* objects passed do not encounter
	// refcounting shenanigans, but if they do, we want to make sure that
	// proper increment/decrement is happening so that the MgUserInformation
	// is released to 0 refcount (and thus deleted) when this function returns

	/*
	Expected release book-keeping:
		MgUserInformation - 1
	 */
	$userInfo = new MgUserInformation();
	functionWithParam($userInfo);
	echo "Locale: " . $userInfo->GetLocale() . "\n";
}

MgInitializeWebTier("C:\\mg-4.0-install\\Web\\www\\webconfig_dev.ini");

echo "=========== BEGIN - exceptionTest() ===========\n";
exceptionTest();
echo "=========== END - exceptionTest() ===========\n";

echo "=========== BEGIN - exceptionTest2() ===========\n";
exceptionTest2();
echo "=========== END - exceptionTest2() ===========\n";

echo "=========== BEGIN - exceptionTest3() ===========\n";
exceptionTest3();
echo "=========== END - exceptionTest3() ===========\n";

echo "=========== BEGIN - connectionServiceTest() ===========\n";
connectionServiceTest();
echo "=========== END - connectionServiceTest() ===========\n";

echo "=========== BEGIN - parameterPassingTest() ===========\n";
parameterPassingTest();
echo "=========== END - parameterPassingTest() ===========\n";

echo "=========== BEGIN - geometryXformTest() ===========\n";
geometryXformTest();
echo "=========== END - geometryXformTest() ===========\n";

?>

The basic tenets behind this test script are:
  • If we new any Mg* class, we expect it to be released (reference count dropped by 1) once the respective variable goes out of scope. We consider the script to not be leaking if every "new" statement has a corresponding release as reported by our refcounting diagnostics.
  • Any non-primitive value returned by any class in the MapGuide API is also released once its respective captured variable goes out of scope
  • If an MgException is thrown and we catch it, it is released once we exit the respective catch block
  • If we have multiple try/catch blocks on MgException and we use the same variable name for the caught exception, we're not leaking on subsequent catch blocks (this used to happen when we were generating the PHP bindings on the older SWIG/PHP version)
  • Finally any MapGuide API object passed as parameters to other functions does not get touched by reference counting. But if it does, we expect that the reference count for that object is the same before and after the function call.
Running this script produces the following output



The key things to notice in the script output is that every variable assignment is being properly released upon exiting the function and anything that we new'd explicitly was released with a 1 -> 0 reference count transition. When an object's reference count drops to 0, it will be de-allocated on the C++ side. The script output is proof that we aren't leaking objects we explicitly new'd up from the PHP script, they are being properly de-allocated. Some objects (like the MgCoordinateSystem) release and do not drop to 0 reference count, but that is okay because I know that the MapGuide API caches MgCoordinateSystem instances for performances reasons, so they aren't leaking instances. They can live beyond the scope of the function call where I initially requested for such objects.

So now knowing that we aren't leaking for the general cases, we can move onto the main PHP test suite.

Running the main PHP test suite it for the first time (in a long time) showed some PHP errors around ambiguous overloaded methods. All of these errors are because PHP 7/8 must've tightened up some of the type checking and passing null as a parameter to a method that has many overloaded signatures may cause confusion. Either that or our heavily modified version of SWIG that we previously used to generate the PHP binding had custom codegen to specifically handle overloaded method resolution.

Regardless, the fix was simple enough. Where passing null would cause ambiguity, we just needed to add the necessary cast (in our case, it was casting to string) to make sure the correct method signature is invoked. Once that fix was applied, the test suite ran to completion ...


... with a metric crapton of test failures :(

Now to figure out why these test cases are failing, which is where we come to the subject of this post.

Anytime in the past, when I need to debug a PHP script, it was a case of sprinkling var_dump and print_r statements everywhere, running it and try to comprehend the now extra verbose output, rinsing and repeating until I found the problem. Basically, printf debugging.

I had heard about a thing called xdebug, but I had no idea how to pair it with an IDE (and what IDE to even use?) to get the nice integrated debugging experience I take for granted with .net, Java or C++. Debugging PHP script felt so primitive because I simply didn't know better and getting proper debugging set up just sounded too hard.

Now, times have changed. We have Visual Studio Code. It has a nice ecosystem of extensions for working with PHP that supports the version of PHP that we are now targeting. If debugging C++ code inside docker containers turned out to be such a piece of cake, then surely the same can be said for using VSCode to debug PHP scripts with breakpoints, variable inspection and the whole works!

So I searched for a VSCode extension that could help me debug PHP, and the first result was this PHP debug adapter. I followed its instructions to the letter.

I went to the xdebug install wizard and pasted by dumped phpinfo details


Once submitted, the wizard figured out what version of PHP I had and gave me the download link for the xdebug extension to download and further instructions on what php.ini changes I needed to make


Then back to the VSCode extension instructions, I also updated php.ini to activate extra xdebug settings

1
2
xdebug.mode = debug
xdebug.start_with_request = yes

Then in VSCode, I opened the PHP script I intend to run and debug, and created a new launch.json file for my active workspace.


The newly installed VSCode PHP debug extension must've kicked in at this point because it created a launch.json with 3 xdebug launch presets ready to go. I modify the "Launch currently open script" preset to add in the extra command-line arguments needed by my PHP script.


Once I saved the launch.json, I open the PHP script again, stuck some breakpoints, switched to the debug tab, making sure to use the "Launch currently open script" preset.


I press the play button to start debugging and lo and behold! We're now breaking and stepping through PHP code and inspecting variables!


Now I can focus on figuring out why there are so many test failures, at a much faster pace than what I would've normally done in the past!

No doubt when MapGuide Open Source 4.0 is finally out the door, I'll have to write up a variation of this post on how to set up VSCode with PHP debugging for your PHP-based MapGuide applications. Because nobody should be sprinking var_dumps and print_r statements to debug PHP code like I have done in the past when this is many orders of magnitude faster and more productive!

Wednesday, 20 April 2022

MapGuide dev diary: An important breakthrough!

Recently I have gotten back into the grind of MapGuide development pushing to clear the final hurdle of having functional PHP bindings for the MapGuide API.

Since I last touched this code, several things have changed.

  • We originally wanted to target PHP 7.x, but this is no longer the latest. PHP 8.1.x is the latest. This release of PHP 8.1 for Windows is also built with the same MSVC compiler we use for MapGuide and FDO (MSVC 2019), which means the stars have aligned again where we can source official windows PHP binaries for bundling.
  • Our last attempt used SWIG 4.0.2 (the most recent stable release). This release unfortunately cannot generate functional bindings for PHP 8, so our hand has been forced and we are now using and assuming current git master of SWIG for generating PHP bindings. The latest git master also no longer generates a [PHP extension .dll + .php proxy script] combination and goes back to generating just a PHP extension .dll, which is great because that how our (now) legacy PHP binding was generated.
Given these environmental changes, we've re-activated the PHP binding work with a semi-clean slate. A lot of hacks and workarounds that were in place because the previous SWIG generated a [PHP extension .dll + .php proxy script] combination have been removed.

The PHP binding work has now advanced to the stage where we can run this basic sanity test script against a binary PHP 8.1.4 distribution on Windows with no errors or crashes.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
<?php

echo "Initializing web tier";
try {
    MgInitializeWebTier("C:\\mg-4.0-install\\Web\\www\\webconfig_dev.ini");
} catch (MgException $initEx) {
    echo "Init failure!";
    die;
} catch (Exception $ex) {
    echo "[php]: Exception: " . $ex->getMessage() . "\n";
    die;
}
echo "[php]: Initialized\n";

// We haven't and shouldn't need to require/include constants.php
// they are now baked into the PHP extension along with the other
// MapGuide API proxy classes
echo "[php]: Testing some constants\n";
echo "  - " . MgMimeType::Agf . "\n";
echo "  - " . MgMimeType::Kml . "\n";
echo "  - " . MgMimeType::Mvt . "\n";
echo "  - " . MgImageFormats::Png . "\n";
echo "  - " . MgImageFormats::Gif . "\n";
echo "  - " . MgLogFileType::Authentication . "\n";
echo "  - " . MgServerInformationProperties::ClientOperationsQueueCount . "\n";

$user = new MgUserInformation("Anonymous", "");
// Basic set/get
$user->SetLocale("en");
echo "[php]: Locale is: " . $user->GetLocale() . "\n";
$conn = new MgSiteConnection();
$conn->Open($user);
// Create a session repository
$site = $conn->GetSite();
$sessionID = $site->CreateSession();
echo "[php]: Created session: $sessionID\n";
$user->SetMgSessionId($sessionID);
// Get an instance of the required services.
$resourceService = $conn->CreateService(MgServiceType::ResourceService);
echo "[php]: Created Resource Service\n";
$mappingService = $conn->CreateService(MgServiceType::MappingService);
echo "[php]: Created Mapping Service\n";
$resId = new MgResourceIdentifier("Library://");
echo "[php]: Enumeratin'\n";
$resources = $resourceService->EnumerateResources($resId, -1, "");
echo $resources->ToString() . "\n";
echo "[php]: Coordinate System\n";
$csFactory = new MgCoordinateSystemFactory();
echo "[php]: CS Catalog\n";
$catalog = $csFactory->GetCatalog();
echo "[php]: Category Dictionary\n";
$catDict = $catalog->GetCategoryDictionary();
echo "[php]: CS Dictionary\n";
$csDict = $catalog->GetCoordinateSystemDictionary();
echo "[php]: Datum Dictionary\n";
$datumDict = $catalog->GetDatumDictionary();
echo "[php]: Coordinate System - LL84\n";
$cs1 = $csFactory->CreateFromCode("LL84");
echo "[php]: Coordinate System - WGS84.PseudoMercator\n";
$cs2 = $csFactory->CreateFromCode("WGS84.PseudoMercator");
echo "[php]: Make xform\n";
$xform = $csFactory->GetTransform($cs1, $cs2);
echo "[php]: WKT Reader\n";
$wktRw = new MgWktReaderWriter();
echo "[php]: WKT Point\n";
$pt = $wktRw->Read("POINT (1 2)");
$coord = $pt->GetCoordinate();
echo "[php]: X: ".$coord->GetX().", Y: ".$coord->GetY()."\n";
$site->DestroySession($sessionID);
echo "[php]: Destroyed session $sessionID\n";
echo "[php]: Test byte reader\n";
$bs = new MgByteSource("abcd1234", 8);
$content = "";
$br = $bs->GetReader();
$buf = "";
while ($br->Read($buf, 2) > 0) {
    echo "Buffer: '$buf'\n";
    $content .= $buf;
}
echo "[php]: Test byte reader2\n";
$bs2 = new MgByteSource("abcd1234", 8);
$content = "";
$br2 = $bs2->GetReader();
$buf = "";
while ($br2->Read($buf, 3) > 0) {
    echo "Buffer: '$buf'\n";
    $content .= $buf;
}
$agfRw = new MgAgfReaderWriter();
echo "[php]: Trigger an exception\n";
try {
    $agfRw->Read(NULL);
} catch (MgException $ex) {
    echo "[php]: MgException caught\n";
    echo "[php]: MgException - Code: ".$ex->GetExceptionCode()."\n";
    echo "[php]: MgException - Message: ".$ex->GetExceptionMessage()."\n";
    echo "[php]: MgException - Details: ".$ex->GetDetails()."\n";
    echo "[php]: MgException - Stack: ".$ex->GetStackTrace()."\n";
    echo "Is standard PHP exception too? " . ($ex instanceof Exception) . "\n";
} catch (Exception $ex) {
    echo "[php]: Exception: " . $ex->getMessage() . "\n";
}
echo "[php]: Trigger another exception\n";
try {
    $r2 = new MgResourceIdentifier("");
} catch (MgException $ex2) {
    echo "[php]: MgException caught\n";
    echo "[php]: MgException - Code: ".$ex2->GetExceptionCode()."\n";
    echo "[php]: MgException - Message: ".$ex2->GetExceptionMessage()."\n";
    echo "[php]: MgException - Details: ".$ex2->GetDetails()."\n";
    echo "[php]: MgException - Stack: ".$ex2->GetStackTrace()."\n";
    echo "Is standard PHP exception too? " . ($ex2 instanceof Exception) . "\n";
} catch (Exception $ex) {
    echo "[php]: Exception: " . $ex->getMessage() . "\n";
}

echo "Press any key to exit\n";
readline();

?>

This script is a "sanity check" in that it is testing the key aspects of the MapGuide API that *must* function properly.
  • We can init the web tier
  • We can create a site connection
  • We can request instances of MgService-derived subclasses with proper downcasting (eg. if we ask for a resource service, we get a MgResourceService and not its parent MgService)
  • We can catch MgExceptions and inspect all relevant properties from it.
  • We can access constants. Notice we aren't require/include-ing constants.php? That's because we've found a way to get SWIG to bake all MapGuide constants into the PHP extension itself so this file is no longer necessary!
  • We can interact with MgByteReader contents properly
Currently the sanity check script is failing on the last point and I strongly suspect it is another case of SWIG typemaps that were relevant for PHP7 but no longer valid for PHP8. We also need to make sure that this script doesn't leak memory. Also I've yet to get this binding building on Linux nor have I ran this sanity check script on Linux.

Once this is taken care of, the long journey begins to make sure everything else is working in order of importance:
  1. Our PHP test suite passes
  2. Our PHP web tier applications (site administrator, schema report, AJAX viewer) are functional
  3. Fusion is functional
  4. We can build updated Apache and PHP end-to-end on Linux
  5. All of the above is functional in both IIS and Apache web server configurations on both Windows and Linux
Then, and only then we can start thinking about a new MapGuide Open Source 4.0 preview/beta release.