Friday, 22 April 2022

MapGuide dev diary: Another (tangential) breakthrough!

Since I got the basic sanity checking PHP script running without crashing, there was one problem remaining around MgByteReader contents being read incorrectly. This turned out to be another out-of-date SWIG typemap around byte arrays. Once that got fixed we are able to read out the contents of the MgByteReader without content corruption. This is an important thing to test because a lot of results from the MapGuide API (XML results, rendered map images, map plots, etc) come in the form of MgByteReaders. If we can't read content out of these readers without data corruption we have major problems, but that is fortunately not the case (so far!).

With that out of the way, it was onto to the current PHP test suite for the MapGuide API and getting it fixed up and passing.

But before we go there, I wanted to check if our new PHP binding was leaking memory in the most obvious cases. To that effect, I built our PHP binding with reference counting diagnostics enabled and wrote this PHP script.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
<?php

function exceptionTest() {
	/*
	Expected release book-keeping:
		MgException - 1
	 */
	try {
		$rid = new MgResourceIdentifier("iamnotvalid");
	} catch (MgException $ex) {

	}
}

function throwingFunction() {
	$agfRw = new MgAgfReaderWriter();
	$wktRw = new MgWktReaderWriter();

	$rid = new MgResourceIdentifier("iamnotvalid");
}

function exceptionTest2() {
	/*
	Expected release book-keeping:
		MgAgfReaderWriter - 1
		MgWktReaderWriter - 1
		MgException - 1
	 */
	try {
		throwingFunction();
	} catch (MgException $ex) {

	}
}

function exceptionTest3() {
	/*
	Expected release book-keeping:
		MgAgfReaderWriter - 1
		MgWktReaderWriter - 1
		MgException - 2
	 */
	try {
		$rid = new MgResourceIdentifier("iamnotvalid");
	} catch (MgException $ex) {

	}

	try {
		throwingFunction();
	} catch (MgException $ex) {
		//Previous iterations of the binding would leak the previous $ex when
		//reusing the variable in this catch block
		//If the release book-keeping for MgException is 2, then this is no
		//longer the case
	}
}

function geometryXformTest() {
	/*
	Expected release book-keeping:
		MgCoordinateSystemFactory - 1
		MgCoordinateSystem - 2
		MgTransform - 1
		MgWktReaderWriter - 1
		MgPoint - 2
		MgCoordinateXY - 2
	 */
	$csFactory = new MgCoordinateSystemFactory();
	$cs1 = $csFactory->CreateFromCode("LL84");
	$cs2 = $csFactory->CreateFromCode("WGS84.PseudoMercator");
	$xform = $csFactory->GetTransform($cs1, $cs2);
	$wktRw = new MgWktReaderWriter();
	$pt = $wktRw->Read("POINT (1 2)");
	$coord = $pt->GetCoordinate();
	echo "Point (".$coord->GetX().", ".$coord->GetY().")\n";
	$pt = $pt->Transform($xform);
	$coord = $pt->GetCoordinate();
	echo "XPoint (".$coord->GetX().", ".$coord->GetY().")\n";
}

function connectionServiceTest() {
	/*
	Expected release book-keeping:
		MgUserInformation - 1
		MgSiteConnection - 1
		MgProxyResourceService - 1
		MgProxyFeatureService - 1
		MgProxyRenderingService - 1
		MgProxyMappingService - 1
		MgProxyKmlService - 1
		MgProxyDrawingService - 1
		MgProxyTileService - 1
		MgProxyProfilingService - 1
	 */

	$userInfo = new MgUserInformation("Anonymous", "");
	$site = new MgSiteConnection();
	$site->Open($userInfo);

	$service1 = $site->CreateService(MgServiceType::ResourceService);
	$service2 = $site->CreateService(MgServiceType::FeatureService);
	$service3 = $site->CreateService(MgServiceType::RenderingService);
	$service4 = $site->CreateService(MgServiceType::MappingService);
	$service5 = $site->CreateService(MgServiceType::KmlService);
	$service6 = $site->CreateService(MgServiceType::DrawingService);
	$service7 = $site->CreateService(MgServiceType::TileService);
	$service8 = $site->CreateService(MgServiceType::ProfilingService);
}

function functionWithParam($userInfo) {
	$userInfo->SetLocale("en");
}

function parameterPassingTest() {
	// We want to check parameters of Mg* objects passed do not encounter
	// refcounting shenanigans, but if they do, we want to make sure that
	// proper increment/decrement is happening so that the MgUserInformation
	// is released to 0 refcount (and thus deleted) when this function returns

	/*
	Expected release book-keeping:
		MgUserInformation - 1
	 */
	$userInfo = new MgUserInformation();
	functionWithParam($userInfo);
	echo "Locale: " . $userInfo->GetLocale() . "\n";
}

MgInitializeWebTier("C:\\mg-4.0-install\\Web\\www\\webconfig_dev.ini");

echo "=========== BEGIN - exceptionTest() ===========\n";
exceptionTest();
echo "=========== END - exceptionTest() ===========\n";

echo "=========== BEGIN - exceptionTest2() ===========\n";
exceptionTest2();
echo "=========== END - exceptionTest2() ===========\n";

echo "=========== BEGIN - exceptionTest3() ===========\n";
exceptionTest3();
echo "=========== END - exceptionTest3() ===========\n";

echo "=========== BEGIN - connectionServiceTest() ===========\n";
connectionServiceTest();
echo "=========== END - connectionServiceTest() ===========\n";

echo "=========== BEGIN - parameterPassingTest() ===========\n";
parameterPassingTest();
echo "=========== END - parameterPassingTest() ===========\n";

echo "=========== BEGIN - geometryXformTest() ===========\n";
geometryXformTest();
echo "=========== END - geometryXformTest() ===========\n";

?>

The basic tenets behind this test script are:
  • If we new any Mg* class, we expect it to be released (reference count dropped by 1) once the respective variable goes out of scope. We consider the script to not be leaking if every "new" statement has a corresponding release as reported by our refcounting diagnostics.
  • Any non-primitive value returned by any class in the MapGuide API is also released once its respective captured variable goes out of scope
  • If an MgException is thrown and we catch it, it is released once we exit the respective catch block
  • If we have multiple try/catch blocks on MgException and we use the same variable name for the caught exception, we're not leaking on subsequent catch blocks (this used to happen when we were generating the PHP bindings on the older SWIG/PHP version)
  • Finally any MapGuide API object passed as parameters to other functions does not get touched by reference counting. But if it does, we expect that the reference count for that object is the same before and after the function call.
Running this script produces the following output



The key things to notice in the script output is that every variable assignment is being properly released upon exiting the function and anything that we new'd explicitly was released with a 1 -> 0 reference count transition. When an object's reference count drops to 0, it will be de-allocated on the C++ side. The script output is proof that we aren't leaking objects we explicitly new'd up from the PHP script, they are being properly de-allocated. Some objects (like the MgCoordinateSystem) release and do not drop to 0 reference count, but that is okay because I know that the MapGuide API caches MgCoordinateSystem instances for performances reasons, so they aren't leaking instances. They can live beyond the scope of the function call where I initially requested for such objects.

So now knowing that we aren't leaking for the general cases, we can move onto the main PHP test suite.

Running the main PHP test suite it for the first time (in a long time) showed some PHP errors around ambiguous overloaded methods. All of these errors are because PHP 7/8 must've tightened up some of the type checking and passing null as a parameter to a method that has many overloaded signatures may cause confusion. Either that or our heavily modified version of SWIG that we previously used to generate the PHP binding had custom codegen to specifically handle overloaded method resolution.

Regardless, the fix was simple enough. Where passing null would cause ambiguity, we just needed to add the necessary cast (in our case, it was casting to string) to make sure the correct method signature is invoked. Once that fix was applied, the test suite ran to completion ...


... with a metric crapton of test failures :(

Now to figure out why these test cases are failing, which is where we come to the subject of this post.

Anytime in the past, when I need to debug a PHP script, it was a case of sprinkling var_dump and print_r statements everywhere, running it and try to comprehend the now extra verbose output, rinsing and repeating until I found the problem. Basically, printf debugging.

I had heard about a thing called xdebug, but I had no idea how to pair it with an IDE (and what IDE to even use?) to get the nice integrated debugging experience I take for granted with .net, Java or C++. Debugging PHP script felt so primitive because I simply didn't know better and getting proper debugging set up just sounded too hard.

Now, times have changed. We have Visual Studio Code. It has a nice ecosystem of extensions for working with PHP that supports the version of PHP that we are now targeting. If debugging C++ code inside docker containers turned out to be such a piece of cake, then surely the same can be said for using VSCode to debug PHP scripts with breakpoints, variable inspection and the whole works!

So I searched for a VSCode extension that could help me debug PHP, and the first result was this PHP debug adapter. I followed its instructions to the letter.

I went to the xdebug install wizard and pasted by dumped phpinfo details


Once submitted, the wizard figured out what version of PHP I had and gave me the download link for the xdebug extension to download and further instructions on what php.ini changes I needed to make


Then back to the VSCode extension instructions, I also updated php.ini to activate extra xdebug settings

1
2
xdebug.mode = debug
xdebug.start_with_request = yes

Then in VSCode, I opened the PHP script I intend to run and debug, and created a new launch.json file for my active workspace.


The newly installed VSCode PHP debug extension must've kicked in at this point because it created a launch.json with 3 xdebug launch presets ready to go. I modify the "Launch currently open script" preset to add in the extra command-line arguments needed by my PHP script.


Once I saved the launch.json, I open the PHP script again, stuck some breakpoints, switched to the debug tab, making sure to use the "Launch currently open script" preset.


I press the play button to start debugging and lo and behold! We're now breaking and stepping through PHP code and inspecting variables!


Now I can focus on figuring out why there are so many test failures, at a much faster pace than what I would've normally done in the past!

No doubt when MapGuide Open Source 4.0 is finally out the door, I'll have to write up a variation of this post on how to set up VSCode with PHP debugging for your PHP-based MapGuide applications. Because nobody should be sprinking var_dumps and print_r statements to debug PHP code like I have done in the past when this is many orders of magnitude faster and more productive!
Posted by Jackie Ng at 02:21 0 comments
Labels: , ,

Wednesday, 20 April 2022

MapGuide dev diary: An important breakthrough!

Recently I have gotten back into the grind of MapGuide development pushing to clear the final hurdle of having functional PHP bindings for the MapGuide API.

Since I last touched this code, several things have changed.

  • We originally wanted to target PHP 7.x, but this is no longer the latest. PHP 8.1.x is the latest. This release of PHP 8.1 for Windows is also built with the same MSVC compiler we use for MapGuide and FDO (MSVC 2019), which means the stars have aligned again where we can source official windows PHP binaries for bundling.
  • Our last attempt used SWIG 4.0.2 (the most recent stable release). This release unfortunately cannot generate functional bindings for PHP 8, so our hand has been forced and we are now using and assuming current git master of SWIG for generating PHP bindings. The latest git master also no longer generates a [PHP extension .dll + .php proxy script] combination and goes back to generating just a PHP extension .dll, which is great because that how our (now) legacy PHP binding was generated.
Given these environmental changes, we've re-activated the PHP binding work with a semi-clean slate. A lot of hacks and workarounds that were in place because the previous SWIG generated a [PHP extension .dll + .php proxy script] combination have been removed.

The PHP binding work has now advanced to the stage where we can run this basic sanity test script against a binary PHP 8.1.4 distribution on Windows with no errors or crashes.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
<?php

echo "Initializing web tier";
try {
    MgInitializeWebTier("C:\\mg-4.0-install\\Web\\www\\webconfig_dev.ini");
} catch (MgException $initEx) {
    echo "Init failure!";
    die;
} catch (Exception $ex) {
    echo "[php]: Exception: " . $ex->getMessage() . "\n";
    die;
}
echo "[php]: Initialized\n";

// We haven't and shouldn't need to require/include constants.php
// they are now baked into the PHP extension along with the other
// MapGuide API proxy classes
echo "[php]: Testing some constants\n";
echo "  - " . MgMimeType::Agf . "\n";
echo "  - " . MgMimeType::Kml . "\n";
echo "  - " . MgMimeType::Mvt . "\n";
echo "  - " . MgImageFormats::Png . "\n";
echo "  - " . MgImageFormats::Gif . "\n";
echo "  - " . MgLogFileType::Authentication . "\n";
echo "  - " . MgServerInformationProperties::ClientOperationsQueueCount . "\n";

$user = new MgUserInformation("Anonymous", "");
// Basic set/get
$user->SetLocale("en");
echo "[php]: Locale is: " . $user->GetLocale() . "\n";
$conn = new MgSiteConnection();
$conn->Open($user);
// Create a session repository
$site = $conn->GetSite();
$sessionID = $site->CreateSession();
echo "[php]: Created session: $sessionID\n";
$user->SetMgSessionId($sessionID);
// Get an instance of the required services.
$resourceService = $conn->CreateService(MgServiceType::ResourceService);
echo "[php]: Created Resource Service\n";
$mappingService = $conn->CreateService(MgServiceType::MappingService);
echo "[php]: Created Mapping Service\n";
$resId = new MgResourceIdentifier("Library://");
echo "[php]: Enumeratin'\n";
$resources = $resourceService->EnumerateResources($resId, -1, "");
echo $resources->ToString() . "\n";
echo "[php]: Coordinate System\n";
$csFactory = new MgCoordinateSystemFactory();
echo "[php]: CS Catalog\n";
$catalog = $csFactory->GetCatalog();
echo "[php]: Category Dictionary\n";
$catDict = $catalog->GetCategoryDictionary();
echo "[php]: CS Dictionary\n";
$csDict = $catalog->GetCoordinateSystemDictionary();
echo "[php]: Datum Dictionary\n";
$datumDict = $catalog->GetDatumDictionary();
echo "[php]: Coordinate System - LL84\n";
$cs1 = $csFactory->CreateFromCode("LL84");
echo "[php]: Coordinate System - WGS84.PseudoMercator\n";
$cs2 = $csFactory->CreateFromCode("WGS84.PseudoMercator");
echo "[php]: Make xform\n";
$xform = $csFactory->GetTransform($cs1, $cs2);
echo "[php]: WKT Reader\n";
$wktRw = new MgWktReaderWriter();
echo "[php]: WKT Point\n";
$pt = $wktRw->Read("POINT (1 2)");
$coord = $pt->GetCoordinate();
echo "[php]: X: ".$coord->GetX().", Y: ".$coord->GetY()."\n";
$site->DestroySession($sessionID);
echo "[php]: Destroyed session $sessionID\n";
echo "[php]: Test byte reader\n";
$bs = new MgByteSource("abcd1234", 8);
$content = "";
$br = $bs->GetReader();
$buf = "";
while ($br->Read($buf, 2) > 0) {
    echo "Buffer: '$buf'\n";
    $content .= $buf;
}
echo "[php]: Test byte reader2\n";
$bs2 = new MgByteSource("abcd1234", 8);
$content = "";
$br2 = $bs2->GetReader();
$buf = "";
while ($br2->Read($buf, 3) > 0) {
    echo "Buffer: '$buf'\n";
    $content .= $buf;
}
$agfRw = new MgAgfReaderWriter();
echo "[php]: Trigger an exception\n";
try {
    $agfRw->Read(NULL);
} catch (MgException $ex) {
    echo "[php]: MgException caught\n";
    echo "[php]: MgException - Code: ".$ex->GetExceptionCode()."\n";
    echo "[php]: MgException - Message: ".$ex->GetExceptionMessage()."\n";
    echo "[php]: MgException - Details: ".$ex->GetDetails()."\n";
    echo "[php]: MgException - Stack: ".$ex->GetStackTrace()."\n";
    echo "Is standard PHP exception too? " . ($ex instanceof Exception) . "\n";
} catch (Exception $ex) {
    echo "[php]: Exception: " . $ex->getMessage() . "\n";
}
echo "[php]: Trigger another exception\n";
try {
    $r2 = new MgResourceIdentifier("");
} catch (MgException $ex2) {
    echo "[php]: MgException caught\n";
    echo "[php]: MgException - Code: ".$ex2->GetExceptionCode()."\n";
    echo "[php]: MgException - Message: ".$ex2->GetExceptionMessage()."\n";
    echo "[php]: MgException - Details: ".$ex2->GetDetails()."\n";
    echo "[php]: MgException - Stack: ".$ex2->GetStackTrace()."\n";
    echo "Is standard PHP exception too? " . ($ex2 instanceof Exception) . "\n";
} catch (Exception $ex) {
    echo "[php]: Exception: " . $ex->getMessage() . "\n";
}

echo "Press any key to exit\n";
readline();

?>

This script is a "sanity check" in that it is testing the key aspects of the MapGuide API that *must* function properly.
  • We can init the web tier
  • We can create a site connection
  • We can request instances of MgService-derived subclasses with proper downcasting (eg. if we ask for a resource service, we get a MgResourceService and not its parent MgService)
  • We can catch MgExceptions and inspect all relevant properties from it.
  • We can access constants. Notice we aren't require/include-ing constants.php? That's because we've found a way to get SWIG to bake all MapGuide constants into the PHP extension itself so this file is no longer necessary!
  • We can interact with MgByteReader contents properly
Currently the sanity check script is failing on the last point and I strongly suspect it is another case of SWIG typemaps that were relevant for PHP7 but no longer valid for PHP8. We also need to make sure that this script doesn't leak memory. Also I've yet to get this binding building on Linux nor have I ran this sanity check script on Linux.

Once this is taken care of, the long journey begins to make sure everything else is working in order of importance:
  1. Our PHP test suite passes
  2. Our PHP web tier applications (site administrator, schema report, AJAX viewer) are functional
  3. Fusion is functional
  4. We can build updated Apache and PHP end-to-end on Linux
  5. All of the above is functional in both IIS and Apache web server configurations on both Windows and Linux
Then, and only then we can start thinking about a new MapGuide Open Source 4.0 preview/beta release.
Posted by Jackie Ng at 02:30 0 comments
Labels: , ,

Friday, 4 March 2022

Announcing: mapguide-react-layout 0.14.7

This release updates OpenLayers to the latest version 6.13 and includes the following changes/fixes:

  • Un-break support for Web Layout resources
  • Use DOMPurify to sanitize all possible manually constructed HTML content. DOMPurify is now also the default HTML cleaning function for the selection panel if no custom HTML cleaning function is specified.
The last release was 0.14.5. What happened to 0.14.6?

The 0.14.6 release was pushed shortly before this announcement and was found to carry a vulnerable version of a package (shortid) that we no longer use. This release (0.14.7) was made shortly after to remove this dependency.

Posted by Jackie Ng at 00:56 0 comments
Labels: , ,

Thursday, 3 March 2022

Announcing: vscode-map-preview 0.5.9

This update tidies up the feature popup display so that overly long name and values are now ellipsized


The original un-ellipsized content is available as a mouseover tooltip


Although I couldn't verify it, the name and content values are now also sanitized with DOMPurify to guard against any potential XSS shenanigans.

This update also includes an updated OpenLayers (6.13.0)

Posted by Jackie Ng at 22:04 0 comments
Labels: , ,

Wednesday, 29 December 2021

Announcing: mapguide-react-layout 0.14.5

 To cap off this year, here's one last release of mapguide-react-layout that includes the following changes:


Posted by Jackie Ng at 20:26 0 comments
Labels: , ,

Wednesday, 15 December 2021

MapGuide and log4shell

The log4shell vulnerability has been making the rounds across the tech space and since MapGuide has Java support, I did a check of our MGOS 3.1.2 and current 4.0 preview installation layouts to see if MapGuide is affected.

MapGuide's Java support comes in the form of:

  • Bundling the Apache Tomcat web server
  • Providing a Java language binding to the MapGuide API in the form of a consumable jar archive.
None of these components carry log4j as a dependency. No log4j jar archives are present in any MGOS installation.

Therefore, you are not affected by log4shell. Everything is good!

Naturally, if your Java-based MapGuide application running on top carries the log4j dependency, you should check if the version you're consuming is affected by log4shell and upgrade that dependency to a non-vulnerable version.

Posted by Jackie Ng at 23:53 0 comments
Labels: , ,

Tuesday, 14 December 2021

All API code samples should be written as interactive notebooks!

Consider this hypothetical if you will.

What if instead of our current static API documentation for MapGuide Maestro, we delivered API documentation as a series of .net interactive notebook documents? Code examples that are not only editable, but also interactive!

Behold! A .net interactive notebook explaining how how to use the Maestro API to render an image of a map.


Want to try this code sample yourself? In addition to having MapGuide Open Source installed locally, you will need Visual Studio Code and the .NET Interactive Notebooks extension installed. Once both are installed, you can download this interactive notebook and save it with a .dib extension. Once downloaded, open the .dib file in VSCode and you can run and play around with the code just like the video above.

I believe this is a game-changer for writing API documentation. I have had the same revelation with interactive notebooks for API documentation as I did with the discovery of storybook for designing, developing and showcasing React components. With tools like this, you are actively incentivized to maintain your API docs and its a win-win for your (developer) userbase who get better quality API docs in the process.

The "killer app" for me is if there was a way to either:

  • Self-host these interactive notebooks on a asp.net core application
  • Export these interactive notebooks to blazor WASM so it can be hosted as static web content on something like GitHub pages
Because right now you need specific tools to consume these interactive notebooks. The tipping point  for me will be if you can consume these interactive notebooks straight from your web browser!

Posted by Jackie Ng at 00:53 0 comments
Labels: , , ,
Subscribe to: Posts (Atom)