In awe of what vscode can do

I was originally hoping to drop the long awaited 2nd Preview of MapGuide Open Source 4.0 this week, but sadly some show-stopping bugs have crept up on the Linux side that means that sadly I have to push back the release until at least this one particular show-stopper for the PostgreSQL FDO provider is addressed.

Because this bug is present only on Linux it means we have to dive into gdb and try to debug through how this provider is producing garbage SRID values that result in broken PostGIS spatial queries.

Now normally I would dread at this prospect because gdb is command-line based and I miss being able to easily debug and step through code graphically with Visual Studio, but that was then and nowadays things are a lot different.

• We now have VSCode, un-doubtedly the most popular code editor that is also multi-platform.
• VSCode has extensions for C++ intellisense and integrated debugging with gdb
• For MGOS 4.0, we now also build MapGuide/FDO for Linux inside docker containers.
• VSCode also has extensions for remote development inside docker containers.

So this show-stopper has presented the perfect opportunity to see how hard or easy it is to tie all these pieces together for a nice debugging experience.

I start by spinning up the FDO build container and a PostgreSQL docker container to run our test code against.

Then after installing the remote extensions, I click the green box which then gives me an option to attach to a running docker container.

Which then shows the list of running docker containers, which includes my FDO build container

This then spawns up a second VSCode instance that allows me to open a folder within the running container. The FDO source code which I want to step through is accessible in this container, so I pick that folder.

Now if the experience here is the same as though I wanted to debug through this code from *outside* the container, I would then need to make sure the C++ extension is installed. I notice in this case that the extension UI shows local and remote installed extensions, so I have to install the C++ extension remotely.

If the debug experience for remote sources is the same as for local, then what should happen next is that I make a launch.json set up to run gdb with the executable that contains our test code. VSCode nicely creates a useful starting launch.json for me to tweak to what I need.

At this point, I make sure gdb is installed in the FDO build container, find some source code to stick some break points and hit the play button on the debug tab to start debugging and lo and behold ...

I am now visually debugging and stepping through the FDO source code! Just like Visual Studio on Windows. There was some small setup involved, but the process was mostly seamless.

VSCode is one truly amazing editor! With the right extensions, it can match any dedicated IDE in capabilities.

Now to tackle the actual show-stopper in question.

FDO road test: SQL Server 2017 on Linux

You can consider this post as the 2017 edition of this post.

So for some background. There's been several annoyances I've been personally experiencing with the SQL Server FDO provider that have given me sufficient motivation to fix the problem right at the source (code). However, before I can go down that road, I needed to set up a local dev installation of SQL Server as my dev environment is more geared towards MapGuide than individual FDO providers.

But just like my previous adventure with the King Oracle FDO provider, I didn't want to have to actually find/download a SQL Server installer and proceed to pollute my dev environment with a whole assortment of junk and bloat. We now live in the era of docker containers! Spinning up a SQL Server environment should be a docker pull away and when I no longer need the environment, I can cleanly blow it away without leaving lots of junk behind.

And it just so happens that with the latest release of SQL Server 2017, not only is running it inside a docker container a first-class user story, it is also the first release of SQL Server that natively runs on Linux.

So through the exercise of spinning up a SQL Server 2017 linux container we can kill multiple birds with one stone:

• We'll know if MapGuide/FDO in its current form can work with SQL Server 2017
• We'll also know how well it works with the Linux version of SQL Server (given its feature set is not at parity with the equivalent Windows version)
• If MapGuide/FDO works, we'd then have a SQL Server environment ready to go which can be spun up and torn down on demand to then start fixing various problems with the FDO provider.

Spinning up the SQL Server 2017 linux docker container

This was easy because Microsoft provides an official docker image. So it was a case of just pulling down the docker image and adjusting some environment parameters to use a custom SQL Server sa login when we go to docker run the container and also define port mappings so we can connect to this container from the docker host OS.

The FDO Toolbox bootstrapping test

This was an easy way to determine if the SQL Server FDO provider works with SQL Server 2017. FDO Toolbox has the ability to:

1. Create a SQL Server data store
2. Bulk Copy spatial data into it
3. Query/Preview data from it

If we can do all 3 things above in FDO Toolbox against the freshly spun up SQL Server 2017 linux container, that's a very good sign that everything works.

Creating the FDO data store

FDO Toolbox has a specialized UI for creating SQL Server data stores that is accessible by right-clicking the FDO Data Sources node and choosing Create Data Store - Create SQL Server

This gives us the UI to set up a new SQL Server data store

The first real test is to see if the FDO provider can connect to our SQL Server container, which is a case of filling in all the required connection properties and clicking the Test button, which gives us:

So far so good. Now that we know the FDO provider can connect to the container, we can fill out the data store parameters and click OK to create the data store, which gave us another good sign:

Now just to be sure that the FDO provider did actually create the database, I connected to this SQL Server instance through alternative tools (such as the new SQL Operations Studio) and we can see that the database is indeed there.

So now we can bulk copy some spatial data into it, which will be a nice solid verification that the feature and schema manipulation functionality of the FDO provider work in SQL Server 2017.

So I set up a bulk copy using a whole bunch of test SHP files. A few moments later, we got another positive sign:

Again, for verification we can look at this database in a different tool and can see that the FDO provider correctly created the database tables.

And that data was actually being copied in

Just as an aside: SQL Operations Studio doesn't do spatial data previews like its big brother SQL Server Management Studio.

A shame really. Oh well, at least we can do that in FDO Toolbox :)

Which is also confirmation that FDO is getting the geometry data out of our SQL Server 2017 linux container without any problems.

So based on all these findings, I feel comfortable in saying that FDO (and applications using it like MapGuide) works just fine with SQL Server 2017, especially its Linux version.

Now to deal with these actual annoyances in the FDO provider itself ...

React-ing to the need for a modern MapGuide viewer (Part 15): Play with it on docker

Today, I found a very interesting website from the tech grapevine:

http://play-with-docker.com

What is this site? It is an interactive docker playground. If you've ever used sites like JSFiddle to try out snippets of JS/HTML/CSS, this is basically the docker equivalent to try out Docker environments.

With PWD, I now have a dead simple way for anyone who wants to try out this viewer to spin up a demo MapGuide instance on PWD for themselves to check out the viewer.

Once you've proved to the site that you're are indeed a human and not a robot, you will enter the PWD console. From here, click + ADD NEW INSTANCE to start a new shell.

Then run the following commands to build the demo docker image and spin up the container

git clone https://github.com/jumpinjackie/mapguide-react-layout
cd mapguide-react-layout
./demo.sh

After a few minutes, you should see a port number appear beside the IP address

This is a link to the default Apache httpd page that is confirmation that the demo container is serving out web content to the outside world.

Now simply append /mapguide/index.php to that URL to access the demo landing page for this viewer. Pick any template on the list to load the viewer using that template.

You now have a live demo MapGuide Server with mapguide-react-layout (and the Sheboygan dataset) preloaded for you to play with to your heart's content for the next 4 hours, after which PWD will terminate your session and all the docker images/containers/etc that you created with it.

This was just one use case that I thought up in 5 minutes after discovering this awesome site! I'm sure there's plenty of other creative uses for such a site like this.

Many thanks to brucepc for his MGOS 3.1 docker image from which the demo image is based from.

My Oracle XE setup adventure with Docker

NOTE: This blog post was left in draft limbo for several months. Some things in this post may feel disjointed and some things are now obviously being referenced in the past tense. 

DISCLAIMER: This is not an authoritative Oracle installation guide. Neither is this post the opinions of someone with expert knowledge of Oracle. This is just a blogger's tale of getting an installation of Oracle XE up and running with Docker with the minimum of pain and unnecessary administration so he can easily have an Oracle XE instance available for testing with MapGuide/FDO and other geospatial software.

So before I start the release cycle for the first beta release of MapGuide Open Source 3.1, I'd thought I'd try to investigate (and try to knock off) some long standing issues with Oracle with the King FDO provider.

Naturally, this requires I have an installation of Oracle (XE at a minimum) lying around, which I don't. So that means I've to install it. Having had the displeasure of working with Oracle in the past and encountering its horrible, UI-from-the-last-millennium setup process and administrative UIs, I really didn't want to go down that path.

And in YEAR(GETDATE()), I should no longer have to install Oracle to a bare metal OS. I shouldn't even have to do it inside a clean test Virtual Machine. We have docker, and there's bound to be an Oracle docker image on the docker hub, that I can pull down, spin up a container and have an Oracle installation ready to go.

So that's what I set out to do. Since Windows is the first priority platform of focus for any MapGuide/FDO issues, I needed to get docker installed on Windows. Unlike Linux, Docker support on Windows is not first-class (they're working on that ^) and currently requires a virtualization layer (eg. VirtualBox) for docker on windows to interface with. Fortunately, docker provides the Docker Toolbox package which makes setting up Docker on Windows or Mac an easy one click installer affair with everything to get docker up and running on Windows or Mac

• Docker engine
• Docker machine
• Docker compose
• VirtualBox to interface with the barebones Linux VM hosting the docker engine
• Kitematic, for downloading of docker images and spinning up of docker containers through an easy to use GUI

With docker installed, it was now a case of finding an appropriate Oracle docker image on the docker hub, and Kitematic makes this dead simple. So after typing in "oracle xe" a few images were available, I gravitated towards the one with the one with the most likes and downloads.

Clicking the Create button starts the downloading process

Once downloaded, Kitematic automatically spins up a running container for the freshly downloaded docker image. Kitematic has a Web Preview option, which I presume points to the Oracle APEX manager, which I recall from my frail memory was the web-based admin for an Oracle XE installation. I don't know what black magic Kitematic does to auto-magically know this container was web-previewable, but the fact this was available was much appreciated. So clicking the preview button opened up Chrome to what I presumed to be the URL of Oracle APEX.

Well, at this point I got an unexpected http authentication prompt. Okay, so I tried the documented login for this container with no luck. What the ...? Okay, so I cancelled the prompt and tried refreshing the URL and then it suddenly takes me to the Oracle APEX login page. Weird!

So anyways, now that I'm at the login page, I login with the given credentials and was greeted with the visual confirmation that I now had a running Oracle XE installation via docker!

Except, this is not the APEX web admin, it's actually the database home page. Turns out, if I put in this address manually in the web browser (http://192.168.99.100:32768/apex) it will take me to APEX, if I open this through the web preview option in Kitematic, it takes me to the database home page. Weird!

Still at the end of the day, I had a running Oracle XE instance. Much more pleasant than my previous Oracle installation experiences!

Now it's time to bootstrap this Oracle database with some spatial data, so in the interest of eating my own dogfood, I gave FDO Toolbox a spin to see if I can load data into the fresh Oracle XE instance. So first things first, we check the running container in Kitematic for the IP and ports of this container and the ports that are exposed.

Then I proceed to use FDO Toolbox to try and create an Oracle data store via the King Oracle provider

But then ...

No dice, I don't know what happened here. I swore recalling some time ago in the long distant past that it should've been possible to bootstrap a clean Oracle database from scratch with the King Oracle FDO provider and FDO Toolbox, but try as I might, I just couldn't get it to work this time round!

So when FDO failed, it was time to make the hot tag and bring OGR into the fold. Running the venerable ogr2ogr utility brought the SHP file data into oracle flawlessly. Except, it is excruciatingly s l o w to get anything out of this Oracle database! A basic spatial table listing from (eg. A FDO feature class listing) takes an eternity.

It was a this point I paged a resident Oracle expert and realized that one shouldn't try connecting and loading in data as the SYS/SYSTEM user as that it means we bring in EVERYTHING when doing the simplest thing like listing spatially-enabled tables from SDO_GEOM_METADATA

So I fire up SQLPlus and connect to this XE container using the documented admin login

sqlplus system/oracle@//192.168.99.100:32769/xe

I then make a MapGuide oracle user

SQL> create user mapguide identified by mapguide;

Then give it sufficient privileges

SQL> grant CREATE SESSION, ALTER SESSION, CREATE DATABASE LINK, -
>   CREATE MATERIALIZED VIEW, CREATE PROCEDURE, CREATE PUBLIC SYNONYM, -
>   CREATE ROLE, CREATE SEQUENCE, CREATE SYNONYM, CREATE TABLE, -
>   CREATE TRIGGER, CREATE TYPE, CREATE VIEW, UNLIMITED TABLESPACE -
>   to mapguide;

Then I exit SQLPlus and proceed to ogr2ogr my sample SHP file into this user's oracle schema

ogr2ogr -f OCI OCI:mapguide/mapguide@192.168.99.100:32769/xe Parcels.shp

Now when I connect with FDO Toolbox again (with emphasis on specifying my oracle user's schema, otherwise it's going to I presume, hit ALL_SDO_GEOMETRY_METADATA un-filtered, which performance-wise means we'd be back to slow square one!)

It is now connecting and performing at the performance level we are expecting.

And if we can connect to this Oracle XE database with FDO Toolbox, it means we can also connect to it with MapGuide proper, which is a case of setting up a new Feature Source, with the same connection parameters.

And from here, it's the standard MapGuide authoring process of creating layers pointing to this Feature Source, styling them up and composing them together on a map.

So what have I learned from this exercise?

1. Docker is just plain awesome. I didn't want to touch the Oracle installation process and I also didn't want to "taint" my host environment with an Oracle installation either. Docker made spinning up a repeatable and disposable Oracle XE environment dead simple. Even docker on windows, which requires a Virtual Machine intermediate layer is still a seamless experience thanks to the Docker Toolbox suite of integrated tools.
2. Either something regressed or FDO Toolbox will need some work to be able to load data into Oracle using the King.Oracle provider, which is strange as FDO is supposed to be an abstraction layer and FDO Toolbox is just simply working against the abstractions provided by the FDO API. If I have to code in Oracle-specific code paths and behaviour to get this functionality to work, then that is a sign of a leaky abstraction. I hope I don't have to do this!
3. Thankfully where FDO failed in this task, OGR was able to pick up the slack. FDO and OGR have this nice duality where one's strengths covers the other's weakness.
4. Always create a new Oracle user after install and load the spatial data into this user's schema, so you don't have to ask yourself why the default SYS/SYSTEM login/schema takes an eternity querying SDO_GEOM_METADATA tables.
5. Be sure to set that OracleSchema connection property in your FDO connection, even if it's a non-SYSTEM user.

Many thanks to Gordon Luckett, for the various pointers on Oracle.

^ Windows 10 and Server 2016 now have native docker hosts which can run both Linux and Windows containers. No more intermediate virtualization layer required. Being able to containerize windows applications is game changer! Older versions of Windows will still have to go the virtualization route via VirtualBox for the Windows Docker client to work.

Docker-izing MapGuide

There was a motivation behind this public service announcement.

It was part of my investigations into being able to deploy MapGuide as a Docker container.

What is Docker you may ask? Have a read of their introduction page.

TL;DR? Docker is a tool that allows you to run and deploy software inside virtualized software containers. This is not a heavyweight VM like VirtualBox and their ilk. This is something that leverages existing features in the Linux Kernel to provide "lightweight" VMs with very low overhead.

A docker-ized application gives us various benefits:

• Your application and its dependencies are all self-contained and will not interfere with anything outside of its container and vice versa.
• Deployment is dead simple. No figuring out what pre-requisites that have to be installed. They'll all be part of the docker image that you can pull down with a single command.
• Your application can run in host environments that the application was not originally compiled/tested for. Build once, run anywhere (where docker is installed :))

There's many more benefits than what's listed here. So you can probably see where a docker-ized MapGuide would really be useful from a developer and administrator perspective.

But there is a catch to all this goodness. Firstly you need a Linux distribution that uses Linux Kernel v3.8 or newer as that contains the required OS virtualization features needed for Docker to work.

Secondly, you need to run a 64-bit Linux distribution as that is what Docker only supports. In order to run MapGuide within a Docker container, we need a way to run a 32-bit MapGuide within a 64-bit Linux environment as Docker can only be run from within a 64-bit Linux host and as previously mentioned, we still don't have a functional 64-bit Linux build of MapGuide yet :( So the workaround is to install the required 32-bit packages, which you can find in my previous post.

So after applying all of this newly acquired knowledge, allow me to introduce my first Docker image for MapGuide.

This is a CentOS 6 base image that has the following software and packages pre-installed:

• MapGuide Open Source 2.6
• FDO 3.9
• Apache HTTPD 2.4.9
• PHP 5.5.11
• msttcorefonts to minimize chances of hieroglyphic labels being rendered.

From this base image, you can build your own docker image that installs your own MapGuide data, applications and configurations and deploy/run that image as a docker container.

Here's a basic example of getting a docker-ized MapGuide up and running with mapguide-rest and some sample data packages using 64-bit Ubuntu 14.04 using the above docker base image:

Firstly, we install the docker package like so

sudo apt-get install docker.io

Then make a new directory and create a file named Dockerfile

Put the following text into the Dockerfile. The comments should be self-explanatory. 

# This image is based from the MapGuide docker base image
FROM jumpinjackie/mapguide-base
# Install additional packages, load your MapGuide applications and data, etc.
# Download Sheboygan dataset
RUN wget -P /usr/local/mapguideopensource-2.6.0/server/Packages http://download.osgeo.org/mapguide/releases/2.6.0/Release/Sheboygan.mgp
# Download Melbourne dataset
RUN wget -P /usr/local/mapguideopensource-2.6.0/server/Packages https://github.com/jumpinjackie/mapguide-sample-melbourne/releases/download/v0.2/Melbourne.mgp
# Download mapguide-rest 0.10
RUN wget https://github.com/jumpinjackie/mapguide-rest/releases/download/v0.10-pre/mapguide-rest-0.10.zip
# Set up install location for mapguide-rest
RUN mkdir -p /usr/local/mapguideopensource-2.6.0/webserverextensions/www/rest
# Extract mapguide-rest
RUN unzip mapguide-rest-0.10.zip -d /usr/local/mapguideopensource-2.6.0/webserverextensions/www/rest
# Fix up permissions of the cache directory (for tiles and smarty templates)
RUN chown daemon:daemon /usr/local/mapguideopensource-2.6.0/webserverextensions/www/rest/cache
# Expose the web server port to the world outside the container. The default port is 8008
EXPOSE 8008
# Run supervisor, that will start the MapGuide Server and Apache httpd server
CMD ["/usr/bin/supervisord"]

Now save the Dockerfile, and run the following command to build the docker image from that file

sudo docker build -t my-mapguide-app .

This command will download the mapguide-base docker image which will take a few moments depending on your download speed. This is only downloaded once and will remain until you explicitly remove this base image.

my-mapguide-app will be the name of the docker image, which you'll need to reference when you will run a container from it, which is what we will do next. But before we do that, we should list our docker images and see if our new image is there.

sudo docker images

Now that we have confirmation that our image has been created, we can create a container from it like so.

sudo docker -d --name mapguide -t my-mapguide-app

The -d switch indicates that this container will run in the background. The --name switch assigns the name mapguide to this container so we don't have to remember a long UUID for referencing this container in future operations, which is what is outputted when the command succeeds.

Finally the -t switch indicates that we want to create a container from the my-mapguide-app image that we just created.

Now that we have started a container, we now have:

• MapGuide Server running
• The Sheboygan and Melbourne sample data packages downloaded to the Packages directory of the MapGuide Server installation directory
• The mapguide-rest extension installed
• Apache HTTP Server running on port 8008 which is exposed

We now just need to know what IP address has been assigned to this container. To do that, we can run the following command

sudo docker inspect mapguide | grep IPAddress

Now that we have an IP address, we can see if things are in order by firing up the Site Administrator

If we login and go to the Manage Packages page, we can see the packages we've downloaded as part of building our my-mapguide-app Docker image are there.

Now load these packages, and go to the mapguide-rest sample apps landing page just to see that the extension was installed.

If the samples on this landing page work, you have just verified that your docker-ized MapGuide container is now fully operational.

So that's a little run-through of how to get MapGuide running in a Docker container. Now the other thing about Docker that is really awesome is that the big players in Cloud Computing already (or starting to) support Docker containers as a PaaS deployment option.

Can you say MapGuide on the cloud?

Now just to clarify, MapGuide on the cloud via IaaS is already relatively easy. I, the AWS noob easily got a MapGuide demo site on Amazon EC2 set up over the GovHack weekend, which stayed up long enough to impress the judges to win some awards.

However the IaaS approach to deployment puts a burden on you to maintain the actual infrastructure (VMs, etc). As a case in point, our GovHack demo site has been taken down because I butchered my Amazon EC2 instance due to a combination of failure to act on an important email alert on this issue from Amazon and my general AWS noob-ery.

The PaaS approach lets you focus solely on deploying the applications, without having to worry about the underlying infrastructure. MapGuide as a Docker container enables the possibility of using this cloud deployment option.

Hopefully this post has sold you on the power and potential of Docker and how having MapGuide as a docker container may open up deployment and development scenarios that were previously not possible or took a lot of manual effort.

As for that Docker base image I linked to, that is just the tip of the iceberg. I'm just getting my feet wet with Docker and foresee many future blog posts on this topic as I learn more about what Docker can and can't do.

I can see the possibility of having different base images in the future:

• Individual Server and Web Tier Docker images. Imagine how easy it would be to set up a load-balanced cluster via Docker containers? I'd have to grok how these containers communicate first, but the possibilities are very interesting.
• Web Tier images tailored for the PHP or Java installation profiles.
• Many others?

I also wouldn't comfortably say that the above docker image is currently production-ready. There's questions that need to be answered:

• How do we handle the case of a mgserver or httpd process falling over on a background docker container?
• How can we easily access and manage log files in a docker container?
• How can we perform repository backup/restoration operations within a docker container?
• How do we perform $MAPGUIDE_SERVER_ADMIN_TASK within the docker container?

If you have an idea on how these questions can be answered, I invite you to help improve the design of this docker image or to enlighten me on the comments below.