Tuesday, 15 April 2014

Announcing: mapguide-rest 0.7

Before we continue our blogging journey into the rest of mapguide-rest's data publishing framework, here's the second release of mapguide-rest.

Here's what this new release offers

ACLs for published data sources

We now let you have some level of control as to what users and groups can access a given published data sources, which is important if you are exposing representations that support POST/PUT/DELETE operations and even more important if this is on a public-facing site. You don't want Joe Anonymous MapGuide user to be able to POST/PUT/DELETE to your data sources willy nilly unless you have configured such data sources to allow for such things to happen.

Consider this example from a recent post

{
    "Source": {
        "Type": "MapGuide",
        "FeatureSource": "Library://Samples/Sheboygan/Data/Parcels.FeatureSource",
        "FeatureClass": "SHP_Schema:Parcels"
    },
    "Representations": {
        "xml": {
            "Adapter": "FeatureSetXml",
            "Methods": {
                "GET": {
                    "MaxCount": 500
                },
                "POST": {},
                "PUT": {},
                "DELETE": {}
            }
        }
    }
}

This configuration will now deny all users because no ACLs have been defined for it. A configuration like this

{
    "Source": {
        "Type": "MapGuide",
        "FeatureSource": "Library://Samples/Sheboygan/Data/Parcels.FeatureSource",
        "FeatureClass": "SHP_Schema:Parcels"
    },
    "Representations": {
        "xml": {
            "Adapter": "FeatureSetXml",
            "Methods": {
                "GET": {
                    "MaxCount": 500,
                    "AllowGroups": ["Everyone"]
                },
                "POST": {
                    "AllowUsers": ["Administrator", "Author"]
                },
                "PUT": {
                    "AllowUsers": ["Administrator", "Author"]
                },
                "DELETE": {
                    "AllowUsers": ["Administrator", "Author"]
                }
            }
        }
    }
}

This will define the following access rules:
  • GET: Anybody can access the data source
  • POST: Only the Administrator and Author users (and related session IDs) can access the data source. Everyone else will be denied access.
  • PUT: Only the Administrator and Author users (and related session IDs) can access the data source. Everyone else will be denied access.
  • DELETE: Only the Administrator and Author users (and related session IDs) can access the data source. Everyone else will be denied access.
Auto-API documentation for published data sources

Thanks to the integration of Swagger UI for our REST API documentation needs, and the fact that all published data sources will offer a "fixed" REST API to interact with it, we now can provide automatic REST API documentation for any published data source.

Simply append a doc/index.html to access a data source's API documentation



New REST API routes

This release includes plenty of new routes to play with
  • GET /library/{resourcePath}.WebLayout/viewer (Load the given Web Layout into the AJAX viewer)
  • GET /library/{resourcePath}.ApplicationDefinition/viewer/{template} (Load the given Flexible Layout in Fusion with the given template)
  • POST /library (Load a package file into the repository)
  • GET /services/getschemamapping.{type} (Get the schema mapping for a given FDO provider and partial connection string)
  • GET /library/{resourcePath}.FeatureSource/preview (Launch a schema report preview on the given Feature Source)
  • GET /library/{resourcePath}.LayerDefinition/preview (Launch a AJAX viewer preview of the given Layer Definition)
  • GET /library/{resourcePath}.MapDefinition/preview (Launch a AJAX viewer preview of the given Map Definition)
  • GET /library/{resourcePath}.SymbolDefinition/preview (Render a preview of the given Symbol Definition)
  • GET /library/{resourcePath}.WatermarkDefinition/preview (Launch a AJAX viewer preview of the given Watermark Definition)
  • GET /session/{sessionId}/{resourceName}.WebLayout/viewer (Load the given Web Layout into the AJAX viewer)
  • GET /session/{sessionId}/{resourceName}.ApplicationDefinition/viewer/{template} (Load the given Flexible Layout in Fusion with the given template)
  • GET /session/{sessionId}/{resourceName}.FeatureSource/preview (Launch a schema report preview on the given Feature Source)
  • GET /session/{sessionId}/{resourceName}.LayerDefinition/preview (Launch a AJAX viewer preview of the given Layer Definition)
  • GET /session/{sessionId}/{resourceName}.MapDefinition/preview (Launch a AJAX viewer preview of the given Map Definition)
  • GET /session/{sessionId}/{resourceName}.SymbolDefinition/preview (Render a preview of the given Symbol Definition)
  • GET /session/{sessionId}/{resourceName}.WatermarkDefinition/preview (Launch a AJAX viewer preview of the given Watermark Definition)
You can use the provided interactive REST API documentation to find out more information about these new routes

Improved HTML representation of the site repository

The default HTML representation of the site repository is very bare-bones and primitive.

With the power of Bootstrap 3, we've brought the HTML representation into the 21st century :)


Most of the resource options now show their results inline




Any Web Layout or Application Definition resources now expose new options for you to launch them in their respective AJAX or Fusion viewer



This is still experimental code, so standard disclaimer: use in production at your own risk.

Download

Tuesday, 1 April 2014

GeoREST as re-imagined by mapguide-rest: Part 2 - The XML adapter

For this post, we are going to talk about one of the adapters provided by mapguide-rest: The XML adapter.

This won't be a comprehensive guide, you can consult the adapter documentation in the github wiki for all the gritty details. This will just be a basic overview of the adapter

The XML adapter lets you provide an XML representation of feature data from any MapGuide Feature Source. The name of this adapter is FeatureSetXml. When reading XML via this adapter (using HTTP GET), the XML is the same format as the XML data returned by a SELECTFEATURES operation from the mapagent.

While the value of consuming MapGuide feature data in this format may be debatable, what the XML adapter special and where it is most valuable is that it is the only adapter to support CRUD through the quartet of HTTP verbs:
  • GET: For reading one or more features
  • POST: For inserting one or more features
  • PUT: For updating one or more features
  • DELETE: For deleting one or more features
All other adapters only provide GET support (to "read" feature data as a particular format). This is the only adapter to support feature manipulation.

Configuration

To enable XML representation support for your feature source, your restcfg.json should look like this:

{
    "Source": {
        "Type": "MapGuide",
        "FeatureSource": "Library://Samples/Sheboygan/Data/Parcels.FeatureSource",
        "FeatureClass": "SHP_Schema:Parcels"
    },
    "Representations": {
        "xml": {
            "Adapter": "FeatureSetXml",
            "Methods": {
                "GET": {
                    "MaxCount": 500
                },
                "POST": {},
                "PUT": {},
                "DELETE": {}
            }
        }
    }
}

The above configuration will enable support for all 4 HTTP methods. If you do not want to support a particular method on this Feature Source, then simply remove the corresponding method key from the Methods section of the above configuration. Attempts to make requests to this data source using such un-configured methods will return HTTP 405 (method not allowed).

Reading XML Features

Assume the above restcfg.json resides in /conf/data/property/restcfg.json

To get the XML of this data source is simply to issue a GET request like this:

GET http://servername/mapguide/rest/data/property/.xml


The number of features in this result will be capped at 500 due to the MaxCount property in the configuration.

You can also request the XML of a single feature by putting the actual ID value of the feature into the URL like so

GET http://servername/mapguide/rest/data/property/45.xml


Inserting XML Features

Assume the above restcfg.json resides in /conf/data/property/restcfg.json

Inserting a new feature requires a HTTP POST to the multiple results URL. Your XML body should be structured as follows:

  • A top-level FeatureSet element
  • With a Features element
  • With a Feature element for each feature you want to insert.
  • Each Feature element should have one or more Property elements
  • Each Property element should have a Name element containing the property name and a Value element containing the property value. Geometry values must be in WKT format

A successful request will return an XML envelope containing the ID values for the inserted feature.



Updating XML Features

There are 2 ways to update features.

  • A PUT request to the multiple results URL
  • A PUT request to the single result URL
Your XML request body must be structured as follows
  • A top-level UpdateOperation element
  • With a Filter element which is the FDO filter for features to be updated. If issuing a PUT request to a single results URL, this element is not required as the filter can be inferred from the URL itself.
  • Also with a UpdateProperties element
  • Containing one or more Property elements
  • Each Property element should have a Name element containing the property name and a Value element containing the property value. Geometry values must be in WKT format. Null values can be specified by omitting the Value element.
Note that on IIS, PUT requests is probably not allowed by default, so you'll probably have to use a POST request and include a X-HTTP-Method-Override = PUT request header.

Sending the request will return an XML envelope indicating the number of features updated by this request.


Deleting Features

Like updating, there are 2 ways to delete features
  • A DELETE request to the multiple results URL
  • A DELETE request to the single result URL
No XML request body is required. You just have to specify a filter parameter that indicates what features will be deleted. If the request is to a single result URL, this parameter is not required as the filter will be inferred from the URL itself.

Like PUT, DELETE is probably not allowed on IIS by default, so you can work around that by using a POST request and include a X-HTTP-Method-Override = DELETE request header.

Sending the request will return an XML envelope indicating the number of features deleted by this request


Transaction Support

For providers that support it, you can configure such CRUD operations to use transactions by specifying UseTransaction: true in the relevant methods in restcfg.json

{
    ...
    "Representations": {
        "xml": {
            "Adapter": "FeatureSetXml",
            "Methods": {
                "GET": {
                    "MaxCount": 500
                },
                "POST": {
                    "UseTransaction": true
                },
                "PUT": {
                    "UseTransaction": true
}, "DELETE": {
                    "UseTransaction": true
                }
} } } }

In Summary

The XML adapter is useful for doing CRUD operations using standard HTTP requests and the specially crafted XML request envelopes as described in this post. No special code or APIs required!

GeoREST as re-imagined by mapguide-rest: Part 1

So after a little tour of the REST API, we now come to the other half of mapguide-rest: The re-imagined GeoREST-style data publishing framework.

If you have used GeoREST before, most of what I will be showing here should be familiar to you conceptually.

Introduction

If you've never used GeoREST before, it is a framework for publishing spatial data (from a MapGuide Feature Source or a raw FDO connection) in one or more various formats and representations, such as:
  • GeoJSON
  • XML
  • OData
  • PNG
  • HTML (via templates)
  • KML (via templates)
  • GeoRSS (via templates)
  • CSV (via templates)
Published spatial data sources follow RESTful principles for interaction and data access.
  • Clients use HTTP GET to "read" features from the data source
  • Clients use HTTP POST to "create" features in the data source
  • Clients use HTTP PUT to "update" features in the data source
  • Clients use HTTP DELETE to "delete" features in the data source
GeoREST enables CRUD for certain Feature Sources without needing to know specifics of the MapGuide API. Accessing this data does not require the use any specialized client libraries and APIs or knowledge of SOAP or other specialized RPC mechanisms. Only knowledge of how to make HTTP requests is required. Based on the type of representation and what's configured for the data source, certain operations may or may not be supported.

The other major feature is that access to such data will generally have clean URLs which assists in bookmark-ability and makes such published data easy to access by humans and machines alike.

While GeoREST offers something unique, for mapguide-rest I wanted to go back to the drawing board on this idea for reasons already stated.

Differences from GeoREST

The main differences in mapguide-rest that distinguish it from GeoREST are:
  • Configuration is done by JSON files instead of XML files
  • Because mapguide-rest is in PHP, it also works in Linux versions of MapGuide/AIMS where GeoREST is windows-only.
  • mapguide-rest is not tied to a specific version of MapGuide/AIMS and can work on any version of MapGuide/AIMS that bundles PHP 5.3 or newer (though we recommend using it on the latest releases).
  • Also because of PHP, mapguide-rest uses the more powerful and flexible Smarty template engine for any template-based representations instead of GeoREST's ctemplate engine allowing for more expressive content templates.
  • The base URL where data sources in mapguide-rest can be accessed is strictly based on the relative location of its configuration file and not based on the value of uripart in restcfg.xml for GeoREST.
  • mapguide-rest provides an easy API to extend it with your own custom formats/representations or geometry/datetime formatters for templated content.
  • mapguide-rest provides coordinate transformation capabilities out of the box.
  • mapguide-rest in its current iteration is yet to support OData like GeoREST currently does.
  • mapguide-rest does not support raw FDO connections as a data source. Only MapGuide Feature Sources are supported.
Configuration

Configuration files are JSON-based and reside under the /conf/data subdirectory of your mapguide-rest installation. These files can reside any levels deep within this folder. These configuration files are named restcfg.json and its relative location determines the base URL in which to access that particular configured data source.

Here's a basic example restcfg.json

{
    "Source": {
        "Type": "MapGuide",
        "FeatureSource": "Library://Samples/Sheboygan/Data/Parcels.FeatureSource",
        "FeatureClass": "SHP_Schema:Parcels"
    },
    "Representations": {
        "xml": {
            "Adapter": "FeatureSetXml",
            "Methods": {
                "GET": {
                    "PageSize": 100,
                    "MaxCount": 500
                }
            }
        },
        "html": {
            "Adapter": "Template",
            "Methods": {
                "GET": {
                    "PageSize": 100,
                    "MaxCount": 500,
                    "MimeType": "text/html",
                    "Templates": {
                        "Single": "property_html_single.tpl",
                        "Many": "property_html_many.tpl",
                        "None": "property_html_none.tpl",
                        "Error": "property_html_error.tpl"
                    }
                }
            }
        }
    }
}

As you can see, the structure of restcfg.json is easy to follow:
  • There is a top-level Source property that indicates the MapGuide Feature Source that we want to make available for RESTful access
  • There is a top-level Representations property that contains one or more supported data formats that specifies the Adapter that will handle this particular representation along with adapter-specific configuration options for each HTTP method that you want to allow through that particular representation (and is supported by the given adapter)
The above restcfg.json enables the Parcels Feature Source to be accessible:
  • As XML data
  • As a set of web pages with master-detail type of navigation
URL Conventions

All URLs follow the following forms:
  • Multiple results: http://servername/mapguide/rest/data/[RELATIVEURIPART]/.[FORMAT]
  • Single results: http://servername/mapguide/rest/data/[RELATIVEURIPART]/[FEATUREID].[FORMAT]
As mentioned above, the location of restcfg.json will determine the base URL which to access data from this particular data source. The RELATIVEURIPART of the above URLs is based on the relative path of this restcfg.json file.

For example, if restcfg.json was stored like so

 - REST_INSTALL_DIR
    - conf
       - data
          - sheboygan
             - property
                - restcfg.json

Then the URL for accessing the detail page of a specific parcel would be:

http://servername/mapguide/rest/data/sheboygan/property/1.html

The FORMAT of the above URL is the name of the extension keyed under the Representations part of restcfg.json and will let mapguide-rest determine which adapter to instantiate to handle this particular request.

For individual feature access, specifying the value of [FEATUREID] will cause the designated adapter to only return the matching feature whose identity property value is [FEATUREID] in the adapter's given format/representation.

All adapters will be processing URLs in these two forms.

Adapters

Adapters are classes in mapguide-rest which will be instantiated based on the given extension in the incoming URL and will be used to handle the particular HTTP request. Adapters will perform specific actions based on the request URL and the particular HTTP method used.

For example, for the FeatureSetXml adapter:

  • A HTTP GET will return one or more features from the Feature Source in XML format
  • A HTTP POST will insert one or more features into the given Feature Source based on the XML content in the request body
  • A HTTP PUT will update one or more features in the given Feature Source based on the filter and properties specified in the XML request body
  • A HTTP DELETE will delete one or more features in the given Feature Source based on the filter specified in the request parameters
Other adapters may only support certain methods (for the record, only FeatureSetXml supports GET/POST/PUT/DELETE. Other adapters only support GET)

mapguide-rest includes the following adapters:
  • FeatureSetXml - Outputs feature data as XML, also supports create/update/delete of features if configured
  • FeatureSetJson - Outputs feature data as GeoJSON
  • FeatureSetCsv - Outputs feature data as a CSV file. Geometry values are written as WKT
  • MapImage - Outputs feature data as a map image with the feature(s) in question being selected
  • Template - Outputs feature data through a user-defined smarty template.
Adapters are the key to how data publishing works in mapguide-rest. I have a feeling that this could be a somewhat lengthy series of posts. So I'll stop here for now. Stay tuned as I go through each adapter in greater detail.

Monday, 31 March 2014

Re-visiting the Maestro layer style editor

Consider the current layer style editor in Maestro


Now, despite being serviceable for our needs it does suffer from several shortcomings:

  • A lot of controls are being created to produce this UI. This is really sluggish when you're editing a style with lots of rules and you need to scroll a lot. We also run the risk of running out of handles if we're editing a really big style.
  • The style previews are Maestro-drawn approximations. Composite styles are blank because we simply have no idea how to draw them in pure .net. Using GETLEGENDIMAGE solves our style preview accuracy problems, but the way this UI has been set up makes it hard to use GETLEGENDIMAGE on say a 100 rule layer (arguments about why you have a layer with 100 theme rules aside) without clogging up your application.
The UI as whole looks like a data grid of sorts. Shouldn't this whole UI be more easier and streamlined if it was grid based?

Yes indeed, so that's what we've done. Meet the new grid-based style editor.


The benefits of the grid-based editor is that because we are using a DataGridView component, we have the ability to not only leverage data-binding against rule objects with very little code plumbing, but also to be able to determine what rules are actually in view on the grid.

As a result, we can issue GETLEGENDIMAGE-based previews only for what you currently see. These requests are also done asynchronously on a background thread, ensuring maximum application responsiveness while not making the MapGuide Server cry since we are only asking for previews for what we can currently see.

Clicking on the Filter cell brings up the Expression Editor. Clicking the LegendLabel cell lets you edit the cell contents. Clicking the Style cell brings up the matching point/line/area style editor and clicking the Label cell brings up the label style editor. Any edits you make are immediately visible on the grid when you close the editor dialog.

When you scroll through the grid, you can see rules that haven't had a preview generated for them yet.


To see the previews for these rules, you can click the Refresh Previews button to refresh the style previews for the rules currently in view.


So now that we have a means for having accurate style previews without obsessively hammering a MapGuide Server, it means that we can also show previews for composite styles. Something that was not possible with the old editor.


Clicking the Style cell for a composite rule in this case still brings up the mega-verbose Symbol Instances dialog for editing composite symbolizations, but at least there's no longer a need to delve into this particular dialog just to get a style preview for a composite rule, because like the other style types, previews for composite rules are also shown in the grid as well.

Moving to a grid-based UI opens up other avenues that were not possible before, like being able to filter the current list of rules by some criteria (handy if you want to focus on some specific rules in a layer with 50+ rules). Not that we have done this already, but these are things that a grid-based UI gives us the ability to explore.

Another improvement that's only available for this grid-based style editor is support for theming composite styles. There are some caveats with this support:
  • The theming algorithm is very naive and basic due to the complexity of Advanced Stylization and the thousands of different possible permutations that a given composite rule could be in. However if you stick to a basic prototype of an un-styled point/line/area the theming algorithm should be able to pick up the relevant color property and apply the theme color ramp to it.
  • On that subject, theming a composite style requires a rule to already exist in the style as a prototype. While this is optional for basic styles, this is required for composite styles.
Now that's not to say we have replaced the old style editor. A new feature like this needs some time to be blooded in as a proper replacement. The old editor is still available if you un-tick the new editor option we've added to the options dialog.


The next release of Maestro will use the grid-based editor by default.

Thursday, 27 March 2014

Announcing: mapguide-rest 0.6

Here's the first release of mapguide-rest

And a general disclaimer: This is still work-in-progress code that's not ready for production use. However, that shouldn't stop you from downloading this release and having a play around :)

I have a bunch of mapguide-rest related posts in the draft queue, but it helps to have an accessible release as a point of reference for these posts rather than tell you to get the latest git clone of mapguide-rest.

One thing that stands out in this release since the initial announcement is the introduction of integrated REST API documentation.

What? REST API documentation? Aren't RESTful APIs self-describing, self-documenting and all this other stuff?

That may be true in an ideal sense, but pragmatism trumps idealism. mapguide-rest is the GET/POST/PUT/DELETE to a set of well-crafted URLs variety of REST API. In such cases, it definitely helps to know the definitive list of allowed URLs you can make requests to and the documented parameters that are allowed for each URLs.

So this is what we've provided for you. The documentation can be accessed at the following URL:

http://yourservername/mapguide/rest/doc/index.html

But this isn't a static document.


It's a fully interactive one!


The interactivity allows you to fully explore the REST API in a very intuitive fashion.

All this was made possible with the use of the Swagger documentation framework. The API reference is built on a modified version of Swagger UI. I've found Swagger to be a wonderful tool for generating REST API documentation. The initial set up takes some effort, but the pay-off of the final result is so worth it!

So now that we have a release on the board, stay tuned for more about the GeoREST-style data publishing framework.

Download

Saturday, 22 March 2014

Generating themes by external lookup values

Ever generate an individual theme on a property and get frustrated at the labels that are generated because your property is basically a "foreign key", where the real descriptive labels come from the lookup table that the foreign key points to?


Sure you could set up a view (if your Feature Source is a relational database) to include the descriptive label properties or (gasp) make an Extended Feature Class that uses Feature Joins to pull in the description properties, but there's overhead involved when theming on strings as opposed to theming on numerical lookup ids.

Wouldn't it be nice if there's a way where we can theme based on description values from a lookup table but retain the ability to theme by the base numerical or string lookup id?

In the next release of MapGuide Maestro, we've solved this problem for you.

Meet the new Theme Generation dialog


The Data Setup portion of the dialog has been split into 2 sections:

  • Theme based on values/ranges from this class: This is the existing theming option
  • Theme based on values from an external class: This is the new theming option where you can source the theme rules from an external class definition (eg. A lookup table)
To illustrate this new feature, consider we're theming against the Parcels Feature Source from the Sheboygan Sample data set (as per the above screenshot). We'd pick the RTYPE property, which tells us that 8 theme rules will be created.


Now unfortunately, the RTYPE property only contains abbreviated values and the theme that gets generated doesn't produce very informative labels (as seen in the first screenshot).

Now suppose we had an external Feature Source or lookup table that has descriptive labels for these RTYPE values.


We can use the new theming option to generate theme rules using the main property as a frame of reference.

Firstly, you would specify the Feature Source containing your lookup table


Then specify the Feature Class that represents the lookup table


Then you specify the key and value properties that will determine the filters and labels that will be generated for your theme rules.


If you want to apply any filtering on the lookup table itself to reduce the number of rules that would be generated, you can specify the FDO filter in the Filter field.

Then click Update to get a feel for the number of rules that will be generated. The rules that will be generated follow this form:
  • Filter: RTYPE = [Value of Default:ParcelType.Type]
  • Label: [Value of Default:ParcelType.Description]

At this point, you can then tweak the color setup and other theme generation settings as before. Once you click OK you can see that the theme rules generated match the values from your lookup table.


As you can hopefully see, this feature should dramatically simplify theming of features from relational databases without requiring any table/view contortions to get the desirable theme rules.

Thursday, 20 March 2014

Creative uses for Watermarks

I give you exhibit A: Injecting useful debugging information into a rendered map.