Fusion redlining revisited.

For the next MapGuide Open Source 2.5 release (not sure if it will be another beta or straight to RC atm), we're giving the Fusion (in particular the Redline widget) a bit more spit and polish.

History

As explained previously, in this author's opinion, the original Redline widget for Fusion was completely useless functionality wise:

• Can't plot a view of your map with your redlines
• Can't control the styling of your redlines
• GML is your only import/export format. Who on earth consumes GML? Certainly not Autodesk's geospatial products!

So we threw it out and rebuilt it based on the original Generic Tasks sample for the AJAX viewer.

Now admittedly, the v1 re-implementation (which was in AIMS 2013) was nothing more than a carbon copy of the Generic Tasks sample, meaning not much real thought was put into usability and user efficiency. The redlining process was a cumbersome multi-step process that didn't really scream productivity.

So for the MGOS 2.4 iteration of this widget, we patched up the usability problems, shortcutting most of the previously mandatory setup steps (with options to revisit these steps if required). We also took advantage of some previously-unknown fusion components (to me anyway) to improve the digitizing process with informative prompts.

This iteration while usable and powerful, had one remaining limitation. SDF was the only import/export format. While round-tripping between Autodesk's geospatial products is not a problem, round-tripping to other geospatial software is a problem because SDF is not exactly a well supported format outside of Autodesk's range of geospatial products.

The widget improvements

So for the MGOS 2.5 iteration of this widget, the Redline widget now supports the following formats:

• Import: SDF, SQLite and SHP (as a zip file)
• Export: SDF, SQLite, SHP (as a zip file), KML and KMZ

That's 2 extra import formats and 4 extra export formats!

Here's how the revised management UI looks

Most of the UI functions as it did before, so we'll just cover the bits that have changed.

What was previously the "New" button has now been separated into its own section 

This may look a bit complex, but it's not really. You pick the types of geometries you want to be able to digitize and hit the button to create the redline data store of the desired format. 

Why did we have to do this? Simple. If we want to support the ever-ubiquitous ESRI SHP file as a redline import/export format, we have to cater to its extremely cumbersome file storage limitations. While SDF/SQLite files can happily store all 3 geometry types for redline data, SHP can only support one geometry type per SHP file. 

So if we want the user to be able to record redlines in SHP files, we have to know what type of SHP file the user wants to create. As you can see from the screenshot, as all 3 geometry types are selected so SHP is not available as an option. But if you only checked one of those types, the ability to create a SHP file for redlines is then made available.

The second difference, is in the actual redline digitizing UI. Geometry types that cannot be digitized will be disabled. So the digitizing UI for a point-only SHP file looks like this (notice how only "Point" is enabled):

The final difference, is the expanded list of download options.

"Download Data" will serve out the redline data store as before. KML and KMZ download options take advantage of MapGuide's built-in KML service APIs to export the selected redline layer out as KML/KMZ files. Unlike the normal usage scenario, this a full KML/KMZ dump of the layer (ie. a GetFeaturesKml call) and not the map (ie. GetMapKml), which is actually a KML skeleton of NetworkLinks back into the mapagent to get the actual layer KML data.

So how does this KML look? Here's some redlines I've prepared earlier

Clicking "Download KML" or "Download KMZ" gives you the download prompt.

Which looks like this when we look at it from Google Earth

As you can see, we lose a bit of visual fidelity because MapGuide styles don't cleanly translate 1:1 to KML styles, but the core geometric shapes and text information are there.

A general note about Importing/Exporting

I guess we failed to document this when re-implementing this widget so it's better now than never.

There are some caveats to take note of when importing/exporting redlines.

• Your export format is the format that you specified when you created/imported the redline data store. If you created a SQLite redline data store, that is what you will get prompted to download (a SQLite file) when exporting. Previously the only supported data store format was SDF, so this wasn't much of an issue.
• If exporting redlines as SHP, you will be prompted to download a zip file. Once again, the shapefile format is not one file, but a set of inter-related files. Since we can't prompt you to download multiple files at once, the widget zips up the files in question and serves that file out for download.
• Similarly for importing redline files. If you want to import a SHP redline, it has to be a zip file of the SHP file and its related DBF, IDX, SHX, etc files.
• On the general subject of importing, the file you choose to upload must meet certain structural requirements in order to be accepted as a redline feature source. Generally speaking, we assume you are uploading redline files created orignally by this widget.
• Exporting only exports the data and not its visual representation (ie. the Layer Definition)
• Similarly, importing will set up a default Layer Definition for your uploaded data.

MapGuide tidbits: RFCs

For this post, I'm going to shed a little insight into those things called RFCs.

Having submitted several RFCs for MapGuide which have been fully realised into new product features, I think I am qualified to share some insight into how RFCs work. Unlike the formal Wikipedia definition, we're going to be talking about RFCs strictly in context of the MapGuide Open Source project.

What are RFCs?

RFCs (Request For Comments) can be generally thought of as detailed proposals for a new feature or enhancement for MapGuide.

Anyone can submit an RFC, but writing an RFC is more than just "MapGuide needs to do X or support Y" as you'll hopefully see below.

How do RFCs differentiate from enhancement tickets?

Enhancement tickets are generally micro-level in nature, RFCs are macro-level in nature that have broad implications to MapGuide as a product and platform. Enhancements that would affect any of the following areas are usually candidates for an RFC proposal and some level of discussion first with the Project Steering Committee:

• API additions/updates/removal. (eg. RFC9)
• A new type of resource. (eg. RFC108)
• A resource XML Schema update. (eg. RFC16)
• Support for new(er) standards. (eg. RFC106)
• Major user-facing product features/enhancements. (eg. RFC24, RFC71)

What RFCs are not

As already mentioned, RFCs are not enhancement tickets, not just in scope but also in purpose.

Enhancement tickets follow a simple formula: "I wish this part of MapGuide could do X" and the rest will then be left as an open exercise for any interested contributor to see if it can be implemented. Or, such tickets will already have a patch included, which makes our jobs much easier :)

RFCs on the other hand, will generally have implementation work done up-front due to their broad scope. Actually I would imagine that most RFCs posted for discussion will have at least 70% of the implementation work already done and the remaining 30% to be filled in as a result of discussions on mapguide-internals or adoption/approval by the Project Steering Committee.

RFCs submitted without any plan of implementation or a 70%-100% working implementation up-front will generally go nowhere. It's not like Autodesk, myself or anybody in the Project Steering Committee is going to help fill in the blanks for you. We'd comment here and there (as needed for an RFC discussion), but ultimately there has to be work done (or to be done) by you (the RFC submitter) or a clear detailed plan of implementation.

RFCs are basically your proposed plan for a new feature/enhancement:

• Here's a idea for a new feature/enhancement for MapGuide
• Here's some scenarios or reasons where we need such a feature/enhancement
• Here's how this new feature/enhancement would be implemented. Where applicable, here's a link to a SVN branch or a build where this feature is already implemented, so you can see for yourself how this feature looks and works.

Such RFCs will have enough meat and potatoes in them for the Project Steering Committee to discuss and vote on for formal adoption into the MapGuide Open Source project.

My process for submitting RFCs

Let me illustrate the life-cycle of an RFC with some examples from RFCs I've personally submitted and seen through to implementation.

1. The idea

When I have an idea, I make a sandbox branch from trunk and start hacking and playing around. As you can see, many RFCs I've submitted started life as sandbox branches. The benefit of doing RFC work on a sandbox branch is that upon adopting by the Project Steering Committee, integration into MapGuide is a simple svn merge away.

2. Drafting the RFC

Once the implementation in my sandbox branch is 70-90% complete, I then move on to drafting up the RFC. This covers the above bullet points in as much detail as possible.

3. mapguide-internals discussion

I then submit this RFC to the mapguide-internals mailing list for discussion. This generally takes 1-2 weeks, which is generally enough time to address any concerns or unknowns in the RFC. After this period, if there's no opposition, the RFC gets called to a vote for adoption. If the RFC does not get adopted (very unlikely in most cases), I'd stop at this point. Otherwise, I move on to the next step.

4. Filling the blanks

The remaining 10-30% of the implementation is then completed based on feedback on the RFC, or if there isn't much feedback, the original implementation plan as stated in the RFC.

5. Merge back into trunk

Since the implementation work is in a sandbox branch, re-integration back into trunk is just a simple merge away. The RFC at this point is complete.

In closing

Hopefully this post has somewhat de-mystified RFCs in the MapGuide Open Source project and may give you some confidence to submit one of your own in the near future.

MapGuide tidbits: Data filters

For this post, we're going to cover briefly about Data Filters

Filters are what can be passed to FDO feature queries for returning/excluding certain features

For example, filters can be used to restrict what gets rendered for a particular layer

They are also used to define what features fall under a certain theme rule

Filters also form the backbone of search commands in the AJAX viewer (and for those wondering, $USER_VARIABLE is the special search command placeholder that gets replaced with whatever the user enters in the search command prompt)

So that's all and good. I'm sure most of you are already aware of what filters are supposed to do. But what some of you are most likely confused by is the actual syntax of these filters.

If you're coming from a MapGuide 6.5 or older background, you are probably assuming these filters are just SQL where clauses. This is not the case with FDO.

FDO is a data abstraction layer after all, meaning we also need a "one size fits all" abstract filtering grammar that can be easily processed and handled in a provider-specific manner (eg. translation to equivalent SQL where clauses)

The BNF of FDO filters can be found in the FDO Developer's Guide. Here's the relevant link for convenience. If this BNF stuff confuses you, the upshot of all this is that for the most part FDO filters are similar, if not identical to SQL where clauses but there are some cases where certain conditions may look a bit un-intuitive.

For example, assuming you know your SQL you might think a NULL condition may look like this:

COLUMN_NAME IS NULL

When in actuality, such a condition does not work as a FDO filter due to incompatibilities with the FDO filter grammar. The equivalent condition in FDO is actually:

COLUMN_NAME NULL

Similarly, you may think that the negated form of the above condition would look like this:

COLUMN_NAME IS NOT NULL

When in actuality, the FDO equivalent is actually:

NOT COLUMN_NAME NULL

If you look at the BNF of the FDO filter grammar, you can see why NULL conditions are structured this way. But as I've already mentioned, such cases are rare and for the most part FDO filters are syntactically similar to most SQL where clauses.

The second thing to note about filters is that some functions may not work for some FDO providers. Not all providers are created equal in terms of features and capabilities and the same differences apply for filters. Now you can safely assume that the key operators will be supported by all FDO providers (A FDO provider would be pretty useless if you can't do a basic A = B comparison filter), but in terms of filter functions, support can vary from provider to provider.

The FDO provider's capabilities (which can be accessed via MgFeatureService.GetCapabilities() ), describe what functions are supported. The good news about this is if you are only using Maestro to construct such filters (for layers, search commands, etc), these differences are insulated from you because Maestro uses the provider's capabilities to present only to you a list of the supported functions in its Expression Editor UI.

Thus assuming you are constructing a syntactically valid FDO filter, it is impossible to actually construct a filter in Maestro that is not supported by the underlying provider. For FDO Toolbox, we use the same technique for its Expression Editor UI as well.

Hope this post sheds more light about filtering.

MapGuide tidbits: "Maps" in Fusion

Chalk this one up to English being the scumbag multi-contextual language that it is.

Here's a little trivia. What does a Map mean in Fusion?

Many things actually. It could be any one of the following:

• A MapGuide Map Definition resource
• A part of the ApplicationDefinition document that defines one logical map (The <mapgroup> element)
• A child <Map> element of the <MapGroup> of the ApplicationDefinition document which is a component layer of a logical map (eg. A MapGuide Map Definition or a Google/Bing/OSM layer).
• A javascript runtime instance of the <MapGroup> element (the Fusion.Widget.Map object)
• A javascript runtime instance of the <Map> child element (the Fusion.Layer.MapGuide or Fusion.Layer.Generic object)
• An OpenLayers.Map object that is wrapped by the Fusion.Widget.Map object

Confused?

Here's some images I've made to hopefully explain all the various "maps" in Fusion.

Here's how the "map" elements in the Application Definition correspond to the editor UI in Maestro.

And here's how said "map" elements correspond to their "map" runtime equivalents.

So when you're talking about a Map in Fusion, it helps to be specific. 

*Really really* specific!

Announcing: MapGuide Open Source 2.5 Beta 1

So with the failed 64-bit Linux experiment out of the way, here's the first beta of MapGuide Open Source 2.5

Get all the information and downloads from here. We already covered what's new in a previous post.

The release comes in the following flavours:

• 32/64-bit Windows Installer
• 32/64-bit Windows InstantSetup bundles
• 32-bit Ubuntu 12.04 deb installer (coming soon)
• 32-bit CentOS 6.x tarball (coming soon)

As we don't actually have a formal FDO 3.8 source branch to build from, this release of MapGuide was built from the FDO 3.8 beta 1 binary SDK which includes the ArcSDE Provider

I'm aiming for a short and aggressive release cycle for 2.5, so it could well be possible that MapGuide Open Source 2.5 is actually released before FDO 3.8 final, in which case so be it. We can always put out a 2.5.1 point release when FDO 3.8 goes final if this is the case.

MapGuide tidbits: Support and User Expectations

I didn't want to have to write this post, but I have no choice. Consider this a Public Service Announcement.

When you buy a licence for Autodesk Infrastructure Map Server, you're not just buying the product, you are also buying for product support from Autodesk or their reseller partners.

For MapGuide Open Source, there is no formal product support.

Your available avenues therefore are as follows:

• The mapguide-users mailing list.
• The MapGuide Open Source trac instance.
• Engaging a MapGuide expert for paid support/consulting.

Being a free and open source project, nobody is obligated to provide support unless you're paying them under a support/consulting agreement. But if you can cooperate and play by our rules of engagement, we are more than happy to provide help and answers wherever possible.

Getting the most out of the mapguide-users mailing list

The mapguide-users mailing list should be your first port of call for any MapGuide question or support query. When I get personal emails asking questions about MapGuide or support questions about MapGuide and:

1. We do not have a pre-existing support/consulting agreement (and if we did, this should've been through my business email, not my personal one).
2. I find that you haven't posted the question/query to the mapguide-users mailing list first.
3. The email is not well written or the email is a really basic or pointless question.

I will either tell you to use the mailing list or ignore the email. Simple as that.

Think about it. If you have a question/problem with Windows, do you go and harass Steve Ballmer? If you have a question/problem with Linux, do you go and harass Linus Torvalds? Of course not! You go through the many channels that are available for Windows/Linux, and for MapGuide Open Source that channel is the mapguide-users mailing list, not my personal email address!

I read the mapguide-users mailing list every day. If you ask a MapGuide question on the mailing list, and I have a fair idea of an answer, I will give you one through the mailing list. Don't go emailing me the same question directly afterwards just because nobody has answered your question on the mailing list. I've most likely already seen your question and don't have an answer. Pestering me personally will not change that fact!

I'm not saying these things from a position of arrogance or superiority. The mailing list is not just a forum for support, it's also a knowledgebase, and the only way the mailing list can be an effective knowledgebase is if everybody asks questions and answers through it. When myself or someone gives you an answer through the mailing list, they are not just answering for you, they are also answering for anybody who might have such a similar problem down the line. Pinging myself or any MapGuide user directly instead of through the mailing list is bad etiquette for the following reasons:

• You're restricting the number of people who could possibly answer your question/query to just myself instead of anyone who's subscribed to the mailing list.
• You're denying potential answers to anyone who's subscribed to the mailing list who might be having the same problem as you.

It's pretty selfish when you think about it.

However, I do give leeway if the personal emails I get are highly confidential or of the more esoteric type (eg. mg-desktop, MapGuide/Maestro code/architecture/feature discussions) and will happily respond to such queries, but otherwise the above rules apply for the reasons already stated.

Now with that out of the way, here's how you can get the most of the mailing list. The key is to ask good questions. We put this same link on the nabble front-end to the mailing list, but it's apparent that nobody reads that. So since you're reading this blog, read that link before you continue. Seriously!

Now to add some MapGuide-specific context, when you need to ask the question please always specify the following where possible:

• The version of MapGuide you're using.
• Your Operating System. For Linux, state your distro.
• Whether you're running a 32-bit or 64-bit version of MapGuide.
• Your MapGuide configuration (IIS/Apache, .net/Java/PHP).
• The version of Maestro you're using (if the question is related to Maestro).

For problems/questions relating to your spatial data, specify which FDO provider you're using. This helps us to better isolate the problem and determine if the problem is specific to MapGuide or the FDO provider.

For problems/questions relating to coding/development, do provide code snippets and explain what you're trying to achieve. Try not to paste your code snippets verbatim in your questions as they're bit ugly to read from the nabble front-end. Consider using something like GitHub gists to put your code snippets in and just link to that gist in your question.

Put screenshots whenever you can. You know the old saying: A picture is worth a thousand words.

Where possible, please try to localize your problem around the Sheboygan sample dataset or publicly accessible WFS/WMS servers. If your problems can be reproduced through the Sheboygan dataset or these WFS/WMS servers, it will make things easier for us to diagnose such problems and there won't be issues of confidentiality around your specific spatial data.

Do become familiar with Firebug and similar debugging tools. For problems relating to the AJAX Viewer or Fusion we'll most likely ask you what Firebug says 80% of the time.

Do keep an eye on what MapGuide is logging. MapGuide may have logged an informative error when your problem occurred.

Understand that the likelihood of getting your MapGuide question/problem resolved decreases with older versions of MapGuide. Simply put, we're not a billion dollar corporation like Autodesk. We simply do not have the resources to support so many different versions of MapGuide. Our default answer is to recommend you upgrade to the latest stable release because:

• It may have resolved your problem
• It fits into our "support window" for reproducing these problems. The window will generally be the current stable release and 1-2 versions back.

Please try to avoid general coding/development questions. This is what sites like Stack Overflow were invented for.

Do actually subscribe to the mailing list proper and not just the nabble front-end. If nabble is down or goes away, so will your questions and/or answers.

Finally, we are not psychics who can read your mind. We only know what you're telling us. The more information you give us, the better it is the chances of us being able to diagnose the problem and give you an answer.

Getting the most out of Trac

So your problem is most definitely a defect in the MapGuide product itself or something that requires enhancement work. This is where you probably want to file a ticket. 

Most of what I said about the mailing list equally applies here. So I'll add things specific to Trac.

Don't use Trac for asking questions! This is what the mailing list is for!

When filing defects, include reproduction steps or test code/data that demonstrates the problem. Anything that reduces the setup time in reproducing the problem will get us fixing the problem itself much faster. Again, I'd like to emphasise the importance of being able to localize the defect against the Sheboygan sample dataset. If the defect is reproducible against this dataset, things will be much simpler.

Take ownership of your filed tickets. Don't just file a ticket, leave and then auto-magically expect someone will then fix the problem. Keep these tickets up do date with any important findings and information. If we ask any questions on that ticket, do follow up on our enquiries. A fair chunk of filed tickets are closed due to ticket inactivity because these tickets didn't contain enough information and the submitter failed to add any extra findings or follow-up on questions that we've put on that ticket.

Keep the wiki up to date. Like the mailing list, it's a knowledgebase that benefits everyone.

Having a milestone assigned to your ticket does not guarantee it will be resolved by that milestone. It usually just means that we'll look at the ticket for the release cycle for that particular milestone, and see if we can fix it for that milestone. Once again, more information on the ticket will make this happen. Otherwise, we'll just shelve that ticket for a later milestone, or close it if there is no activity for a certain period of time.

Be specific about enhancement tickets. Don't just file an enhancement ticket stating we wish MapGuide would do XYZ, be specific. If you have an idea about how such an enhancement would be implemented, state your plan of attack.

When submitting patches, understand how to create patches properly. We're happy to accept patches, but it makes it much easier for us if these patches are diffs that can be cleanly applied to a svn working copy. Pay special attention to whitespace (use 4 spaces instead of a tab).

In closing...

Hopefully, this post explained the expectations from you (the user) and from us (the developers/contributors).

Please always go to the mailing list first for any MapGuide questions/support. My personal email address is not for your MapGuide support or basic enquiries!

Not happening :(

So my joy of getting the MapGuide Server executable running in a 64-bit Linux environment has been placated with a new sobering reality.

If we were to ship MapGuide Open Source 2.5 for 64-bit Linux in is current form, at least half the FDO providers included would be completely broken. Running the FDO test suites that I was able to run drove this point home. There's obviously a whole assortment of linux code paths in the FDO core and its providers that make invalid assumptions in a 64-bit environment. The volume of problems is such that getting them addressed in my desired 2.5 release time frame is not going to happen.

So 64-bit Linux support will be on the back-burner for at least one more release cycle :(

With that said, onward we go to beta 1!