Improving the MapGuide API Documentation (part 3)

This post has a bit of a misleading title (it's more the wrapper API we're improving than the documentation) but whatever, we'll run with it.

So just to explain the previous screenshot (and probably my java-related tirades on Twitter and Google Plus for anyone who happens to follow me there) ...

That screenshot was "visual" confirmation of some MapGuide API wrapper enhancement work I've been doing to address some of the pain points of using the MapGuide API (namely Java, but this work has some cross-pollinating benefits for .net as well)

The Problem

For the 15 people in the room who actually use the MapGuide Java wrapper API in its current form (I kid, though it could possibly be true), you've probably had to deal with these annoying issues:

• The Java proxy of MgException being a checked exception. Thus having to pollute your java code with "throws MgException" in calling methods or having to try/catch an exception that probably isn't thrown 99% of the time and if you do catch it, you're probably just gonna log or display that exception anyway.
• Java class method naming being a verbatim transplant of their C++ counterparts (ie. They're in UpperCamelCase instead of lowerCamelCase)
• Java proxies of MapGuide collection classes being verbatim wrapper classes, and not behaving like a Java collection. You can't even for-each through these collections due to key java interfaces not being implemented in the proxy classes.
• Needing to have the web-based API reference on hand as doxygen commentary is lost in translation to the target language. Auto-complete in your Java IDE of choice gives you no documentation, because none is transferred across.
• Needing to have the web-based API reference on hand to also find out if a given class or method is deprecated, because nothing in the proxy classes will tell you that to trigger any compiler warnings
• Hardcore memory leaks in the Java wrapper

Most of the above points, pretty much summed up in this 6-year old ticket.

Though the memory leaks were plugged for the 2.4 release, the other problems are still present. Despite my own reservations about Java (the language), Java (the platform/ecosystem) can't be ignored. Just because we're using SWIG to churn out bindings to 3 different languages doesn't mean we can't do some tweaking here and there for a given language. We already do this for .net (to support properties and serializable exceptions), so why not give the Java wrapper some love as well?

The Solution

For those who are wondering how we actually build bindings to the MapGuide API in 3 different languages, here's a high-level overview:

As already mentioned, SWIG is the tool used to generate the language bindings to the MapGuide API, but in order to enforce some level of encapsulation in the wrapper APIs (because there's C++ internals in some MapGuide API classes that shouldn't be exposed in the wrapper classes), we run a custom pre-processor (IMake.exe) through the MapGuide C++ headers to generate a "sanitized" input file for SWIG to do its thing.

If you ever look at a any MapGuide C++ header and wonder what those PUBLISHED_API, INTERNAL_API and EXTERNAL_API markers are for, they are for IMake to generate an encapsulated view of that particular class for SWIG. Similarly, the __get and __set markers you see at the end of some method declarations are hints to IMake to generate .net properties for that particular method.

The same IMake tool is also used to generate the constants.php for PHP and the necessary constants source files for .net and Java

The solution consists of 3 parts:

• Modifying SWIG to support our java-specific requirements
• (Ab)using some SWIG directives for purposes beyond their original scope
• Modifying the IMake preparation tool to take advantage of these (ab)used SWIG directives

Modifying SWIG

This part was simpler than I thought. The SWIG source code was not as hostile as I originally thought and hacking the java module to support what we need was relatively straightforward. SWIG was basically modified to support 2 optional java-only flags:

• To omit the "throws MgException" clause in any java method generated
• To output java methods in lowerCamelCase. Since all MapGuide API C++ methods are guaranteed to be in UpperCamelCase, this is a simple case of lower-casing the first letter of each method that SWIG writes out.

These flags are optional, because if we are going to provide a new enhanced Java wrapper API, we'd still want to keep the existing one around for compatibility purposes, with the enhanced wrapper being an opt-in choice.

(Ab)using SWIG directives

The directives in question are:

• %typemap(javaclassmodifiers) (.net equivalent is %typemap(csclassmodifiers))
• %javamethodmodifiers (.net equivalent is %csclassmodifiers)

Normally, these 2 directives are used to alter class/method visibility of the respective generated proxy classes

For example, if we apply this SWIG directive:

 %javaclassmodifiers MgMap::MgMap() "protected"  

We get the following java proxy class method declaration:

 public class MgMap
 {  
   ...  
   protected MgMap()  
   {  
     ....  
   }  
 }  

Similarly if we apply this typemap:

 %typemap(javaclassmodifiers) MgMap "private"  

We get the following java proxy class declaration:

 private class MgMap  
 {  
   ...  
 }  

The trick with these directives is that that SWIG does no validation of the visibility string, meaning we can put any arbitrary content in there, say things like javadoc comments and annotations (or XML comments and attributes in .net). In fact, this technique is what the authors of SWIG recommend as a way of being able to document your proxy classes

So since we all know that the blank MgMap constructor is a deprecated API that you should stay away from. We can specify SWIG directives like this:

 //Java  
 %javamethodmodifiers MgMap::MgMap() %{/**  
  * This constructor is deprecated. Use MgMap(MgSiteConnection) instead  
  */   
 @Deprecated public%}  
 //C#  
 %csmethodmodifiers MgMap::MgMap() %{///<summary>  
 /// This constructor is deprecated. Use MgMap(MgSiteConnection) instead  
 // </summary>  
 [Obsolete(\"This API is deprecated\")] public%}  

Which will then turn into the following proxy class method declarations when SWIG processes them:

 //Java   
  public class MgMap   
  {   
   /**  
   * This constructor is deprecated. Use MgMap(MgSiteConnection) instead  
   */  
   @Deprecated public MgMap() {   
    ...   
   }    
  }   
  //C#   
  public class MgMap   
  {   
   ///<summary>
   /// This constructor is deprecated. Use MgMap(MgSiteConnection) instead  
   ///</summary>
   [Obsolete("This API is deprecated")] public MgMap() {   
    ...   
   }    
  }   

Meaning with these directives, it is now possible for us to not only pass down (converted) doxygen comments, but also pass down deprecation intent as well, because sometimes an API reference is not enough to tell you something is deprecated. You ideally want csc or javac to throw a compiler warning at your face while compiling against the wrapper API library to truly drive the point home that you should not be using this class or method.

Modifying IMake

So we have the means to transfer documentation and deprecation from the C++ classes, but surely we aren't going to manually transcribe the documentation fragments of the hundreds of classes that are in the MapGuide API? Of course not!

The IMake tool already does a pass through all the MapGuide C++ headers. And if you look at the constants.php generated by IMake, you can see that it does pick up and collect doxygen commentary along the way (albeit dumped verbatim into the target language, making it un-usable as javadoc or .net documentation). So the IMake tool was modified to do the following:

• Translate these collected doxygen fragments to their javadoc or .net counterparts
• Write these fragments out as the aforementioned SWIG directives for the target language

Other assorted codegen magic

Sadly, Java being the dinosaur of a programming language that it is compared to C# (in my opinion), does not support something similar to partial classes, which would've made augmentation of generated java proxy classes so much simpler, so in order to add java.util.Collection and java.util.Iterable support, we have to once again look at SWIG directives for ways we can augment the generated java classes.

Fortunately SWIG also supports a javacode typemap, that basically allows us to inject vast bodies of java code into our SWIG generated java proxy classes. This allows us to easily implement java.util.Collection support for the collection classes and java.lang.Iterable support allowing for these objects to be iterated in a for-each fashion.

In a way, these SWIG directives are kind of like pseudo-templates for Java, minus the pain of having to deal with the half-assed Java generics, as their way of implementing generics completely blows (I can't make an array of generic type T? GTFO!)

The MgException class (from which all MapGuide exceptions inherit from) extends a pre-defined java AppThrowable class, which itself extends from Exception. As a result, all proxy exceptions are checked as a result. Making these exception classes un-checked as a simple case of re-basing the AppThrowable class to extend from RuntimeException instead.

The End Result

So how well do these changes fare? We can verify by running javadoc against the generated proxy Java class source files. Have another look at that same screenshot of the javadoc from our "enhanced" Java wrapper API:

What can we see here?

• Java method names are now properly named in lowerCamelCase
• Doxygen \deprecated markers have been properly translated to @Deprecated annotations (and ObsoleteAttribute in .net)

And also a simple reminder from IMake that we've still got lots of room for improvement in the API documentation content

If we dive down to the method level, here's what we currently have in doxygen:

And here's what gets transplanted to javadoc

As you can see, most of the key doxygen directives have matching/compatible javadoc equivalents allowing for easy conversion by the IMake tool.

All MapGuide exceptions are now un-checked, as they extend RuntimeException

For .net, we get the same result as going through the DoxyTransform tool. In fact, this API enhancement work will render the existing .net documentation approach obsolete, as this approach does the same thing but supports multiple target languages.

So the javadoc output is proof that we have working transplanted documentation, now comes the important bit: making this documentation easily consumable from within a Java IDE, which is incidentally where I'm currently scratching my head. 

What is the official/recommended way to consume Javadoc in a Java IDE (Eclipse, Netbeans, etc)? Is there an official way? Or does each IDE have its own way? I'd appreciate some pointers from someone more knowledgeable in Java tooling than myself. Feel free to comment below.

At the moment, from what I can gather there's 2 ways to bundle up this documentation for consumption within a Java IDE:

• Pack the java proxy class source files into a separate -sources jar file
• Pack the javadoc documentation files into a separate -javadoc jar file 

I tried both approaches through IntelliJ IDEA and it seems to pick up both forms:

Here's the result of ctrl+Q with the -sources jar

And the result of ctrl+Q with the -javadoc jar

But I'd still prefer to only have to choose one method that will work on all the major Java IDEs. I'm just not sure which one it is. I'm currently leaning on the -source version, but I'd still like an "official" answer.

But what about PHP?

You might have noticed that PHP is a conspicuous absence from this MapGuide API enhancement work.

Well the truth is that PHP poses the following problems:

• Besides constants.php there is no actual PHP source files to "compile" into. So there's no actual place to drop our translated doxygen comments into.
• There is no officially defined standard for documenting PHP code. Even if there was, the real aim of this API enhancement work is to improve integrated API documentation (ie. From within an IDE). If all the PHP documentation solutions ultimately produce something similar to what we already have in our API reference (a web-based documentation source), then what is the real point? We already have that!

What's left?

Besides answering the Java IDE documentation question, this API enhancement will need a requisite test run on Linux (especially the SWIG/IMake stuff). All the viewers still work as before on Windows post-SWIG/IMake surgery, indicating the wrapper APIs haven't been negatively affected in any way.

Also I have to draft up an RFC for these enhancements, for eventual inclusion into the next major release of MapGuide Open Source (2.5). If everything pans out, this next release will have 4 Web API options:

• PHP
• .net (with XML documentation files for Visual Studio)
• Java (the old crufty one, kept for compatibility purposes) with Java IDE documentation support (whatever it will be)
• Java (the enhanced one with the changes described here) with Java IDE documentation support (whatever it will be)