Saturday 30 June 2012

Thoughts on MapGuide scalability

I've been doing some automated stress testing on the MapGuide 2.4 beta. The tool I used was Bruno Scott's scripts for the Grinder load testing tool. Running these stress tests have given me more insight into tweaking serverconfig.ini values for greater scalability, which I am sharing in this particular post.

For the record, the system I'm running the grinder test from (and where all these images in this post are sourced from) is a Quad Core desktop running 64-bit Windows 7 with 8GB RAM, which also happens to a have a Google Chrome browser with several tabs open (one for writing this post :-)), and the box isn't breaking a sweat.

Salt. Take a few grains of it


Like my post about MapGuide's undocumented expression functions, don't take my word as gospel. Most of my insights are from a 200 user automated stress test that is running as I'm writing this. Normal users do not operate like java threads following pre-determined scripts in a grinder test run, you will have a mix of heavy users, light users and users in between. Rather, consider this post as a data point from which you can extrapolate from to form your own conclusions about the optimal server configuration settings for a high volume environment.

The default settings will not cut it

Let me say off the bat that the default serverconfig.ini is definitely not optimized for serving high volumes of users. My first grinder test run simulating 50 users breached the session repository limit within 2 minutes! Gradual tweaks to serverconfig.ini has made the MapGuide Server more resilient to high volume requests. A 200 user grinder test is running as I write this post with no crashes or noticeable errors (the only errors thus far are insignificant IO errors due to the fact that I've capped the size of the request logs).

Read on to see what needs changing.


# of Session Repositories = Your "available memory"


Each time a new session is created (eg. Login with Studio/Maestro or launch the map viewer), a new session repository file is created. The SessionRepositoriesLimit setting in serverconfig.ini defines the maximum number of session repository files that can exist at any one time. If the MapGuide Server breaches this limit, it will deny any further CreateSession requests until the number of session repositories goes under this limit.

You can actually count the number of sessions that are active by looking at the Repositories/Session folder of your MapGuide Server install directory. Each session takes 2 files (a .db file and a .dbxml file), so the math is not rocket science.




How does this number go down then? Through MapGuide's session expiry mechanism. Every time a session expires, MapGuide will delete the related session repository file. So to ensure a stable MapGuide Server in a high volume environment, you need to keep this number in check, and the way to do that is to strike a fine balancing act between session creation and session expiry to ensure that this number stays steady.


This is somewhat analogous to memory allocation in a managed programming language. If you are allocating memory at a rate faster than the garbage collector can clean up, you will eventually get an out-of-memory error thrown by the runtime. Same principle here. MapGuide's session expiry is your "garbage collector"


Simply put, you will need to be more aggressive with your session expiry settings. Here's the default values in serverconfig.ini


  • SessionTimeout                     = 1200
  • SessionTimerInterval               = 400


A session is considered expired if nothing is done with that session id within 1200s (20mins) and that expiry check is done every 400s (6m40s). In the case of our automated stress test, this is way too long a period to wait to start cleaning up, which is why the grinder test was able to hit the limit in only 2 minutes. We lowered the SessionTimeout to 200s (3m20s) and the SessionTimerInterval to 60s (1min). This allowed the MapGuide Server to reclaim expired sessions at a rate that the automated stress test was creating them resulting in a steady number of active session repositories.

From a user perspective, there shouldn't be any problems with unexpected expired session errors. Maestro and the AJAX/Fusion viewers already have keep-alive mechanisms built in to ensure your session is not cleaned up due to aggressive session expiry settings.


Server memory use correlates to # of active sessions.


The memory allocation analogy for active sessions is somewhat rooted in fact. Each active session will take up memory, but each expired session will also release it. If you take a look at the task manager




The graph of memory goes in a steady up-and-down fashion in line with the session creation from the stress test (going up) and session expiry/cleanup (going down). The default session timeout settings would result in sharper drops in memory usage, but you would probably already hit the session repository limit before that happens.

Don't bother with 32-bit MapGuide

Here's the memory usage from the 200 user automated grinder test currently running as I'm writing this post



4GB like 640KB before it, was thought to be enough for everybody. But as you can see by the memory usage, a 32-bit MapGuide Server will be hitting its upper memory limit quite easily on a 32-bit system. So there is no real point in having a 32-bit MapGuide Server for a high volume environment. You should be running 64-bit MapGuide which can support and address an astronomical amount of memory (so yeah you're in a bit of a pickle if you are using or considering Linux as we still don't offer 64-bit MapGuide linux builds yet).

Conclusion (TL;DR)
  • The default serverconfig.ini will not cut it. You will have to edit this file.
  • Crank up the session repository limit as high user volume can easily breach the default limit (my stress test uses a SessionRepositoriesLimit of 4000)
  • Lower the session expiry timeout and checking interval (my stress test uses a SessionTimerInterval of 60 and a SessionTimeout of 200) to ensure expired sessions are reclaimed faster.
  • If the numbers are steady (memory usage, number of session repo files), that's a good thing!
  • Use 64-bit MapGuide. Don't bother with the 32-bit one. Throw as much RAM as you can. More users = More RAM.

6 comments:

Unknown said...

Love your post on scalability of mapguide, i have been using this to setup AIMS 2014.

Do you have any other advice regarding scalability?

We are utilising AIMS 2014 to visually display live election results during the 2014 South African elections and the site will run on a public domain with a lot of users.

Jackie Ng said...

MapGuide has (had?) load balancing capabilities, but nobody (not even myself) knows much about it.

The only references to load balancing is this wiki article:

http://sandbox.mapguide.com/index.php/Pragmatic_Load_Balancing

And a small tidbit from the best practices guide

http://images.autodesk.com/adsk/files/mapguide_enterprise_implementation_best_practices.pdf

If/when I get around to playing around with load balancing in MapGuide, you can be sure I'll be blogging something about it :)

Unknown said...

Thanks for the information.
keep up the good work with your blog!

Unknown said...

Hi Jackie,

Is normal that the map server only delete some repositories?
I find in my case only the initiated sessions with Autodesk Infrastructure Studio are cleaned but the sesions initiated by the viewers never die.
¿It´s a related theme with the developed application? Because when the clients goes to the navigator and loads a sdf file with my app, a map is created at "the moment".
¿I need to add a specific instruction to force the end of the every sesion when the clients close the map? If the answer is yes, how can I do that?
I will appreciate very much any comment.

Thanks !!
Julieta Mtz

Jackie Ng said...

Hi Julieta,

If your viewers have keep-alive mechanisms on (Fusion does, AJAX viewer will only keep-alive if you enabled the setting in the WebLayout), their session dbxml files won't be deleted by the MapGuide Server while the client still has the viewer open, as they will be regularly sending ping requests that tell MapGuide that that particular session is still alive and should not be cleaned up.

If you want clients to properly clean themselves up when the browser window closes down instead of waiting for the MapGuide Server to clean up expired sessions, you could look into listening to window.onunload (see: https://developer.mozilla.org/en-US/docs/Web/API/Window.onunload) and have that issue a fire-and-forget AJAX request to a PHP/Java/.net script that calls MgSite.DestroySession() on that session id. This approach may cause problems if you want your client application to re-use the same session id between browser refreshes.

Hope that helps.

- Jackie

Unknown said...

so we got 285 000 hits in one day during the South African elections and kinda broke AIMS 2014 :) the map server just stops responding to requests after a while.
we followed the best practices guide, and config settings and tried the load balancer setup but the server fails to start with the second site connection IP address.
but overall it performed well.
check our site at elections.sabc.co.za

Thanks for all the tips on mapguide :)