Well in Visual Studio 2010 (Beta 2), this is no longer the case. Invoking the Add References dialog is super fast! The reasons being:
1) .NET and COM references are now loaded asynchronously in the background.
2) The default tab is the Projects tab (which is the one I go for most of the time)
Truth be told, I would happily upgrade to VS2010 just on this one feature. But I'd probably lose out on the coffee breaks :-)