This article ventures into the world of Outlook COM Interop in .NET. This is an introduction to some of the better practices in the general use of COM objects in .NET, as well as some of the surprises you may encounter along the way. Outlook is merely exploited as a real world example here; the contents of this article can also be applied to COM Interop in general.

This article also presents a solution I have devised for a very common problem (proper and safe release of referenced COM objects) for critical review. I have found much discussion on the Internet on the topic, but the advice is typically poor on how to handle it.

Before anything else, I would like to state that this article applies to applications that have heavy interaction with Outlook doing non-trivial things, not simple programs.

So, let’s assume you are required to consume the Outlook API to further some purpose. Perhaps you are developing an add-in for Outlook. Let’s say you’re doing this in .NET. Great, great….now, are you using VSTO, or were you smart enough to make your own COM shim?

Well, regardless, when dealing with Outlook, you will have to tango with Outlook’s COM objects. Hopefully you can tango well, as Outlook, like many other of its kind, is not a forgiving partner. Stepping on its toes in any shape or form will result in discomfort on your end.

Before we get too hot and bothered about all of that, let’s step back and examine what we’re exactly dealing with when we are using an “Outlook COM object”.

Primary Interop Assembly Objects

Unless you are very special, if you are using .NET, then you are most likely consuming the managed interfaces for Outlook COM objects made available through the Outlook Primary Interop Assemblies.

Many newer developers might have never had to deal with COM. Certainly, these days one can use .NET without knowing a wink about COM, and create well functioning applications in the process.

To create an object using COM, you would have a client call CoCreateInstance. In .NET, we have the new and New keywords built into the language.

COM clients keep track of an internal reference count, which is provided by IUnknown. This reference count comes into play when the client wishes to free a coclass. In .NET, we have the CLR garbage collector do all the nasty work for us in terms of garbage collection.

So what are Interop assemblies?

You must be new here! Interop assemblies are simply CLR assemblies that contain converted type definitions originating from a COM type library. They are commonly created auto-magically by Visual Studio, or through the use of tlbimp.

tlbimp MyUniqueLibrary.dll /out:Interop.MyUniqueLibrary.dll

There’s a name for this kind of interop assembly: Alternate Interop Assembly.

If you are not familiar with COM Interop, then this article is probably not the best place to start. Luckily, the Internet has plenty of information waiting for you on the topic.

The Primary Interop Assemblies are provided by Microsoft as the official way to interact with Office COM types.

How are Primary Interop Assemblies different?

I’m so very glad you asked. The notion of a “Primary Interop Assembly” is a typical kind of creature we tend to see from the folks at Microsoft, in that the Primary Interop Assemblies are the “official type definitions” for a library. They are meant to protect you from “counterfeit definitions that could only be used by the most unscrupulous of individuals.

There’s a bit more to it than just referring to it as something else, however. The process for creating a PIA is slightly different from creating an AIA, in that you have to sign your assembly and add a PrimaryInteropAssembly attribute to it as well.

If you know Microsoft, you know there’s a bit more to it than that. Things aren’t referred to differently if they aren’t treated differently by some process. Indeed, your suspicions are well founded, as the CLR and Visual Studio do treat PIA’s differently. For example, if I have a registered PIA on a system, and I have some software loaded in Visual Studio referencing the type library directly, Visual Studio will secretly just use the PIA instead of reimporting the type library.

That’s useful. So, if you have COM components you want customers to use, I guess it makes sense to create PIA’s to hand out, as a PIA will guarantee your interop assembly being used as opposed to an unofficial AIA (as long as your PIA is on that system, of course).

How PIA’s Work at Run Time

The way Primary Interop Assemblies work in regards to their distribution has changed greatly with the release of .NET 4.0. We don’t really care about all of that for this article. So put that to the side, it doesn’t matter.

What matters is what happens at run time. We’re dealing with COM here people, so clearly something strange and disturbing is going to happen. The amorphous blob of terror that manifests itself during execution is known as the Runtime Callable Wrapper.

The RCW is a very special proxy that exposes COM objects for use by managed code by marshaling calls between a .NET client and COM object.

It helps us out by marshaling method parameters and return values between .NET and COM clients. In order to do this, each RCW contains pointers to the COM object.

During run time, an RCW will be created for each coclass our managed client interacts with. The important thing here is that it only creates one RCW per coclass, regardless of its use. This frees us from having to worry about the COM object’s lifetime if it’s being used heavily, like we would have in the past.

When an RCW wraps a COM object, what it is doing is incrementing the COM object’s reference count by 1. The RCW then maintains an internal reference count that is incremented for each variable using the RCW. This internal reference count allows multiple instances of the same COM object without having to increment the COM object’s reference count past 1.

For example, we can get the ActiveExplorer in Outlook like so:

Explorer activeExplorer = Application.ActiveExplorer();

The Explorer we’re getting back here is actually a RCW wrapped object that points to the Explorer COM object. This increments the Explorer COM object’s reference count by 1, as well as incrementing the newly created RCW’s reference count by one.

Now, if we get the active explorer again without leaving scope like so:

Explorer secondActiveExplorer = Application.ActiveExplorer();

This will increment the RCW’s internal reference count by 1 again; however, the COM object’s reference count stays at 1.

Garbage Collection and the RCW

With COM, cleaning up your references involved explicitly releasing the COM object. Explicit action on behalf of the developer was required. This requirement was very easy for many developers to screw up, which brings about a situation where bugs and memory leaks can surface.

.NET is very different, of course. It does all the garbage collection work for us. So how do these reference COM objects get cleaned up by the CLR then?

Looking at the above example: setting the explorer variables to null isn’t going to immediately cause a release of the COM object. Recall that the RCW maintains an internal reference count. Garbage collection is not going to occur for the RCW and the COM object it wraps until the RCW’s internal reference counter reaches 0.

The memory actually used by the COM object is unmanaged. Thus, the COM object will be the one cleaning up that memory space, and that will happen when the COM object’s reference count goes to 0. The RCW has a finalizer which will release this COM pointer and then destroy the COM component.

The RCW’s finalizer won’t run until some time after the RCW is garbage collected. If you’re familiar with .NET garbage collection, then you would know that this means that there is a high probability of being stuck with unmanaged memory being allocated for extended periods of time.

If an RCW is somehow promoted to Generation 2 by the garbage collector, we could find ourselves in a situation where we have a lot of unmanaged memory allocated for a very long time.

What does this all mean? Well, it depends on the COM object, but in general, we don’t want to keep open references to COM lying around in memory. This is especially true of Office, and even more so with Outlook. Basically, we need to get back in the driver’s seat with regards to garbage collection and explicitly destroy these COM objects when we no longer need them.

Microsoft provides us with the following to release an underlying COM object:

Marshal.ReleaseComObject(explorer);

If you’ve Google’d this topic before, you have probably seen many sites recommending the use of:

Marshal.FinalReleaseComObject(explorer);

The difference between the two is that ReleaseComObject will release one of the COM interface pointers mapped to the RCW. Garbage collection doesn’t occur until all of those pointers are released. In the case of the two Explorer’s, we would need to call ReleaseComObject twice before the actual COM object was released. FinalReleaseComObject only requires one use, and it will clean up all interface pointers.

The use of FinalReleaseComObject is frequently recommended by people. It is unfortunate that its use is so commonly recommended, as it is actually very dangerous if not used correctly to your application’s stability. We’ll look at that in a bit.

RCW Reference Counting Rules

The exact circumstances of when the RCW reference count is incremented is a common source of confusion among developers. Here is a brief listing of the most important rules in this regard:

  1. Each time an object is requested from another COM object, the RCW reference count is incremented by one.
  2. Casting the referenced COM object into another COM interface does not increment the RCW reference count.
  3. Moving the referenced COM object around in your managed code does not increment the RCW reference count. For example, setting a class field to reference the existing COM object, or passing it as a parameter to a method.
  4. Events originating from COM that provide a COM object as a parameter result in that object’s RCW reference count being incremented.
So basically, we’re only going to see a reference count increase when dealing with COM. Nothing else we do in managed-land will result in that being incremented.

So, this brings us to the meat of this article….

Proper Usage of Outlook Interop

The following may apply to general COM interop, however I am focusing on Outlook here.

Outlook is a very special beast, as mentioned before. If your interaction with Outlook is complex, and you do not use its provided COM object’s correctly, you will suffer in many, undocumented ways.

I have encountered many problems with Outlook that were the result of other individuals’ improper usage of its COM objects. Sometimes the effects of improper use are not easily reproducible, or even noticeable until the usage has scaled up a bit. These problems, left unchecked, would easily compromise the user’s experience in Outlook.

I’ll give one example. Let’s say you are maintaining a reference to an Outlook appointment item. Now, if you keep that reference for too long, issues may arise. One such issue is during the background send/receive synchronization that goes off every now and then. What can happen is that a message box can pop up out of nowhere complaining that the item cannot be saved because it has been changed by another. The user then may have problems actually closing the appointment. That sort of thing will wreck the user experience and make your product very unpopular.

In fact, maintaining references to many Outlook objects for too long causes very strange issues to occur. Many times, when Outlook needs to do some processing on something, it may very well not ever get around to it as long as we hold a reference to an object.

Another example relates to Outlook shutting down. This doesn’t apply as much anymore with Outlook 2010, but previously, if everything wasn’t all tidied up, Outlook would hang forever.

This article is not intended to focus on what can actually go wrong if you misuse Outlook interop — many of those scenarios are simply not talked about, and don’t exist in any form in documentation. The point of this article is how to avoid these problems. I can say, from much experience, that if you use Outlook in any sort of advanced capacity, you will run into serious problems if you don’t take the proper precautions, regardless if you’re using .NET or not.

So, this brings us to the two main guidelines I have for proper Outlook Interop:

Outlook Interop Main Guidelines

  1. Interact with Outlook COM objects as little as possible
  2. Release all Outlook COM objects immediately after use, as much as possible

Only use Outlook COM objects when you have to. If you have other, less experienced perhaps, developers working with you, then what YOU want to do is ensure that THEY interact with the COM objects as little as possible.

For my work I have developed a very robust Outlook facade library that provides a set of entities known as Agents, which act as middle men between high level developer code and Outlook. Basically, the Outlook facade protects developers’ code by preventing them from having to directly interact with Outlook objects. The Agents themselves have rigorous operating procedures compiled from all my experience that ensure safe usage of Outlook’s objects.

I’m not posting that code as it is proprietary, however we’re going to go over some of the techniques you would find in there that encourage well behave Outlook interaction.

#1 – Dot me once, shame on you; dot me twice, shame on me…

You may have heard of this before. Also known as the “double-dot syndrome”.

Basically, if you are accessing an Outlook object through another object, and the desired object is more than one level deep, you have just failed.

Example, the following is fine:

Explorer explorer = Application.ActiveExplorer();

Great! The following, is bad:

Selection selection = Application.ActiveExplorer().Selection;

What’s so bad about that? Well, if you can recall the general guidelines, we should be releasing references as soon as we can. Therefore, in the first example here, we can release the Explorer whenever we are ready. In the second example, we can release the Selection whenever we are ready. But what about the ActiveExplorer() occurring after the first dot there? What happened to that?

Well, it didn’t just blink out of existence. By using two dots, you have created a hidden RCW and you will never be able to release it manually. You have just failed.

How to remedy?

Explorer explorer = Application.ActiveExplorer();

Selection selection = explorer.Selection;

Now you can release both of them successfully. Even if the object you want is 16 levels deep, you have to create 16 variables (assuming all the levels are COM objects) and then release them when appropriate.

Letting one instance of this go, results in a compromised application where stability cannot be guaranteed! Outlook COM interop is very unforgiving.

#2 -Retrieve required information in alternative ways when possible

What information do you require? Are you looking for the SMTP identity of the current MAPI profile? Do you need to know whenever a new item has entered a MAPI folder? Are you trying to determine the permissions the user has in the folder the user is currently operating in?

There are a lot of pieces of data we have and may require access to. Do you need to use Outlook for any of the above?

Outlook is a very successful software product. But, an extended amount of experience with it leads me to believe that it can be very unreliable and unsafe. Different versions of Outlook may behave in very different ways; these differences may be undocumented, and not commonly encountered.

All of the data points listed above can be retrieved without using the Outlook API. You can easily use something like Redemption in its place. Redemption is also a COM library, and requires proper usage. That aside, do I have greater faith in the reliability of Redemption as opposed to the reliability of Outlook? Yes. It’s going to be much simpler, and is most likely much simpler than the huge code base that is probably Outlook.

It is important to remember that, for the most part, Outlook will be operating in cached mode. You need to be aware of where you are getting your information from when using Redemption. This article is not going to cover all of that, but I thought I’d just mention it.

When do we HAVE to use Outlook?

Well we might HAVE to use Outlook if we need to know about data that wouldn’t be available from Exchange, such as a position of a window, or whether or not the user is preparing to send an email out in regards to an updated meeting.

Perhaps your product is required, by business rules, to prevent that change from occurring, due to some real world conflict, or something similar. You can’t rely on simply detecting changes in the user’s information store, as you have to prevent those changes from occurring. The reason why that is important is that if the change has already occurred, then an email has already gone out. If you revert those changes later, you’d have to send out another email, which would be a show stopper for most large organizations.

You’re going to need to use Outlook for items like that. Even then, you may not have to. Perhaps you can detect these things through the use of the Win32 API. Sound hacky? Well, that is good to feel that way; we want our code to be responsibly developed so it behaves adequately.

Unfortunately, you also want to use what’s most reliable. Sometimes, using Outlook’s API is just not reliable. For example:

In Outlook, when the user drags a meeting to a new time, it will ask if the user wants to send an update to the attendees. Usually they will say yes. The email then appears, and it can then be sent regarding the new time.

In Outlook 2003, in cached mode, as soon as the item is dragged and dropped, an ItemChange event goes off in the folder, right when the dialog which is asking the user if they want to update all attendees appears. So if you needed to prevent the change, doing so was very easy in this case; you could just set the cancel parameter provided to the change event to true, and there you go.

In Outlook 2007, this changed drastically. In cached mode, when the item was dragged and dropped, the item would appear to physically move to the new time. Despite this, no ItemChange event would fire yet. The user would be prompted if they wanted to send an update to attendees, and then the email would appear.

At that point in time, the ItemChange event still would have not fired, even though the appointment has moved to a new location. When would the item change event actually? Not until either:

  • A background Send/Receive synchronization had occurred
  • The user clicked Send to send the email

This is a bad place to be in, and makes the whole method unfeasible for use. Most emails are going to be sent out before a background Send/Receive (and even if they weren’t, relying on that is an unacceptable way of doing things).

In the second case, once the user has clicked Send, then it is already too late.

This change in behavior, by the way, was completely undocumented. I can’t find any acknowledgment of this change anywhere on the web either. I forwarded this to Microsoft support, and we pretty much ended up with a “too bad”.

No complaints, just saying: if you use the Outlook API at all, then your code can be considered hacky to some degree already. So, if you can find a consistent and alternative solution, you should follow through with it.

#3 – Release all Outlook COM references as soon as possible in an intelligent manner

This shouldn’t be news to you. You need to release all of your Outlook COM references manually. You should release them as soon as possible to avoid problems.

What may be news to use is that the commonly suggested way: “Always use FinalReleaseComObject“, is actually horrible advice depending on how complex your application is.

And by the way, before we continue, let me say I believe in the obvious and practically common-sense concept that things should only be as complicated as they have to be. However, in the real world, while things can remain simply beautiful in isolation, they are going to interact with complex systems, if you solving any sort of real problem. In many cases, “simple” things are going to fail miserably. The nature of complication in a program operation can, but certainly does not always reflect on the quality of the code itself. So, none of that drivel please, it is highly redundant.

FinalReleaseComObject will result in the release wrapped COM object right then and there. If you need to open an appointment to check something, you will want to use that method on that appointment once you have what you need, for example.

In the case of the two explorers we created before:

Marshal.FinalReleaseComObject(activeExplorer);

This will clean up this object for us then and there.

So always use this FinalReleaseComObject without abandon right?

Wrong.

Very wrong, you will fail in many situations if you do.

The Problem

For example, say we have an object that requires keeping an reference to an Explorer at various times during operation. Perhaps it monitors the events and changes having to do with the window properties of the Explorer. Let’s call it ExplorerMonitor:

Explorer activeExplorer = Application.ActiveExplorer();

ExplorerMonitor explorerMonitor = new ExplorerMonitor(activeExplorer);

Ok, cool. Cool. Now, let’s say we have another object that instead deals with watching folders. It responds to folder events and changes in the folder’s properties. It also requires a reference to an Explorer at various times, as it may need to know what the current folder is, or when a folder switch has occurred in the folder. Let’s call it FolderMonitor.

Explorer anotherActiveExplorer = Application.ActiveExplorer();

FolderMonitor folderMonitor = new FolderMonitor(activeExplorer);

I could actually just have used the “activeExplorer” instance I did in the first example. That won’t change the outcome of what we’re about to see; I’m creating a “separate” one to illustrate a point.

Now, if you are you intelligent, you would have designed both of these objects to deal with COM object release in their Dispose procedures. You shouldn’t restrict COM cleanup to just Dispose, by the way. You should do it whenever you no longer need the Outlook object.

Anyways, let’s assume that the folder monitor we created is in the process of being Disposed due to whatever reason (let’s assume the program isn’t shutting down, but it has been determined that the folder monitor is no longer needed, so we just don’t use it anymore, and .NET eventually garbage collects it).

So, let’s assume that the Dispose method for the folder monitor does a FinalReleaseComObject(_explorer) call. Ok, great, we cleaned up our COM reference in there as we should be doing.

Unfortunately, we’re going to get a nasty exception the next time the explorer monitor does anything with its own Explorer references: InvalidComObjectException. The exception will complain about the RCW being separated from the underlying object.

So what happened here? FinalReleaseComObject guarantees COM release, and thus it will release every single interface pointer to that COM object in the application domain. Recall how we first got the explorers:

Explorer activeExplorer = Application.ActiveExplorer();

Explorer secondActiveExplorer = Application.ActiveExplorer();

These could be in separate methods or classes for all we care. Even though it looks like we’re dealing with separate entities here, we are not. Both variables point to the same RCW. The RCW would have an internal reference count of 2 here.

So, this is sort of a big problem. Basically, we can say that running FinalReleaseComObject will kill access to that COM object globally in your application. This is further compounded by the problem that when you are dealing with a supposedly RCW-wrapped COM object, there is no easy way to tell prior to using it if it is actually mapped to a COM object or not. There is a way, but it requires reflection, so it is far too expensive, and would require messy code all over the place.

Sort of a big problem here.

The Solution

We have several avenues of corrective action to explore here. We could either just use ReleaseComObject, which would only release one of the references and not all, we could restrict our use of FinalReleaseComObject, or we could be clever.

The first avenue, the use of ReleaseComObject, is a very bad avenue. The reason for that is that RCW referencing counting rules are not equivalent to COM referencing counting rules. In fact, they are not documented, frankly in my opinion not documented AT ALL. There’s a couple sentences about how that all works on the MSDN, but no concrete, granular explanations.

So, you have no way of knowing how many times you should call ReleaseComObject. There may be additional references out there generated for reasons not obvious to you, if you didn’t release them, you might as well not release any. So don’t go there. Use FinalReleaseComObject only, unless you completely understand the reference counting rules of RCW.

The second avenue, restrict our use of FinalReleaseComObject, also does not work. We need to be calling that consistently. The alternative is to design some complex tracking system that goes along with the first avenue in figuring out how many references there are.

Any tracking system like that is going to fail miserably. There’s a much better way.

If you have an object that needs to hold on to a reference to an Outlook COM object, and that class was designed properly so that it calls FinalReleaseComObject in its Dispose method, you need to somehow make sure that by calling FinalReleaseComObject on a Outlook object, you aren’t stripping its use from anyone else who still may require its use.

In other words, you need to create a unique RCW for each separate reference to a COM object that you know will be responsibly released when the lifetime for that object dies (or if that object decides the reference isn’t needed anymore).

How do we do this? In order to do this, we will want to add some logic to either the constructor or setter of a class that takes an Outlook object, like so:

public ExplorerMonitor(Explorer explorer)

{

//The explorer parameter is an RCW wrapped COM object

//Let's get the explorer's IUnknown interface

//(I never use Hungarian notation, but I just love how "pUnk" looks!)

IntPtr pUnkExplorer = Marshal.GetIUnknownForObject(explorer);

//Using this managed pointer, we can create a unique RCW

_explorer = ((Explorer) Marshal.GetUniqueObjectForIUnknown(pUnkExplorer));

Marshal.Release(pUnkExplorer);
}

Bingo. Now you’re in business.

So, assuming we did the same for folder monitor, we can initialize like this:

Explorer activeExplorer = Application.ActiveExplorer();

ExplorerMonitor explorerMonitor = new ExplorerMonitor(activeExplorer);

FolderMonitor folderMonitor = new FolderMonitor(activeExplorer);

Marshal.FinalReleaseComObject(activeExplorer);

The release of the active explorer at the end there will not affect the usage of the explorer in explorer monitor and folder monitor, and it the release of the object in either monitor won’t affect the other.

One thing to note here before I go is that simply having a COM release in the Dispose method isn’t adequate at all if it can be done sooner. If you are ever in a place where you can conclude that “I don’t need this reference” then release it. The ExplorerMonitor and FolderMonitor classes would have a public method that would do that for you. Don’t wait for either of them to be disposed. You just want it to be handled in the Dispose method for extraordinary circumstances.

Matt Weber

I'm the the Senior Software Architect at Emergingsoft where I lead the software development team. I am also the owner of this website. I enjoy well-designed code, independent thought, and the application of rationality in general. You can reach me at matt@badecho.com.

  4 Responses to “Outlook COM Interop and Reference Counting: Or How I Learned to Stop Worrying and Love the RCW”

  1. [...] been swimming in a sea of COM Interop lately. Some time ago, I wrote an article that had some important tidbits regarding the nature of Runtime Callable Wrappers, or [...]

  2. [...] ** Great blog post on this subject can be found here. [...]

  3. Great post !!!

  4. Great post

 Leave a Reply

(required)

(required)

You may use these HTML tags and attributes: <a href="" title="" rel=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

 
   
© 2012-2013 Matt Weber. All Rights Reserved. Terms of Use.