{"id":2613,"date":"2023-02-07T22:36:42","date_gmt":"2023-02-08T03:36:42","guid":{"rendered":"https:\/\/badecho.com\/?p=2613"},"modified":"2023-02-28T12:08:55","modified_gmt":"2023-02-28T17:08:55","slug":"safehandles-in-structs","status":"publish","type":"post","link":"https:\/\/badecho.com\/index.php\/2023\/02\/07\/safehandles-in-structs\/","title":{"rendered":"Using SafeHandles in Structs via Stateful Marshalling"},"content":{"rendered":"\n
\"Using<\/figure><\/div>\n\n\n\n

SafeHandles<\/a> provide reliable and secure means of managing the lifetime of unmanaged resources. It’s commonly recommended to use SafeHandle<\/code> instead of IntPtr<\/code> whenever we need to acquire and deal with unmanaged system handles.<\/p>\n\n\n\n

The .NET runtime makes this recommendation easy to follow, as the SafeHandle<\/code> class is integrated with platform invoke, allowing for the correct type of SafeHandle<\/code> to be used interchangeably with IntPtr<\/code> as parameters in P\/Invoke method signatures.<\/p>\n\n\n\n

Unmanaged handles aren’t only used as method parameters, however, as they are also present as members of the many kinds of C-style structs that unmanaged functions accept as parameters. Unfortunately, with structs, the use of SafeHandle<\/code> is not<\/em> interchangeable with IntPtr<\/code>.<\/p>\n\n\n\n

This means that we’re still stuck using dangerous IntPtr<\/code> handles when dealing with structs. However, with .NET 7’s introduction of source generation for platform invokes<\/a>, we now have a way to safely make use of SafeHandle<\/code> types in our structs. This article will demonstrate how.<\/p>\n\n\n\n

A Sample Handle<\/h2>\n\n\n\n

We’re going to take a look at an example SafeHandle<\/code> and its current limitations, and then look at how we can overcome them as far structs are concerned.<\/p>\n\n\n\n

One kind of unmanaged resource we encounter frequently is a handle to a window. It is created when a window is created, and it is used to interact with said window throughout its lifetime.<\/p>\n\n\n\n

WindowHandle.cs<\/a><\/h6>\n\n\n
\n\/\/\/ <summary>\n\/\/\/ Provides a level-0 type for window handles.\n\/\/\/ <\/summary>\n\/\/\/ <suppressions>\n\/\/\/ ReSharper disable UnusedMember.Local\n\/\/\/ <\/suppressions>\npublic sealed class WindowHandle : SafeHandle\n{\n    \/\/\/ <summary>\n    \/\/\/ Initializes a new instance of the <see cref="WindowHandle"\/> class.\n    \/\/\/ <\/summary>\n    \/\/\/ <param name="handle">The handle to the window.<\/param>\n    \/\/\/ <param name="ownsHandle">\n    \/\/\/ Value indicating if this safe handle is responsible for releasing the provided handle.\n    \/\/\/ <\/param>\n    public WindowHandle(IntPtr handle, bool ownsHandle)\n        : this(ownsHandle)\n    {\n        SetHandle(handle);\n    }\n\n    \/\/\/ <summary>\n    \/\/\/ Initializes a new instance of the <see cref="WindowHandle"\/> class.\n    \/\/\/ <\/summary>\n    public WindowHandle()\n        : this(true)\n    { }\n\n    \/\/\/ <summary>\n    \/\/\/ Initializes a new instance of the <see cref="WindowHandle"\/> class.\n    \/\/\/ <\/summary>\n    \/\/\/ <param name="ownsHandle">\n    \/\/\/ Value indicating if this safe handle is responsible for releasing the provided handle.\n    \/\/\/ <\/param>\n    private WindowHandle(bool ownsHandle)\n        : base(IntPtr.Zero, ownsHandle)\n    { }\n\n    \/\/\/ <summary>\n    \/\/\/ Gets a default invalid instance of the <see cref="WindowHandle"\/> class.\n    \/\/\/ <\/summary>\n    public static WindowHandle InvalidHandle\n        => new(false);\n\n    \/\/\/ <inheritdoc\/>\n    public override bool IsInvalid\n        => handle == IntPtr.Zero;\n    \n    \/\/\/ <inheritdoc\/>\n    protected override bool ReleaseHandle()\n        => User32.DestroyWindow(handle);\n}\n<\/pre>\n\n\n
The WindowHandle<\/code><\/a> class provides SafeHandle<\/code>-like functoinality for unmanaged window handles. <\/p>\n\n\n\n
It’ll be initialized via the default constructor by P\/Invoke whenever it’s used as a return value for a function that normally returns a window handle.<\/p>\n\n\n\n
P\/Invoke Function Returning WindowHandle<\/h6>\n\n\n
\n[LibraryImport(LIBRARY_NAME, \n               EntryPoint = "CreateWindowExW", \n               StringMarshalling = StringMarshalling.Utf16, \n               SetLastError = true)]\npublic static unsafe partial WindowHandle CreateWindowEx(int dwExStyle,\n                                                         string lpClassName,\n                                                         string lpWindowName,\n                                                         int style,\n                                                         int x,\n                                                         int y,\n                                                         int width,\n                                                         int height,\n                                                         IntPtr hWndParent,\n                                                         IntPtr hMenu,\n                                                         IntPtr hInstance,\n                                                         void* lpParam);\n<\/pre>\n\n\n
Normally, the signature for  CreateWindowEx<\/a><\/code> would be written to return an IntPtr<\/code> handle to the window. But, as you can see, we can easily swap that with our custom SafeHandle<\/code> and it all works like magic.<\/p>\n\n\n\n
Our custom SafeHandle<\/code> can also be used as a parameter for methods that expect a window handle and are normally written to accept an IntPtr<\/code>.<\/p>\n\n\n\n
P\/Invoke Function Accepting WindowHandle as Parameter<\/h6>\n\n\n
\n[LibraryImport(LIBRARY_NAME, SetLastError = true)]\n[return: MarshalAs(UnmanagedType.Bool)]\npublic static partial bool GetWindowRect(WindowHandle hWnd, out RECT lpRect);\n<\/pre>\n\n\n
And everything works just perfectly, with the added security and reliability that using a SafeHandle<\/code> grants us.<\/p>\n\n\n\n
Unfortunately, that’s where the magic just about stops.<\/p>\n\n\n\n
Structs and SafeHandles Don’t Mix Well<\/h2>\n\n\n\n
If we continue to expand our repertoire of window-related P\/Invoke functions, we’ll eventually run into some methods expecting window handles to be provided as members of a C-style struct.<\/p>\n\n\n\n
An example is the Shell_NotifyIcon<\/code><\/a> function, used to register an icon with the taskbar’s status area. This function accepts a NOTIFYICONDATA<\/a><\/code> struct as one of its parameters, which has a member that gets set to the handle of the window meant to receive taskbar notifications.<\/p>\n\n\n\n
If we try to define this structure so that it uses a WindowHandle<\/code>, we will run into problems.<\/p>\n\n\n\n
A Problematic NOTIFYICONDATA<\/h6>\n\n\n
\n\/\/\/ <summary>\n\/\/\/ Represents information that the system needs to display notifications in the notification area.\n\/\/\/ <\/summary>\n[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]\ninternal unsafe struct NOTIFYICONDATAW\n{\n    \/\/\/ <summary>\n    \/\/\/ A handle to the window that receives notifications associated with an icon i\n    \/\/\/ n the notification area.\n    \/\/\/ <\/summary>\n    public WindowHandle hWnd;\n\n    \/\/ Insert the rest of the members..\n}\n<\/pre>\n\n\n
The presence of a WindowHandle<\/code> in this struct will immediately result in compile-time errors similar to the following if we try to use the struct as a parameter to Shell_NotifyIcon<\/code>:<\/p>\n\n\n\n
SYSLIB10: The type 'BadEcho.Interop.NOTIFYICONDATAW' is not supported by source-generated P\/Invokes.\nThe generated source will not handle marshalling of parameter 'lpData'.<\/code><\/pre>\n\n\n\n
This is an error you’d get when using the new source generator system \u00e0 la the LibraryImportAttribute<\/code>. If you’re using the older DllImportAttribute<\/code>, then your program may compile, however it isn’t going to work at runtime.<\/p>\n\n\n\n
So what can we do? Well, let’s start with the more obvious, less-than-perfect, workaround.<\/p>\n\n\n\n
The Bad Way: Getting a Handle Out of a SafeHandle<\/h2>\n\n\n\n
We can revert our struct definition to use IntPtr<\/code> instead of our WindowHandle<\/code>. This will resolve the compile-time errors.<\/p>\n\n\n\n
All we need to do then is squeeze the SafeHandle<\/code>‘s internal handle out from it, and use that when populating the struct’s members.<\/p>\n\n\n\n
A Dangerous Squeeze<\/h6>\n\n\n
\nvar iconData = new NOTIFYICONDATAW\n               {\n                   cbSize = (uint) sizeof(NOTIFYICONDATAW),\n                   uCallbackMessage = (uint) _TrayEvent,\n                   uFlags = NotifyIconFlags.Message | NotifyIconFlags.Guid,\n                   guidItem = _id,\n                   hWnd = windowHandle.DangerousGetHandle()\n                   \/\/ Everything else...\n               };\n\nShell32.Shell_NotifyIcon(NotifyIconMessage.Add, ref iconData);\n<\/pre>\n\n\n
Getting the internal handle out from the SafeHandle<\/code> requires the calling of DangerousGetHandle<\/a><\/code>, a method with an appropriately ominous name.<\/p>\n\n\n\n
There are numerous articles out there which go into why using this method is, for lack of a better word, “dangerous”. It opens ourselves up to runtime errors due to stale handles, as well as potential security issues via handle recycling security attacks.<\/p>\n\n\n\n
To use this method safely, we need to use the DangerousAddRef<\/a><\/code> and DangerousRelease<\/a><\/code> methods — and the best place to use these is right in the middle of the whole marshalling procedure, somewhere that’s been out of reach to us until .NET 7’s release.<\/p>\n\n\n\n
This leads us to the better way to “handle” our SafeHandle<\/code>-in-structs conundrum: by implementing a custom stateful marshaller for this struct.<\/p>\n\n\n\n
The Good Way: A Stateful Marshaller<\/h2>\n\n\n\n
We’re going to leverage some of .NET 7’s new P\/Invoke source generation technology so that we can keep our WindowHandle<\/code>, and a few other SafeHandle<\/code> instances that we require, where we want them in the call to Shell_NotifyIcon<\/code>.<\/p>\n\n\n\n
A custom marshaller<\/a> allows for fine-grained control over how a type is marshalled (transformed when crossing between managed and native code). <\/p>\n\n\n\n
Most (this is a bit of an assumption) marshallers are ‘stateless’, meaning they are static classes that maintain no state regarding the object being marshalled. You can see an example of one here<\/a>.<\/p>\n\n\n\n
Custom Marshaller Components<\/h3>\n\n\n\n
We’ll be making a marshaller that keeps track of the object and its constituent parts during the marshalling process. Maintaining state is required so that we can correctly update the reference counters on our SafeHandle<\/code> instances. <\/p>\n\n\n\n
Custom marshalling requires the following components:<\/p>\n\n\n\n