7

I have shell extension dll written in C++ and COM The dll is registered and loaded into memory. My upgrade setup program will do the following:

  • Unregister the shell extension dll, kill the explorer.exe
  • Copy the later version of the shell extension dll (Step 2)
  • Start t he explorer.exe

It works fine. But the issue is:

If the user opened any other applications ( Internet explorer, sometimes windows task manager, notepad etc), the step Step 2 is failing.

Is there any way to close all shell extension dll hooks while upgrading the dll.

In the dll I am using GetOverlayInfo, context menu, database connection etc

Xearinox
  • 3,224
  • 2
  • 24
  • 38
JChan
  • 1,411
  • 4
  • 24
  • 34
  • 6
    It's a fallacy thinking that killing `explorer.exe` will help anything. First of all it is a very invasive thing to do (Windows Installer will instead *move* the file in use into another place on the same volume to install the new file in the original location) and secondly it may not work at all because the user could be using a completely different file manager that makes use of shell extensions. I do that myself and installers making those assumptions are just annoying. You could close the references remotely, causing all kinds of havoc. Don't do it! – 0xC0000022L Jul 06 '12 at 16:20
  • I'm with @0xC0000022L. I hate restarts just like everyone else, but I think this calls for one. You can complete your installation when the system is started again, before explorer is launched. – Eran Jul 06 '12 at 16:28
  • @eran: just requires a restart of the affected applications (or a logoff), actually. – 0xC0000022L Jul 06 '12 at 16:32
  • Your shell extension DLL gets loaded into other processes through the shell dialogs. Like File + Open in Notepad. You can rename the original DLL and put your new one into place but a logout is required to make it effective. – Hans Passant Jul 06 '12 at 16:37
  • 1
    @HansPassant: just wondering, do you know the definitive answer to this one: will the new shell extension be used by new application instances right away or not? I'm not 100% sure. – 0xC0000022L Jul 06 '12 at 16:41
  • Thanks to all for your valuable suggestions. Let me check which of the above solution is best fit with my problem – JChan Jul 06 '12 at 16:45

2 Answers2

9

Simply put: don't do it (i.e. do not unload it forcibly). You would have to enumerate all processes that have your shell extension loaded and then "restart" those. This is very invasive and quite frankly: bad behavior (for an installer). This also requires particular privileges which your installer may not otherwise need.

What most people still don't seem to be aware of is that MoveFile (and MoveFileEx) can be used to move DLLs or EXE files that are currently in use by a running application.

This is the approach Windows Installer takes. Have you ever noticed the folder \Config.msi in the root of a given drive after installing some .msi? This folder actually contains (moved and usually renamed to some unique "temporary" name) the original files that have been moved out but were still in use at the time. They are then usually scheduled for deletion upon boot (MoveFileEx with MOVEFILE_DELAY_UNTIL_REBOOT).

You can use the same mechanism in your homebrew installer and move the old file that is in use out of its original location and move the other one in right away. Any new application instance will then make use of the new shell extension (*), while the old one will continue to be used by any of the running applications that loaded it at one point or another.

(*) I'm not 100% sure about this because of the rules that apply to DLL loading and prevent modules with the same name to be loaded again under some circumstances.

0xC0000022L
  • 20,597
  • 9
  • 86
  • 152
  • We are using bitrock installer. I will try your suggestion. – JChan Jul 06 '12 at 16:51
  • If there is a DLL lock, Bitrock Installer copy the new file appeneded with some unique numbers. But user must reboot the system. During the reboot, it will delete the old dll and also remove the unique numbers from the dll. – JChan Jul 06 '12 at 22:12
  • @JChan: sounds like they are using the technique that Windows Installer is using and that I described. – 0xC0000022L Jul 09 '12 at 13:46
  • The solution is not foolproof, ie, during the upgrade process, we will unregsiter the old dll, copy the new dll and register it. If dll lock happened, it will never register the new dll. but it will replace the old dll with new one. This is an issue. The new dll need to be registered. I am wondering how I can resolve this issue – JChan Jul 09 '12 at 16:07
  • 1
    @JChan: if you mean self-registration you should stop doing that altogether. There is a reason MS recommends against using that for its own technology (Windows Installer). Instead set the registry values yourself. Besides, during an upgrade registration and deregistration will only rarely be needed because most of the values are likely shared between the two versions of the DLL. – 0xC0000022L Jul 09 '12 at 21:00
  • @oxC0000022L - thansks for your help – JChan Jul 10 '12 at 22:05
1

Depending on your installer, you may be able to make use of the Restart Manager support in Windows Vista+. This will allow your setup to query for every application that is using your DLL, and try and gracefully shut them down. If they can't be, then you will need to register it for replacement on a restart. Once the setup has finished, the Restart Manager will try and restart any programs it shut down (that support being restarted).

Deanna
  • 23,876
  • 7
  • 71
  • 156