How do I make context menus for my software?

Question

So the idea is to make an encryption software which will work only on .txt files and apply some encryption functions on it and generate a new file. To avoid the hassle of user having to drag-and-drop the file, I have decided to make an option similar to my anti-virus here.

I want to learn how to make these for various OS, irrespective of the architecture :)

1. What are these menus called? I mean the proper name so next time I can refer to them in a more articulate way
2. How to make these?

My initial understanding:

What I think it will do is: pass the file as an argument to the main() method and then leave the rest of the processing to me :)

Answer 1

Probably not exactly the answer you were hoping for, but it seems that this is a rather complicated matter. Anyway, I'll share what I know about it and it will hopefully prove enough to (at least) get you started.

Unfortunately, the easiest way to create a context menu using Java is editing the Registry. I'll try to summarize the milestones of the overall requirements and steps to achieve our objective.

<UPDATE>
See at the end of the post for links to sample code and a working demo.
</UPDATE>

What needs to be done

1. We need to edit the Registry adding an additional entry (for our java-app) in the context menus of the file-types we are interested in (e.g. .txt, .doc, .docx).

   1. We need to determine which entries in Registry to edit, because our targeted file-extensions might be associated with another 'Class' (I couldn't test it on XP, but on Windows 7/8 this seems to be the case). E.g. instead of editing ...\Classes\.txt we might need to edit ...\Classes\txtfile, which the .txt Class is associated with.

   2. We need to specify the path to the installed jre (unless we can be sure that the directory containing javaw.exe is in the PATH variable).

   3. We need to insert the proper keys, values and data under the proper Registry nodes.

2. We need a java-app packaged as a .JAR file, with a main method expecting a String array containing one value that corresponds to the path of the file we need to process (well, that's the easy part - just stating the obvious).

All this is easier said than done (or is it the other way around ?), so let's see what it takes to get each one done.

First of all, there are some assumption we'll be making for the rest of this post (for the sake of simplicity/clarity/brevity and the like).

Assumptions

1. We assume that the target file-category is .TXT files - the same steps could be applied for every file-category.
2. If we want the changes (i.e. context-menus) to affect all users, we need to edit Registry keys under HKCR\ (e.g. HKCR\txtfile), which requires administrative priviledges. For the sake of simplicity, we assume that only current user's settings need to be changed, thus we will have to edit keys under HKCU\Software\Classes (e.g. HKCU\Software\Classes\txtfile), which does not require administrative priviledges. If one chooses to go for system-wide changes, the following modifications are necessary:
   1. In all REG ADD/DELETE commands, replace HKCU\Software\Classes\... with HKCR\... (do not replace it in REG QUERY commands).
   2. Have your application run with administrative priviledges. Two options here (that I am aware of):
      • Elevate your running instance's priviledges (can be more complicated with latest windows versions, due to UAC). There are plenty of resources online and here in SO; this one seems promising (but I haven't tested it myself).
      • Ask the user to explicitely run your app "As administrator" (using right-click -> "Run as administrator" etc).
3. We assume that only simple context-menu entries are needed (as opposed to a context-submenu with more entries). After some (rather shallow) research, I have come to believe that adding a submenu in older versions of Windows (XP, Vista), would require more complex stuff (ContextMenuHandlers etc). Adding a submenu in Windows 7 or newer is considerably more easy. I described the process in the relevant part of this answer (working demo provided ;)).

That said, let's move on to...

Getting things done

1. You can achieve editing the Registry by issuing commands of the form REG Operation [Parameter List], with operations involving ADD, DELETE, QUERY (more on that later). In order to execute the necessary commands, we can use a ProcessBuilder instance. E.g.

   String[] cmd = {"REG", "QUERY", "HKCR\\.txt", "/ve"};
new ProcessBuilder(cmd).start();
// Executes: REG QUERY HKCR\.txt /ve

   Of course, we will probably want to capture and further process the command's return value, which can be done via the respective Process' getInputStream() method. But that falls into scope "implementation details"...

   1. "Normally" we would have to edit the .txt file-class, unless it is associated with another file-class. We can test this, using the following command:

      // This checks the "Default" value of key 'HKCR\.txt'
REG QUERY HKCR\.txt /ve

      // Possible output:
(Default) REG_SZ txtfile

      All we need, is parse the above output and find out, if the default value is empty or contains a class name. In this example we can see the associated class is txtfile, so we need to edit node HKCU\Software\Classes\txtfile.

   2. Specifying the jre path (more precisely the path to javaw.exe) falls outside the scope of this answer, but there should be plenty of ways to do it (I don't know of one I would 100% trust though).
      I'll just list a few off the top of my head:

      • Looking for environment-variable 'JAVA_HOME' (System.getenv("java.home");).
      • Looking in the Registry for a value like HKLM\Software\JavaSoft\Java Runtime Environment\<CurrentVersion>\JavaHome.
      • Looking in predifined locations (e.g. C:\Program Files[ (x86)]\Java\).
      • Prompting the user to point it out in a JFileChooser (not very good for the non-experienced user).
      • Using a program like Launch4J to wrap your .JAR into a .EXE (which eliminates the need of determining the path to 'javaw.exe' yourself).

_{Latest versions of Java (1.7+ ?) put a copy of javaw.exe (and other utilities) on the path, so it might be worth checking that as well.}

3. So, after collecting all necessary data, comes the main part: Inserting the required values into Registry. After compliting this step, our HKCU\Software\Classes\txtfile-node should look like this:

HKCU
|_____Software
      |_____Classes
            |_____txtfile
                  |_____Shell
                        |_____MyCoolContextMenu: [Default] -> [Display name for my menu-entry]
                              |_____Command: [Default] -> [<MY_COMMAND>]*

*: in this context, a '%1' denotes the file that was right-clicked.

Based on how you addressed step (1.2), the command could look like this:
"C:\Path\To\javaw.exe" -jar "C:\Path\To\YourApp.jar" "%1"
_{Note that javaw.exe is usually in ...\jre\bin\ (but not always only there - recently I've been finding it in C:\Windows\System32\ as well).}

Still being in step (1.3), the commands we need to execute, in order to achieve the above structure, look as follows:

REG ADD HKCU\Software\Classes\txtfile\Shell\MyCoolContextMenu /ve /t REG_SZ /d "Click for pure coolness" /f
REG ADD HKCU\Software\Classes\txtfile\Shell\MyCoolContextMenu\Command /ve /t REG_SZ /d "\"C:\Path\To\javaw.exe\" -jar \"C:\Path\To\Demo.jar\" \"%%1\" /f"

        // Short explanation:
        REG ADD  <Path\To\Key>  /ve  /t REG_SZ  /d "<MY_COMMAND>"  /f
        \_____/  \___________/  \_/  \_______/  \_______________/  \_/
 __________|_______   |          |       |___           |           |
|Edit the Registry |  |   _______|________   |   _______|_______    |
|adding a key/value|  |  |Create a no-name|  |  |Set the data   |   |
--------------------  |  |(default) value |  |  |for this value.|   |
                      |  ------------------  |  |Here: a command|   |
       _______________|______________        |  |to be executed.|   |
      |Edit this key                 |       |  -----------------   |
      |(creates the key plus         |   ____|_________    _________|_____
      | any missing parent key-nodes)|  |of type REG_SZ|  |No confirmation|
      --------------------------------  |(string)      |  -----------------
                                        ----------------

Implementation Considerations:

• It is probably a good idea to check if our target class (e.g. txtfile), does already have a context-menu entry named "MyCoolContextMenu", or else we might be overriding an existing entry (which will not make our user very happy).
• Since the data part of the value (the part that comes after /d and before /f) needs to be enclosed in "", keep in mind that you can escape " inside the string as \". You also need to escape the %1 so that it is stored in the Registry value as-is (escape it like: %%1).
• It is a good idea to provide your user with an option to "un-register" your context-menu entry. The un-registering can be achieved by means of the command:
  REG DELETE HKCU\Software\Classes\txtfile\Shell\MyCoolContextMenu /f
• Omitting the /f at the end of the commands may prompt the "user" (in this case your app) for confirmation, in which case you need to use the Process' getOutputStream() method to output "Yes" in order for the operation to be completed. We can avoid that unnecessary interaction, using the force flag (/f).

Almost, there !

Finding ourselves at step (2), we should by now have the following:

1. A context-menu entry registered for our files in category txtfile (note that it is not restricted to .TXT files, but applies to all files pertained by the system as "txtfiles").
2. Upon clicking that entry, our java-app should be run and its main() method passed a String array containing the path to the right-clicked .TXT file.

From there, our app can take over and do its magic :)

That's (almost) all, folks !

Sorry, for the long post. I hope it turns out to be of use to someone.
I'll try to add some demo-code soon (no promises though ;)).

UPDATE

The demo is ready !

• I created a tiny demo-project.
• Here is the source code.
• Here is a ready-to-go JARred App.