use custom python module in script run from c#

Question

I am trying to run a python script from a c# console application. This c# code works fine until I import our custom python module that in turn imports other modules, e.g. arcpy, keyring, etc within the python script. The script runs fine from the cmd line but once in c# it just hangs. In the end, a system deadlock error pops up. How do I get my python to run? The goal is to get the password from python to c#. I am open to other approaches...

c#

        private static string getPassword(string database)
        {
            ProcessStartInfo start = new ProcessStartInfo();
            start.FileName = @"C:\Python27\ArcGIS10.6\python.exe";
            string script = @"C:\src\xxxx\etw_password.py";
            start.Arguments = $"\"{script}\" \"{database}\"";
            start.UseShellExecute = false;
            start.CreateNoWindow = false;
            start.RedirectStandardOutput = true;
            start.RedirectStandardError = true;
            string errors;
            string password;
            Console.WriteLine(start.Arguments);
            using(Process process = Process.Start(start))
            {
                errors = process.StandardError.ReadToEnd();
                password = process.StandardOutput.ReadToEnd();
            }


            return password;
        }

python: not working-- I replaced our custom python security calls with "x"

import sys
import xxxx

database = sys.argv[1]
password = xxxx.xxxx(database, 'xxxx')
print password

python: this works fine

import sys

database = sys.argv[1]
print database

I also run the test below to check if it was just our module or if the problem extended to others. I tested with numpy and arcpy. Both failed.

python: not working

import sys
import numpy

database = sys.argv[1]
print database

Answer 1

The issue lies with your script, not C#. Your C# code is well-formed and working (I tested it out by replacing your script with a hello world script that takes in one argument, and it printed the output correctly).

If you're saying it works on the command line, then it might be important to note what path you are running the script from. Your script might contain relative path dependencies. This would, in turn, affect the C# code because you may need to start in that directory first.

If you want my test, here it is:

script.py:

import sys

database = sys.argv[1]
print("Hello, " + database)

Program.cs:

        static void Main(string[] args)
        {
            var pwd = getPassword("World!");
            Console.WriteLine($"OUTPUT: {pwd}");
        }

        private static string getPassword(string database)
        {
            ProcessStartInfo start = new ProcessStartInfo();
            start.FileName = @"C:\Users\<path-to-my>\python.exe";
            string script = @"C:\<path-to-my>\script.py";
            start.Arguments = $"\"{script}\" \"{database}\"";
            start.UseShellExecute = false;
            start.CreateNoWindow = false;
            start.RedirectStandardOutput = true;
            start.RedirectStandardError = true;
            string errors;
            string password;
            Console.WriteLine(start.Arguments);
            using (Process process = Process.Start(start))
            {
                errors = process.StandardError.ReadToEnd();
                password = process.StandardOutput.ReadToEnd();
            }

            return password;
        }

Output:

Answer 2

I had the same problem, plus I was using a virtual python environment that I had created before installing my dependencies. So if you are not working with a virtual environment, you will only have to set the environment variable PYTHONPATH to the directory where your custom modules are installed. Btw. I am using Python 3.7.9. on Windows.

In my case I had created a virtual environment using

python -m venv .meinvenv

I figured that I would need to set the same environment variables in my ProcessStartInfo that would be set when I called the activate script for my virtual environment from the command line. So I set the environment variable for my virtual environment (using the variable start from your code snippet):

start.EnvironmentVariables["VIRTUAL_ENV"] = "C:\\Users\\mattgl\\.meinvenv";

And I added to the environment variables a modified path, which consisted of the Scripts folder of my virtual environment prepended to the original path:

start.EnvironmentVariables["PATH"] = "C:\\Users\\mattgl\\.meinvenv\\Scripts;" + start.EnvironmentVariables["PATH"];

After that, although my script was executed, I got a ModuleNotFoundError: No module named 'spacy' (because spacy happened to be the first special module to be imported in my script).

That I solved by specifying the environment variable PYTHONPATH in my ProcessStartInfo object like so:

start.EnvironmentVariables["PYTHONPATH"] = "C:\\Users\\mattgl\\.meinvenv\\Lib\\site-packages";

The path I specified was the path in my virtual Python environment where all modules had been installed before. Yours will be different.

After that, my python script could resolve all its dependencies and ran just fine.