How to run authentication on a mlFlow server?

Question

As I am logging my entire models and params into mlflow I thought it will be a good idea to have it protected under a user name and password.

I use the following code to run the mlflow server

mlflow server --host 0.0.0.0 --port 11111 works perfect,in mybrowser i type myip:11111 and i see everything (which eventually is the problem)

If I understood the documentation and the following https://groups.google.com/forum/#!topic/mlflow-users/E9QW4HdS8a8 link here correct, I should use nginx to create the authentication.

I installed nginx open sourcre and apache2-utils

created sudo htpasswd -c /etc/apache2/.htpasswd user1 user and passwords.

I edited my /etc/nginx/nginx.conf to the following:

server {
        listen 80;
        listen 443 ssl;

        server_name my_ip;
        root NOT_SURE_WHICH_PATH_TO_PUT_HERE, THE VENV?;
        location / {
            proxy_pass                      my_ip:11111/;
            auth_basic                      "Restricted Content";
            auth_basic_user_file /home/path to the password file/.htpasswd;
        }
    }

but no authentication appears.

if I change the conf to listen to listen 11111 I get an error that the port is already in use ( of course, by the mlflow server....)

my wish is to have a authentication window before anyone can enter by the mlflow with a browser.

would be happy to hear any suggestions.

For those reading this that don't want to set this up themselves just create a repo on https://dagshub.com and start logging to `https://dagshub.com//.mlflow` with a username and password. — Nomios, May 23 '21 at 07:50

score 9 · Accepted Answer · answered Dec 13 '19 at 16:37

the problem here is that both mlflow and nginx are trying to run on the same port...

first lets deal with nginx:

1.1 in /etc/nginx/sites-enable make a new file sudo nano mlflow and delete the exist default.

1.2 in mlflow file:

server {
    listen YOUR_PORT;
    server_name YOUR_IP_OR_DOMAIN;
    auth_basic           “Administrator’s Area”;
    auth_basic_user_file /etc/apache2/.htpasswd; #read the link below how to set username and pwd in nginx

    location / {
        proxy_pass http://localhost:8000;
        include /etc/nginx/proxy_params;
        proxy_redirect off;
    }
}

1.3. restart nginx sudo systemctl restart nginx

on your server run mlflow mlflow server --host localhost --port 8000

Now if you try access the YOUR_IP_OR_DOMAIN:YOUR_PORT within your browser an auth popup should appear, enter your host and pass and now you in mlflow

now there are 2 options to tell the mlflow server about it:

3.1 set username and pwd as environment variable export MLFLOW_TRACKING_USERNAME=user export MLFLOW_TRACKING_PASSWORD=pwd

3.2 edit in your /venv/lib/python3.6/site-packages/mlflowpackages/mlflow/tracking/_tracking_service/utils.py the function

def _get_rest_store(store_uri, **_):
    def get_default_host_creds():
        return rest_utils.MlflowHostCreds(
            host=store_uri,
            username=replace with nginx user
            password=replace with nginx pwd
            token=os.environ.get(_TRACKING_TOKEN_ENV_VAR),
            ignore_tls_verification=os.environ.get(_TRACKING_INSECURE_TLS_ENV_VAR) == 'true',
        )

in your .py file where you work with mlflow:

import mlflow
remote_server_uri = "YOUR_IP_OR_DOMAIN:YOUR_PORT" # set to your server URI
mlflow.set_tracking_uri(remote_server_uri)
mlflow.set_experiment("/my-experiment")
with mlflow.start_run():
    mlflow.log_param("a", 1)
    mlflow.log_metric("b", 2)

A link to nginx authentication doc https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-http-basic-authentication/

Thanks for the solutionPlease replace the curly double quotes from your code. That character is not recognizable in linux and is throwing errors. — pushd93, Jul 09 '20 at 16:58
Instead of changing the _get_rest_store, one can use os.environ['MLFLOW_TRACKING_USERNAME'] = 'userid' os.environ['MLFLOW_TRACKING_PASSWORD'] = 'password' before setting the tracking uri — ni_i_ru_sama, Jan 21 '21 at 16:57
i am planning to run similar setup on aws ecs. how secure would it be to run mlflow with only nginx's authentication like this on production? — Naveen Reddy Marthala, Jun 17 '22 at 14:23

score 4 · Answer 2 · answered Aug 06 '20 at 12:57

If you just want MLFlow installed with some basic authentication you can use mlflow-easyauth to get a Docker container with HTTP basic auth (username/password) setup integrated. It uses Nginx under the hood. Authentication details are configured using environment variables.

Disclaimer: I am the maintainer of that project

score 0 · Answer 3 · answered May 14 '21 at 12:00

In order for you to setup authentication for mlflow Tracking Server using nginx, you essentially need to do the following;

should be nginx/nginx plus (but nginx will serve this purpose)
you need two ports to be opened one for tracking server to run by default(11111 in your case) other one to run airflow with password protection(say 8080 and it could be any port which has to be opened by firewall)
create a auth file by using htpasswd utility under the /etc/nginx directory by using the command sudo htpasswd -c /etc/nginx/.htpasswd user_name and enter the password when it prompted.
Make sure you have changed the permission to 644 to this file, otherwise your proxy redirection will work, but you might hit the 500 error after you enter the username and password, this is because of auth file is not accessible by the service.

Now, you can go to sudo nano /etc/nginx/sites-enabled/default file comment everything inside the file and create a separate server block and put down the below configuration, you wonder why you need to edit this file alone? then i highly recommend to check this out this discussion Difference between sites-enabled and sites-available? After you made the change, your configuration file typically looks like this

server {
    listen 8080;

    location / {
        proxy_pass http://localhost:11111;

        auth_basic           "Administrator’s Area";
        auth_basic_user_file /etc/nginx/.htpasswd;

    }
}

Once you finished the above, you can check the diagnosis of the configuration then you need to restart the nginx server

sudo nginx -t
sudo service nginx restart

Now, you can check with your new port which is 8080 in your case, hopefully it should work.

You have to also set the environment variables for mlflow to use the updated credentials while you run your training jobs. In your code add the below lines,

import os

# Set username and password when authentication was added
os.environ['MLFLOW_TRACKING_USERNAME'] = <MLFLOW_TRACKING_USERNAME>
os.environ['MLFLOW_TRACKING_PASSWORD'] = <MLFLOW_TRACKING_PASSWORD>

Additional Tip:

You can also add ssl in the configuration, so that you can use https protocal instead of http, assume you have certificates. If you don't have you can create self signed one or use some of the free tools like certibot, etc..

Then your configuration would be similar like this, you have to add this certificates beneath port listening part;

listen 8080 ssl;

#server_name YOUR_IP_OR_DOMAIN;
ssl_certificate /etc/nginx/certificate/certificate.crt;
ssl_certificate_key /etc/nginx/certificate/certificate.key;

Sometimes, though you did everything as per the procedure, but authentication might not reflect. In such case, you need to change the owner of the auth file to 'www-data user' from root.

Hope this post will helps while setting up first time and in debugging.

Thank you.

score -1 · Answer 4 · answered Dec 03 '19 at 08:44

-1

I think you're forwarding to 11111 after authenticating at port 80. So, you can try my_ip:80 in browser

answered Dec 03 '19 at 08:44

ksh

1

How to run authentication on a mlFlow server?

4 Answers4