How to set different keras backend in different conda environments

Question

How do I set different keras backends in different conda environments? Because in a specific environment, if I change the backend to tensorflow in keras.json, then in another python environment, keras backend will be tensorflow too. There is only one keras.json in my documents.

Answer 1

For using different keras backends in different environments in Anaconda - 'env1' and 'env2'

• Activate the first environment 'env1'
• Import keras from python with the default backend (if it fails to load tensorflow for example, install tensorflow in that environment) successfully
• In the ~ folder, a '.keras' folder will be created which will contain keras.json file
• For the other environment create a copy of the '.keras' folder as '.keras1'
• Change the keras.json file in that folder as per requirements (the 'backend' field)
• For using that config in 'env2' go to '~/anaconda3/envs/env2/lib/pythonx.x/site-packages/keras/backend' and edit the __init__.py file
• Make the changes marked with ##
• You will be able to import keras with different backends in env1 and env2

from __future__ import absolute_import
from __future__ import print_function
import os
import json
import sys
import importlib
from .common import epsilon
from .common import floatx
from .common import set_epsilon
from .common import set_floatx
from .common import cast_to_floatx
from .common import image_data_format
from .common import set_image_data_format

# Set Keras base dir path given KERAS_HOME env variable, if applicable.
# Otherwise either ~/.keras or /tmp.
if 'KERAS_HOME' in os.environ:
    _keras_dir = os.environ.get('KERAS_HOME')
else:
    _keras_base_dir = os.path.expanduser('~')
    if not os.access(_keras_base_dir, os.W_OK):
        _keras_base_dir = '/tmp'
    _keras_dir = os.path.join(_keras_base_dir, '.keras1')##

# Default backend: TensorFlow.
_BACKEND = 'tensorflow'

# Attempt to read Keras config file.
_config_path = os.path.expanduser(os.path.join(_keras_dir, 'keras.json'))
if os.path.exists(_config_path):
    try:
        with open(_config_path) as f:
            _config = json.load(f)
    except ValueError:
        _config = {}
    _floatx = _config.get('floatx', floatx())
    assert _floatx in {'float16', 'float32', 'float64'}
    _epsilon = _config.get('epsilon', epsilon())
    assert isinstance(_epsilon, float)
    _backend = _config.get('backend', _BACKEND)
    _image_data_format = _config.get('image_data_format',
                                     image_data_format())
    assert _image_data_format in {'channels_last', 'channels_first'}

    set_floatx(_floatx)
    set_epsilon(_epsilon)
    set_image_data_format(_image_data_format)
    _BACKEND = _backend

# Save config file, if possible.
if not os.path.exists(_keras_dir):
    try:
        os.makedirs(_keras_dir)
    except OSError:
        # Except permission denied and potential race conditions
        # in multi-threaded environments.
        pass

if not os.path.exists(_config_path):
    _config = {
        'floatx': floatx(),
        'epsilon': epsilon(),
        'backend': _BACKEND,
        'image_data_format': image_data_format()
    }
    try:
        with open(_config_path, 'w') as f:
            f.write(json.dumps(_config, indent=4))
    except IOError:
        # Except permission denied.
        pass

# Set backend based on KERAS_BACKEND flag, if applicable.
if 'KERAS_BACKEND' in os.environ:
    _backend = os.environ['KERAS_BACKEND']
    _BACKEND = _backend

# Import backend functions.
if _BACKEND == 'cntk':
    sys.stderr.write('Using CNTK backend\n')
    from .cntk_backend import *
elif _BACKEND == 'theano':
    sys.stderr.write('Using Theano backend.\n')
    from .theano_backend import *
elif _BACKEND == 'tensorflow':
    sys.stderr.write('Using TensorFlow backend.\n')
    from .tensorflow_backend import *
else:
    # Try and load external backend.
    try:
        backend_module = importlib.import_module(_BACKEND)
        entries = backend_module.__dict__
        # Check if valid backend.
        # Module is a valid backend if it has the required entries.
        required_entries = ['placeholder', 'variable', 'function']
        for e in required_entries:
            if e not in entries:
                raise ValueError('Invalid backend. Missing required entry : ' + e)
        namespace = globals()
        for k, v in entries.items():
            # Make sure we don't override any entries from common, such as epsilon.
            if k not in namespace:
                namespace[k] = v
        sys.stderr.write('Using ' + _BACKEND + ' backend.\n')
    except ImportError:
        raise ValueError('Unable to import backend : ' + str(_BACKEND))


def backend():
    """Publicly accessible method
    for determining the current backend.

    # Returns
        String, the name of the backend Keras is currently using.

    # Example
    ```python
        >>> keras.backend.backend()
        'tensorflow'
    ```
    """
    return _BACKEND

Answer 2

Here is what I did for my own purposes, same logic as Kedar's answer, but on a Windows install (and Keras version) for which locations and file names may differ :

1/ Set a specific keras.json file, in a folder of your targeted Anaconda environment. Modify the "backend" value.

2/ Then force the 'load_backend.py' (the one specific to your anaconda env.) to load this specific keras.json. Also, force de "default backend" to the one you want in that very same file.

=======================================================

IN DETAILS :

1.1 Open the Anaconda environment folder for which you want a specific backend. In my case it's C:\ProgramData\Anaconda3\envs\[MyAnacondaEnvironment]\

1.2 Here create a folder .keras, and in that folder copy or create a file keras.json (I copied mine from C:\Users\[MyWindowsUserProfile]\.keras\keras.json).

Now in that file, change the backend for the one you want, I've chosen 'cntk' for some tests. The file's content should now look like that :

{
    "floatx": "float32",
    "epsilon": 1e-07,
    "backend": "cntk",
    "image_data_format": "channels_last"
}

And the file's name and location looks like C:\ProgramData\Anaconda3\envs\[MyAnacondaEnvironment]\.keras\keras.json

2.1 Now open the file 'load_backend.py' specific to the environment you are customizing, located here (in my case) C:\ProgramData\Anaconda3\envs\[MyAnacondaEnvironment]\Lib\site-packages\keras\backend

2.2 Here line 17 to 25 in my Keras version (2.3.1), the file usually loads the backend from the configuration file it locates with the help of your environment variables or of your current Windows user for instance. That's why currently your backend is cross environment.

Get rid of this by forcing 'load_backend.py' to look which backend to load directly in your environment specific configuration file (the one you created at step 1.2)

For instance, line 26 of that 'load_backend.py' file (line 26 in my case, anyway right after the attempt to load the configuration file automatically) add that line (and customize it for your own location) :

_keras_dir = 'C:\ProgramData\Anaconda3\envs\[MyAnacondaEnvironment]\.keras' ##Force script to get configuration from a specific file

3.1 Then replace (line 28 in my case, anyway right after you forced _keras_dir path) the default backend _BACKEND = 'tensorflow' by _BACKEND = 'cntk'.

You should be done

Answer 3

One solution is to create different users for different environments and put different keras.json files for both:

$HOME/.keras/keras.json

This way you'll be able to change any keras parameter independently.

---

If you only need to change the backend, it is easier to use KERAS_BACKEND env variable. The following command will use tensorflow, not matter what's in keras.json:

$ KERAS_BACKEND=tensorflow python -c "from keras import backend"
Using TensorFlow backend.

So can you start a new shell terminal, run export KERAS_BACKEND=tensorflow in it and all subsequent commands will use tensorflow. You can go further and set this variable per conda env activation as discussed in this question (if you need it permanently):

$PREFIX/etc/conda/activate.d