Sacred - pass all parameters as one

Question

When using Sacred it is necessary to pass all variables from the experiment config, into the main function, for example

ex = Experiment('iris_rbf_svm')

@ex.config
def cfg():
  C = 1.0
  gamma = 0.7

@ex.automain
def run(C, gamma):
  iris = datasets.load_iris()
  per = permutation(iris.target.size)
  iris.data = iris.data[per]
  iris.target = iris.target[per]
  clf = svm.SVC(C, 'rbf', gamma=gamma)
  clf.fit(iris.data[:90],
          iris.target[:90])
  return clf.score(iris.data[90:],
                   iris.target[90:])

As you can see, in this experiment there are 2 variables, C and gamma, and they are passed into the main function.

In real scenarios, there are dozens of experiment variables, and the passing all of them into the main function gets really cluttered. Is there a way to pass them all as a dictionary? Or maybe as an object with attributes?

A good solution will result in something like follows:

@ex.automain
def run(config):
    config.C      # Option 1
    config['C']   # Option 2

Not sure about sacred, but in python you can! this is what is often refered to **kwargs you can check here for some example: http://book.pythontips.com/en/latest/args_and_kwargs.html — RomainL., Nov 22 '18 at 12:52
Yep, I'm aware of that. The problem is that I am not the one passing the variables from the `cfg` function into the `run` function so I'm not responsible for sending them as positional / key-word arguments — bluesummers, Nov 22 '18 at 12:55

score 4 · Accepted Answer · answered Mar 04 '19 at 11:44

4

Yes, you can use the special value _config value for that:

ex = Experiment('iris_rbf_svm')

@ex.config
def cfg():
  C = 1.0
  gamma = 0.7

@ex.automain
def run(_config):
  C = _config['C']
  gamma = _config['gamma']

answered Mar 04 '19 at 11:44

Qlaus

885
5
15

1

Best solution imo, unfortunately the docs state that the special values might change in the future. If so, I hope that sacred implements a fixed pattern for accessing all config parameters at once. – saper0 Jan 30 '20 at 17:08

score 1 · Answer 2 · answered Dec 04 '18 at 17:17

Most elegant solution I could find

ex = Experiment('iris_rbf_svm')

@ex.config
def cfg():
    C = 1.0
    gamma = 0.7

    params = {'C': C, 'gamma': gamma}

@ex.automain
def run(params):
    iris = datasets.load_iris()
    per = permutation(iris.target.size)
    iris.data = iris.data[per]
    iris.target = iris.target[per]
    clf = svm.SVC(params['C'], 'rbf', gamma=params['gamma'])
    clf.fit(iris.data[:90],
            iris.target[:90])
    return clf.score(iris.data[90:],
                     iris.target[90:])

But you still need to put the variables one by one into the `params` dictionary in this solution. — Raven Cheuk, Jun 16 '21 at 07:46

score 0 · Answer 3 · answered Dec 23 '19 at 19:26

There is a related example in the git repository: https://github.com/IDSIA/sacred/blob/master/examples/08_less_magic.py

You can specify a dictionary directly (without creating a decorated function)

from sacred import Experiment
from sacred.observers import FileStorageObserver
from sklearn import svm, datasets, model_selection

ex = Experiment("svm")

ex.add_config(
    {  # Configuration is explicitly defined as dictionary.
        "C": 1.0,
        "gamma": 0.7,
        "kernel": "rbf",
        "seed": 42,
    }
)

and use it as follows:

def get_model(C, gamma, kernel):
    return svm.SVC(C=C, kernel=kernel, gamma=gamma)


@ex.main  # Using main, command-line arguments will not be interpreted in any special way.
def run(_config):
    X, y = datasets.load_breast_cancer(return_X_y=True)
    X_train, X_test, y_train, y_test = model_selection.train_test_split(
        X, y, test_size=0.2
    )
    clf = get_model(
        _config["C"], _config["gamma"], _config["kernel"]
    )  # Parameters are passed explicitly.
    clf.fit(X_train, y_train)
    return clf.score(X_test, y_test)


if __name__ == "__main__":
    ex.run()

Sacred - pass all parameters as one

3 Answers3