1

MetaFlow permits you to set the maximum number of concurrent tasks using the --max-workers CLI flag (ref: https://docs.metaflow.org/metaflow/scaling#safeguard-flags). However, I would like to avoid setting this every time.

Is it possible to set the --max-workers flag from the Python definition of the FlowSpec (without the CLI)?

crypdick
  • 16,152
  • 7
  • 51
  • 74

1 Answers1

2

The CLI flag sets the METAFLOW_RUN_MAX_WORKERS environment variable. You can set this within Python by setting this environment variable before you define your FlowSpec:

# force MetaFlow to use only 1 concurrent task during a foreach fanout
os.environ.update({"METAFLOW_RUN_MAX_WORKERS": "1"})

Source: chat with @Savin

crypdick
  • 16,152
  • 7
  • 51
  • 74