Can I access python variables within a `%%bash` or `%%script` ipython notebook cell?

Question

Is there a way to access variables in the current python kernel from within a %%bash or other %%script cell?

Perhaps as command line arguments or environment variable(s)?

Answer 1

Python variables can be accessed in the first line of a %%bash or %%script cell, and so can be passed as command line parameters to the script. For example, with bash you can do this:

%%bash -s "$myPythonVar" "$myOtherVar"
echo "This bash script knows about $1 and $2"

The -s command line option allows you to pass positional parameters to bash, accessed through $n for the n-th positional parameter. Note that what's actually assigned to the bash positional variable is the result of str(myPythonVariable). If you're passing strings containing quote characters (or other bash-sensitive characters), you'll need to escape them with a backslash (eg: \").

The quotes are important - without them the python variables (string representations) are split on spaces, so if myPythonVar was a datetime.datetime with str(myPythonVar) as "2013-10-30 05:04:09.797507", the above bash script would receive 3 positional variables, the first two with values 2013-10-30 and 05:04:09.797507. It's output would be:

This bash script knows about 2013-10-30 and 05:04:09.797507

If you want to name the variables and you're running linux, here's an approach:

%%script env my_bash_variable="$myPythonVariable" bash
echo myPythonVariable\'s value is $my_bash_variable

You can specify multiple variable assignments. Again beware of quotes and other such things (here bash will complain bitterly!). To grok why this works, see the env man page.

Answer 2

To include python variables within bash commands run using the syntax !<some command> you can use {<variable>} as follows:

In [1]: for i in range(3):
   ...:     !echo {i+1}
   ...:     
1
2
3

While this is slightly different from what the OP asked, it is closely related and useful in performing a scripting task. This post has more great tips and examples of using shell command within IPython and Jupyter notebooks.

Answer 3

One problem is, if the variable that you want to give to your bash is a list, it does not work as expected.

For example, in one python cell:

l = ['A', 'B', 'C']

then if you give it directly to the magic option the next cell:

%%bash -s "$l"
for i in $1
do
echo $i
done

It will be oddly split like this:

['A',
'B',
'C']

The simplest answer is to put code inside braces {} to transform your python list in bash list, like the following:

%%bash -s "{" ".join(l)}"
for i in $1
do
echo $i
done

Which give the expected output:

A
B
C

Answer 4

If someone like me ends up here looking for how to use Python variables when running commands with !, just add prefix the variable with $ and that should do it:

!echo $foobar

Answer 5

Just to note a variation, if you need to pass something other than a simple variable to the bash script:

%%bash -s $dict['key1'] $dict['key2'] $dict['key3']

goes gruesomely wrong, but

%%bash -s {dict['key1']} {dict['key2']} {dict['key3']}

works nicely.

Answer 6

You can use Python string templates if you are willing to define a new magic with:

from IPython import get_ipython
from IPython.core.magic import register_cell_magic

ipython = get_ipython()


@register_cell_magic
def pybash(line, cell):
    ipython.run_cell_magic('bash', '', cell.format(**globals()))

And then if you define a variable in Python like:

test = 'Python variables'

you can use it:

%%pybash
echo '{test} will be expanded'
echo '{{double braces will be replaced with single braces}}'

Resulting in:

Python variables will be expanded
{double braces will be replaced with single braces}

Answer 7

No, %%script magic are autogenerated and don't do any magic inter-process data communication. (which is not the case for %%R but which is a separate magic in its own class with extra care from R peoples)

But writing your own magic that does it is not too hard to do.

Answer 8

Inspired by the answer from @krassowski, the following works to interpret arbitrary expressions in curly braces, with the semantics of f-strings. That is, you can use expressions such as {1+val} and not just {val}, given that the variable val is defined in the global python context (such as in a Jupyter notebook).

@register_cell_magic
def pybash(line, cell):
    cell_replaced = eval("f" + repr(cell))
    # print("Evaluating:\n{}\n-----------".format(cell_replaced))
    ipython.run_cell_magic('bash', '', cell_replaced)

As an example, define a global variable in a python cell:

val = 2

Then, in a new cell:

%%pybash
echo \\
echo {1+val}

This outputs

\
3

This is safe as long as the cell content is not controlled by an attacker.