How do I provide inline input to an IPython (notebook) shell command?

Question

I want to put together an IPython notebook with some shell commands and their input. In the bash prompt I can use "here-document" syntax:

bash-3.2$ mysql -u root <<END_IPUT
> use mydb;
> show tables;
> END_INPUT

How do I get the same effect in IPython, and specifically in a jupyter notebook? I know how to execute shell commands as IPython as "line magics" or "cell magics", e.g.:

In [7]:  !! ls -tF
Out[7]:  ['Demo-notebook.ipynb',
          'createdb.sql',
          ...

I've looked at IPython as a system shell, which shows how to enable some syntactic niceties. After the following, I can run system commands without prepending ! or !!

# Turn everything in $PATH into an alias; 
# then enable calling aliases without ! or %
%rehashx      
%autocall 2

But none of this helps with providing input to these commands inline: The here-document syntax is invalid in IPython, and results in a python SyntaxError. So how do I do it?

Have you tried the bash cell magic? Start a cell with the line `%%bash`, and put bash code in the rest of it. — Thomas K, May 17 '16 at 17:07
Good suggestion, thanks! I'd already found `%%sx`, which does the same I think. But the documentation for `%bash` led me to `%%script`, which is _exactly_ what I was looking for. In the meantime I also found an extension specifically for `sql`, which solved my immediate problem even better. But if you want to write an answer expanding on your suggestion, I'll accept it. — alexis, May 17 '16 at 21:28
@Thomas, before you delete my tag for a third time, look at how many people follow each one. If [tag:ipython-notebook] is such a problem, please take up the process to have the tags merged. — alexis, May 22 '16 at 23:17
I am trying to merge the tags, but I can't do that until the new tag has 80% as many uses as the old. I am periodically sweeping new questions over to the new tag to boost its numbers. Thanks. — Thomas K, May 23 '16 at 10:21
I see. Well, please stop deleting my tags. Having both should be good enough for you. (And just **add** [tag:jupyter-notebook] to other questions.) And making the tags themselves synonyms, of course. — alexis, May 23 '16 at 10:25
I made jupyter-notebook a synonym for ipython-notebook, but you'll have to do it the other way around. (I don't have enough privileges in the new tag.) — alexis, May 23 '16 at 10:33

score 8 · Accepted Answer · answered May 18 '16 at 08:37

8

The bash cell magic allows you to enter multiple lines of bash. Just start any cell with %%bash:

%%bash
echo "This is bash"
ls

If you're using a heredoc to pipe text into another program, you can also use the script cell magic:

%%script bc
2+3

There may also be other cell magics which address your problem more specifically!

answered May 18 '16 at 08:37

Thomas K

39,200
7
84
86

1

there is few small differences between `!` and `%%bash` when running a command. 1. `%%bash` doesn't display ASCII colors, while color appear fine using`!`. 2. `%%bash` collect the output and flush them on the screen once while `!` displays the output one by one as it executes. – Mahmoud Nov 19 '21 at 01:10

alexis · Answer 2 · 2018-02-26T22:41:54.200

With some more research, and with the help of @ThomasK's tip (see accepted answer), I found a couple of ways to do this.

One solution is the cell magic %sx. It executes the cell contents as a bash script, and captures and returns the output as a list of lines. Sometimes handy, sometimes not.
```
In[1]:  %%sx
        echo Hello, world
        cat -n <<DATA
        this
        and that
        DATA

Out[1]: ['Hello, world', '     1\tthis', '     2\tand that']
```
Change that to %%bash, as @ThomasK suggested, and the output is printed out, not returned. To capture it for further processing, ipython provides the --out parameter. But then I need to print it myself-- it won't show up by default.
```
In[1]:  %%bash --out var
        echo Hello, world
        echo "   Again"

In[2]:  var

Out[2]: 'Hello, world\n   Again\n'
```
%%bash is actually shorthand for %%script bash, billed as working "like the #! line in a script" (see the help with %%script?). Which it does. So here's how to dispense with the here-document syntax and have any program read the cell contents as input. %%script also accepts the --out parameter.
```
        %%script --out var mysql -u root -p XYZ
        USE somedb;
        SELECT * FROM users
             WHERE passwd IS NULL\G
```

This is nice enough, but I ended up not using it because my use case is mysql, and eventually I discovered that there's a great third-party ipython-sql extension that I can use instead:

%load_ext sql
%sql mysql+pymysql://user:passwd@localhost/somedb

This presupposes installing the extension, with pip install ipython-sql. For mysql, I also needed pip install pymysql. After the above setup, I just talk to the database like this

In[1]:  %%sql 
        SELECT * FROM users
            WHERE passwd IS NULL

Out[1]: 1 rows affected.
        ...

The returned table is actually drawn as a notebook table with borders, which is kinda nice. And it is made available as a Pandas dataframe (a smart list), which I can grab from _.

Could I do: `%%bash /usr/bin/env ipython` in order to run an exported `notebook.py` at the command line? — jtlz2, Aug 10 '17 at 08:15

How do I provide inline input to an IPython (notebook) shell command?

2 Answers2

Linked