5

Thanks in advance for any help. I am fairly new to python and even newer to html.

I have been trying the last few days to create a web page with buttons to perform tasks on a home server.

At the moment I have a python script that generates a page with buttons:

(See the simplified example below. removed code to clean up post)

Then a python script which runs said command and outputs to an iframe on the page:

(See the simplified example below. removed code to clean up post)

This does output the entire finished output after the command is finished. I have also tried adding the -u option to the python script to run it unbuffered. I have also tried using the Python subprocess as well. If it helps the types of commands I am running are apt-get update, and other Python scripts for moving files and fixing folder permissions.

And when run from normal Ubuntu server terminal it runs fine and outputs in real time and from my research it should be outputting as the command is run.

Can anyone tell me where I am going wrong? Should I be using a different language to perform this function?

EDIT Simplified example:

initial page:

#runcmd.html

<head>
    <title>Admin Tasks</title>
</head>

<center>
<iframe src="/scripts/python/test/createbutton.py" width="650" height="800" frameborder="0" ALLOWTRANSPARENCY="true"></iframe>
<iframe width="650" height="800" frameborder="0" ALLOWTRANSPARENCY="true" name="display"></iframe> 
</center>

script that creates button:

cmd_page = '<form action="/scripts/python/test/runcmd.py" method="post" target="display" >' + '<label for="run_update">run updates</label><br>' + '<input align="Left" type="submit" value="runupdate" name="update" title="run_update">' + "</form><br>" + "\n"

print ("Content-type: text/html")
print ''
print cmd_page

script that should run command:

# runcmd.py:

import os 
import pexpect
import cgi
import cgitb
import sys 

cgitb.enable()

fs = cgi.FieldStorage()

sc_command = fs.getvalue("update")

if sc_command == "runupdate":
    cmd = "/usr/bin/sudo apt-get update"

pd = pexpect.spawn(cmd, timeout=None, logfile=sys.stdout)

print ("Content-type: text/html")
print ''
print "<pre>"

line = pd.readline()  
while line:
    line = pd.readline()

I havent tested the above simplified example so unsure if its functional.

EDIT:

Simplified example should work now.

Edit:

Imrans code below if I open a browser to the ip:8000 it displays the output just like it was running in a terminal which is Exactly what I want. Except I am using Apache server for my website and an iframe to display the output. How do I do that with Apache?

edit:

I now have the output going to the iframe using Imrans example below but it still seems to buffer for example:

If I have it (the script through the web server using curl ip:8000) run apt-get update in terminal it runs fine but when outputting to the web page it seems to buffer a couple of lines => output => buffer => ouput till the command is done.

But running other python scripts the same way buffer then output everything at once even with the -u flag. While again in terminal running curl ip:800 outputs like normal.

Is that just how it is supposed to work?

EDIT 19-03-2014:

any bash / shell command I run using Imrans way seems to output to the iframe in near realtime. But if I run any kind of python script through it the output is buffered then sent to the iframe.

Do I possibly need to PIPE the output of the python script that is run by the script that runs the web server?

ButtzyB
  • 77
  • 2
  • 2
  • 9
  • yes. Here's [code example](http://stackoverflow.com/a/11729467/4279) – jfs Mar 15 '14 at 13:04
  • That looks confusing. To me it looks like that sends a command to the terminal and displays it there?. what I want is to display the output of the command in an iframe as it would appear in the terminal. @J.F.Sebastian – ButtzyB Mar 15 '14 at 14:08
  • your case is even simpler then in the code example where messages are sent back and forth between `javascript` code in the browser and a subprocess on the server. To answer your question: the stdout from the subprocess is sent to the browser. – jfs Mar 15 '14 at 14:50
  • Yes with the code I have after the command is completed the iframe is populated with the entire output from the command. what I would like is for it to be populated with each line like you would see as the command runs in a terminal. But I don't know how. @J.F.Sebastian – ButtzyB Mar 15 '14 at 14:54
  • [the code example that I provided](http://stackoverflow.com/a/11729467/4279) sends the output as soon as the subprocess flushes its internal stdout buffer (the example python program uses `-u` flag, so there is no buffer; each byte is sent immediately) – jfs Mar 15 '14 at 15:56
  • But doesn't that example create a new web-server? I am already running my site using Apache server so how would I do that with Apache instead of running a seperate server? As in go to page on my apache server site => click button (to run command) => output to my iframe. @J.F.Sebastian – ButtzyB Mar 15 '14 at 16:06
  • You could try to run wsgi application from [@Imran's answer](http://stackoverflow.com/a/22424925/4279) using mod_wsgi in Apache. Or try this [cgi script](http://jfsz.kd.io/python.py) – jfs Mar 16 '14 at 16:01
  • script is empty. @J.F.Sebastian – ButtzyB Mar 16 '14 at 16:52
  • [try this](https://gist.github.com/zed/46a210950c95f73a75a6) – jfs Mar 16 '14 at 17:01
  • Never mind was formatting error @J.F.Sebastian – ButtzyB Mar 17 '14 at 10:32
  • nope not outputting to the web browser at all @J.F.Sebastian – ButtzyB Mar 17 '14 at 10:38
  • What happens if you run `curl http://youhost.com/path/to/script.py`? – jfs Mar 17 '14 at 12:47
  • using curl from terminal works fine. @J.F.Sebastian – ButtzyB Mar 17 '14 at 17:11
  • check that the response is `HTTP/1.1` and not `HTTP/1.0` – jfs Mar 17 '14 at 17:17
  • yeah response seems to be right POST /scripts/python/runcmds.py HTTP/1.1" 200 489 is what shows up in the access log. @J.F.Sebastian – ButtzyB Mar 17 '14 at 17:40
  • POST is a *request*. Look at the *response*. Start a network sniffer and look for `HTTP/1.1 200 OK` (the first line in the response from the server) – jfs Mar 17 '14 at 17:46
  • I cleared out the log and tried again and there is no get after clicking a button just the post and the page waits for the script to finish but nothing is output to the iframe. @J.F.Sebastian – ButtzyB Mar 17 '14 at 17:54
  • try: `curl -v http://youhost.com/path/to/script.py |& grep 'HTTP/'` What do you see HTTP/1.0 or HTTP/1.1 200 OK? – jfs Apr 26 '14 at 03:30
  • I have managed to get it working since my last message. The only issue im haveing now is it waits for the python buffer to fill before it starts printing. Bash commands seem to output almost immediatly to the iframe but other python scripts run through it seem to buffer first then output. @J.F.Sebastian – ButtzyB Apr 27 '14 at 12:45
  • if you think you found the solution; you could [post it as your own answer](http://stackoverflow.com/help/self-answer). To fix buffering issue, have you tried `python -u`? Here are [several workarounds for the block-buffering issue for subprocess' stdout](http://stackoverflow.com/a/20509641/4279). – jfs Apr 27 '14 at 12:51

1 Answers1

7

You need to use HTTP chunked transfer encoding to stream unbuffered command line output. CherryPy's wsgiserver module has built-in support for chunked transfer encoding. WSGI applications can be either functions that return list of strings, or generators that produces strings. If you use a generator as WSGI application, CherryPy will use chunked transfer automatically.

Let's assume this is the program, of which the output will be streamed.

# slowprint.py

import sys
import time

for i in xrange(5):
    print i
    sys.stdout.flush()
    time.sleep(1)

This is our web server.

2014 Version (Older cherrpy Version)

# webserver.py

import subprocess
from cherrypy import wsgiserver


def application(environ, start_response):
    start_response('200 OK', [('Content-Type', 'text/plain')])
    proc = subprocess.Popen(['python', 'slowprint.py'], stdout=subprocess.PIPE)

    line = proc.stdout.readline()
    while line:
        yield line
        line = proc.stdout.readline()


server = wsgiserver.CherryPyWSGIServer(('0.0.0.0', 8000), application)
server.start()

2018 Version

#!/usr/bin/env python2
# webserver.py
import subprocess
import cherrypy

class Root(object):
    def index(self):
        def content():
            proc = subprocess.Popen(['python', 'slowprint.py'], stdout=subprocess.PIPE)
            line = proc.stdout.readline()
            while line:
                yield line
                line = proc.stdout.readline()
        return content()
    index.exposed = True
    index._cp_config = {'response.stream': True}

cherrypy.quickstart(Root())

Start the server with python webapp.py, then in another terminal make a request with curl, and watch output being printed line by line

curl 'http://localhost:8000'
Jonathan
  • 6,741
  • 7
  • 52
  • 69
Imran
  • 87,203
  • 23
  • 98
  • 131
  • Sorry as I said I am fairly new to this so bare with me. Im using apache as my web server on ubuntu server 13.10. With using your code or similar will that output to an iframe on the same page as the button? as in click button => runs subprocess on server => iframe gets populated with output as it happens? @Imran – ButtzyB Mar 15 '14 at 14:27
  • 1
    You don't have to run Apache, this script itself is a web server. If you have this running, you could set iframe['src'] = 'http://localhost:8000' with the command you want to execute as querystring, and parse the querystring out of 'environ' in application code. – Imran Mar 15 '14 at 14:37
  • Just to clarify I am running apache as I am am hosting a basic website on the server. Im trying to have a page with some buttons that run specific tasks (like apt-get update) and output to the iframe as it happens line by line. Instead of after the command has completed. Is there a way i could do this with apache? @Imran – ButtzyB Mar 15 '14 at 14:43
  • When I run your example in terminal I get the output as if I had typed the command into the terminal I would like that output to appear in the iframe like the command was running in the terminal. Sorry if I sound confusing. @Imran – ButtzyB Mar 15 '14 at 14:56
  • 1
    just tried your example and opened browser to the ip:8000 and it displays exactly how I want Excellent. now how do i do that with apache and the iframe? Thanks. @Imran – ButtzyB Mar 15 '14 at 15:03
  • Apache is unrelated in this case, iframe's allow you to embed an external source in a webpage, so you just drop an iframe into your html. (Be aware that some Cross-Site security may block it) – Jonathan May 30 '17 at 19:48
  • I'm trying to implement the same thing. I used the 2018 verison code form above but it is not printing the output line by line. I'm getting the output all at once (after 5 seconds). Has something changed? @Imran – Hemabh Aug 26 '20 at 21:07