Python soup error

Question

Hi guys this is my code:

#! /user/bin/env python
import os
import lxml
from bs4 import BeautifulSoup
string = os.system("curl -i https://it.wikipedia.org/wiki/Coldplay")
soup = BeautifulSoup(string, features="xml")
tag = soup.find_all("tbody")

and when i will execute it this is my error

  enter code hereTraceback (most recent call last):
  File "script_wiki.py", line 6, in <module>
  soup = BeautifulSoup(string, features="xml")
  File "build/bdist.macosx-10.12-intel/egg/bs4/__init__.py", line 192, in __init__
  TypeError: object of type 'int' has no len()

i'm beginner and i haven't a idea what is the problem sorry

`os.system()` doesn't return the output of the command, it returns the exit code, which is a number. — Barmar, May 17 '17 at 23:12
`os.system` returns the exit code of the command you ran, not a string of whatever that command printed. — user2357112, May 17 '17 at 23:13

Taku · Accepted Answer · 2017-05-17T23:45:00.450

os.system will return the exit code of the command. Not the return data. If the command is ran successfully, string will be 0.

You should instead use subprocess.call:

subprocess.call([
    'curl',
    '-i',
    "https://it.wikipedia.org/wiki/Coldplay"
    ], shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

But you shouldn't use curl command when using python, use urllib instead:

import urllib.request 
url = "https://it.wikipedia.org/wiki/Coldplay"
soup = BeautifulSoup(urllib.request.urlopen(url), features="xml")
tag = soup.find_all("tbody")

if you're using python 2, do import urllib2 and use urllib2.urlopen instead.

A more popular way over using urllib is to use the requests library, but you will need to install it, pip install requests.

import requests
r = requests.get('https://it.wikipedia.org/wiki/Coldplay')
soup = BeautifulSoup(r.content, features="xml")
tag = soup.find_all("tbody")

Thanks you for say me step by step what i do. – DMine May 17 '17 at 23:33 — DMine, May 17 '17 at 23:33

score 1 · Answer 2 · answered May 17 '17 at 23:20

You were correctly told in the comments that system does not return the program output. But there is no need to use curl at all. Python module urllib.request works just fine:

from urllib.request import urlopen
URL = "https://it.wikipedia.o‌rg/wiki/Coldplay"
soup = BeautifulSoup(urlopen(URL), f‌eatures="xml")

score 1 · Answer 3 · edited May 23 '17 at 12:34

To use curl, you need to import and run subprocess.check_output or subprocess.run (Python 3.5+ only) to capture the output of a system command. As stated already, os.system returns only the error status of the command you run, not its output, which goes to console.

See this discussion:

Running shell command from Python and capturing the output

There may be better ways to achieve your goal, but to understand how to make your existing code work, try subprocess.

Python soup error

3 Answers3