1

My code is just making a database which stores links. But due to some reason when i open the database, it is an empty table. I have no idea why does this happen.

Not Causing any trouble

#!/usr/bin/python
import urllib2
from BeautifulSoup import *
import sqlite3

count = 0
ecount = 0
aj = list()
error = list()
run = 1

The sql connect

conn = sqlite3.connect('example.db')
cur = conn.cursor()
sql = '''CREATE TABLE URL (LINKS CHAR(512), ID INT(512))'''
cur.execute(sql)

Not Causing any trouble

def retriver(durl):
    newurl = list()
    url = durl
    notpar = {"#", "None", "NONE", "javascript:void(0)"}


    hl = urllib2.urlopen(url, None)
    html = hl.read()
    hl.close()
    soup = BeautifulSoup(html)
    tags = soup('a')

THIS IS THE SCRAPER

    for tag in tags:
        new = tag.get('href', None)
        newurl.append(new)
        for i in newurl:
            try:
                if i[0] == "/" and i[0:3] != "www":
                    if i[len(a)] == "/":
                        i = url[0:len(url)-1] + i
                    else:
                        i = url + i
                if i[0:3] == "www":
                    i = "/" + i
                if "." not in list(i):
                    continue
                if i[0] == "/":
                    i = i[1:len(i)]
                if i[0:4] != "http":
                    if i[len(i)] != "/":
                        i = "/" + i

THIS IS WHERE THE PROBLEM STARTS

                if i not in aj:
                    if i not in notpar:
                        aj.append(i)
                        print i
                        count += 1
                        sql = '''INSERT INTO URL(LINKS, ID) VALUES (%s, %d)'''
                        sqldata = (i, count)
                        cur.execute(sql, sqldata)
                        conn.commit()
            except:
                error.append(i)
                ecount += 1
                global ecount
    global count
    global aj

u = raw_input('Enter the url :: ')
aj.append(u)
retriver(u)

Not Causing any trouble

while True:

    try:
        retriver(aj[run])
        run += 1
        if run > 20:
            break
        print run
    except:
        run += 1
        retriver(aj[run])
        print run

2 Answers2

1

In this line:

sql = '''INSERT INTO URL(LINKS, ID) VALUES (%s, %d)'''

Try changing it to this:

sql = '''INSERT INTO URL(LINKS, ID) VALUES (?, ?)'''

This post here references that the database's API for parameter substitution uses a ? rather than the %s or %d symbol.

Community
  • 1
  • 1
qwertyuip9
  • 1,522
  • 2
  • 16
  • 24
0

I think you should not use query like sql = '''INSERT INTO URL(LINKS, ID) VALUES (%s, %d)'''

For inserting variables directly in your query use format strings like this :

sql = cur.execute("INSERT INTO URL(LINKS, ID) VALUES (?,?)",(x,y))
Reza Tanzifi
  • 572
  • 4
  • 13