Use line from text file as data for function

Question

I'm totally new at this so please bear with my lack of understanding.

What I want to ulitmatley do is get a line of text from a file, then place it into a url (already have that code and understand that), that will give a result. The value from the text file needs to be deleted after loaded.This is what I have so far, it is for a script to find instagram photos using the instagram API.

import time, random
import urllib,json,urllib2

def getuserId(userId):
userId_Dict = {}
try:
    list_of_users = open('superlikelist.txt', "r")
    for line in list_of_users:
        print line

    return userIdlist

I will later use the defined value for this

def userspics(userId):
numuserspics=0
urlUserMedia = "https://api.instagram.com/v1/users/%s/media/recent/?access_token=%s" % (getuserId,auth_token)
values = {
          'client_id' : client_id}    
try:
    print urlUserMedia
    data = urllib.urlencode(values)
    req = urllib2.Request(urlUserMedia,None,headers)
    response = urllib2.urlopen(req)
    result = response.read()                      
    dataObj = json.loads(result);
    picsForUser = random.randint(1, 3)
    print "Pics for user %s" % (picsForUser, userId)
    countPicViews=0
    for picture in dataObj['data']:
        countPicViews = countPicViews+1
        numLikesFollows = numLikesFollows+1
        if(countPicViews == picsToLike):
            break
except Exception, e:
    print e

score 1 · Accepted Answer · answered Dec 18 '13 at 01:55

The only way to remove a line from the start of a text file is to rewrite the whole file, skipping over that line. You can do that in Python, or by using an external tool like sed, but no matter what you do, that's what's going to happen.

As Eli says, it will be much better to first copy the items into some kind of format that allows random-access modification. For example, you can use a dbm database. (Of course a dbm acts like a dictionary, not like a set… but you can always simulate a set with a dictionary with meaningless values.)

First, use this script to convert the flat text file into a database:

from contextlib import closing
import dbm
db = dbm.open('superlikelist.db', 'n')
with open('superlikelist.txt') as f, closing(db):
    for line in f:
        dbm[line] = ''

Now, you can do this:

from contextlib import closing
import dbm

def process_everything():
    db = dbm.open('superlikelist.db', 'w')
    with closing(db):
        for url in db.keys():
            do_something_with(url)
            del db[url]

Another alternative, if you insist on using a text file, is to avoid rewriting the file over and over by, instead just keeping track of the last line number read in a separate place. Like this:

def process_file():
    try:
        with open('lastread.txt') as lr:
            lastread = int(lr.read())
    except:
        lastread = -1
    with open('superlikelist.txt') as f:
        for i, line in enumerate(file):
            if i > lastread:
                do_stuff_with(line)
                with open('lastread.txt', 'w') as lr:
                    lr.write(str(lastread))

If you must rewrite the original file for some reason, you can at least avoid rewriting it for each line, and instead just rewrite it once at the end of the process. Obviously you want to make sure this happens even if the end of the process is, say, the user hitting control-C when you're only half-way done, but you can deal with that with a try/finally or an atexit. (Of course that won't help if someone pulls the plug on the computer… but then if someone pulls the plug while you're rewriting the whole file, you'll have an even worse problem. And this change makes that problem less likely.) So:

import os

def process_file():
    with open('superlikelist.txt') as f:
        try:
            for line in f:
                do_stuff_with(line)
        finally:
            with open('superlikelist.new', 'w') as fout:
                fout.writelines(f)
            os.rename('superlikelist.new', 'superlikelist.txt')

This is a somewhat hacky implementation. First, you probably want to use, e.g., a tempfile.NamedTemporaryFile rather than hardcoding the name. Second, on Windows, you can't just move one file over another one if either of them are open; in fact, there's really no good way to "atomically" replace a file. The best you can do is to keep track of the temp file's name, then, after closing both files, as quickly as possible os.remove('superlikelist.txt') then do rename.

score 0 · Answer 2 · edited May 23 '17 at 11:43

0

I would not recommend doing things this way. This is not what file IO should be used for. Much better to read items from a Queue or a database and delete them as you process each one.
If you must do it this way, things get a bit messy and definitely not pythonic, but it'll work.

Do something like:

f = open('superlikelist.txt', 'rw') 
first_line = f.readline()
#remove one line from file.
os.system("sed -i -e '1d' " + f)

Similar discussion here.

edited May 23 '17 at 11:43

Community

1
1

answered Dec 18 '13 at 01:05

Eli

36,793
40
144
207

The `open` mode should be `'r'`, not `'rw'`. Also, you need to `close` the file (or, better, use a `with` statement). Also, you really ought to use `subprocess`, not `os.system`. And really, there's no reason to use `sed` when you can do it just as easily in Python, unless you deliberately want to make your code not work on Windows. – abarnert Dec 18 '13 at 01:33

Use line from text file as data for function

2 Answers2