How to open and search large txt files in Python/ flask

Question

So i am currently making a flask app that is a part of speech tagger, and part of the app uses a couple of txt files to check if a word is a noun or a verb, by seeing if that word is in the file. for example, here is my object I use for that:

class Word_Ref (object):
    #used for part of speech tagging, and word look up.

def __init__(self, selection):
    if selection == 'Verbs':
        wordfile = open('Verbs.txt', 'r')
        wordstring = wordfile.read()
        self.reference = wordstring.split()
        wordfile.close()
    elif selection == 'Nouns':
        wordfile = open('Nouns.txt', 'r')
        wordstring = wordfile.read()
        self.reference = wordstring.split()
        wordfile.close()
    elif selection == 'Adjectives':
        wordfile = open('Adjectives.txt', 'r')
        wordstring = wordfile.read()
        self.reference = wordstring.split()
        wordfile.close()
    elif selection == 'Adverbs':
        wordfile = open('Adverbs.txt', 'r')
        wordstring = wordfile.read()
        self.reference = wordstring.split()
        wordfile.close()
    elif selection == 'Pronouns':
        self.reference = ['i', 'me', 'my', 'mine', 'myself', 'you', 'your', 'yours', 'yourself', 'he', 'she', 'it', 'him', 'her'
                          'his', 'hers', 'its', 'himself', 'herself', 'itself', 'we', 'us', 'our', 'ours', 'ourselves',
                          'they', 'them', 'their', 'theirs', 'themselves', 'that', 'this']
    elif selection == 'Coord_Conjunc':
        self.reference = ['for', 'and', 'nor', 'but', 'or', 'yet', 'so']
    elif selection == 'Be_Verbs':
        self.reference = ['is', 'was', 'are', 'were', 'could', 'should', 'would', 'be', 'can', 'cant', 'cannot'
                          'does', 'do', 'did', 'am', 'been']
    elif selection == 'Subord_Conjunc':
        self.reference = ['as', 'after', 'although', 'if', 'how', 'till', 'unless', 'until', 'since', 'where', 'when'
                          'whenever', 'where', 'wherever', 'while', 'though', 'who', 'because', 'once', 'whereas'
                          'before']
    elif selection =='Prepositions':
        self.reference = ['on', 'at', 'in']
    else:
        raise ReferenceError('Must choose a valid reference library.')
def __contains__(self, other):
    if other[-1] == ',':
        return other[:-1] in self.reference
    else:
        return other in self.reference

And then here is my flask app py document:

from flask import Flask, render_template, request
from POS_tagger import *

app = Flask(__name__)


@app.route('/', methods=['GET', 'POST'])
def index(result=None):
    if request.args.get('mail', None):
        retrieved_text = request.args['mail']
        result = process_text(retrieved_text)
    return render_template('index.html', result=result)


def process_text(text):
    elem = Sentence(text)
    tag = tag_pronouns(elem)
    tag = tag_preposition(tag)
    tag = tag_be_verbs(tag)
    tag = tag_coord_conj(tag)
    tag = tag_subord_conj(tag)
    tagged = package_sentence(tag)
    new = str(tagged)
    return new


if __name__ == '__main__':
    app.run()

So, when ever the process_text function in the flask app uses any function that uses open() and then .read(), it causes an internal server, even if I use it with the Word_Ref object or not. Also, I also tested this with a txt file with 3 lines, and it still caused the same internal server error. All the other functions of my POS_tagger work within the flask app, and all of these functions, even the open() work in the interpreter.

Any alternate solution to the open() way of looking in txt files for this purpose?

EDIT: here are the tracebacks:

File "/Users/Josh/PycharmProjects/Informineer/POS_tagger.py", line 174, in tag_avna
    adverbs = Word_Ref('Adverbs')
File "/Users/Josh/PycharmProjects/Informineer/POS_tagger.py", line 91, in __init__
    wordfile = open('Adverbs.txt', 'r')
FileNotFoundError: [Errno 2] No such file or directory: 'Adverbs.txt'

The txt files are in the same directory though as the flask app

What do the flask logs say? If you run the app in debug mode you'll get python tracebacks: `app.run(debug=True)` These will help both you and us — Simon Fraser, Sep 22 '15 at 09:19
ok posted them, strange thing is the txt file is in the same directory as the flask app py file. — Josh Weinstein, Sep 22 '15 at 09:26
But that's not the [working directory](https://code.google.com/p/modwsgi/wiki/ApplicationIssues#Application_Working_Directory) of the python interpreter as it runs the file. This is a [find data file](https://stackoverflow.com/questions/779495/python-access-data-in-package-subdirectory) type problem. — Yann Vernier, Sep 22 '15 at 09:46

score 0 · Accepted Answer · edited May 23 '17 at 11:51

0

Maybe try something like this in your Flask app.py program :

import os

_dir = os.path.abspath(os.path.dirname(__file__))
adverb_file = os.path.join(_dir, 'Adverbs.txt')

You may need to modify depending on where you want _dir to point to but it will be a bit more dynamic.

Also consider using a Context Manager for File IO. It will condense the code a bit and also guarantees that the file is closed in case of Exceptions, etc.

For example:

with open(adverb_file, 'r') as wordfile:
    wordstring = wordfile.read()
    self.reference = wordstring.split()

edited May 23 '17 at 11:51

Community

1
1

answered Sep 22 '15 at 14:34

siegerts

461
4
11

Ok I added your approach, and the tagger works if i run it in the python console, but if i run it in the flask project i get this: File "/Users/Josh/PycharmProjects/Informineer/POS_tagger.py", line 101, in __init__ wordfile = open(adverb_file, 'r') FileNotFoundError: [Errno 2] No such file or directory: '/Applications/PyCharm.app/Contents/bin/Adverbs.txt' – Josh Weinstein Sep 22 '15 at 17:24
For some reason it's going to the pycharm bin rather than my project – Josh Weinstein Sep 22 '15 at 17:24
Actually in the end, when i run the Flask app through the commandline it all works, but not through PyCharm.... :/ Thanks for your help. – Josh Weinstein Sep 22 '15 at 17:35
@JoshWeinstein no prob! Glad that you got it working. – siegerts Sep 22 '15 at 18:50

How to open and search large txt files in Python/ flask

1 Answers1