0

I have a module I am working on that has multiple smaller packages and directories built into it. A part of the module includes a webscraper and I have a couple different text files to send information through multiple different processes. I have a package that scrapes the data, and dumps it into the pool, then it opens a .txt file and writes which pool it dumped the information into. Then a separate package opens up that file to determine which pool to clear out and put into a SQL database, storing the information. My module works completely fine when I run it inside the package, but when I try to import the module into other projects it raises a FileNotFoundError

FileNotFoundError: [Errno 2] No such file or directory: 'InRoute/limit.txt'

How can I make it so that the .txt files are recognized from outside the main package?

Tristen
  • 362
  • 1
  • 18
  • what is the directory structure of the project ? – Surajano Apr 10 '17 at 04:54
  • I have 1 main package, which is titled ScrapeDate, then inside the main one I have 6 mini-packages that serve different purposes. The .txt folder is in one of the modules titled InRoute, which is inside the main package ScrapeData, So, ScrapeDate --> __init__.py --> InRoute --> limit.txt would be the order is was run in. – Tristen Apr 10 '17 at 04:57
  • Are you sure you have `__init__.py` file in the directory containing your module? (I know you said 'package', still!) – 0xc0de Apr 10 '17 at 04:57
  • Yes, it is in the right place, and my first thoughts were that I needed to mess with something like if __name__ == '__main__' then continue with that, but the problem I am having is when I try to import it and run the package from outside the module, I am thinking it is trying to open the file from the directory I am running the script out of from where I import it, not from the directory the scripts are in. – Tristen Apr 10 '17 at 05:00
  • Is using the absolute path of the file not an option here? Or possibly os.path.join() if the destination of the path is dynamic? – JacobIRR Apr 10 '17 at 05:05
  • Why are you storing data in the same place as your code instead of some sanctioned location dedicated to data? – Ignacio Vazquez-Abrams Apr 10 '17 at 05:07
  • I want to make it OS independent because people might install it in any number of different folders, and so that makes things difficult. I'm using the files to prevent collisions since it will scrape the data and then clean it and then store it in a database, and so I have multiple pools that I can divert the scraper to dump into, and so one script is dumping info into a pool while another script is pulling info out from another pool. The pools are text files. But then another text file contains an instruction set that tells the "workers" which files to clear out/dump into. – Tristen Apr 10 '17 at 05:14
  • The reason that I don't just put it directly into the database is because the info needs to be cleaned first and in order to prevent the connection closing I dump everything into an initial pool, and then from there I pull it out, clean it, throw out useless stuff, and then direct it to the appropriate table in the database with another script, and then empty out the pool. – Tristen Apr 10 '17 at 05:17

0 Answers0