Long story short, after a crash course in Python/BeautifulSoup, I managed to create a script to take an input text file that contains a list of URLs (1 on each line), scrape the URL, and write output to a database. There are some cases where I want an error to exit the script (including some trapped errors as well as unexpected), but as the list of URLs to scrape is pretty large, it would be handy if I could edit the input text file (or create a copy and edit that) to remove each URL as it is successfully processed. The idea being that if the script exits (by trap or crash), I'd have a list of the URLs left to be processed. Is something like this possible? I can find code samples to edit the text file, but I'm getting stuck at how to take out the row just processed.
Asked
Active
Viewed 27 times
-2
-
Not sure I understand the point of downvoting when I'm trying to understand a hypothetical aspect of the python framework, but to clarify, I'm trying to understand if it would be possible to open a CSV, delete the top row of said CSV, then save. I have found docs and posts about deleting a given row using search, but haven't come across if you can do it by index position or the like. – Moebius Aug 09 '20 at 15:39
1 Answers
0
Finally came across the post here that achieves the answer, though I'm not positive it's the most efficient way as it's reading the entire file and saving each time, but that may be the best that can be done in Python. In my case, the file is in the 1200 lines range, so it easily fits into memory.
Moebius
- 23
- 1
- 3