0

My code consists of two parts:

  1. part: Scraping content from a webpage
  2. part: Evaluating the content

In the 1st part everything is running without any problems. This part takes about 5min.

But I get errors in the 2nd part. Every time I rerun the code to check whether I succeeded in solving the problem in the 2nd part I have to wait 5min. Is there a more efficient way of debugging?

Is there a way to store a variable from the 1st part of session 1 and reuse it in session 2? I want to skip the 1st part and go directly to the 2nd part.

Moinuddin Quadri
  • 46,825
  • 13
  • 96
  • 126
lioli
  • 191
  • 1
  • 13
  • 1
    Its common to decouple programs using intermediate storage. Part 1 could write its scrape results to a file or database and Part 2 could consume that data. How you save it depends on your data. Perhaps an easy way to to it is to pickle each scrape result to a file named by current time so that it can be processed in order later. – tdelaney Jan 15 '17 at 19:52
  • Depending on what you mean by "scrape" you could save entire web pages to files instead of pickling anything. – tdelaney Jan 15 '17 at 19:53

0 Answers0