my issue is nearly identical to the issue posted here:
That thread has been inactive for years and if there's a different protocol for "re-opening" the issue please advise - I'm posting this question in the mean time, and I apologize ahead of time if I should be doing this differently.
I can't post the code, but here are some details I can share - I'm executing a script that contains many iteratively generated print statements to track progress over the several hours the script takes to execute. While monitoring my CPU usage in Task Manager, I can see that periodically the usage drops to 0% and only resumes when I enter any kind of key stroke in the actual command prompt that the script is running in.
This has happened on my laptop and on the server that I've tried running the script on. The operating systems are Windows 8.1 and Windows Server 2012r2, I'm using Anaconda 2.2 with Python 3.4.3. The only non standard python libraries I'm using are pandas 0.15.2, numpy 1.9.2, statsmodels 0.6.1, and scikit-learn 0.16.1.
I'm not sure if I can nail down whether or not this always occurs at a specific line, but I'll try - potentially I can trace it to a particular package I'm using if I can do that? If anyone has any ideas what could cause something like this, please share, otherwise any advice on how to troubleshoot this problem on my own would greatly appreciated.
UPDATE: I ran the following code to try to reproduce the error:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import statsmodels.api as sm
from sklearn.linear_model import LogisticRegression
from datetime import datetime
num_rows = 1000
i = 1
t_init = datetime.now()
while True:
with open('temp_stage_1.txt','w') as file:
file.write('current stage 1 iteration number: %d' % i)
X = np.random.randint(2, size=(num_rows,25))
y = np.random.randint(2, size=num_rows)
with open('temp_stage_2.txt','w') as file:
file.write('current stage 2 iteration number: %d' % i)
clf = LogisticRegression()
clf.fit(X,y)
clf.score(X,y)
with open('temp_stage_3.txt','w') as file:
file.write('current stage 3 iteration number: %d' % i)
logit = sm.Logit(y,X)
results = logit.fit(disp=False)
with open('temp_stage_4.txt','w') as file:
file.write('current stage 4 iteration number: %d' % i)
for j in range(10000):
waste_time_str = 'wasting some time'
if i % 1000 == 0:
t_now = datetime.now()
t_delta = (t_now-t_init).seconds
t_init = t_now
print(t_delta)
print(i)
i += 1
I was able to reproduce the error and by opening the temporary files that were created I could see that the error occurred after the 4th temporary file was updated on the 26000th iteration. I second time running it, the error occurred on another multiple of 1000 according to the 4th temporary file. Another interesting observation is that after I hit a keystroke and the execution resumes, the printed out time delta reflects the time it spent sitting there waiting. This is also consistent with the original script that I saw this error with, however, in that instance it only ever printed what appeared to be normal time ranges, so I know that the error occurred after the time values were assigned. In both cases, it looks like the error is occurring at one of the print statements.
