0

I construct a list of dataframe by:

def compute_list_of_comprehensive_SP_series(instrument_now):
    series_now = normalized_price_history[instrument_now].copy()
    series_now.name = series_now.name + "_Settle"
    comprehensive_SP_series = compute_comprehensive_time_series_pd_based_on_time_series_now(series_now,MA_list_to_consider, resample_to_day = False)
    return comprehensive_SP_series

list_of_comprehensive_SP_series =  []
for j in to_trade_instruments:
    list_of_comprehensive_SP_series.append(compute_list_of_comprehensive_SP_series(j))

The memory usage has gone up from 

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                   
 3672 ********  20   0 1457m 453m  26m S  0.0  0.4   0:05.23 python

to 

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                   
 3672 ********  20   0 1789m 785m  26m S  0.0  0.6   0:29.41 python

The size of all data frame adding together is indeed 300+M, by doing:

total_size = 0
for j in list_of_comprehensive_SP_series:
    total_size+=sys.getsizeof(j)
print total_size/1e9

outputting:

0.382917456

However, after I delete the list of dataframe and collect the memory back:

del list_of_comprehensive_SP_series
import gc
gc.collect()

The memory usage does not drop at all. This is quite frustrating in my memory intensive application. Any help?

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                   
 3724 ********  20   0 1788m 784m  26m S  0.0  0.6   0:29.35 python
cs95
  • 379,657
  • 97
  • 704
  • 746
user40780
  • 1,828
  • 7
  • 29
  • 50
  • 1
    Are you using IPython or a jupyter notebook? Usually, the most recent computations are cached. – cs95 Nov 10 '17 at 21:02
  • yep, I am using jupiter notebook.... how do i release the memory? – user40780 Nov 10 '17 at 21:03
  • Hmm, I think this is a duplicate... Use `%reset`. – cs95 Nov 10 '17 at 21:04
  • `del list_of_comprehensive_SP_series` doesn't "delete" the *object* referenced by the name, it *deletes the name itself*. If there are references sticking around to that object, or in this case, to the objects contained in the list, they will not be garbage collected. – juanpa.arrivillaga Nov 10 '17 at 21:06

0 Answers0