MemoryError, Killed

Question

I am trying to create two matrices key_mat and c_mat c_mat initially is (1,847) fixd_len = 847

while the number is not 0, Im trying to pile up key_mat and c_mat

so this could become some(n,847) Col :1 --> n

Each element in Cols of key_mat and c_mat is either 1 or 0 int type;

its not dot product.. specifically i need bit-wise so I am doing element-wise operation;

number = int(''.join(map(str,key_mat.tolist()[0])),2)
        #Build matrix of Keys
        #print key_mat
        #print c_mat
        while number >= 0:
            #Divide integer>> by 2 until it covers all rows
            number >>= 1
            key_mat = np.concatenate((key_mat, np.matrix(list((np.binary_repr(number)).zfill(fixd_len)), dtype=int)))
            c_mat = np.concatenate((c_mat, c_mat))

Does, this increase memory alot?

I suppose c_mat and key_mat are in shared RAM memory and key_mat and c_mat assignment happens over there and not cache.

im running it in a VPS and on my machine which is core i7, 8Gb Win 8;

PS: Consider 847 bit places..

938439603600587528746394711938657107663969949193687942084737423845328945327403963493426274822541422606069252398088182827397836333287780407720182613329988145004965865323862822167078543736143176539997470989737828269291292380585577139908076735904949708259328L

possible bits and that number being the highest possible decimal number;

you start dividing >> by 2, and you re assign in the same variable.

Notice this quick n dirty code, there is no strict type assigned and var are reused;

large matrix var is re assigned dynamically.

What is occupying memory??

I am not sure.

UPDATE: I found this interesting. still reading it.

UPDATE2: The above update gave good insight (esp answers and comments)

and if someone could tell me how to approach this problem (847 is still not very big). how do you tackle memory tight situation here? Would be grateful..

I'm not sure what your code is doing (string join and concatenate are not happily mixed with numpy), but I believe that in Win8 you are still limited to 4GB per process. — mdurant, Aug 19 '14 at 16:43
@mdurant: 64bit windows is limited to 8TB (http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778%28v=vs.85%29.aspx) — Gerrat, Aug 19 '14 at 16:44
well take c_mat to be (1,847) col of 1s and 0s elements only; do same with key_mat; — user2290820, Aug 19 '14 at 16:46
@Gerrat number = int(''.join(map(str,key_mat.tolist()[0])),2) — user2290820, Aug 19 '14 at 16:56
Yes, I can see that...I wanted to know the approximate magnitude of it. Is it around 100...1000000...1000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 ?? — Gerrat, Aug 19 '14 at 16:57
@Gerrat Since key_mat is (1,847) matrix of [1,0,1,0...], it depends whats the input bit. assuming highest input bit in key_mat is not more than 255589106 in decimal; you could try concatenating row by row same bit string but i dont think that should give memory error — user2290820, Aug 19 '14 at 17:01
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/59596/discussion-between-user2290820-and-gerrat). — user2290820, Aug 19 '14 at 17:06

score 2 · Accepted Answer · answered Aug 19 '14 at 17:10

2

Let's assume that n is initially 255589106 (yes, worst case scenario, but bear with me).

Your loop halves n each time through, and since log(255589106, 2) ~= 27, you could end up looping about 27 times. Each time through the loop, you're doubling the size of c_mat (and maybe doubling key_mat as well - not sure). If c_mat starts off at a mere 847 bytes, after 27 doublings, it would be (847 * 2**27) bytes in size...or over 100Gigs. This doesn't even include the size of k_mat.

I'm not exactly sure what your program is doing, but it looks like its simply trying to do way too much in memory.

answered Aug 19 '14 at 17:10

Gerrat

28,863
9
73
101

847 * 2**27. interesting memops analysis; c_mat alone is taking up over 1xx Gigs! So not piling up, and doing someoperation(c_mat,k_mat) as (1,847) 27 times sounds more feasible? some_operation is xoring – user2290820 Aug 19 '14 at 17:13
1

int = 4 bytes. Its actually (847 * 4)* (2**27) - 454729662464 bytes ~ 454.7 Gb – user2290820 Aug 19 '14 at 17:20

score 0 · Answer 2 · answered Aug 20 '14 at 14:20

Now it is working.

I checkd for c_mat.shape and key_mat.shape and both were supposed to increase row by 1 on each loop but c_mat was increasing by 2**c_mat.shape[0] because c_mat was replacing its last value.

c_mat originally was c_mat = np.copy(C) and I wanted to actually concatenate (c_mat,C);

Now it's not an exponential increase but a linear addn of Columns by 1 row;

number = int(''.join(map(str,key_mat.tolist()[0])),2)
        #Build matrix of Keys
        #print key_mat
        #print c_mat
        while number >= 0:
            #Divide integer>> by 2 until it covers all rows
            number >>= 1
            key_mat = np.concatenate((key_mat, np.matrix(list((np.binary_repr(number)).zfill(fixd_len)), dtype=int)))
            c_mat = np.concatenate((**Original_C**, c_mat))

Issue is fixed.

MemoryError, Killed

2 Answers2