0

i have some code where I use arrays to hold large amounts of data(sometimes over 1million data points), I use arrays because they can be dynamically defined in 2 directions and are able to insert values at a specific position. However I found that when I use Redim to change to dimensions of the array it creates a duplicate of the array in the RAM causing me to eventually run out of RAM. I have found that Erase can be used to remove the array and save RAM, then I can use Redim to create a new array. However this means I have no where to hold my data in the mean time, unless I create a second array and manually copy the data across.

Is there a way to Redim an array to without losing RAM but with a more efficient way of hold the data rather than manually copy to a new array and back again. Or is there another class I can use to hold the data instead of an array that fulfills the requirements above.

Please ask for sample code if you would like some.

Many thanks for your help

Pezzzz
  • 738
  • 4
  • 14
  • 33
  • ...eventually run out of RAM? Are you talking about a memory leak? Could you show us how you are doing this (code)? If not a memory leak, are you saying that when your datasets get large you do not have room for two copies in RAM? – J... Nov 08 '12 at 11:32
  • I am certain this isn't a memory leak. I was using `Redim Preserve` and `Redim` on an array with over a million records several times. I have updated the code to only `Erase` and `Redim` the array and now it works for twice the amount of data I used to get a RAM error. – Pezzzz Nov 08 '12 at 12:35

1 Answers1

1

Using an array in this way is not very memory efficient.

You would be better to use a List(of T) Have a look at this answer for reasons why

In this case you can use a class to hold each data point. Using a List(of T) allows you to insert into the middle of the list without a copy of the entire data being created.

'Define your datapoint class
Private Class DataPoint
    Public Property X As Long
    Public Property Y As Long
    Public Sub New(ByVal X As Long, ByVal Y As Long)
        Me.X = X
        Me.Y = Y
    End Sub
End Class

'create and initialise the list
Dim MyDataPoints As New List(Of DataPoint)
'add some data to it
MyDataPoints.Add(New DataPoint(1, 2))
MyDataPoints.Add(New DataPoint(1, 2))
MyDataPoints.Add(New DataPoint(4, 3))
'Now insert a new datapoint at an arbitrary position in the list
MyDataPoints.Insert(2, New DataPoint(5, 6))
Community
  • 1
  • 1
Matt Wilko
  • 26,994
  • 10
  • 93
  • 143
  • This is definitively not memory efficient. You'll be creating millions of objects and putting quite a big pressure on the GC, besides the fact that you'll be using at least 8 bytes more per instance - so instead of using 16 bytes (2 Longs) you'll be using 24 bytes per instance, a 50% increase. But if you make DataPoint a Structure instead it'll become a lot better. – Rolf Bjarne Kvinge Nov 08 '12 at 09:27
  • I don't need to be memory efficient holding the data, I need a memory efficient way of re-dimensioning the class holding the data. I am currently in the process of writing a new class as a list of lists, which seems to be the solution to the problem. – Pezzzz Nov 08 '12 at 12:33