1

I am loading in a lot of CSV files into a struct using Golang. The struct is

type csvData struct {
    Index   []time.Time
    Columns map[string][]float64
}

I have a parser that uses:

csv.NewReader(file).ReadAll()

Then I iterate over the rows, and convert the values into their types: time.Time or float64.

The problem is that on disk these files consume 5GB space. Once I load them into memory they consume 12GB!

I used ioutil.ReadFile(path) and found that this was, as expected, almost exactly the on-disk size.

Here is the code for my parser, with errors omitted for readability, if you could help me troubleshoot:

var inMemoryRepo = make([]csvData, 0)

func LoadCSVIntoMemory(path string) {
    parsedData := csvData{make([]time.Time, 0), make(map[string][]float64)}
    file, _ := os.Open(path)
    reader := csv.NewReader(file)
    columnNames := reader.Read()
    columnData := reader.ReadAll()
    for _, row := range columnData {
        parsedData.Index = append(parsedData.Index, parseTime(row[0])) //parseTime is a simple wrapper for time.Parse
        for i := range row[1:] {                                       //parse non-index numeric columns
            parsedData.Columns[columnNames[i]] = append(parsedData.Columns[columnsNames[i]], parseFloat(columnData[i])) //parseFloat is wrapper for strconv.ParseFloat
        }
    }
    inMemoryRepo = append(inMemoryRepo, parsedData)
}

I tried troubleshooting by setting columnData and reader to nil at end of function call, but no change.

Marc
  • 19,394
  • 6
  • 47
  • 51
Devesh Shukla
  • 43
  • 5
  • 1
    Well, the main reason is that you are not storing the data as is, you are trying to store it as a `map`. – Eldar Aug 28 '20 at 06:22
  • Can you tell me more? Am I missing important knowledge on how maps are stored in memory? I will read up once more, but I would appreciate it if you could give me your insights – Devesh Shukla Aug 28 '20 at 06:24
  • 2
    I read up and I seem to have fixed some of the run off by calling runtime.GC(). Now it consumes ~8GB in memory! thanks – Devesh Shukla Aug 28 '20 at 06:34

2 Answers2

5

There is nothing surprising in this. On your disk there are just the characters (bytes) of your CSV text. When you load them into memory, you create data structures from your text.

For example, a float64 value requires 64 bits in memory, that is: 8 bytes. If you have an input text "1", that is 1 single byte. Yet, if you create a float64 value equal to 1, that will still consume 8 byes.

Further, strings are stored having a string header (reflect.StringHeader) which is 2 integer values (16 bytes on 64-bit architectures), and this header points to the actual string data. See String memory usage in Golang for details.

Also slices are similar data structures: reflect.SliceHeader. The header consists of 3 integer values, which again is 24 bytes on 64-bit architectures even if there are no elements in the slice.

Structs on top of this may have padding (fields must be aligned to certain values), which again adds overhead. For details, see Spec: Size and alignment guarantees.

Go maps are hashmaps, which again has quite some overhead, for details see why slice values can sometimes go stale but never map values?, for memory usage see How much memory do golang maps reserve?

icza
  • 389,944
  • 63
  • 907
  • 827
  • Hi, so I called runtime.GC() and that helped a lot and cut memory usage from 12GB to 8GB. And generally calculating as 8 * len(column) * number of columns still roughly corroborates the on-disk size. As long as this is expected behaviour, it is fine, I will try optimizing in other ways. Thanks for your help! – Devesh Shukla Aug 28 '20 at 06:38
0

Reading an entire file into memory rarely is a good idea.

What if your csv is 100GiB?

If your transformation does not involve several records, maybe you could apply the following algorithm:

open csv_reader (source file)
open csv_writer (destination file)
for row in csv_reader
    transform row
    write row into csv_writer
close csv_reader and csv_write
Fulldump
  • 783
  • 5
  • 10