How to sort large multidimensional array from csv data?

Question

I am importing a csv that could potentially have around 100,000 rows. Each row has 5 columns. The first column of each row will have a sentence and the other 4 columns have numeric values. I need to loop through the csv data and grab each word from each row and add it as a it's own row in a new array. So for example:

| big red truck   | 5 | 2 | 5 | 1 |
| small red truck | 4 | 2 | 0 | 0 |
| big fast truck  | 3 | 2 | 4 | 1 |

becomes

| truck | 12 | 6 | 9 | 2 |
| red   | 9  | 4 | 5 | 1 |
| fast  | 3  | 2 | 4 | 1 |
| small | 4  | 2 | 0 | 0 |

This is what I'm currently doing. It works fine with smaller files but at around 50,000 rows I run into issues and start getting back server errors.

function get_csv_terms($csvdata){
    $terms = array();
    $csv_rows = count($csvdata);
    $x = 0;
    //get terms
    while($x <= $csv_rows){
        $groupTerm = explode(' ', $csvdata[$x][0]);
        foreach( $groupTerm as $term ){
            if($term != NULL){
                if(!in_array($term, $terms)){
                    $terms[] = $term;
                }
            }
        }
        $x++;
    }

    return $terms;
}

//filter csv and create data for table output
function filter_csv($csvdata){
    $sortedData = array();
    $csv_rows = count($csvdata);
    $terms = get_csv_terms($csvdata); 
    $terms_count = count($terms);
    $x = 0;

    while($x <= $terms_count){
        $y = 0;
        while($y <= $csv_rows){
            $termWords = explode(" ", $csvdata[$y][0]);
            $termWordCount = count($termWords);
            $z = 0;
            while($z <= $termWordCount){
                if($terms[$x] != NULL){
                    if($termWords[$z] == $terms[$x]){
                        $sortedData[$terms[$x]][0] +=  intval($csvdata[$y][1]);
                        $sortedData[$terms[$x]][1] +=  floatval($csvdata[$y][2]);
                        $sortedData[$terms[$x]][2] +=  floatval($csvdata[$y][3]);value
                        $sortedData[$terms[$x]][3] +=  floatval($csvdata[$y][4]);
                    }
                }
                $z++;
            }
            $y++;
        }
        $x++;
    }

    return $sortedData;
}

Answer 1

As mentioned in the comment trails, the resolution should be to either increase the memory limit far enough (if possible), or else reduce data duplication and switch to processing the CSV one record at a time to reduce memory usage.

There's also a possibility that using a defined object class could give you better memory efficiency, if not just better code readability. For example, you might try this:

class Term
{
  // Instance properties
  public $UsageCount = 0;
  public $Weight     = 0.00;
  public $Value      = 0.00;
  public $OtherFloat = 0.00;

  // -----------------------

  // Term dictionary
  private static $_terms = array();

  public static Get($term)
  {
    if(!isset($_terms[$term]))
    {
      $_terms[$term] = new Term();
    }
    return $_terms[$term];
  }

  public static GetAll()
  {
    return $_terms;
  }
}

and in your CSV loop:

$termWords = explode(" ", $csvdata[$y][0]);
...
foreach($termWords as $termWord)
{
  $Term = Term::Get($termWord);
  $Term->UsageCount += intval($csvdata[$y][1]);
  $Term->Weight     += floatval($csvdata[$y][2]);
  $Term->Value      += floatval($csvdata[$y][3]);
  $Term->OtherFloat += floatval($csvdata[$y][4]);
}

Recent versions of PHP (7 and later) are great at memory efficiency for objects. Depending on what other properties and code you might have, the object-oriented approach can provide a lot of nice benefits (e.g. automatic pass by reference, easier-to-read-and-maintain code, memory efficiency, clear defaults, data management, etc...).