Dumping a RETS Data Feed

Question

So I have a RETS data feed from Paragon I am trying to dump all this data into a CSV using phRETS library. I have tried many different ways to dump data from this RETS feed via dmql or libRETS and none of them are working. Here is my php code using phRETS to try to dump the data feed.

Right now it just generates an empty csv. This is what is displayed in the terminal.

• Connecting to http://RETS url here/login?rets-version=rets/1.5 as user
  
  • Connected
    
• Property:RE_1
  
  • Query: (LIST_87=1980-01-01T00:00:00+) Limit: 1000 Offset: 0
    
  • Total found: 0
    
  • done
    
• Disconnecting
  

I wont include the correct url to the RETS feed in here but it is using version 1.5

 <?php

    $rets_login_url = "http://Insert RETS feed link /login?rets-version=rets/1.5";
$rets_username = "**************";
$rets_password = "**********";

// use http://retsmd.com to help determine the SystemName of the DateTime field which
// designates when a record was last modified
$rets_modtimestamp_field = "LIST_87";

// use http://retsmd.com to help determine the names of the classes you want to pull.
// these might be something like RE_1, RES, RESI, 1, etc.
$property_classes = array("RE_1");

// DateTime which is used to determine how far back to retrieve records.
// using a really old date so we can get everything
$previous_start_time = "1980-01-01T00:00:00";

//////////////////////////////

require_once("phrets.php");

// start rets connection
$rets = new phRETS;

echo "+ Connecting to {$rets_login_url} as {$rets_username}<br>\n";
$connect = $rets->Connect($rets_login_url, $rets_username, $rets_password);

if ($connect) {
        echo "  + Connected<br>\n";
}
else {
        echo "  + Not connected:<br>\n";
        print_r($rets->Error());
        exit;
}

foreach ($property_classes as $class) {

        echo "+ Property:{$class}<br>\n";

        $file_name = strtolower("property_{$class}.csv");
        $fh = fopen($file_name, "w+");

        $maxrows = true;
        $offset = 0;
        $limit = 1000;
        $fields_order = array();

        while ($maxrows) {

                $query = "({$rets_modtimestamp_field}={$previous_start_time}+)";

                // run RETS search
                echo "   + Query: {$query}  Limit: {$limit}  Offset: {$offset}<br>\n";
                $search = $rets->SearchQuery("Property", $class, $query, array('Limit' => $limit, 'Offset' => $offset, 'Format' => 'COMPACT-DECODED', 'Count' => 1));

                if ($rets->NumRows() > 0) {

                        if ($offset == 1) {
                                // print filename headers as first line
                                $fields_order = $rets->SearchGetFields($search);
                                fputcsv($fh, $fields_order);
                        }

                        // process results
                        while ($record = $rets->FetchRow($search)) {
                                $this_record = array();
                                foreach ($fields_order as $fo) {
                                        $this_record[] = $record[$fo];
                                }
                                fputcsv($fh, $this_record);
                        }

                        $offset = ($offset + $rets->NumRows());

                }

                $maxrows = $rets->IsMaxrowsReached();
                echo "    + Total found: {$rets->TotalRecordsFound()}<br>\n";

                $rets->FreeResult($search);
        }

        fclose($fh);

        echo "  - done<br>\n";

}

echo "+ Disconnecting<br>\n";
$rets->Disconnect();

Part of the issue was I was getting an error for this line of code

require_once("phrets.php");

I did not have this file so I googled it and found a phrets.php file from https://github.com/dangodev/PHRETS-Example/blob/master/lib/phrets.php

This could be a bad version of that file but I do not know.

If anyone knows how to perform a data dump on a RETS feed then please let me know. If there is a better way than using phRETS then let me know. I really just need to dump the data from my RETS feed into a csv or into something. Thanks!

EDIT: I changed my LIST_87 to L_UPDATEDATE and it is now finding data but not writing it to file. I also changed the Limit to 200. Here is what my terminal is displaying now.

• Connected
  
  • Property:RE_1
    
  • Query: (L_UPDATEDATE=1980-01-01T00:00:00+) Limit: 200 Offset: 0
    
  • Total found: 1193
    
  • Query: (L_UPDATEDATE=1980-01-01T00:00:00+) Limit: 200 Offset: 200
    
  • Total found: 1193
    
  • Query: (L_UPDATEDATE=1980-01-01T00:00:00+) Limit: 200 Offset: 400
    
  • Total found: 1193
    
  • Query: (L_UPDATEDATE=1980-01-01T00:00:00+) Limit: 200 Offset: 600
    
  • Total found: 1193
    
  • Query: (L_UPDATEDATE=1980-01-01T00:00:00+) Limit: 200 Offset: 800
    
  • Total found: 1193
    
  • Query: (L_UPDATEDATE=1980-01-01T00:00:00+) Limit: 200 Offset: 1000
    
  • Total found: 1193
    
  • done
    
  • Disconnecting
    

my csv file that is created still just has an empty table in it.

Answer 1

1. Use the official PHRETS Github page: https://github.com/troydavisson/PHRETS. He has updated the library which includes an updated branch of code that is more object-oriented.

2. Check that the field names in your search query are correct. For example, you have "LIST_87" which may not be correct. Use RETS MD to verify.

3. From this documentation, Offset and Limit are both strings. You have them as integers in your search query.

4. If you just want to download the RETS feed as a CSV file, then this can be easily done using: RETS Connector, a GUI program requiring no coding.

Answer 2

In your code, you need to change the initial value of $offset to 1.

$offset = 1;

Since the value of $offset is 0, the following condition is failed,

if ($offset == 1) {
    // print filename headers as first line
    $fields_order = $rets->SearchGetFields($search);
    fputcsv($fh, $fields_order);
}

header of the CSV not written and $fields_order array is still empty.

Then followed by the while loop,

foreach ($fields_order as $fo) {
    $this_record[] = $record[$fo];
}

$this_record remains empty, since $fields_order array is empty.

Thus your CSV becomes empty, even-though data is present.