0

I have been trying all day to convert dbf files to CSV and cannot seem to get it. I have looked at various options and cannot seem to get one that will work. Here is one that I have been trying.

  import arcpy
  import dbf
  from arcpy import env
  import os

    def DBFtoCSV(path):
 '''Convert every DBF table into CSV table. 
'''
env.workspace = path
tablelist = arcpy.ListTables() # list tables in file
for table in tablelist: # iterate through every table
    #make sure you are just working with .dbf tables 
    if table.endswith('.dbf'):
        with dbf.Table(os.path.join(path, table)) as current_table:
            print current_table
            dbf.export(current_table)
    print "\n Processing ",table[:-4]+".csv table complete."
  if __name__ == '__main__':  
     path=r'path'
DBFtoCSV(path)

The error I am getting now is:

       Processing  name.csv table complete.

    Table:         F:/name.dbf
    Type:          Visual Foxpro
    Codepage:      cp1252 (Windows ANSI)
    Status:        read-write
    Last updated:  2014-02-24
    Record count:  4887170
    Field count:   23
    Record length: 235
    --Fields--
      0) respondent I binary
      1) report_yr I binary
      2) report_prd I binary
      3) sys_key I binary
      4) tr_id C(24)
      5) tr_contrac I binary null
      6) tr_begin_d T binary null
      7) tr_end_dat T binary null
      8) tr_timezon C(2) null
      9) tr_delv_ct C(4) null
     10) tr_delv_sp C(48) null
     11) tr_class_n C(4) null
     12) tr_term_na C(4) null
     13) tr_inc_nam C(4) null
     14) tr_inc_pea C(4) null
     15) tr_prod_na C(49) null
     16) tr_quantit B binary null
     17) tr_price B binary
     18) tr_units C(9) null
     19) tr_tot_tra B binary null
     20) tr_tot_tr2 B binary null
     21) tr_other M
     22) tr_revised T binary

     array('c', '\x00\x00')
    16
    (2, 0)
(235, array('c', '      \x8f\x04\x00\x00\xd9\x07\x00\x00\x03\x00\x00\x00\x01\x00\x00\
  x001Q09                    \x04\x00\x00\x001u%\x00\xe5\x03\x00\x00\x8au%\x00\x18
    X&\x05MPPNM PNM Switchyard                                  F   LT  M   FP  CAPA
 CITY                                         \x00\x00\x00\x00\x80+\x18A\xba\xda\
    x8a\xfdew\x0f@$/KW-MO  \x00\x00\x00\x00\x00\x00\x00\x00\xcd\xcc\xcc\xccR\xc47A\x
 00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'))
  ('0', 233, 2, 235, 0, 5, <function none at 0x110DF9B0>, <function none at 0x110D
 F9B0>)
array('c', '\x00\x00')
  Traceback (most recent call last):
  File "dbf_convert_stack.py", line 20, in <module>
DBFtoCSV(path)
  File "dbf_convert_stack.py", line 16, in DBFtoCSV
dbf.export(current_table)
  File "C:\Python27\ArcGIS10.4\lib\site-packages\dbf\ver_2.py", line 7859, in ex
  port
data = record[fieldname]
    File "C:\Python27\ArcGIS10.4\lib\site-packages\dbf\ver_2.py", line 2541, in __
getitem__
return self.__getattr__(item)
  File "C:\Python27\ArcGIS10.4\lib\site-packages\dbf\ver_2.py", line 2508, in __
 getattr__
value = self._retrieve_field_value(index, name)
    File "C:\Python27\ArcGIS10.4\lib\site-packages\dbf\ver_2.py", line 2693, in _r
etrieve_field_value
if ord(null_data[byte]) >> bit & 1:
IndexError: array index out of range
Cœur
  • 37,241
  • 25
  • 195
  • 267
bab354
  • 39
  • 2
  • 5

2 Answers2

0

Instead of using dbfpy use my dbf module instead:

import dbf  # instead of dbfpy

def DBFtoCSV(path):
    '''Convert every DBF table into CSV table. '''
    env.workspace = path
    tablelist = arcpy.ListTables() # list tables in file
    for table in tablelist: # iterate through every table
        #make sure you are just working with .dbf tables 
        if table.endswith('.dbf'):
            with dbf.Table(table) as current_table:
                dbf.export(current_table)
        #keep track of processing
        print "\n Processing ",table[:-4]+".csv table complete."
Ethan Furman
  • 63,992
  • 20
  • 159
  • 237
  • Thanks! I am having trouble installing your dbf module. I have downloaded the zip file and put into the work directory, but when I type "pip install dbf", it cannot find the file. Thanks again for your help, its been 3 years since I have worked with Python and that was basic stuff. – bab354 Jul 26 '16 at 11:58
  • Check [this answer](http://stackoverflow.com/a/30148383/208880) for help (I've never used pip with a zip file). – Ethan Furman Jul 26 '16 at 12:01
  • It seems that IntEnum is not available for Python 2.7. Is there any work around as I am getting Import Error for IntEnum. I am pretty glued to 2.7 for work reasons due to the licensed software of ARCGIS that I possess. – bab354 Jul 26 '16 at 12:16
  • @jollygood18: `IntEnum` should be there. Can you update your question with the traceback you are getting? – Ethan Furman Jul 26 '16 at 12:38
  • Just did. Updated the code with what you gave me and the error I am getting – bab354 Jul 26 '16 at 12:45
  • @jollygood18: Ah, I think I see the problem: [`enum34`](https://pypi.python.org/pypi/enum34) is required, and it looks like I forgot to update the `setup.py` with that info. On the down-side it also looks like you have a different `enum` package already installed, so installing `enum34` could break other working code. Looking at my options... – Ethan Furman Jul 26 '16 at 12:53
  • Okay, gotcha. I just installed Enum this morning because before I was getting an Enum import error. I think I should be okay to remove it if that is what it takes to get it working – bab354 Jul 26 '16 at 12:58
  • Ah, excellent! Uninstall that one and then install `enum34` and you should be good to go. – Ethan Furman Jul 26 '16 at 12:59
  • Thanks so much Ethan! I have not gotten passed the import error. It still is not quite right. I really appreciate your time and help. I think I am close, but for some reason it is not finding the file? It gives me an error on with dbf.Table(table) as current_table: -Saying that the file name cannot be found, even though it is the exact same spelling as in the file directory. Any thoughts? – bab354 Jul 26 '16 at 13:15
  • @jollygood18: I would guess that your script is not being run in the same location as the dbf files, and the `table` variable does not include the path. To verify, add a `print table` right after the `for table in tablelist:`. If `table` does not include the path you can try `with dbf.Table(os.path.join(path, table)) ...` (don't forget to `import os` ;) . – Ethan Furman Jul 26 '16 at 13:22
  • Yeah you were right! Thanks again. Last question ( I hope considering it's the last line of the program). I am now getting AttributeError: 'VfpTable' object has no attribute export. Is this a problem with my file format? – bab354 Jul 26 '16 at 13:28
  • Thanks! Lol, I just can't get this code to work. No, I am getting an IndexError: Array index out of range. Is there a size limit because some of these files are large and some I believe are empty. Could that be throwing it off? – bab354 Jul 26 '16 at 13:34
  • @jollygood18: Add the traceback to the end of your question. Also, add a `print current_table` just before the export, and add that as well (just for the one that bombs) -- that will give us some basic info about the table. – Ethan Furman Jul 26 '16 at 13:39
  • Just did, It seems its the first one that bombs – bab354 Jul 26 '16 at 13:47
  • Huh. Can you send me a copy of that table? Empty is fine (and quicker! That's a huge table! ;) . – Ethan Furman Jul 26 '16 at 14:24
  • Sure thing. I am new to Stack Exchange(as you probably can tell). What's the best way to send things (ex pm)? Also, I not sure how to create an empty copy especially because I haven't found a suitable program to open these dbf's. I was using OpenOffice, but installing that changed the file extension dbf to OpenOffice so I got rid of it. – bab354 Jul 26 '16 at 14:29
0

This can be fairly straightforward with SearchCursor. All you really need to do is get the field names, pass that into the cursor, then write the complete row to a csv with Python's csv module.

import arcpy
import csv

dbf = table_name # Pass in the table you've identified
outputFile = '{}.csv'.format(dbf.split('.dbf')[0])

# Get the fields in the dbf to use for the cursor and csv header row.
fields = []
for field in arcpy.ListFields(dbf):
    fields.append(field.name)

# Make the csv.
with open(outputFile, 'wb') as output:
    dataWriter = csv.writer(output, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)

    # Write header row.
    dataWriter.writerow(fields)

    # Write each row of data to the csv.
    with arcpy.da.SearchCursor(dbf, fields) as cursor:
        for row in cursor:
            dataWriter.writerow(row)

print('Finished creating {}'.format(outputFile))
Evan
  • 456
  • 7
  • 16
  • Thanks! I have copied what you gave me and modified the script but am getting the following error: TypeError: Fields must be string or non-empty sequence of strings. Happy to show you the code. An additional question: When you say pass the table that you identified, should I just loop through arcpy.TableList or continue to use dbf.Table – bab354 Jul 26 '16 at 17:19
  • I would use arcpy.TableList if possible. Regarding your error, do any of your fields have unusual characters in them? – Evan Jul 26 '16 at 19:24
  • Yeah, some underscores and numbers. In addition to others, haven't scanned them all. I tried playing around with ValidateFieldName, but could not get it work. Is there an easy fix using a loop? – bab354 Jul 26 '16 at 20:14
  • There are probably several things to try, but I would start by ensuring you're pushing a string into the fields list: `fields.append(str(field.name))` – Evan Jul 26 '16 at 20:20
  • I tried that and then I printed fields at the end of the loop and for some reason the list is coming up empty. Trying to figure out why. – bab354 Jul 26 '16 at 20:44
  • Have you tried with another file? Something very basic (e.g. a table with a few fields) just to validate your workflow and make sure you're not getting caught up by a data issue. – Evan Jul 27 '16 at 18:09