3

I have a huge collection of visual foxpro dbf files that I would like to convert to csv. (If you like, you can download some of the data here. Click on the 2011 link for Transaction Data, and prepare to wait a long time...)

I can open each table with DBF View Plus (an awesome freeware utility), but exporting them to csv takes a few hours per file, and I have several dozen files to work with.

Is there a program like DBF View plus that will allow me to set up a batch of dbf-to-csv conversions to run overnight?

/Edit: Alternatively, is there a good way to import .dbf files straight into SQL Server 2008? They should all go into 1 table, as each file is just a subset of records from the same table and should have all the same column names.

Zach
  • 29,791
  • 35
  • 142
  • 201
  • How about something like: http://www.whitetown.com/dbf2csv/ – Eddy Jan 12 '12 at 22:33
  • Do you HAVE Visual Foxpro available? If so, I could post a simple program code you could run.... – DRapp Jan 13 '12 at 16:36
  • @DRapp Unfortunately no. I also know very little about VFP. Is it available for free (like SQL Express is)? – Zach Jan 13 '12 at 16:37
  • What programming access tools DO you have... Visual Studio? C#??? And do you know how many records are in each file??? roughly? – DRapp Jan 13 '12 at 16:40
  • One other know of caution.. We too have used a DBF View utility, don't know if its the same one or not. However, if you DO use it on a dbf that has an auto-increment column for ID purposes in whatever system, it can kill that setting and your app could have other problems after that since all new IDs would be created as zero. – DRapp Jan 13 '12 at 16:58
  • @DRapp I have visual studio but am not terribly comfortable with it. We're looking at about 1-2 million rows per file. I can probably get away with dealing with just 5 files, so we're looking at 5-10 million rows. I'm very comfortable with SQL server, which may be a better destination for these files. – Zach Jan 13 '12 at 17:09
  • @DRapp: I'm not building an ap. I'm basically interested in 1 table (Transactions) that I want to aggregate by seller and year. Unfortunately, to do this I need to deal with these unwieldy, gigantic VFP dbf files. – Zach Jan 13 '12 at 17:11

4 Answers4

6

Load up your list of FoxPro files in an array/list then call the ConvertDbf on each to convert them from FoxPro to csv files. See the c# console application code below...

Credit c# datatable to csv for the DataTableToCSV function.

using System;
using System.Data;
using System.Data.OleDb;
using System.IO;
using System.Linq;
using System.Text;

namespace SO8843066
{
    class Program
    {
        static void Main(string[] args)
        {
            string connectionString = @"Provider=VFPOLEDB.1;Data Source=C:\";
            string dbfToConvert = @"C:\yourdbffile.dbf";
            ConvertDbf(connectionString, dbfToConvert, dbfToConvert.Replace(".dbf", ".csv"));

            Console.WriteLine("End of program execution");
            Console.WriteLine("Press any key to end");
            Console.ReadKey();
        }

        static void DataTableToCSV(DataTable dt, string csvFile)
        {
            StringBuilder sb = new StringBuilder(); 
            var columnNames = dt.Columns.Cast<DataColumn>().Select(column => column.ColumnName).ToArray(); 
            sb.AppendLine(string.Join(",", columnNames)); 
            foreach (DataRow row in dt.Rows) 
            { 
                var fields = row.ItemArray.Select(field => field.ToString()).ToArray(); 
                for (int i =0;i < fields.Length;i++)
                {
                    sb.Append("\"" + fields[i].Trim() );
                    sb.Append((i != fields.Length - 1) ? "\"," : "\"");
                }
                sb.Append("\r\n");
            } 
            File.WriteAllText(csvFile, sb.ToString());
        }

        static void ConvertDbf(string connectionString, string dbfFile, string csvFile)
        {
            string sqlSelect = string.Format("SELECT * FROM {0}", dbfFile);
            using (OleDbConnection connection = new OleDbConnection(connectionString))
            {
                using (OleDbDataAdapter da = new OleDbDataAdapter(sqlSelect, connection))
                {
                    DataSet ds = new DataSet();
                    da.Fill(ds);
                    DataTableToCSV(ds.Tables[0], csvFile);
                }
            }
        }
    }
}
Community
  • 1
  • 1
DaveB
  • 9,470
  • 4
  • 39
  • 66
2

This works very well and thanks for the solution. I used this to convert some visual foxpro dbf tables to flat files. With these tables, there is the additional challenge of converting fields of type Currency. Currency fields are a 64-bit (8 byte) signed integer amidst a 36 element byte array starting at the 27th position. The integer is then divided by 1000 to get 4-decimal precision equivalent.

If you have this type of field, try this inside the fields FOR loop

if (("" + fields[i]).Equals("System.Byte[]"))
{
    StringBuilder db = new StringBuilder();
    byte[] inbytes = new byte[36];
    inbytes = ObjectToByteArray(fields[i]);
    db.Append("" + (double)BitConverter.ToInt64(inbytes,27)/1E4);
    sb.Append("\"" + db);
}

With the following helper method

private static byte[] ObjectToByteArray(Object obj)
    {
        BinaryFormatter bf = new BinaryFormatter();
        using (var ms = new MemoryStream())
        {
            bf.Serialize(ms, obj);
            return ms.ToArray();
        }
    }
Jeff Gracz
  • 21
  • 1
2

In that case, SQL-Server I think has a capability of connecting to foxpro tables. I'm not exactly sure how as I've never done it recently (last time using SQL-Server about 8+ yrs ago). I'm sure there are other threads out there that can point you to connecting SQL-Server to VFP.

I quickly searched and saw this thread

In addition, you might need the latest OleDb provider to establish the connection which I've also posted in a thread here. This thread also shows a sample of the connection string information you may need from SQL-Server. The data source information should point to the PATH where the .DBF files are found, and not the specific name of the .DBF you are trying to connect to.

Hope this helps you out.

Community
  • 1
  • 1
DRapp
  • 47,638
  • 12
  • 72
  • 142
  • The problem is, I have a 64 bit version of SQL server, and the FoxPro driver is 32 bit, so I can't get them to play together. – Zach Jan 13 '12 at 17:22
  • @Zach, AAAggggghhhh... hate that... Sorry I can't help further. – DRapp Jan 13 '12 at 17:40
  • Thanks anwyays! Man, this is turning out to be a lot more difficult than I initially estimated... – Zach Jan 13 '12 at 17:44
1

Check out my answer to Foxbase to postrgresql data transfer. (dbf files reader).

Community
  • 1
  • 1
Ethan Furman
  • 63,992
  • 20
  • 159
  • 237