0

I finished creating an accounting web application for an organization using codeignter and mysql db, and I have just submitted it to them, they liked the work, but they asked me how they would transfer their old manual data to the new one online, so that their members would be able to see their account balances and contributions history.

This is a major problem for me because most of my tables make use of 'referential integrity' to ensure data synchronization and would not support the style of their manual accounting.

I know a lot of people here have faced cases like this and I would love to know the best way to collect users history, and I also know this might probably be flagged as not a real question, but I really really have to ask people with experience.

I would appreciate all answers. Thanks (And vote downs too)..

Ramos1116
  • 23
  • 1
  • 6
  • How manual are we talkin': pen-and-paper, Excel spreadsheet? – Chris Forrence Jul 10 '14 at 16:44
  • Excel Spreadsheet actually, but they use manual computation, no formulas on the spreadsheet – Ramos1116 Jul 10 '14 at 16:45
  • I was thinking maybe I create another table for the old data, but this means I would have to go back to my code and Include balances from to each balance computation – Ramos1116 Jul 10 '14 at 16:52
  • Could you clarify what you're asking for by "collect users history"? My guess is that you're looking to parse the Excel spreadsheet, match the user in the spreadsheet to one in your application, massage the values into a format that your application can understand and insert those into your own database. Is that correct? – Chris Forrence Jul 10 '14 at 16:55
  • That is exactly what I want to do, but in my application for instance, if a loan is requested and accepted it is logged into a table called financial_transactions, so all transactions have a parent, but there is no way to do this in my situation – Ramos1116 Jul 10 '14 at 17:03
  • I think the extra table for the old data makes the most sense, considering the data is undoubtedly in a different format. But not looking at both systems, its hard to say. – developerwjk Jul 10 '14 at 17:08
  • I know the question is opinion based and there are probably a lot of ways around it as smcjones have shown, am just looking for the best opinion to follow. Thanks – Ramos1116 Jul 11 '14 at 09:34

1 Answers1

2

No matter what the case is, data conversions are very challenging and almost always time consuming. Depending on the consistency of the data in question, it could be a case that about 80% of the data will transfer over neatly if you create a conversion program using PHP. That conversion code in and of itself may be more time consuming than it is worth. If you are talking hundreds of thousands of records and beyond, it is probably a good idea to make that conversion program work. Anyone who might suggest there is a silver bullet is certainly not correct.

Here are a couple of suggested steps:

  1. (Optional) Export your Excel spreadsheets to Access. Access can help you to standardize data and has tools in place to help you locate records which have failed in some way. You can also create filters in Access if you need to. The benefit of taking this step, if you are familiar with Access, is that you have already begun the conversion process to a database. As a matter of fact, if you so desire, you can import your MySQL database information into Access as well. The benefit of this is pretty obvious: You can create a query and merge your two separate tables together to form one table, which could save you a great deal of coding.

  2. Export your Access table/query into a CSV file (note, if you find it is overkill or if you don't have Access, you can skip step 1 and simply save your .xls or .xlsx file to type .csv. This may require more legwork for your PHP conversion code but that is probably a matter of preference. Some people prefer to avoid Access as much as possible, and if you don't normally use it you will be wasting time trying to learn it just to save yourself a little bit of time).

  3. Utilize PHP's built-in str_getcsv function. This will convert a CSV file into a PHP array.

  4. Create your automated program to parse through each record. Based on the column and its requirements, you can either accept or reject records. You can then export your data, such as was done in this SO answer, back to CSV. You can save two different CSV files, one with accepted records, and one with rejected records.

With rejected records, which are all but inevitable when transferring from a spreadsheet, you will need to have a course of action. The simplest way for your clients is probably to give them a procedure to either manually import records into the database, if you've given them an interface to do so, or - probably simpler but requiring more back-and-forth - to update the records in Excel to be compliant with the new system.

Edit Based on the thread under your question which sheds more light on what you are trying to do (i.e., have a parent for each transaction that is an accepted loan), you should be able to contrive a parent field, even if it is not complete, by creating a parent record for each set of transactions based around an account. You can do this via Access, PHP, or, more likely, a combination.

Conclusion Long story short, data conversions take time. If you put the time in up front, it will be far easier to maintain a standardized series of information in the long run. If you find something which takes less time in the beginning, it will mean additional work for you in the long run in order to make this "simple" fix work over time.

Similarly, the closer you can get legacy data to conform to your new data, the easier it will be for your clients to perform queries etc. While this may mean that some manual entry will be required on the part of you or your client, it is better to inform the client of the pros and cons of each method fully and let them decide. My recommendation would always be to put extra work in at the front-end because it almost always ends up cheaper than having to deal with a quick fix in the long run, but that is not always practical given real world constraints.

Community
  • 1
  • 1
smcjones
  • 5,490
  • 1
  • 23
  • 39