If I want to build a custom database, how could I?

Question

I built a previous program that took client info and stored it in a folder of txt files (impractical much) but now I want to upgrade the program to be more efficient and put the info into a database of some sort...

How can I take the info from the text files and add them to the new database without having to manually do each one. I know this is vague but I need more so the method/logic instead of the exact code, Also if I don't use SQL what is another method for making a db (Not using another commercial Db)

btw the txt files are in simple format (name,city,age) all on separate lines for easy iteration

There's JSON, XML, SQLite, simple pickling. What do you have in mind? — Santa, Apr 28 '11 at 18:27
well im on path to learning sql in python but for this soft upgrade to the program i could use pickling or shelve...which is better, i tried doing a dictionary of dictionary's in a pickle file but i could find a way to get what i needed out of it — Isov5, Apr 28 '11 at 18:32
You might find the question [_What is the best data structure for storing a set of four (or more) values?_](http://stackoverflow.com/questions/15418386/what-is-the-best-data-structure-for-storing-a-set-of-four-or-more-values) (and my answer to it) of interest. — martineau, Aug 17 '16 at 17:23

martineau · Answer 1 · 2011-06-06T13:08:03.613

3

The free and portable Python shelve module in the standard library is probably all you need. It allows you to create and use what are essentially persistent dictionaries, so there's a very gradual learning curve. Converting your text files into one should be fairly easy, although won't be automatic -- you'll probably need to write a simple script to do it.

edited Jun 06 '11 at 13:08

answered Apr 28 '11 at 19:19

martineau

119,623
25
170
301

Although `shelve` is not secure against malicious data, it is very suitable to serve as building block to learning how to build a database from scratch. – RandyTek Aug 17 '16 at 16:28

score 2 · Answer 2 · answered Apr 28 '11 at 18:27

2

Well, you could read your txt* files using the csv module in Python.

I'm afraid that knowledge of SQL is a must for any sort of database manipulations, unless you have the comfort of an ORM eg. Django's ORM.

*they aren't called that on anything but Windows.

answered Apr 28 '11 at 18:27

aviraldg

9,531
6
41
56

Knowledge of SQL is a must even when using an ORM. Database normalization is substantially different from object-oriented programming when doing anything more complex than simple records. – André Caron Apr 28 '11 at 18:46
2

@Andre: The OP wants to store *just* "name,city,age". I guess that qualifies as "simple records." – aviraldg Apr 28 '11 at 18:48

score 2 · Answer 3 · edited Oct 21 '17 at 17:19

I would just get list of all *.txt files is directory using os.listdir() then read and parse all of them and finally put all information in some "database". Python has few such "database" kind of modules

csv module (http://docs.python.org/library/csv.html)
bsddb (this is only in Python 2.x, https://docs.python.org/2/library/bsddb.html)
sqlite3 module (http://docs.python.org/library/sqlite3.html)

score 1 · Answer 4 · answered Apr 28 '11 at 18:43

1

In addition to the suggestions above, take a look at Elixir, which is an abstraction layer over SQLAlchemy that makes building database interfaces (and inserting / querying information) rather simple.

answered Apr 28 '11 at 18:43

Mike Pennington

41,899
19
136
174

score 0 · Answer 5 · answered Mar 15 '13 at 21:05

The main reason for DB to have a SQL is to make it separate and generic from the application that you are developing.

To have your own DB built you need to have a storage mechanism could be files on the hard disk, with search options so that you can access data immediately with keywords that you are interested in. on top of this you have to have a layer that initiates queues, reads them and translates to the lower file read and write functions. you need to have this queue layer because lets say you have 100 applications and all are trying to read and write from the same file at the same time and you can imagine what can happen to the file . there will be access denied , somebody using it, data corrupted etc etc.. so you need to put all these in queue and let this queue layer translate things for you.

to start with start from different ways of reading/writing/sorting of data into the file, and a queue layer. From there you can build applications.

The queue layer here is similar to the client that is trying to push the data into the communication port in most of the available databases.

If I want to build a custom database, how could I?

5 Answers5

Linked