5

As a relatively new programmer, I have several times encountered situations where it would be beneficial for me to read and assemble program data from an external source rather than have it written in the code. This is mostly the case when there are a large number of objects of the same type. In such scenarios, object definitions quickly take up a lot of space in the code and add unnecessary impediment to readability.

As an example, I've been working on text-based RPG, which has a large number of rooms and items of which to keep track. Even a few items and rooms leads to massive blocks of object creation code.

I think it would be easier in this case to use some format of external data storage, reading from a file. In such a file, items and rooms would be stored by name and attributes, so that they could parsed into an object with relative ease.

What formats would be best for this? I feel a full-blown database such as SQL would add unnecessary bloat to a fairly light script. On the other hand, an easy method of editing this data is important, either through an external application, or another python script. On the lighter end of things, the few I heard most often mentioned are XML, JSON, and YAML.

From what I've seen, XML does not seem like the best option, as many seem to find it complex and difficult to work with effectively.

JSON and YAML seem like either might work, but I don't know how easy it would be to edit either externally. Speed is not a primary concern in this case. While faster implementations are of course desirable, it is not a limiting factor to what I can use.

I've looked around both here and via Google, and while I've seen quite a bit on the topic, I have not been able to find anything specifically helpful to me. Will formats like JSON or YAML be sufficient for this, or would I be better served with a full-blown database?

Brian Tompsett - 汤莱恩
  • 5,753
  • 72
  • 57
  • 129
George Osterweil
  • 1,564
  • 3
  • 12
  • 21

8 Answers8

5

Though there are good answers here already, I would simply recommend JSON for your purposes for the sole reason that since you're a new programmer it will be the most straightforward to read and translate as it has the most direct mapping to native Python data types (lists [] and dictionaries {}). Readability goes a long way and is one of the tenets of Python programming.

mVChr
  • 49,587
  • 11
  • 107
  • 104
  • Thanks. JSON seems promising, and I could write a small script to easily add new items and such, rather than editing manually. Can JSON data be translated easily into objects in a somewhat procedural way? – George Osterweil Jun 21 '12 at 01:11
4

I'm a big fan of using Python Pickles to store data in files.

A Pickle can serialize any kind of Python object properly, in particular "Complex" things -- any kind of Python Class, Functions -- any kind of Object at all!

It is not limited to the relatively simple structures like "lists", "dicts", "strings" and "numbers" available in data formats like JSON.

Felix
  • 541
  • 2
  • 7
  • The OP said, "an easy method of editing this data is important." Pickles aren't editable at all. – Ned Batchelder Jun 21 '12 at 14:18
  • I'll grant you the point that Pickles are not Human-Readable or Human-Editable. But you forgot to finish his quote: "... either through an external application, or another python script" "Easy to edit" doesn't necessarily mean "easy to edit by hand in my favorite text editor". Pickles are very easy to edit in python, by loading the Pickle, modifying the objects, and re-saving it. – Felix Jun 22 '12 at 02:10
  • This is pretty much correct. I don't necessarily need to be able to easily edit in the file. As long as I can write a fairly simple interface to write these structures it should work fine. It's just a matter of the fact I have a lot of data in a very specific format, so I'm trying to avoid cluttering my code when I could read just as easily from an external data file, and thus have much better readability and make the code more modular. – George Osterweil Jul 02 '12 at 04:04
3

Well it depends on your use case.

If your file is relatively small and almost static, either YAML or JSON can serve the purpose. Check out What is the difference between YAML and JSON? When to prefer one over the other for more info

If your file is huge, or dynamic, or there will be some concurrency control involved, then you 'd better let a database to handle it.

Community
  • 1
  • 1
xvatar
  • 3,229
  • 17
  • 20
2

I've had similar needs for a couple applications and settled on JSON using jsonpickle.

To make the json output more human readable/editable I use these formatting settings:

jsonpickle.set_encoder_options('simplejson', sort_keys=True, indent=4)

Then to encode / decode data:

text = jsonpickle.encode(data)
...
data = jsonpickle.decode(text)

The nice thing about jsonpickle is it lets you store class objects without needing to manually convert everything to dicts (as you would if you used plain JSON). jsonpickle also includes hooks to let you define your own converters, if you need more control over how things get converted.

Relational databases have their place of course, particularly for large multiplayer games; if a lot of your game logic involves frequent small updates to a huge number of objects the database approach is going win.

[Update] One further note, if you're going to be hand-editing json files a lot, make yourself a little json-checker script you can run on your edited files to find syntax errors, it'll save you a fair bit of time.

bryce
  • 468
  • 5
  • 23
  • 1
    The data will be mostly static, and never need to be updated during runtime, so this seem like a good way of doing it. One thing is, though, that the data might not be in object format to begin with, unless I create it via a second script. – George Osterweil Jun 21 '12 at 01:17
  • That's ok, jsonpickle will handle dicts, lists, strings, etc. just as well. – bryce Jun 21 '12 at 01:20
  • For my own data, I used a plain text file for the initial content and made a throwaway script that parsed the text, set up the object, and turned it into json via jsonpickle. After that, I'd just hand edit the json files. – bryce Jun 21 '12 at 01:27
1

If you want editability, YAML is the best option of the ones you've named, because it doesn't have <> or {} required delimiters.

Ned Batchelder
  • 364,293
  • 75
  • 561
  • 662
1

XML, JSON or YAML are more "loose" formats than what a relational database would give you. Relational databases are table-oriented and will impose some constraints in the way you store data.

From your description, I'd stick to JSON or YAML. With them, you can express fairly complex object graphs (XML would be my option if you require more "formal" typing or schemas).

For reading or writing operations, you usually consider serialiszing/deserializing to/from objects (like http://docs.python.org/library/json.html does).

jjmontes
  • 24,679
  • 4
  • 39
  • 51
0

I would be tempted to research a little into some GUI that could output graphviz (DOT format) with annotations, so you could create the rooms and links between them (a sort of graph). Then later, you might want another format to support heftier info.

But should make it easy to create maps, links between rooms (containing items or traps etc..), and you could use common libraries to produce graphics of the maps in png or something.

Just a random idea off the top of my head - feel free to ignore!

Jon Clements
  • 138,671
  • 33
  • 247
  • 280
0

For a small and closed echosystem, pickle is a great solution. according to my experience and Ilia Zaitsev's article, a more robust, faster and lighter files, I would suggest the combination of pandas DataFrame for relational data structure and pyarrow.feather for read / write.

for example:

import pandas as pd
from dataclasses import make_dataclass
import pyarrow.feather as feather

Point = make_dataclass("Point", [('tag', str), ('x', float), ('y', float)])
df = pd.DataFrame([Point('tl', 1.0, 1.0), Point('br', 3.0, 4.0)])
feather.write_feather(df, '/tmp/point_data.arrow)
Assaf-ge
  • 464
  • 4
  • 6