Techniques for object-oriented or modular file and data I/O?

Question

It seems like every single time I go to write any code that deals with fetch, push, read, or write operations, the entire swath of code is ad hoc, ugly, and completely unusable outside the context of that exact application. Worse yet, it feels like I have to reinvent the wheel every time I design these things. It seems to me that the nature of I/O operations is very linear, and not well suited to modular or object-oriented patterns.

I'm really hoping somebody can tell me I'm wrong here. Are there techniques/patterns for object-oriented or modular file and data I/O? Is there some convention I can follow to add some code re-usability? I know that various tools exist to make reading individual files easier, like XML parser and the like, but I'm referring to the larger designs which use those tools.

The problem isn't limited to a single language; I hit the same wall in Java, C, Matlab, Python, and others.

A sub-topic of this is addressed by this question about what object should invoke saving. That question seems to refer to a factory pattern, where a file's contents are built up, then finally written to disk. My question is about the overall architecture, including factories for write operations, but also (Insert Pattern Here) for reading/fetching operations.

The best thing I've been able to come up with is a facade pattern... but holy smokes is the code in those facades ugly.

Somebody please tell me a pattern exists wherein I could either re-use some of my code, or at least follow a template for future read-writes.

Somebody asked about Modular Design here, but the answers are specific to that asker's problem, and aren't altogether useful.

---

Example

This is only an example, and is based on a project I did last year. Feel free to provide a different example.

Our program is a physics sandbox. We want to load XML data which describes physical attributes of objects in that sandbox. We also need to load .3DS files which contain 3D rendering information. Finally, we need to query an SQL database to find out who owns what objects.

We also need to be able to support 3D model formats when they come out. We don't know what those files will look like yet, but we want to set up the code framework in advance. That way, once we get the new data schema, the loading routine can be implemented quickly.

Data from all 3 sources would be used to create instances of objects in our software.

Later, we need to save physics information, like position and velocity, to a database, and also save custom texture information to local files. We don't know what type of file the texture will be, so we just want to lay out the code structure so we can drop in the saving code later.

Without some sort of design pattern, even a small number of objects quickly leads to a closely-coupled network.

A facade can decouple the objects/data from the corresponding files, but all that does is centralize the problem within the input and output facades, which can turn into a nightmarish mess in a hurry. Moreover, the objects/data are now closely coupled to the facade, so no modularity is really gained.

---

Edited from 3 weeks ago...

Previously, I had provided a bunch of psuedocode for the issue I was facing when I first asked this question, but I have since decided it obfuscated my main question. Suffice to say: I had to use about 2000 lines of really wonky code for that particular set of read operations, it did remarkably little in terms of processing and organization, and I'll never be able to use any of it on another project again.

I would like to avoid writing such code in the future.

Answer 1

There may be particulars of your problem or question that are not entirely addressed by this solution, but I would like to share the approach I have taken to unifying I/O Operations. I have used this before in a few different high-level languages. This strategy scales best when used in conjunction with serialization.

It seems that the two basic I/O operations are Save/Put and Load/Get. Here is the most abstract universal interface that represents that:

public interface ObjectRepository
{
    <T> void save(string resourceId, T obj);
    <T> T load(string resourceId);
}

This strategy is sufficient for all types of I/O operations, where the resource id (a database UUID/GUID/String, a file path, a web URL...etc) is known.

The simplest implementations vary somewhat depending on language and framework, but I find that the most universally applicable are implementations that rely on standard forms of serialization, namely binary, XML, and JSON. When using proprietary objects, I most commonly use an XmlFileRepository, which translates my simple data object to/from Xml, and saves/loads at the chosen filepath.

Furthermore, if used in conjunction with the Abstract Factory pattern, it is rather trivial to use diverse data format inputs and outputs for even a single type of object. Sample code:

public NewtonianObject load(string respositoryType, string resourceId) 
{
    ObjectRepository repo = RepositoryFactory.create(respositoryType);
    return (NewtonianObject)repo.load(resourceId);
}

public void exportAsXml(string fileName, NewtonianObject obj)
{
    ObjectRepository repo = RepositoryFactory.create("XmlFileRepository");
    repo.save(fileName, obj);
}

Answer 2

Holub makes usage of a Builder Pattern for exporting data from objects (rendering) and something he named an Inverse Builder pattern for initialization.

These Exporter/Importer objects as he calls them are part of the domain model model itself, it is a way a model should persist and create itself using different data sources.

https://youtu.be/CYCNRCrX1zE?t=45m6s

http://www.javaworld.com/article/2072302/core-java/more-on-getters-and-setters.html

Answer 3

Template Pattern

Here in the Template Pattern, create an abstract class related to I/o operations which hold set of abstract methods and few other methods if any re-usability exist. And all external system like database, Text... can extend this abstract class and do the specific job.