TDD with large data in Python

Question

I wonder if TDD could help my programming. However, I cannot use it simply as most of my functions take large network objects (many nodes and links) and do operations on them. Or I even read SQL tables. Most of them time it's not really the logic that breaks (i.e. not semantic bugs), but rather some functions calls after refactoring :)

Do you think I can use TDD with such kind of data? What do you suggest for that? (mock frameworks etc?) Would I somehow take real data, process it with a function, validate the output, save input/output states to some kind of mock object, and then write a test on it? I mean just in case I cannot provide hand made input data.

I haven't started TDD yet, so references are welcome :)

Answer 1

You've pretty much got it. Database testing is done by starting with a clean, up-to-date schema and adding a small amount of known, fixed data into the database. You can then do operations on this controlled environment, knowing what results you expect to see.

Working with network objects is a bit more complex, but it normally involves stubbing them (i.e. removing the inner functionality entirely) or mocking them so that a fixed set of known data is returned.

There is always a way to test your code. If it's proving difficult, it's normally the code design that needs some rethinking.

I don't know any Python specific TDD resources, but a great resource on TDD in general is "Test Driven Development: A Practical Guide" by Coad. It uses Java as the language, but the principles are the same.

Answer 2

most of my functions take large network objects

Without knowing anything about your code, it is hard to assess this claim, but you might want to redesign your code so it is easier to unit test, by decomposing it into smaller methods. Although some high-level methods might deal with those troublesome large objects, perhaps low-level methods do not. You can then unit test those low-level methods, relying on integration tests to test the high-level methods.

---

Edited:

Before getting to grips with TDD you might want to try just adding some unit tests.

• Unit testing is about testing at the fine-grained level: see my answer to the question How do you unit test the real world.
• You might have to introduce some indirection into your program, to isolate parts that are impossible to unit test.
• You might find it useful to decompose your data into an assembly of smaller classes, which can be tested individually.