Using a Python Class to share data between Python scripts

Question

I am wondering if it makes sense in Python to use a class containing a list as if it is an in-memory database.

class MyData(object):
    def __init__(self):
        self._data = ['apple', 'orange', 'banana']

    @property
    def data(self):
        return self._data

    @classmethod
    def append(self, item):
        self._data.append(item)

Here's the scenario:

A first python script a.py will import this class MyData and use the data in its list.

A second python script b.py continuously poll an online API in an infinite loop, imports this same class and appends more items to the list.

When a.py runs another time, it should be able to read the newly added items in the list.

Question: Is this a good idea, or are we better off using a real database like MySQL/redis?

Update: The data does not need to be persisted to disk

I don't get how you will store state here. If `a.py` runs another time, it will here simply start again with the initial list. You can use code generators to alter the Python code, but that would be horrible design. — Willem Van Onsem, Dec 18 '17 at 21:16
This is what databases were designed to do. You will have a much easier time using SQL. — Myles Hollowed, Dec 18 '17 at 21:17
Furthermore in many cases a database is meant to store data. It is done in such a way to make it easy to insert, update, remove, ... data, and run aggregates over it. Here you only have a list, so you would miss a lot of important functionality. Most databases detach data from a specific application: the database can be queried in a variety of ways each time retrieving other interesting subsets of data. — Willem Van Onsem, Dec 18 '17 at 21:18
Finally whether it is stored in memory or on a disk does not depend on whether you use SQL. There are database managers for SQL that can store a database completely (or partially) in RAM memory. By default most database management systems will store a large portion of the indexes in memory. — Willem Van Onsem, Dec 18 '17 at 21:20
why do you have a `classmethod` that expects a `self` argument? That's not how class methods work... — Tadhg McDonald-Jensen, Dec 18 '17 at 21:33
That doesn't help your case, either way is `b.py` expected to indicate to `a.py` that there is new data at all? you could probably just `print` the data from `b.py` then pipe that to `a.py` which it'd take with `input`. — Tadhg McDonald-Jensen, Dec 18 '17 at 21:36

score 1 · Answer 1 · answered Dec 18 '17 at 21:18

1

This will not work how you described it. a.py and b.py will be run be different python interpreters, thus not sharing memory (i.e. they won't share the class).

answered Dec 18 '17 at 21:18

Kiersten Arnold

1,840
1
13
17

score 1 · Answer 2 · answered Dec 18 '17 at 21:38

If the idea is about running a.py and b.py on different python processes, you'd need to persist your data somewhere on disk, using interprocess communication or similars... My best advice? Use just your database through some of the zillion python modules allowing you to communicate to it, that's what they're made for.

Now, if the question is about sharing an object state between modules, here's a simple example which shows you how to achieve that:

storage.py:

class Storage(object):

    def __init__(self):
        self._data = ['apple', 'orange', 'banana']

    @property
    def data(self):
        return self._data

    def append(self, item):
        self._data.append(item)

Storage = MyData()

a.py:

from storage import Storage


class A():

    def __init__(self):
        self.count = 0

    def __str__(self):
        self.count += 1
        return f"read count={self.count} data={','.join(Storage.data)}"

b.py:

from storage import Storage


class B():

    def __init__(self):
        self.count = 0

    def write(self, item):
        Storage.append(item)

test.py:

from a import A
from b import B

if __name__ == "__main__":
    reader = A()
    writer = B()

    print(f"Initial state of storage {reader}")
    for i, item in enumerate(["pomegranate", "watermelon", "strawberry"]):
        writer.write(item)
        print(reader)

For more info about this topic you can check this Is there a simple, elegant way to define singletons? or similars SO threads talking about python&singletons.

score 1 · Answer 3 · answered Dec 18 '17 at 21:43

assuming there is nothing else that a.py needs to output, you can output the data you need to transfer by just printing it, for example:

#a.py
for i in range(6):
    print("data:{}".format(i))

Then by piping the output to b.py which can receive the input with input (raw_input with python2)

#b.py
try:
    while True:
        data = input()
        print("recieved data:", repr(data))
except EOFError: #raised when input stream is closed
    print("data stopped coming")

Then you can run them together using the | in a command prompt to pipe the data:

Tadhgs-MacBook-Pro:codes Tadhg$ python3 a.py | python3 b.py
recieved data: 'data:0'
recieved data: 'data:1'
recieved data: 'data:2'
recieved data: 'data:3'
recieved data: 'data:4'
recieved data: 'data:5'
data stopped coming

Using a Python Class to share data between Python scripts

3 Answers3