Read Mysql Database in Tensorflow

Question

I can't find the way that using MySQL database in TensorFlow.

I made the table and sensor data for reference.

Here is the question.

What should I use to read MySQL database in TensorFlow?
I find the way that making CSV file in MySQL, and read it in TensorFlow. But it is not real-time data. I want to use the data in real-time.

Please help me. Thank you.

score 3 · Answer 1 · answered Mar 17 '20 at 07:29

Late to the party. But i did it like that:

Read data from mysql in chunks (limit+offset) with pandas df.read_sql() in a separate thread.
yield the data with a generator
create a tf.Dataset with .from_tensor_slice() from the pandas dataframe.
Use model.fit() on the dataset.
Get the next chunk of data from the generator and train on that.

So, while the model is training on the data chunk, the next chunk is being loaded in the background. Ideally, the GPU will never sit idle. This can be adjusted by choosing a good chunk size and defining how many epochs the model should be trained on each chunk.

Relevant links: Make python generator run in background https://stackoverflow.com/a/29528804/5292996 https://www.tensorflow.org/tutorials/load_data/pandas_dataframe

That sounds great! If you ever find the time to paste a code example. That would be awesome. — PiTheNumber, Jul 22 '20 at 20:41

score 0 · Accepted Answer · edited May 23 '17 at 12:10

0

You'd probably need to just use a standard python interface for mysql and feed it into tensorflow. Here are some examples:

edited May 23 '17 at 12:10

Community

1
1

answered Dec 22 '16 at 21:19

Shan Carter

2,425
2
14
10

But this is inefficient is not it? What if you have a big data? – ARAT Nov 24 '18 at 04:08

Read Mysql Database in Tensorflow

2 Answers2