0

In my previous question, (How to Animate multiple columns as dots with matplotlib from pandas dataframe with NaN in python), I managed to animate multiple dots from a dataframe as an animation.

However, I wanted to set a background for the animation as a network graph, so that it seems that the dots are moving on the lines of the network.

Using the code from How to Animate multiple columns as dots with matplotlib from pandas dataframe with NaN in python

I've created a new MCV example, the code is listed below:

import random
import networkx as nx
import matplotlib.pyplot as plt
import numpy as np
import math
import pandas as pd
from matplotlib import animation
#from JSAnimation import IPython_display
%matplotlib inline

# initialise graph object
G = nx.Graph()
color_map =[]
G.add_node(1, pos=(1, 0)); color_map.append('r')
G.add_node(2, pos=(2, 0)); color_map.append('r')
G.add_node(3, pos=(3, -1)); color_map.append('r')
G.add_node(4, pos=(3, 1)); color_map.append('r')
G.add_node(5, pos=(4, -1)) ;color_map.append('r')
G.add_node(6, pos=(4, 1)); color_map.append('r')
G.add_node(7, pos=(5, 0)); color_map.append('r')
G.add_node(8, pos=(6, 0));  color_map.append('r')

e = [(1, 2, 1), 
    (2, 3, 1), 
    (2, 4, 2), 
    (3, 5, 5), 
    (4, 6, 2), 
    (5, 7, 1), 
    (6, 7, 2), 
    (7, 8, 1)]

G.add_weighted_edges_from(e)
labels = nx.get_edge_attributes(G,'weight')

nx.draw(G,nx.get_node_attributes(G, 'pos'))
nx.draw_networkx_edge_labels(G,nx.get_node_attributes(G, 'pos'),edge_labels=labels)
nx.draw_networkx_labels(G,nx.get_node_attributes(G, 'pos'))

df_x = pd.DataFrame(data=
    np.array(
    [[np.NaN, np.NaN, np.NaN, np.NaN], 
     [1, np.nan, np.NaN,np.NaN], 
     [1.5, 4, np.NaN,np.NaN], 
     [2, 5, 3,4]]
        ), index= [1, 2, 3, 4], columns=[1, 2, 3, 4])
print(df_x)

df_y = pd.DataFrame(data=np.array(
    [[np.NaN, np.NaN, np.NaN, np.NaN], 
     [0, np.nan, np.NaN,np.NaN], 
     [0, -1, np.NaN,np.NaN], 
     [0, 0, 1,1]]
    ), index= [1, 2, 3, 4], columns=[1, 2, 3, 4])

%matplotlib notebook
from matplotlib import animation
#from JSAnimation import IPython_display
#from IPython.display import HTML

fig = plt.figure(figsize=(10,10))
ax = plt.axes()
nx.draw(G,nx.get_node_attributes(G, 'pos'),node_size = 10)
n_steps = df_x.index 

graph, = plt.plot([],[],'o')

def get_data_x(i):
    return df_x.loc[i]

def get_data_y(i):
    return df_y.loc[i]


def animate(i):
    x = get_data_x(i)
    y= get_data_y(i)
    graph.set_data(x,y)
    return graph,

animation.FuncAnimation(fig, animate, frames=n_steps, repeat=True, blit = True)

This creates a workable animation, which works. But however, when I use a very large dataset ( pandas dataframe index is ~8000 rows * 800 columns instead of the example pandas dataset I posted), the animation takes very long(hour or so) to render and most of the times my browser( google chrome) crashes.

So I thought is maybe due to it needs to redraw the networks graph each frame? How can I set the background as the networkx graph? From there on it is just plotting points right? My actual graph is a bit larger (~5000 nodes, ~6000 edges).

Hopes anyone can help me speed the rendering of the animation up!

Jeroen123
  • 55
  • 8
  • if you're going to use HTML, then you'd better work with a javascript library like HighCharts or d3. Python generating maptplotlib plots exporting to HTML sounds a little crazy. – MrE Oct 25 '18 at 03:27
  • @MrE Yeah you're right! But for this animation, it doesn't do a lot. If you comment it out, the animation still works. I saw it in another example which was close to mine example, so I imported that package. But it runs completely fine without from - JSAnimation import IPython_display - from IPython.display import HTML I updated the example with this, and it still runs fine. But not for very large datasets.. – Jeroen123 Oct 25 '18 at 07:40
  • 1
    it doesn't work for large data sets because it's generating DOM elements (or maybe SVG, which goes into the DOM the same way), as opposed to using modern tools like canvas to draw the elements. Rendering millions of DOM elements will be slow in any browser. – MrE Oct 25 '18 at 15:35
  • @MrE Yeah, I couldn't find any 'simple' solution to the problem. Therefore I thought of a workaround; it would be just as good to create an MP4 file with the writerclass of matplotlib animation to analyse the results of my simulation. This works significantly faster and is now done in under 2 minutes. I'll post my solution aswell! Thanks for your help anyway! :-) – Jeroen123 Oct 26 '18 at 09:55
  • EDIT : * under 5 minutes – Jeroen123 Oct 26 '18 at 10:00

1 Answers1

0

After some digging around, I found no 'easy' solution to this problem when trying to animate large datasets into an animation with matplotlib in a jupyter notebook. I just decided to write everything to an mp4 file, which works just as good for animations.

My code for this including the MVC example:

import random
import networkx as nx
import matplotlib.pyplot as plt
import numpy as np
import math
import pandas as pd
from matplotlib import animation
#from JSAnimation import IPython_display
%matplotlib inline

# initialise graph object
G = nx.Graph()
color_map =[]
G.add_node(1, pos=(1, 0)); color_map.append('r')
G.add_node(2, pos=(2, 0)); color_map.append('r')
G.add_node(3, pos=(3, -1)); color_map.append('r')
G.add_node(4, pos=(3, 1)); color_map.append('r')
G.add_node(5, pos=(4, -1)) ;color_map.append('r')
G.add_node(6, pos=(4, 1)); color_map.append('r')
G.add_node(7, pos=(5, 0)); color_map.append('r')
G.add_node(8, pos=(6, 0));  color_map.append('r')

e = [(1, 2, 1), 
    (2, 3, 1), 
    (2, 4, 2), 
    (3, 5, 5), 
    (4, 6, 2), 
    (5, 7, 1), 
    (6, 7, 2), 
    (7, 8, 1)]

G.add_weighted_edges_from(e)
labels = nx.get_edge_attributes(G,'weight')

nx.draw(G,nx.get_node_attributes(G, 'pos'))
nx.draw_networkx_edge_labels(G,nx.get_node_attributes(G, 'pos'),edge_labels=labels)
nx.draw_networkx_labels(G,nx.get_node_attributes(G, 'pos'))

df_x = pd.DataFrame(data=
    np.array(
    [[np.NaN, np.NaN, np.NaN, np.NaN], 
     [1, np.nan, np.NaN,np.NaN], 
     [1.5, 4, np.NaN,np.NaN], 
     [2, 5, 3,4]]
        ), index= [1, 2, 3, 4], columns=[1, 2, 3, 4])
print(df_x)

df_y = pd.DataFrame(data=np.array(
    [[np.NaN, np.NaN, np.NaN, np.NaN], 
     [0, np.nan, np.NaN,np.NaN], 
     [0, -1, np.NaN,np.NaN], 
     [0, 0, 1,1]]
    ), index= [1, 2, 3, 4], columns=[1, 2, 3, 4])

def get_data_x(i):
    return df_x.loc[i]

def get_data_y(i):
    return sdf_y.loc[i]

def animate(i):
    x = get_data_x(i)
    y= get_data_y(i)   
    graph.set_data(x,y)
    return graph,

# Set up formatting for the movie files
Writer = animation.writers['ffmpeg']
writer = Writer(fps=15, metadata=dict(artist='Me'), bitrate=1800)

fig = plt.figure(figsize=(20,20))
ax = plt.axes()

nx.draw(G,nx.get_node_attributes(G, 'pos'),node_size = 1)

n_steps = df_x.index 

graph, = plt.plot([],[],'o')

ani = animation.FuncAnimation(fig, animate, frames= n_steps, interval=1, repeat=True, blit = True)
ani.save('path/file.mp4', writer=writer)
Jeroen123
  • 55
  • 8