Speeding up pandas profiling analysis using check_correlation?

Question

Using pandas profiling to generate a report. the size of the dataset is very large to speed up the processing im trying to turn off correlations so i used check_correlations from another post I saw, ValueError: Config parameter "check_correlation" does not exist. is then the issue I get from using this line

a = prof.ProfileReport(df, title='Downloads', check_correlation=False)

which generates this issue of

ValueError: Config parameter "check_correlation" does not exist.

prof = pandas_profiling ? – Mayeul sgc Oct 09 '19 at 08:11 — Mayeul sgc, Oct 09 '19 at 08:11

Levent · Accepted Answer · 2019-10-10T13:51:16.933

Since they have changed the configurations on version 2, you could use it as:

import pandas_profiling

profile = df.profile_report(check_correlation_pearson=False,
correlations={'pearson': False,
'spearman': False,
'kendall': False,
'phi_k': False,
'cramers': False,
'recoded': False})

to turn off correlations. However, it is still not as fast as version 1.4. You could also investigate other configurations here.

score 4 · Answer 2 · answered Aug 27 '20 at 00:47

4

This way didn't work for me and I used:

a = prof.ProfileReport(df, title='Downloads', minimal=True)

answered Aug 27 '20 at 00:47

Romeu Fronzaroli

129
5

score 0 · Answer 3 · answered Oct 09 '19 at 08:53

0

Please, see this issue in pandas-profiling project.

answered Oct 09 '19 at 08:53

knagaev

2,897
16
20

score 0 · Answer 4 · answered Jan 02 '23 at 19:51

As of version 3.6+ you can do this:

profile = df.profile_report(
    title="Report without correlations",
    correlations={
        "auto": {"calculate": False},
        "pearson": {"calculate": False},
        "spearman": {"calculate": False},
        "kendall": {"calculate": False},
        "phi_k": {"calculate": False},
        "cramers": {"calculate": False},
    },
)

# or using a shorthand that is available for correlations
profile = df.profile_report(
    title="Report without correlations",
    correlations=None,
)

Speeding up pandas profiling analysis using check_correlation?

4 Answers4