1

I'm working on committing a project I have been working on for awhile that we have not yet uploaded to GitHub. Most of it is Python Pandas where we are doing all our ETL work and saving to CSV's and Pickle files to then use in creating dashboards/running metrics on our data.

We are running into some issues with version control without using GitHub so want to get on top of that. I don't need version control on our CSV or Pickle files, but I can't change the file paths or everything will break. When I try to initially commit to the repo it won't let me because our pickle and CSV files are too big. Is there a way for me to commit the project and not upload the whole CSV/pickle files (the largest is ~10 GB).

I have this in my gitignore file, but still not letting me get around it. Thanks for any and all help!

*.csv
*.pickle
*.pyc
*.json
*.txt
__pycache__/MyScripts.cpython-38.pyc
.Git
.vscode/settings.json
*.pm
*.e2x
*.vim
*.dict
*.pl
*.xlsx
Tyler Kraupp
  • 21
  • 3
  • 1
    If you have *already committed* the large files, they're *already in commits*. Since commits are literally read-only—no commit can *ever be changed*, not even by Git itself—they will remain in those commits as long as those commits live. Commits live forever *by default* but *that* can be changed: see instructions for removing large files. – torek Nov 10 '21 at 15:33
  • You said you don't need to version the large files, but if you decide that you *do* need to version them, you can use the Git-LFS "wrapper" software around Git, which secretly hides the large files away from Git so that Git doesn't "see" them, then secretly provides the large files to *you* when you use Git-LFS instead of Git. I'm not a huge fan of Git-LFS but it does solve a real problem with versioning large files, if you decide that you *do* need to do that. – torek Nov 10 '21 at 15:35
  • Otherwise, see, e.g., [How to remove/delete a large file from commit history in the Git repository?](https://stackoverflow.com/q/2100907/1256452) – torek Nov 10 '21 at 15:36
  • @torek thanks! Looks like that will work. Looked into LFS and looks like it will be a great solution for what we need to do. Thanks! – Tyler Kraupp Nov 11 '21 at 16:42

0 Answers0