files don't exist yet "this exceeds GitHub's file size limit of 100.00 MB"

Question

I am trying to git add, commit, push an update made to some Python code, where I changed the naming convention of the files.

NB: I want my local branch to replace the remote version.

I have also deleted these files from data/ folder. However, git push and git push --force yield the same error:

remote: error: File workers/compositekey_worker/compositekey/data/20210617-031807_dataset_.csv is 203.87 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: File workers/compositekey_worker/compositekey/data/20210617-032600_dataset_.csv is 180.20 MB; this exceeds GitHub's file size limit of 100.00 MB

But data/ only contains example datasets from online:

$ ls
MFG10YearTerminationData.csv  OPIC-scraped-portfolio-public.csv

Is the problem to do with caching? I have limited understanding of this.

git status:

On branch simulate-data-tests
Your branch is ahead of 'origin/simulate-data-tests' by 6 commits.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean

git rm --cached 20210617-031807_dataset_.csv:

fatal: pathspec '20210617-031807_dataset_.csv' did not match any files

git log -- <filename> in data/:

$ git log -- 20210617-031807_dataset_.csv
commit 309e1c192387abc43d8e23f378fbb7ade45d9d3d
Author: ***
Date:   Thu Jun 17 03:28:26 2021 +0100

    Exception Handling of Faker methods that do not append to Dataframes. Less code, unqiueness enforced by 'faker.unique.<method>()'

commit 959aa02cdc5ea562e7d9af0c52db1ee81a5912a2
Author: ***
Date:   Thu Jun 17 03:21:23 2021 +0100

    Exception Handling of Faker methods that do not append to Dataframes. Less code, unqiueness enforced by 'faker.unique.<method>()'

You're got two possibly independent questions here. 1) files too large: although you don't see them with ls, they may still exist in your commit history. 2) rejected push: as the message says, you need to run `git fetch` first, and merge or rebase you local changes to where that branch is on remote right now. — joanis, Jun 17 '21 at 12:09
To see the commit history, do `git log --stat` and look for those files in older commits. You can also do `git log -- `, that will show you the commits that exist for ``, even if it's been deleted since. — joanis, Jun 17 '21 at 12:11
As for the rejected push because you need to fetch, that question has been asked and answered before, probably many times, but see https://stackoverflow.com/q/4684352/3216427 — joanis, Jun 17 '21 at 12:13
@joanis I want my local to completely overwrite my remote branch. How do I enforce this? — StressedBoi69420, Jun 17 '21 at 12:17
Short answer: add `--force` to your push command. Long answer: that's often not a good idea, because it removes stuff that's been pushed before, and if others have fetched it and worked from it, they'll hate you, but if you're sure, `--force` is how you do it. (PS: that info is in the link I shared above.) — joanis, Jun 17 '21 at 12:19
This is a safe option for me as I am the only person using this branch. Worst case, I have these files of interest in personal storage anyway. Will report back — StressedBoi69420, Jun 17 '21 at 12:21
Which one, the file too large or the rejected push? If it's the file too large, did you try the log commands I suggested? What did you find out? — joanis, Jun 17 '21 at 12:37
@LovekeshKumar `git: 'lfs' is not a git command. See 'git --help'.` — StressedBoi69420, Jun 17 '21 at 12:40
They are non-existing in your current commit, but do they exist in the history? When you push, the history will get pushed too, not just the latest snapshot. — joanis, Jun 17 '21 at 12:43
Oh ok. So how do I make sure what I currently have in my local branch gets pushed to replace remote? @joanis — StressedBoi69420, Jun 17 '21 at 12:49
Help me help you, dude! Do the large files exist in your history or not? How to fix things depends on the answer to that question! If they exist in the history, you need to rewrite the history to remove them. — joanis, Jun 17 '21 at 12:53
OK, so indeed, the history has the file. you need to rewrite the history to remove it. Two tools are commonly used for this, filter-repo and filter-branch. See the answers here: https://stackoverflow.com/q/43762338/3216427 for details. — joanis, Jun 17 '21 at 13:10
@joanis I tried `git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch " HEAD` on both files. Yet `git log -- ` still shows the same details in my post. — StressedBoi69420, Jun 17 '21 at 13:26
@joanis I've solved the problem, but by using GitHub online. I will explain in answer — StressedBoi69420, Jun 17 '21 at 13:38
@joanis Thank you very much for your efforts. I've no learnt about caching, history, and more on commits today as well as several commands. — StressedBoi69420, Jun 17 '21 at 13:47

score 0 · Answer 1 · answered Jun 17 '21 at 13:44

A bit of a round about way but works for this situation effectively.

If you are sure that you want your local branch files to be in your remote branch; and have been experiencing these issues of once committed but since deleted files.

On GitHub Online, go to your folder, select you're branch.

Then "Add file" > "Upload files" file manually that you initially wanted pushed.

Then on your machine:

git checkout master

git branch -d local_branch_name

git fetch --all

I was successfully able to make a git push thereafter.

files don't exist yet "this exceeds GitHub's file size limit of 100.00 MB"

1 Answers1