GIT LFS tracking too many files

Unreal Courses Ask
,
#1

Hello everyone,

I have an issue with GIT LFS: my project got to a point in which I’m tracking 1.5GB of LFS data. I started investigating this issue and it looks like once you add a file into LFS it will always stay there, so my guess is that I have been tracking the same files over and over again in different directories?

To solve the issue I deleted my remote repository (by following this guide ), so right now I re-created my repository but when I try to push (I already ignored the files from my dynamic folder on LFS) I still try to push 1.5GB of data into my remote!

Is there a way to clean out the whole LFS tracking from my local branch and push only the files I need to track?, GitHub says I cannot use more than 1GB of data (unless I pay), tried GitLab but had a ton of errors while trying to push 1.5GB.

Please help :slight_smile:, do you know why I’m tracking 1.5GB of LFS data?

Thanks in advance.

Edit: I will try to run git lfs uninstall once I get back home and reinstall GIT LFS, maybe that will clean the whole LFS history and track the current static folder

#2

@sampattuzzi perhaps you could help out with this.

1 Like
#3

I recommend you start a new repository with the dynamic/static split. This way you shouldn’t be tracking as many trackable assets. You will still have the history in the other repo if you need it.

1 Like
#4

Ok so, now I went through a lot of pain to try to get this working, LFS is a nightmare if you don’t know what you are doing and I still haven’t fixed the issue.

So I managed to use BFG to cleanup the whole repository doing this:
(First of all you need to download BFG from the link above, it’s on the right side, call it from the location you place it in, in my case I renamed it “bfg.jar” and placed it in my repo’s root folder, where the .git is located)

  • java -jar bfg.jar --delete-files .umap “D:\Projects\UnrealProjects\MasterOfShadows”
  • java -jar bfg.jar --delete-files .uasset “D:\Projects\UnrealProjects\MasterOfShadows”
  • git reflog expire --expire=now --all && git gc --prune=now --aggressive

And after that last call to call the garbage collector, I tried to re-track all my files:

(deleted my .git/lfs folder and .gitattributes file)

  • git lfs uninstall
  • git lfs install
  • git lfs track “Content/Static/**”
  • git rm --cached -r .
  • git add .

After this I thought that my tracked files would be all good now (which they do seem to be ok) but I noticed that on my .git/lfs folder I track some empty folders, and after trying to push I get this error:

Trying to push "Content/Static/Geometry/Meshes/1M_Cube.uasset" with OID f471a994148db424160e6589509afbe29b9205f5db07ab4c231811aefccff955. Not found in D:\Projects\UnrealProjects\MasterOfShadows\.git\lfs\objects\f4\71.

Which indeed is not on that folder, so re-tracking is not really working, looks like the file in LFS is keeping it’s old OID instead of generating a new one.

I do keep a backup but my .git/lfs folder has a size of 2.55 GB, compared to my new one that is just 97 MB so I could say that I’m indeed duplicating a lot of unnecessary assets.

Any help on how to re-track my files properly would be appreciated, I don’t think that starting a new repository as you suggested would be of any help since the tracking remains on the .git/lfs, or can you explain what you mean by creating a new repository? (do you mean that I should create a backup -> delete .git repo and init a new one?)

Sorry for the long text, I just wanted to share my issues with the rest of the group in case anyone else happens to have the same issue.

Can no longer push to my GitHub repo
#5

After spending the last two days trying to understand LFS and how it works I just wanted to say that I feel like it’s not worth it, LFS is very complicated and can cause tons of issues when moving files around.

I will leave some of the questions I asked and resources I read here in case anyone wants to take a look at them, I ended up contacting Github to ask them to re-enable LFS on my repository and now I’m tracking only the static content, I had to restart my git repo and lose all the history.

I tried cleaning up the project using their recommended approach but that tutorial didn’t work smoothly for me, if anyone else manages to go through the same issue and fixes it please let me know.

1 Like
#6

I think you went for the simplest approach by restarting the repo. Unfortunately, LFS is still a fairly early project (compared with Git itself). I think in trying to hide the complexity for the simple cases it makes it much harder in the corner cases.

#7

Yes, I still find it insane that I ended up with more than 1.5GB of data when I just have around 100MB to track, and they are charging money on GitHub for it.

#8

Is that because you change the static assets frequently? You could always try storing the LFS files somewhere cheaper. I believe they have some servers that are backed by Amazon S3 storage. But this might be overkill. Doesn’t sound like you have that much to store. You could alway just store it in git without LFS.

#9

I refactored my project one or two times before we ended up moving our files to the static folder, I didn’t add or remove any assets along the way, and my lfs/objects folder got to 2.43GB!

Could it be that retracking the files to LFS adds more space to that folder?

#10

Very possible that LFS doesn’t recognise the moves and just sees it as you adding the files again.

#11

Hello Sam
I split my folders in DYNAMIC and STATIC parts as you did and delete unwanted empty folders I did the same in CD. but when I open Unreal Editor again delete unwanted folder was returned with content and now I have duplicate folders. this is more complicate unwanted files is depend on DYNAMIC and STATIC files.
and now I am afraid if I force delete unwanted folders and files, then regenerated project it may be crashed !!!
can you advise me something ??

#12

Did you move and delete inside the editor. If so, try removing the intermediates, etc. and rebuilding the project

#13

I force delete they, fortunately the project don’t crash.
As Dan told me the problem reason was that I didn’t push my commit in GitHub, but push was impossible because there was problem of memory location LFS. and then Dan advise me to import my project into GitLab, but no reason I could not import it and could not upload directly.
at last I remove remote repository and now I use only SourceTree
if you have any idea how to connect my project to GitHub or GitLab please write me.
My project’s memory in CD is 35GB I don’t why its so big, I have a few more Texture files than you?
Such size is the problem for GitLab?

#14

Yes, that’s huge. Use a utility to figure out where all your space is getting used up.

#15

what do you mean “Utility” can I compress my file and such way it’s use less memory in CD ?
If it’s possible please explain how can i do it

#16

If you’re on Windows use something like WinDirStat to see which files are large and you can try to reduce the size of your project by eliminating large files.

#17

Privacy & Terms