1

I would like to make a simple VCS for my graphic and asset files I use in my code; I tried GIT but our company is pretty small so we have limitations about what can be stored on those servers.

Have been looking around to see if I can write my own, but I never did such thing so I think it is much faster to use an existing product; and I assume there are other developers that get in a situation where they can't just upload gb and gb of data that is not code, and is looking for better solutions.

All I need fundamentally is to save versions of the files, associate metadata to each file and be able to upload and download those files on a server space (probably a NAS or a simple linux machine that host a ton of mechanical hard drives). I don't need much more than this to get started, and as far as files, I have zip files containing assets for my project (scientific test data and renders of math models; they can be either 3d format or 2d format as plot or plain images).

Any guidance would be appreciated, I am currently using Python, so anything I can write that would be fitting my requirements, or anything I can use via Python, is what I have in mind as goal.

rataplan
  • 149
  • 4

1 Answers1

1

I use Subversion on a Ubuntu 20.04 64 bit OS at home. I have Apache web server running with the webDAV setup and that allows me to use Subversion through a browser or use http to access my subversion repository from a Windows 10 machine using Visual Studio and the ankh subversion plug-in.

Works great. There are other possibilities such as Mercurial. Really kind of depends on your goals and the organization.

I went this way after doing something similar for a small team at a university running Ubuntu with Subversion and Apache on a Linux virtual machine on the university network for two different teams on two different projects with one team of some three people and the other team with about 5 people.

See enabling Subversion access via Apache web server and DAV on Ubuntu

I see mentions of an open source version control system Boar that looks to be in python and handles large binary files. The source seems to be on GitHub and they would probably be happy to have help. See https://github.com/mekberg/boar

For binary files see the following:

Do version control systems use diffs to store binary files?

How do different version control systems handle binary files?

Version control for large binary files and >1TB repositories?

Binary Delta Storage

See also Git FAQ: Handling Large Binary Files with LFS

Large binary files are a tough problem for every version control system: every little change to a large binary file will add the complete (large) file to the repository once more. This results in huge repository sizes very quickly.

The "Large File Storage" extension for Git deals with exactly this problem.

You can also set up your own Git server. See How to Set up the HTTP Git Server for Private Projects

One possibility is the HTTP Git Server. This open source project uses NGINX to serve up Git repositories over your Local Area Network (LAN). HTTP Git Server is surprisingly easy to setup and manage.

I’m going to walk you through the process of installing and configuring HTTP Git Server on Ubuntu 18.04. Once complete, you’ll have a repository that anyone on your LAN can use.

Richard Chambers
  • 16,643
  • 4
  • 81
  • 106
  • Thanks; do you get any issue with large binary files? I was told by our devop person that none of the big names like git, svn or similar are suited for binary files; which is what prompted me to start exploring other venues. – rataplan Feb 11 '21 at 04:34
  • 1
    @rataplan since most of the version control systems are intended for source code or other text and store the various versions of a file as deltas or differences rather than the complete text file they don't do such a good job with binary files which these systems would just store as a whole rather than deltas. I'll update my post with some additional info. – Richard Chambers Feb 11 '21 at 04:40
  • In How to Answer, see the section Answer Well-Asked Questions, and the final bullet point within. Tool recommendation requests are explicitly off-topic. – Charles Duffy Feb 11 '21 at 05:47
  • @CharlesDuffy I originally read this as a "how do I make a simple version control system using python" and that was the answer I was going to give. However the first thought that came to mind was to describe my simple setup as an alternative rather than trying to roll his own.. Then as I looked further I found additional SO posts on the subject of binary files version control and added that material to indicate possible problems. Then ended with the Boar GitHub where someone had already done something similar. I did not plan to start with recommendations, it just ended that way. – Richard Chambers Feb 11 '21 at 14:37
  • Thanks @RichardChambers for the extensive reply; sadly on SO anything that ask something that is not code, end up being considered "suggestions". Then you can browse on SO posts and see that most of the time it depends from who read the questions, and if they are in a bad mood, they mark it as off topic. I am glad your reply ended up posted, since it is very helpful; and I don't really care if the question is closed... This is the byproduct of having a site handled by people, which read rules as they see fit. Cheers – rataplan Feb 12 '21 at 00:37