1

If a project depends on some libraries, there are basically two options available:

  1. Locate the libraries in a well known global path (like /usr/src/linux-headers-4.13.0-1-amd64/
  2. Copy the sources into the project folder.

Using the global location approach takes minimum space and time (download time). If the dependencies are well versioned (tagged, like v3.1, v3.1.1, etc...), this option just works well.

If our project needs to use the latest commit of the library, versioning is not an option. If we just pull the latest commit into the well known location and let our project use the library, chances are we will be unable to compile our project 6 months later, which is unacceptable.

If we add the dependency as a subproject into our project, we will always be able to compile our project. This is the safest approach. Problem is that if the library is around 100MB, copying sources into the project folder is just a waste of disk usage, download time etc...

How do the people handle this problem?

ceremcem
  • 3,900
  • 4
  • 28
  • 66

2 Answers2

1

You can't have your cake and eat it:

  • either you are going to rely on a stable, versioned library, then reference that (whereever it is located)

  • or you are going to rely always on the "latest and the greatest" - then reference a copy of that

Disk space is not really an issue - obviously you need to have at least one local copy of the lib you are using once on your developer machine, it does not make a difference if it is a "proper versioned" code or "head of trunk". And when it is always "head of trunk", you will probably use some version control system like git or svn to get the library, so whenever you update your local copy, you will pull only the changes from the repo, not the "full 100 MB".

However, to make your builds reproducible, with each version of your source code, you should version all dependencies as well together with your source code. If you are using third party libs, you can either

  • rely on the vendor providing you with exactly the versions you used during the project, and expect them to provide you with these versions for the next 10 years or so

  • or, keep your own copy of each version of the third party libs you used (probably put them in your own source code repo as well). That is the approach I always prefer for professional development, who knows if the library vendor is maintaining his stuff still next week, next month, next year?

Doc Brown
  • 19,739
  • 7
  • 52
  • 88
  • So this is a little bit the dependency provider's call, as he/she prefers to include all its dependencies as "copy/paste" manner where we have no chance to fetch only a portion of that library. For example, ChibiOS (a RTOS library for microcontrollers) includes all CMSIS libraries at once, so I have to include all of it (150MB in total) even though I would need only 1MB of the code. Preparing an appropriate mirror in this case seems the way to go where the library is splitted accordingly. – ceremcem Dec 15 '17 at 13:51
  • @ceremcem: of course it is the dependency providers call - it makes surely a difference if they provide you with a new release each day, or a new release once a year. But it is also your call how often you think you need to update to the latest version. Just because you could get a new version every day, it does not necessarily mean you need to update more than once a year. – Doc Brown Dec 15 '17 at 14:25
  • Thinking again, the problem resides that we don't store the exact commit hash of the dependencies we compiled our code against to. Another solution might be taking an automatic note to somewhere in the project (eg. `./dependencies.txt`) on compilation that declares which commit hash of what dependency the project was compiled against to. That information might be used as a "safe point to return" and if project rejects to compile at some point in the future, we can checkout the dependency/dependencies to that hash. – ceremcem Jan 03 '18 at 12:00
  • @ceremcem: though you did not mention it, it seems you are using git, I guess? And where are the libraries you are talking of? On Github? Please clarify. – Doc Brown Jan 03 '18 at 12:28
  • ... and if the answer is "yes" and "yes", please look [here](https://softwareengineering.stackexchange.com/questions/110093/how-would-one-handle-external-dependencies-in-an-open-source-project), Git submodules may be what you are looking for. – Doc Brown Jan 03 '18 at 12:37
  • I'm using git and fan of git submodules already. The dependency I referred is also using git, so I may add it as a submodule, that's possible. The problem is that library is 150MB in size and it won't be wise to add 150MB library into every 100KB project. – ceremcem Jan 03 '18 at 14:31
  • @ceremcem: so you are looking for git specific advice - you should have stated this in first place, not everone uses git, you know? Maybe https://stackoverflow.com/questions/10914022/how-to-check-out-specific-version-of-a-submodule-using-git-submodule contains what you are looking for? – Doc Brown Jan 03 '18 at 14:49
0

Here is how you can both have and eat your cake at the same time (see quantum physics):

  1. Maintain your dependencies (libraries) in global locations with their own distributed version control system (git, hg, bazaar, etc.). (See note 1)
  2. Maintain a dependency version database (a text file) inside the project folder and update it on each build. (Format: MyLibrary commit-hash-or-version)

Project will always use the latest version of the dependencies. If compilation fails at some point (when you try to compile the project 3 years later, for example),

  1. Go and get which version of a dependency was used at the last successful build point
  2. Create a checkout of the dependency with that hash in a temporary location.
  3. Point the temporary location as your dependency path in your build process.

Obviously this "return-to-successful-point" process can be easily automated.

Note 1: Dependencies are not meant to be only source codes, they might be simply some tools in binary format. In this case the dependency manager hook should invoke the tool with --version parameter (or what is appropriate for that tool), get the version, append to the dependencies.txt file. Obviously you should be prepared to be able to get the older version of the tool in case you'll need it at some point in the future.

ceremcem
  • 3,900
  • 4
  • 28
  • 66