104

What is "vendoring" exactly? How would you define this term?

Does it mean the same thing in different programming languages? Conceptually speaking, not looking at the exact implementation.

einpoklum
  • 118,144
  • 57
  • 340
  • 684
Niels Bom
  • 8,728
  • 11
  • 46
  • 62
  • 2
    I've found it being used in Python, PHP and the Ruby worlds. But it's not a very well-known concept. That's why I'm asking the question :-) – Niels Bom Oct 06 '14 at 13:47
  • 2
    The fact that I haven't seen it used that much also contributes to my incomplete understanding of the concept. – Niels Bom Oct 06 '14 at 14:46

5 Answers5

85

Based on this answer

Defined here for Go as:

Vendoring is the act of making your own copy of the 3rd party packages your project is using. Those copies are traditionally placed inside each project and then saved in the project repository.

The context of this answer is in the Go language, but the concept still applies.

Community
  • 1
  • 1
17xande
  • 2,430
  • 1
  • 24
  • 33
  • 35
    It is worth adding that vendoring is the conceptual opposite of using a dependency manager – konstin Jan 20 '18 at 22:43
  • It is also worth adding that a vendor's package is by definition not free. A good example is Fullcalendar (excellent piece of software, not very expensive). – Brice Coustillas Sep 27 '19 at 10:14
  • 2
    One reason for vendoring is to avoid version conflicts ([longer explanation](https://medium.com/plain-and-simple/dependency-vendoring-dd765be75655)) – Martin Thoma Jan 03 '21 at 23:13
38

If your app depends on certain third-party code to be available you could declare a dependency and let your build system install the dependency for you.

If however the source of the third-party code is not very stable you could "vendor" that code. You take the third-party code and add it to your application in a more or less isolated way. If you take this isolation seriously you should "release" this code internally to your organization/working environment.

Another reason for vendoring is if you want to use certain third-party code but you want to change it a little bit (a fork in other words). You can copy the code, change it, release it internally and then let your build system install this piece of code.

Niels Bom
  • 8,728
  • 11
  • 46
  • 62
  • 3
    I've read this a couple of times now, still not sure I understand what *vendoring* might be. Surely it must be more than establishing a dependency on a fork of someone else's code or there wouldn't be a strange new word for *it*, whatever *it* might be. But please, don't rewrite your definition on my account, I know enough words already and suspect that I don't need to learn what *vendoring* means, if anything. – High Performance Mark Oct 06 '14 at 13:56
  • 2
    I think "establishing a dependency on a fork of someone else's code" comes quite close to what I'm saying, so thanks for that. However I have the feeling it's a little more than that, it's also "building/releasing your own version of someone else's code" for a specific use. – Niels Bom Oct 06 '14 at 14:45
  • 10
    I would like to add my understanding of the etymology of this term: the ‘vendor’ in this case is the creator of the software (presumably you, since we're on Stack Overflow); and you're shipping a ‘vendor’-selected version of a dependency (as opposed to a user-selected, or more likely, user's-system-selected, version). The term has been around a *very* long time, to the best of my knowledge. – ELLIOTTCABLE Jan 26 '16 at 20:37
  • Uhm... wouldn't it be better to fork a repository of our choice and use that as a dependency, instead of including the whole code of this dependency inside another project? – Ricky Robinson Sep 22 '21 at 14:29
  • @RickyRobinson you're operating under the assumption "one app == one repository". Many developers use multiple repositories to break down their applications into a core part and libraries/plugins. And then the difference between an external and an internal dependency isn't that big. – Niels Bom Sep 23 '21 at 07:21
  • "wouldn't it be better to fork a repository of our choice and use that as a dependency" it depends on how flexible is your dependency manager. – Luiz Felipe Apr 09 '23 at 15:17
12

Vendoring means putting a dependency into you project folder (vs. depending on it globally) AND committing it to the repo.

For example, running cp /usr/local/bin/node ~/yourproject/vendor/node & committing it to the repo would "vendor" the Node.js binary – all devs on the project would use this exact version. This is not commonly done for node itself but e.g. Yarn 2 ("Berry") is used like this (and only like this; they don't even install the binary globally).

The committing act is important. As an example, node_modules are already installed in your project but only committing them makes them "vendored". Almost nobody does that for node_modules but e.g. PnP + Zero Installs of Yarn 2 are actually built around vendoring – you commit .yarn/cache with many ZIP files into the repo.

"Vendoring" inherently brings tradeoffs between repo size (longer clone times, more data transferred, local storage requirements etc.) and reliability / reproducibility of installs.

Borek Bernard
  • 50,745
  • 59
  • 165
  • 240
2

As far as I know the term comes from Ruby on Rails.

It describes a convention to keep a snapshot of the full set of dependencies in source control, in directories that contain package name and version number.

The earliest occurrence of vendor as a verb I found is the vendor everything post on err the blog (2007, a bit before the author co-founded GitHub). That post explains the motivation and how to add dependencies. As far as I understand the code and commands, there was no special tool support for calling the directory vendor at that time (patches and code snippets were floating around).

The err blog post links to earlier ones with the same convention, like this fairly minimal way to add vendor subdirectories to the Rails import path (2006).

Earlier articles referenced from the err blog, like this one (2005), seemed to use the lib directory, which didn't make the distinction between own code and untouched snapshots of dependencies.

The goal of vendoring is more reproducibility, better deployment, the kind of things people currently use containers for; as well as better transparency through source control.

Other languages seem to have picked up the concept as is; one related concept is lockfiles, which define the same set of dependencies in a more compact form, involving hashes and remote package repositories. Lockfiles can be used to recreate the vendor directory and detect any alterations. The lockfile concept may have come from the Ruby gems community, but don't quote me on that.

The solution we’ve come up with is to throw every Ruby dependency in vendor. Everything. Savvy? Everyone is always on the same page: we don’t have to worry about who has what version of which gem. (we know) We don’t have to worry about getting everyone to update a gem. (we just do it once) We don’t have to worry about breaking the build with our libraries. […]

The goal here is simple: always get everyone, especially your production environment, on the same page. You don’t want to guess at which gems everyone does and does not have. Right.

There’s another point lurking subtlety in the background: once all your gems are under version control, you can (probably) get your app up and running at any point of its existence without fuss. You can also see, quite easily, which versions of what gems you were using when. A real history.

Tobu
  • 24,771
  • 4
  • 91
  • 98
  • The downside of vendoring without the concept of a lockfile is another kind of dependency hell, which is experienced by C++ programmers that usually "copy n paste" vendor code because dependency management in C++ is basically broken (yeah, I know about vcpkg, I still vendor my vcpkg ports, ironically, even thou it does have a lockfile of sorts) – Luiz Felipe Apr 09 '23 at 15:02
1

Summarizing other, (too?) long answers:

Vendoring is hard-coding the often forked version of a dependency.

This typically involves static linking or some other copy but it doesn't have to.

Right or wrong, the term "hard-coding" has an old and bad reputation. So you won't find it near projects openly vendoring, however I can't think of a more accurate term.

MarcH
  • 18,738
  • 1
  • 30
  • 25
  • "bad reputation" is a stupid concept. In engineering there are only trade-offs. Sometimes hard-coding a core dependency of your project is the way to go, because everything's on top of that load bearing part, and it must not fail because of external environment. If that means if its outdated, or has a security bug, it won't have its rug pulled by an external (probably automated) sysadmin tool that goes updating things nilly-willy. Because reliability is more important, and if there's a security flaw, you can update the vendor and the continuous delivery system will update it either way. – Luiz Felipe Apr 09 '23 at 15:12
  • Its about control of changes, you don't suddenly update things just because a security vulnerability was reported, its like removing a wooden pillar because someone reported its flammable and can catch fire, if you remove it now the ceiling falls over your head, doesn't matter how flammable the pillar is. It doesn't matter how secure your app is if it users can't use it because the service is down. – Luiz Felipe Apr 09 '23 at 15:13
  • 1
    Dependency management is indeed a very complicated topic and simplifications like "removing a wooden pillar" are stu... not helpful. https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527 If you're lucky and a very specific and focused update targeting only a severe vulnerability is available then yes of course you "suddenly" update. But of course you're not always that lucky, life isn't always that simple so yes there are always trade-offs. – MarcH Apr 10 '23 at 16:33