Direct dependency vs Transitive dependency in build system

Question

I was studying Maven's build system and it adds a lot of transitive dependencies because of its transitive dependency system (maven itself does not add dependencies but transitive dependency system does). I see issues with it like major version conflicts and unknown dependencies coming in.

I was thinking why is the system designed this way and why not take direct dependencies. My library does not need to depend on something which my dependency is using but not my library (I mean I understand why it needs to be included in the build list, my dependencies need to build using those, but why does it needs to cause major version conflict?). Am I missing something fundamental here? One thing that I can think of is that my library's build dependency list can grow to be very big because of all the direct dependencies I will need to take, but that does seem to be as big of a problem as problems with transitive dependency system.

I am new to build systems, so please don't be too harsh. I also tried to google this question but didn't find useful answers but please feel free to comment anything that I might have missed.

Thanks

Answer 1

If you need library A to run, and library A needs library B to run, and this needs C to run, it is very tedious to figure this out and add all the relevant dependencies to your project.

Before Maven and Gradle, many people worked that way and found out, that it is much easier to let a build tool figure out the transitive dependencies.

Answer 2

My library does not need to depend on something which my dependency is using but not my library [...]

This is your major misconception. There are two possibilities:

1. The direct dependency of your library exposes types from the transitive dependency in its public API. To use this public API you need to access these types, so you need the transitive dependencies during compile time.

2. The direct dependency of your library only uses its own dependency internally, but not in its public API. In this case, your library does not need to depend on the transitive dependency during compile time. But as soon as your library code runs (even in a test), it may use some functionality of its direct dependency that internally uses functionality of the transitive dependency, causing your library code to fail.

[...] I mean I understand why it needs to be included in the build list, my dependencies need to build using those [...]

There is no actual build list (or order) for external dependencies, because they are used when they are already built (the downloaded .jar files contain compiled .class files). But as I mentioned above, you will need the transitive dependencies either during compile time or during runtime (e.g. tests), so your build system (Maven or Gradle) will fetch them for you.

[...] but why does it needs to cause major version conflict?

@khmarbaise already explained in his comment, why and how version conflict between transitive dependencies may occur:

You are using two libs X and Y. Both of them using another lib (A) So X is using A in version 1.0.0 but Y is using A in version 2.0.0. In the end you can't have both on the classpath there must be done a decision for one version. So depending on how X,Y are implemented either X can fail while using A in V1.0.0 or Y can fail in using A in V1.0.0 or with V2.0.0 the same... This can happen if X or Y are being updated. This is also true for different version combinations like A in 1.0.0 and 1.1.0 (if compatibility is not 100%)