21

The way my team's project is developed, we generate a Shared Object library for our application from all all of our .o object files. My task (hopefully it is specific enough but also general enough to be of use to others!) is to link in only the object files that have changed since the last time the executable was created. For example, here is the command line that i use to build the .so:

g++34 -shared -rdynamic -m64 -Wl,-rpath,'$ORIGIN' MyObject1.o MyObject2.o MyObject3.o MyObject4.o   -o libMySharedLibrary.so

Which works as expected! :) My goal is to be able to link in only the changed object files from now on, to speed up the concurrent linking process. An example command would be:

g++34 -shared -rdynamic -m64 -Wl,-rpath,'$ORIGIN' MyObject1.o MyObject3.o   -o libMySharedLibrary.so

Which would update libMySharedLibrary.so with the newer object files, while maintaining the older object files in libMySharedLibrary.so also. In actuality, when I generate libMySharedLibrary.so using the above command, the file-size is much smaller than that of when all object files are included, so I can almost be sure that the above command isn't doing what I want.

Through my research I've found that there is a -i option for the linker which is the same as the -r option, which appears to just combine all the object files into one large object file as well. Unfortunately it does not appear that this is what I want.

In short, I'd like to link in only the changed object files after the initial link, resulting in quicker linking process for the future links. Is there a way to do this?

EDIT: An example of what I've tried with -i/-r:

Example command: g++34 -Wl,-r -nostdlib -rdynamic -m64 -Wl,-rpath,'$ORIGIN' MyObject1.o MyObject2.o MyObject3.o MyObject4.o -o AllMyObjects.o

I had to add the -nostdlib tag to stop it from yelling at me about needing it, and removed -shared because shared objects are not allowed with the -r tag.

This command would appear to slam all of my .o files into one big .o file. So If I could just update that .o file from here on out with only the changed .o files, that would be great. After AllMyObjects.o was initially created, I tried this command: g++34 -Wl,-r -nostdlib -rdynamic -m64 -Wl,-rpath,'$ORIGIN' MyObject1.o MyObject3.o -o AllMyObjects.o, but it would also create a much smaller (file-size-wise) AllMyObjects.o, so I'm assuming it can't possibly have all of the object files. I feel like this is something that I'm likely making a small mistake on. Anyone have any advice? Thanks in advance.

Raphael Raven
  • 213
  • 2
  • 5
  • Is the link phase really dominating your incremental builds? If not, why bother? – dmckee --- ex-moderator kitten Nov 22 '10 at 19:46
  • Can you show us what you've tried with `-i`/`-r`? It looks like it *is* the option you want, though I expect you'll run into symbol conflicts (considering there is no ‘replace’ option). – Stéphan Kochen Nov 22 '10 at 19:48
  • The link phase is not the most time consuming part of my builds. However, this is the task that my team leader has given me. We're already good on only compiling what has changed. Currently, we compile only what has changed, while relinking every single object file each time. – Raphael Raven Nov 22 '10 at 19:55
  • Updated with edit of what I've tried so far with -r. Thanks for the help. – Raphael Raven Nov 22 '10 at 20:05
  • Here is a minimal runnable example and synthetic benchmark without `-static`: https://stackoverflow.com/questions/29391965/what-is-partial-linking-in-gnu-linker/53959624#53959624 Maybe we should edit this question title to say "shared objects". – Ciro Santilli OurBigBook.com Dec 28 '18 at 14:05

2 Answers2

7

It looks like you're right about -shared and -r not working together. I was skeptical about your old GCC version, but even on Ubuntu 10.10 I can see the same:

$ ld -shared -r
/usr/bin/ld.bfd.real: -r and -shared may not be used together

Unfortunately, that means you've reached a dead-end if you absolutely need shared objects. The binutils linker simply doesn't implement it.

If static libraries are an option for you, they are simply archives that can easily be manipulated with the ar utility.

Otherwise, you'll have to look at different linkers or compiler suites. I can't guarantee that you'll find this feature, though, it seems exotic.

Stéphan Kochen
  • 19,513
  • 9
  • 61
  • 50
  • Ok. Thank you very much for your quick responses. I'll definitely talk with my team leader and see if using static libraries would be an option. Also in my research, I discovered an alternate linker called "gold" which was developed by Ian Lance Taylor at google. It was added to the binutils in March of 2008. I may also look into this. I was hoping there would be a way to do this using the default linker, but it doesn't appear that's the case. Again, Thanks for your help! – Raphael Raven Nov 22 '10 at 20:32
  • 4
    I appear to have `gold` installed, but `gold -shared -r` gives the same result. – Stéphan Kochen Nov 22 '10 at 20:36
  • Haha of course that'd be the case. :) Thanks again for your help. – Raphael Raven Nov 22 '10 at 20:39
  • 1
    `-r` does not do what you seem to think it does. It tells the linker to create an object file instead of an executable or a shared library. So ld does not support `-r` and `-shared` together because it makes zero sense. That would mean 'create a shared library that is a regular object file'. – Maarten Sep 26 '14 at 15:38
  • If that's true, then the manpage for GNU binutils `ld` doesn't really describe it as such. But it looks like the manpage for OSX `ld` is more clear: “-r Merges object files to produce another mach-o object file with file type MH_OBJECT.” – Stéphan Kochen Sep 28 '14 at 08:46
5

You can sort of get the behavior you're after using archive/static libraries, but the initial link will still take the same amount of time.

Using an archive file:

# Initially create the archive
ar r libmylib.a <all object files>

# Create your shared object (re-use this line after libmylib.a is updated)
g++ -shared -rdynamic -m64 -Wl,-rpath,'$ORIGIN' libmylib.a -o libmylib.so     

# Update the archive file
ar r libmylib.a updated1.o updated2.o

As I said, it will still take the same amount of time to actually link the .so as it did before.

Brian Vandenberg
  • 4,011
  • 2
  • 37
  • 53
  • 1
    Even better, you can use `ar T` thin archives to avoid the bloat of copying all objects into the archives: https://stackoverflow.com/questions/2157629/linking-static-libraries-to-other-static-libraries/27676016#27676016 This is what the Linux kernel started doing: https://unix.stackexchange.com/questions/5518/what-is-the-difference-between-the-following-kernel-makefile-terms-vmlinux-vml/482978#482978 – Ciro Santilli OurBigBook.com Dec 28 '18 at 13:48