2

Error message:

Undefined symbols for architecture arm64:
  "std::__1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__1::allocator<unsigned short> >::shrink_to_fit()", referenced from:
      base::UTF8ToUTF16(char const*, unsigned long, std::__1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__1::allocator<unsigned short> >*) in libbase.a(utf_string_conversions.o)
      base::WideToUTF16(wchar_t const*, unsigned long, std::__1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__1::allocator<unsigned short> >*) in libbase.a(utf_string_conversions.o)

ld: symbol(s) not found for architecture arm64

subprocess.CalledProcessError: Command '['clang++', '-B', '/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/', '-shared', '-Xlinker', '-install_name', '-Xlinker', '@rpath/Cronet.framework/Cronet', '-Xlinker', '-objc_abi_version', '-Xlinker', '2', '-arch', 'arm64', '-Werror', '-isysroot', '/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS15.0.sdk', '-stdlib=libc++', '-miphoneos-version-min=8.0', '-fembed-bitcode', '-Wl,-ObjC', '-o', 'obj/components/cronet/ios/arm64/Cronet', '-Wl,-filelist,obj/components/cronet/ios/arm64/Cronet.rsp', '-framework', 'UIKit', '-framework', 'CoreFoundation', '-framework', 'CoreGraphics', '-framework', 'CoreText', '-framework', 'Foundation', '-framework', 'JavaScriptCore', '-framework', 'CFNetwork', '-framework', 'MobileCoreServices', '-framework', 'Security', '-framework', 'SystemConfiguration', '-lresolv']' returned non-zero exit status 1
Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
yfainaer
  • 41
  • 3
  • 2
    I’m aware of one report of nearly identical errors (undefined symbols used by these same two functions) when building Chromium from QtWebEngine 5.15.7 on an Apple Silicon Mac running macOS 12.0 Monterey. – chrstphrchvz Nov 03 '21 at 23:51
  • Is there a workround? – yfainaer Nov 05 '21 at 08:45
  • Not that I am aware of. Consider subscribing to https://trac.macports.org/ticket/63725 for updates. – chrstphrchvz Nov 05 '21 at 22:43
  • Did you 0) update libbase, 1) include `` before libbase? [This comment](https://chromium.googlesource.com/aosp/platform/system/libbase/+/refs/heads/upstream/master/include/android-base/utf8.h#33) seems to indicate those functions are only needed on Windows – viraltaco_ Nov 26 '21 at 12:13
  • Can you provide a minimal working example that causes this issue? – Joan Marcè i Igual Nov 28 '21 at 11:48
  • Be aware that issue is for slightly older chromium base library code which predates `string16` being [replaced](https://bugs.chromium.org/p/chromium/issues/detail?id=911896) by `std::u16string`. This may explain why similar reports of this issue are rare. – chrstphrchvz Dec 09 '21 at 11:52
  • This is the subject of [a meta question](https://meta.stackoverflow.com/questions/413602/should-i-recommend-to-delete-a-partial-answer-from-a-reputable-external-source) (an answer, but it ought to affect this question as well). (Note: The meta question may be automatically be deleted within a month or so, depending on votes, answers to it, etc.) – Peter Mortensen Dec 09 '21 at 15:39
  • For reference, the QtWebEngine 5.15.x issue: https://bugreports.qt.io/browse/QTBUG-98813 – chrstphrchvz Dec 13 '21 at 21:58

3 Answers3

3

This was pretty tricky to track down, but I think I found the issue. First of all, string16.ii can be simplified to:

template <class T>
struct basic_string {
    __attribute__((internal_linkage))
    void shrink_to_fit();
};

template <class T>
void basic_string<T>::shrink_to_fit() { }

template class basic_string<char>;

utf_string_conversions.ii can be simplified to:

template <class T>
struct basic_string {
    __attribute__((internal_linkage))
    void shrink_to_fit();
};

template <class T>
void basic_string<T>::shrink_to_fit() { }

extern template class basic_string<char>;

int main() {
    basic_string<char> s;
    s.shrink_to_fit();
}

Since shrink_to_fit has internal linkage, the compiler doesn't even bother emitting it inside string16.o, because it is not used in that TU. If it did emit it in string16.o, it would be dead code since no other TU can refer to it anyway.

Then, from utf_string_conversions.o, we see an extern template instantiation declaration, which basically promises that we will be able to find shrink_to_fit() in some other TU (presumably string16.o). Of course, that can't be the case, since another TU can't "export" shrink_to_fit, which has internal linkage as explained above.

As a result, we end up in a situation where utf_string_conversions.o expects to see shrink_to_fit() in some other TU, but the other TU that should provide it doesn't. Furthermore, if we compile the above, we can actually see that the compiler is warning us from exactly that:

<stdin>:4:10: warning: function 'basic_string<char>::shrink_to_fit' has internal linkage but is not defined [-Wundefined-internal]
    void shrink_to_fit();
         ^
<stdin>:14:7: note: used here
    s.shrink_to_fit();
      ^
1 warning generated.
Undefined symbols for architecture x86_64:
  "basic_string<char>::shrink_to_fit()", referenced from:
      _main in main.o
ld: symbol(s) not found for architecture x86_64

This warning does not show up in the original code because warnings inside system headers are suppressed. This really tripped me up initially, and I think it would make sense for the compiler to warn if the explicit template instantiation declaration appears outside of a system header, regardless of whether the class itself is declared inside a system header. I actually removed the system header directives from the original reproducer and I was able to get the same warning, see this.

Now, you might be wondering why that doesn't happen with other methods of basic_string? Well, apparently, most if not all other methods of basic_string are either

  1. defined inside the class (and are hence implicitly inline), or
  2. defined outside the class but marked with inline explicitly, or
  3. not marked with __attribute__((internal_linkage))

I believe that's the reason why this issue is only showing up with shrink_to_fit(): it is a non-inline function because it is defined outside its class definition, despite the class being a template.

Concretely:

  1. Chromium should consider removing their explicit instantiation -- those are brittle, as explained here.
  2. I will fix libc++'s shrink_to_fit() so it is inline. (edit: here)
  3. I will file a Clang bug to discuss the possibility of emitting that warning when the explicit template instantiation declaration appears outside of a system header. (edit: here)

Thanks for reporting this tricky issue.

Louis Dionne
  • 3,104
  • 1
  • 15
  • 35
1

I don’t know why the output object file for base/strings/string16.cc doesn’t contain base::string16::shrink_to_fit() (my current guess is that it has something to do with the include/c++/v1/string header file from libc++ in the macOS 12 and iOS 15 SDKs), but it does appear to contain base::string16::reserve(unsigned long). For std::basic_string in C++17 and earlier (not C++20 or later), shrink_to_fit() is equivalent to reserve(0). Since the affected version of string16.cc is presumably being compiled with -std=c++14, a workaround is to replace (in UTFConversion() in base/strings/utf_string_conversions.cc):

  bool res = DoUTFConversion(src_str.data(), src_len32, dest, &dest_len32);

  dest_str->resize(dest_len32);
  dest_str->shrink_to_fit();

  return res;
}

with:

  bool res = DoUTFConversion(src_str.data(), src_len32, dest, &dest_len32);

  dest_str->resize(dest_len32);
  dest_str->reserve(0);

  return res;
}
chrstphrchvz
  • 657
  • 5
  • 18
  • 1
    This suggestion isn't great, instead we should figure out why `string16.cc` doesn't contain `shrink_to_fit`. Can you please share a self-contained reproducer for creating that object file? – Louis Dionne Dec 07 '21 at 22:13
  • 1.checkout chromium source code 2.setup Cronet iOS build environment 3.compile with Xcode 13.1 and iOS 15.0, MacOS 12.0.1 – yfainaer Dec 08 '21 at 08:56
  • This is a lot of work just to get my hands just on one object file. If you have that setup locally, can you share a Gist with the preprocessed content of the file and the command-line invocation used to compile it? That should be enough. – Louis Dionne Dec 08 '21 at 14:19
  • @LouisDionne Here is the preprocessed string16.cc (from qtwebengine-chromium with macOS 12 SDK rather than cronet with iOS 15 SDK, but exhibits the same issue): https://gist.github.com/chrstphrchvz/f9d486d060bf4d1726dfc82d4f47bb17 – chrstphrchvz Dec 09 '21 at 02:20
  • Thanks @chrstphrchvz. Can you show me a minimal translation unit (preprocessed) that, when linked against the object file, fails with the missing symbol error? And also the compiler invocation you're using. – Louis Dionne Dec 09 '21 at 21:26
  • Working theory: `string16.o` does not contain `shrink_to_fit()` because that method is marked with `__attribute__((internal_linkage))`. If you use `shrink_to_fit()` from your code, if it is annotated with `internal_linkage`, the compiler wouldn't try to use the one from `string16.o` (because it knows it won't be there). I suspect that somehow you are building without `internal_linkage` on `shrink_to_fit()`, and the compiler expects to see the method defined in `string16.o`. – Louis Dionne Dec 09 '21 at 21:28
  • Are you compiling with AppleClang or LLVM Clang? What is the definition of `_LIBCPP_HIDE_FROM_ABI`? If you can show me the output of this with the compiler you're having issues with, it would be helpful: `echo "#include <__config>\n_LIBCPP_HIDE_FROM_ABI" | -xc++ - -E -P`. – Louis Dionne Dec 09 '21 at 21:30
  • @LouisDionne Here is utf_string_conversions.cc preprocessed with macOS 12 SDK: https://gist.github.com/chrstphrchvz/acd74ee34b69384eee876c9e72fd9d44. Compiling string16.ii and utf_string_conversions.ii separately and then linking with `clang++ -std=c++14 -stdlib=libc++ string16.o utf_string_conversions.o` outputs mostly unrelated errors (missing `main()` and dependencies `base::IsStringASCII()`, `base::BasicStringPiece()`, etc.) but should say `shrink_to_fit()` is needed by two instances of `UTFConversion()` (which were inlined to `UTF8ToUTF16()` and `WideToUTF16()` in this question’s errors). – chrstphrchvz Dec 10 '21 at 10:15
  • @LouisDionne I used AppleClang 1300.0.29.30 (Xcode 13.2 beta 2); the output for your command is `"typedef __char16_t char16_t;\ntypedef __char32_t char32_t;\nnamespace std { inline namespace __1 { } }\n__attribute__ ((__visibility__("hidden"))) __attribute__ ((internal_linkage))"`. – chrstphrchvz Dec 10 '21 at 10:18
  • @LouisDionne Since utf_string_conversions.cc may be compiled separately from string16.cc (i.e. the explicit instantiation of `base::string16`), should `_LIBCPP_HIDE_FROM_ABI_PER_TU` be defined to 0 when compiling each of these files, or is there still a better alternative? (Maybe chromium’s jumbo build also sometimes compiles string16.cc and utf_string_conversions.cc in the same translation unit, if the files aren’t excluded; the asker’s cronet build doesn’t appear to use jumbo build, as the linker errors say `libbase.a(utf_string_conversions.o)` instead of e.g. `libbase.a(base_jumbo_20.o)`.) – chrstphrchvz Dec 10 '21 at 11:17
0

Here is my feedback to Apple and their reply:

Basic information

Please provide a descriptive title for your feedback:

undefined symbols of shrink_to_fit() when compiling Cronet on iOS 15 with Xcode 13

Which area are you seeing an issue with?

Xcode

What type of feedback are you reporting?

Incorrect/unexpected behavior

Details

What version of Xcode are you using?

13.1

Description

Please describe the issue:

When compiling Cronet on the iOS platform with Xcode 13.1 and iOS 15.0, an error happens:

Undefined symbols for architecture arm64:

"std::__1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__1::allocator >::shrink_to_fit()", referenced from: base::UTF8ToUTF16(char const*, unsigned long, std::__1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__1::allocator >) in libbase.a(utf_string_conversions.o) base::WideToUTF16(wchar_t const, unsigned long, std::__1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__1::allocator >*) in libbase.a(utf_string_conversions.o)

ld: symbol(s) not found for architecture arm64

subprocess.CalledProcessError: Command '['clang++', '-B', '/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/', '-shared', '-Xlinker', '-install_name', '-Xlinker', '@rpath/Cronet.framework/Cronet', '-Xlinker', '-objc_abi_version', '-Xlinker', '2', '-arch', 'arm64', '-Werror', '-isysroot', '/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS15.0.sdk', '-stdlib=libc++', '-miphoneos-version-min=8.0', '-fembed-bitcode', '-Wl,-ObjC', '-o', 'obj/components/cronet/ios/arm64/Cronet', '-Wl,-filelist,obj/components/cronet/ios/arm64/Cronet.rsp', '-framework', 'UIKit', '-framework', 'CoreFoundation', '-framework', 'CoreGraphics', '-framework', 'CoreText', '-framework', 'Foundation', '-framework', 'JavaScriptCore', '-framework', 'CFNetwork', '-framework', 'MobileCoreServices', '-framework', 'Security', '-framework', 'SystemConfiguration', '-lresolv']' returned non-zero exit status 1

Please list the steps you took to reproduce the issue

  1. checkout Chromium source code
  2. set up Cronet build environment
  3. compile with Xcode 13.1 and iOS 15.0, macOS v12.0.1 (Sierra)

What did you expect to happen?

Compile successfully like Xcode 12 does

What actually happened?

Undefined symbols for architecture arm64: "std::__1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__1::allocator >::shrink_to_fit()", referenced from: base::UTF8ToUTF16(char const*, unsigned long, std::__1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__1::allocator >) in libbase.a(utf_string_conversions.o) base::WideToUTF16(wchar_t const, unsigned long, std::__1::basic_string<unsigned short, base::string16_internals::string16_char_traits, std::__1::allocator >*) in libbase.a(utf_string_conversions.o)

Apple's reply:

It looks like you are using std::basic_string<char16, base::string16_char_traits>, which comes from Chromium. Since there is an explicit instantiation declaration for that type 2, the instantiation of std::basic_string<char16, base::string16_char_traits> appears to be provided in a library that probably ships with Chromium. You need to make sure you link against it, otherwise such link errors are to be expected.

We think this link may also be relevant: https://stackoverflow.com/a/17484003/627587. It doesn't pertain to libc++, however the issue is most likely the same.

Also, in case you are wondering why this started happening with Xcode 13.1, we suspect this is simply because std::basic_string<char16, base::string16_char_traits>::shrink_to_fit was previously being inlined in your code, which means that the fact that you're not linking to the required Chromium library was not triggering any error. We made some changes to shrink_to_fit and the compiler is probably deciding not to inline it anymore because it is more efficient to use the instantiation announced by the string16.h header, which uncovers the fact that you haven't been linking against that library.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
yfainaer
  • 41
  • 3
  • I disagree that the cause is “not linking the required Chromium library”. But presumably Apple was provided too little information to suggest a more likely alternative… – chrstphrchvz Dec 09 '21 at 11:31
  • 1
    …The objects for string16.cc and utf_string_conversions.cc are linked simultaneously into libbase.a (this is when the linking error has appeared). I can replace `shrink_to_fit()` usage with `reserve(0)` and not get a similar undefined symbols error for `reserve(unsigned long)`, which I think suggests that the required “library” for `string16` *is* being linked in. And, there was no similar linking error for previous macOS versions’ SDKs (assuming chromium build toolchain behaving differently on each macOS version isn’t a factor). – chrstphrchvz Dec 09 '21 at 11:31
  • The ***first part*** (without the literal questions (but some or all *could* be transformed into sub section titles)) belongs in the question. [Stack Overflow is not a forum](http://meta.stackexchange.com/a/92115). You can edit (change) both: [change your answer](https://stackoverflow.com/posts/70284124/edit) and [change your question](https://stackoverflow.com/posts/69520633/edit). Or in other words, this ought to be changed into the Q&A format. Thanks in advance. – Peter Mortensen Dec 09 '21 at 15:25
  • This answer is being discussed on [meta](https://meta.stackoverflow.com/questions/413602). – cigien Dec 09 '21 at 17:14
  • 1
    I wrote that Apple reply. I was indeed taking a guess at what might be happening, and the most likely from looking at the Chromium sources was that you were not linking against the required `.a`. I'll take a look at the preprocessed output above. – Louis Dionne Dec 09 '21 at 18:59
  • @LouisDionne Now that fix has landed in LLVM, do you have any idea when we can expect Apple clang to be updated/fixed? – rgehan Jan 09 '22 at 12:14
  • Unfortunately I can't make comments on future releases at all, but I'll try to post here when we release the fix, which shouldn't be too long. – Louis Dionne Jan 10 '22 at 16:18