How to include prebuilt shared library with NDK using CMake?

Question

I'm trying to use a pre-built shared library in my android app using CMake. I have a source tree looking like this (only relevant parts included):

app
 |-- build.gradle
 |
 |-- libs
 |    |-- arm64-v8a
 |    |    |-- libexternal.so
 |    |
 |    |-- x86_64
 |         |-- libexternal.so
 |
 |-- src
      |-- main
           |-- cpp
                |-- CMakeLists.txt

I added this to my CMakeLists.txt

set( external_DIR ${CMAKE_SOURCE_DIR}/../../../libs)
add_library( libexternal SHARED IMPORTED )
set_target_properties( libexternal
        PROPERTIES IMPORTED_LOCATION
        ${external_DIR}/${ANDROID_ABI}/libexternal.so )

... stuff the creates shared myapp from project files

target_link_libraries(myapp
    libexternal
    android
    native_app_glue
    ${log-lib})

and this to build.gradle

android {
...
    defaultConfig {
        ndk {
            abiFilters 'arm64-v8a', 'x86_64'
        }
        ...
    }
    ...
    sourceSets {
        main {
            // let gradle pack the shared library into apk
            jniLibs.srcDirs = ['libs/']
        }
    }
}

When I run this from Android Studio using my phone, it crashes with this error:

2019-12-08 21:17:10.920 28951-28951/com.example.<app-name> E/AndroidRuntime: FATAL EXCEPTION: main
    Process: com.example.<app-name>, PID: 28951
    java.lang.UnsatisfiedLinkError: Unable to load native library "/data/app/com.example.<app-name>-MNTQOYsTwhDHYCzD7w_eWQ==/lib/arm64/libmyapp.so": dlopen failed: library "<path-to-project>/app/src/main/cpp/../../../libs/arm64-v8a/libexternal.so" not found

It seems like the app on the phone is trying to fetch the library from its path in my computer rather than from the APK.