Correctly deploying unmanaged binaries in IIS

Question

I'm having trouble deploying unmanaged dll's with IIS.

I've read Embedding unmanaged dll into a managed C# dll but I do not want to embed my binaries as I need the system to be extensible.

Edit to clarify extensibility To reduce complexity of building plugins to my library I want people to be able to drop binaries including unmanaged ones onto the bin folder and have it centrally managed.

I've also read Unmanaged DLLs fail to load on ASP.NET server but I cannot manually copy the files to PATH visible folders.

Using the code below I attempted to shadowcopy the unmanaged binaries to the executing folder to then P/Invoke the LoadLibrary method to load them into memory.

This throws the following error when it hits File.Copy.

Access to the path 'MyPath\bin\x86' is denied.

Is there a surefast method to do this? I'm struggling to find any coherent information. I cannot manually change IIS permissions as this is part of a library that could be installed anywhere.

I also cannot use

<hostingEnvironment shadowCopyBinAssemblies="false" />

My Code

private void RegisterNativeBinaries()
{
    if (NativeBinaries.Any())
    {
        return;
    }

    string folder = Is64Bit ? "x64" : "x86";
    string sourcePath = 
    HttpContext.Current.Server.MapPath("~/bin/" + folder);
    Assembly assembly = Assembly.GetExecutingAssembly();
    string targetBasePath = new Uri(assembly.Location).LocalPath;

    DirectoryInfo directoryInfo = new DirectoryInfo(sourcePath);
    if (directoryInfo.Exists)
    {
        foreach (FileInfo fileInfo in directoryInfo
                                               .EnumerateFiles("*.dll"))
        {
            if (fileInfo.Name.ToUpperInvariant()
                             .StartsWith("IMAGEPROCESSOR"))
            {
                IntPtr pointer;
                string targetPath = Path.GetFullPath(
      Path.Combine(targetBasePath, "..\\" + folder + "\\" + fileInfo.Name));

                // This is where the error is thrown.
                File.Copy(sourcePath, targetPath, true);

                try
                {
                    // Load the binary into memory.
                    pointer = NativeMethods.LoadLibrary(sourcePath);
                }
                catch (Exception ex)
                {
                    throw new ApplicationException(ex.Message);
                }

                if (pointer == IntPtr.Zero)
                    {
                        throw new ApplicationException(
                        "Cannot load " + fileInfo.Name);
                    }

                    // Store the pointer for freeing later.
                    NativeBinaries.Add(pointer);
                }
            }
        }
    }