asp.net core - how to simultaneously read file without throwing IOException

Question

I'm trying to write an action that will serve files. What I've tried (File method comes from ControllerBase):

string fileServer = @"\\FileServer\Attachments";
[HttpGet("{fileName}")]
public async Task<IActionResult> Index([FromRoute] string fileName)
{
    var filePath = Path.Combine(fileServer, fileName);

    try
    {
        var fs = new FileStream(filePath, FileMode.Open);
        return File(fs, "application/octet-stream");
    }
    catch (Exception e)
    {
        return BadRequest(new { ExitMessage = e.Message });
    }
}

Alternative approach as suggested by someone else:

try
{
    var memory = new MemoryStream();
    using (var stream = new FileStream(filePath, FileMode.Open))
    {
        await stream.CopyToAsync(memory);
    }
    memory.Position = 0;

    return File(memory, "application/octet-stream");
}

Both work fine but when a second request is triggered whilst the first request didn't finish yet they get

System.IO.IOException
The process cannot access the file '\FileServer\Attachments{fileName}' because it is being used by another process.

How can I serve the same file simultaneously?