Instances of Azure Functions are sharing variables?

Question

Not sure if the question makes sense, but it's what I'm observing. My Azure Function uses a BlobTrigger to process PDF files that are uploaded to a Blob Storage. Things work fine, until I upload several blobs at once, in which case, using the code below I observe the following:

• The first context.getLogger() correctly logs each blob that triggers the Function.

• In the Azure File Share, each PDF file is correctly saved.

• The second context.getLogger() in many cases returns incorrect results (from one of the other files), as if variables are being shared between instances of my Function. Note that lines[19] is unique for each PDF.

• I notice similar behavior later on in my code where data from the wrong PDF is logged.

EDIT: to be clear, I understand logs aren't going to be in order when multiple instances run in parallel. However, rather than getting 10 unique results for lines[19] when I upload 10 files, the majority of the results are duplicates and this issue worsens later on in my code when based on X I want to do Y, and 9 out of 10 invocations produce garbage data.

Main.class

public class main {
   @FunctionName("veninv")
       @StorageAccount("Storage")
       public void blob(
           @BlobTrigger(
                   name = "blob",
                   dataType = "binary",
                   path = "veninv/{name}") 
               byte[] content,
           @BindingName("name") String blobname,
           final ExecutionContext context
           ) {

         context.getLogger().info("BlobTrigger by: " + blobname + "(" + content.length + " bytes)");

           //Writing byte[] to a file in Azure Functions file storage
               File tempfile = new File (tempdir, blobname);
               OutputStream os = new FileOutputStream(tempfile);
               os.write(content);
               os.close();

               String[] lines  = Pdf.getLines(tempfile);
               context.getLogger().info(lines[19]);
           }
    }

Pdf.class

   public static String[] getLines(File PDF) throws Exception {
           PDDocument doc = PDDocument.load(PDF);
           PDFTextStripper pdfStripper = new PDFTextStripper();
           String text = pdfStripper.getText(doc);
           lines = text.split(System.getProperty("line.separator"));
           doc.close();
           return lines;
   }

I don't really understand what's going on here, so hoping for some assistance.

Answer 1

Yes. Azure function invocations can share variables. I'd need to see all the code to be 100% certain, but it looks like the lines object is declared as static and it could be shared across invocations. Let's try changing from a static String[] to String[] and see if the problem goes away?

Azure functions are easy to get off the ground, it's easy to forget about the execution environment. Your functions invocations aren't as isolated as they appear. There is a parent thread calling your function, and many static variables aren't "thread safe." Static variable represents a global state so it is globally accessible. Also, it is not attached with any particular object instance. The "staticness" of the variable relates to the memory space it sits in not it’s value. So, the same variable is accessible from all class instances in which it is referenced.

PS. You've solved the issue in your answer here by reducing concurrency, but that may come at a cost to scalability. I'd recommend load testing that. Also static variables can be useful. Many are thread-safe and you want to use them in Azure functions, such as your httpClient or sqlClient DB connections! Give number three a read, here.

Answer 2

No, it's quite hard to believe that function can have such a serious issue. I see some potential problems which might be causing this in your case:

1. Are you sure you are uploading to a different unique blob for each file every time? You can check by logging the blobname param.
2. Since you store the file in temp directory File tempfile = new File (tempdir, blobname);, if the blob name is same as mentioned in #1, it would overwrite with last write wins. If it's possible to construct pdf directly from bytes or stream, you can consider that instead of creating an intermediate file in filesystem. If I am not wrong you are using PDFBox which has support to load from byte[] https://pdfbox.apache.org/docs/2.0.3/javadocs/index.html?org/apache/pdfbox/pdmodel/PDDocument.html (check the load method overload which accepts byte[]). I have also answered your another question related to this.
3. Check if you have static field causing this.
4. You don't need to use a separate queue which you are thinking to introduce. Though you won't need it at all if the actual issue is fixed, Blob trigger already uses internal queue, default concurrency is 24, but you can configure it in host.json. https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-blob-trigger?tabs=java#concurrency-and-memory-usage

UPDATE:

Looks like in your pdf class you declared 'lines' somewhere outside the method as static which is the root cause of this problem. It's nothing to do with function, but the devil of static :)

Below is the correct code (notes 'lines' variable is now local to the method):

public static String[] getLines(File PDF) throws Exception {
           PDDocument doc = PDDocument.load(PDF);
           PDFTextStripper pdfStripper = new PDFTextStripper();
           String text = pdfStripper.getText(doc);
           String[] lines = text.split(System.getProperty("line.separator"));
           doc.close();
           return lines;
   }

Answer 3

Just wanting to share that changing host.json to the following, to stop concurrent function invocation, appears to have fixed my issue:

{
    "version": "2.0",
    "extensions": {
        "queues": {
            "batchSize": 1,
            "newBatchThreshold": 0
        }
    }
}

Massive thanks to @KrishnenduGhosh-MSFT for their help. I'm still unsure why concurrent function invocation led to the issues I was experiencing, but given that my program also connects to a SQL database and Sharepoint site (both which are throttled) sequential-processing is the best solution regardless.