1

I have a general question, context:

I want to take external data from my own api to cognitive search, since it is not possible I want to do it through the Azure Functions App, in this way I process my data and take it to Cognitive Search. Is that possible? Investigating a bit about Functions it seems that it is because I can convert my data into bloob or jsons which is just what Cognitive would need.

I want to do this because I already have my data in an external place so for ease of updating I want to get it into my architecture I need to handle it this way. Anyone an expert on the subject? I will implement my application in nodejs

Thanks in advance for all your help

Izlia
  • 235
  • 1
  • 2
  • 18

1 Answers1

2

You can get your external API with HTTP trigger using Javascript function and then download that data in your blob storage or locally and add it in your cognitive search index via this Azure Search Javascript sample or Azure Rest API and Azure Portal like below:-

My HTTP Trigger function that gets the data from external API and gives response in the browser:-

My functions, index.js code:-

module.exports  =  async  function (context, req) {

const  url  =  'https://reqres.in/api/users?page=2';

const  response  =  await  fetch(url);

const  data  =  await  response.json();

context.res  = {

body:  data

};

};

enter image description here

External Data:-

enter image description here

Now, I referred this Azure Search JavaScript sample to create index in cognitive search with Javascript like below:-

npm init
npm install @azure/search-documents
npm install dotenv

Output:-

enter image description here

Created search.env and added my API Key and Cognitive search endpoint like below:-

enter image description here

index.js:-


const { SearchIndexClient, SearchClient, AzureKeyCredential, odata } =
require("@azure/search-documents");

// Load the .env file if it exists require("dotenv").config();

// Getting endpoint and apiKey from .env file const endpoint =
process.env.SEARCH_API_ENDPOINT || ""; const apiKey =
process.env.SEARCH_API_KEY || "";

async function main() {
    console.log(`Running Azure Cognitive Search JavaScript quickstart...`);
    if (!endpoint || !apiKey) {
        console.log("Make sure to set valid values for endpoint and apiKey with proper authorization.");
        return;
    }

    // remaining quickstart code will go here }

main().catch((err) => {
    console.error("The sample encountered an error:", err); });

Now, You can add your external API from functions to blob storage using Azure Functions Blob trigger and then import the external API's data to Search index via that Blob or directly download the data via HTTP trigger like above and add it in the Index like below:-

You can also upload the external data file to Blob by using the code in this Document.

index.js:-


const { SearchIndexClient, SearchClient, AzureKeyCredential, odata } =
require("@azure/search-documents"); const indexDefinition =
require('./searchquickindex.json'); const indexName =
indexDefinition["first_name"];

Complete source code

Importing the data via Blob in Portal:-

I uploaded my external data file in blob like below:-

enter image description here

And imported that blob in my search with Import Data like below:-

enter image description here

enter image description here

SiddheshDesai
  • 3,668
  • 1
  • 2
  • 11
  • Thanks a million for this explanation. Thanks for the explanation, I'll work on it and share how it turned out in my use case. – Izlia May 22 '23 at 15:25
  • I have a question. There's something that isn't clear to me. Once I configure my index to fetch my data, I would need to create and configure this trigger for blob files. However, I don't understand how it works. Who is listening the trigger? Could you please explain its function in the context of my use case? – Izlia May 22 '23 at 17:44
  • What I don't understand is how the function I create is connected to the trigger for creating the blob. I'm unsure about how the file processing works. In my function, I receive JSON data. Let's say this trigger converts the response from my function into a blob, which I then call in Cognitive. Now, suppose the response in my function changes for some reason. Will the trigger automatically capture those changes, or would I need to refresh it manually? Thank you in advance for your assistance. – Izlia May 22 '23 at 17:54
  • 1
    Yes, Blob trigger will trigger if any new blob arrives in the storage or any changes gets updated in the blob, This if there's a new response the blob function will trigger and save the response in the storage account – SiddheshDesai May 22 '23 at 17:58
  • So far, I have successfully completed the integration up to my intended goal. what I did was two functions: an HTTP function and a Blob Trigger function. The HTTP function connects to my external API and filters the data until I get the desired JSON, here I call the Blob Trigger function and pass it the context and data as parameters. Inside the Blob Trigger function, I do the validations until I connect to the blob storage container and check if a blob with the name I'm trying to write already exists. If the blob already exists, return null; if not, convert the JSON to a buffer and update it. – Izlia May 24 '23 at 16:36
  • Thanks to your help, I gained a clearer understanding of what needed to be done, and I was able to find the documentation I needed more easily. I truly appreciate your help. I have a question regarding the flow you suggested: Do you think it is best to maintain a cognitive search? Considering that I have over 100 JSON files to create, I am wondering if this flow is scalable and sustainable. I would appreciate your perspective on this matter. – Izlia May 24 '23 at 16:38
  • 1
    Glad it was helpful, Azure Function with consumption plan will scale up and down base on the requirement of Function triggers deployed in it and it will scale up and down depending on your cognitive search responses, One way to optimize the performance of your function app is to use dedicated function premium plan which uses dedicated app service plan for function, The advantage is reduced cold start and a dedicated resource serving your requests. Premium plan reference -https://learn.microsoft.com/en-us/azure/azure-functions/functions-premium-plan?tabs=portal – SiddheshDesai May 24 '23 at 18:24