0

I am working on a new project which contain 2 main parts:-

  1. Scheduled service which reads files from SharePoint online and send them to external API.

  2. The external API will process the received documents using Artificial Intelligent models >> extract data from the documents >> then Push the extracted data back to our system >> where our system will update SharePoint with the extracted data.

So currently I am defining the architecture and technologies that I am going to use, where I define the following at this stage: -

  1. For point one, I am going to build a .net core console application >> hosted it inside windows server >> schedule it using windows tasks scheduler. This service will read the files from SharePoint using SharePoint API >> send them to the external API.

  2. But I am not sure about point 2? How I am going to received the pushed data from the external API inside our system? Can I define an API end point inside my console application? Or for this I need to build something like ASP.NET Core MVC hosted in IIS which exposed some API endpoints to get the pushed data? Keeping in mind that the external API and our system will be installed inside the same windows server. OR I can build an ASP.NET Core MVC web application which will expose a controller class as an API end point + use its console application program.cs to be scheduled using windows tasks scheduler?

Thanks

marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
John John
  • 1
  • 72
  • 238
  • 501
  • 2
    There's lots of ways to achieve inter process communication besides HTTP. There's message queues, databases, lots of OS specific methods. – mason Oct 09 '21 at 12:47
  • @mason will you be able to advice more on this please? – John John Oct 09 '21 at 14:32
  • @mason in my case i am going to use the `WebClient()` to do the calls,, but my concern is how i can have an application which expose an API +have a scheduled job? for example should i create 2 separate applications ; one created as .net core console application and the other created as asp.net core mvc ? or we can achieve this inside one application either using console application or asp.net core web application.. did you get my point? thanks – John John Oct 09 '21 at 14:45
  • You can architect your app however you like. Try something. If it doesn't work, then provide a [mcve] and explain what's not working about your approach. Stack Overflow is here to answer specific programming questions and answers, not architect your application. – mason Oct 09 '21 at 15:49
  • Building on the comment from @mason, I would definitely use a back-end processor for this use case. I'm a big fan of Rebus and my blog article in the next link shows how to build a long-running workflow in Rebus. The workflow is started by an API call, which sounds pertinent to your scenario: https://seankearon.me/posts/2020/12/rebus-sagas-csharp/ – Sean Kearon Oct 16 '21 at 07:41

3 Answers3

1

After the external API handles the data, when it returns data to your system, there should have an endpoint for the external API to call and send data. So, you could create an Asp.net core MVC application and expose the related endpoint method.

For the console application, it is used to create the Scheduled service, right? If that is the case, you can create background tasks with hosted services, and add it in your MVC application, then there is no need to create the console application.

You can also refer this thread: How can I execute a method every year using BackgroundService in ASP.NET core Application?

Edit:

The Asp.net core application can be hosted on Windows as a Windows Service, you can create the API application and host it as a windows service to receive the data from external API refer the Host ASP.NET Core in a Windows Service.

Besides, you can also search "Create a Worker Service with an API door in .NET Core 3.1" using Google or Bing, there has multiple tutorial or sample to create a worker service with API.

Zhi Lv
  • 18,845
  • 1
  • 19
  • 30
  • there is always a debate if we should be running long running processes and background processes inside IIS web application,, as IIS web applications are not designed for running long running processes nor background processes.. am i correct? not sure what are the capabilities of .net core specifically but i am talking about .net and asp.net applications.. am i correct? – John John Oct 12 '21 at 10:59
  • are you able to advice more on my above comment? thanks – John John Oct 13 '21 at 08:21
  • For the Hosted Services, they are services/logic that you host within your host/application/microservice. Note that in this case, the hosted service simply means a class with the background task logic. The Asp.net core application can be hosted on Windows as a Windows Service, you can host the API application as a windows service to receive the data from external API, see [Host ASP.NET Core in a Windows Service](https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/windows-service?view=aspnetcore-5.0&tabs=visual-studio) and [this link](https://stackoverflow.com/questions/61138759/). – Zhi Lv Oct 14 '21 at 08:21
0

My thoughts on this, having built similar solutions:

  1. The scheduler feels fine unless you want to cluster it, in which case you could use a component such as Quartz.Net to ensure that each job only runs on one server at a time.

  2. This will depend on how the External API can return results. The most common option would be to for you to provide an internet API endpoint, and for the external API to act as its client. You would not use an MVC app since you are not providing a Web UI for the external API - here is an example .Net Core API.

You will need to ensure that the API endpoint is secured, as in the above example. The simplest option might be an API key, but again it will depend on the External API, its options for returning results, and perhaps its developers.

CONNECTIVITY, HOSTING AND DATA

This feels like a key consideration. You can host HTTPS endpoints in IIS or any other process type, eg Windows service or console app. But then you need to expose that process to the internet so that the Extermal API can call it right?

It is worth seeing if you can use a cloud provider such as Azure to reduce costs for some of this:

That will of course depend on where the data itself is hosted and whether you can move some or all of it. That is the design area I would focus on first.

Whichever option you choose, it feels like you are likely to need to enable inbound connectivity back to your main system.

Gary Archer
  • 22,534
  • 2
  • 12
  • 24
0

You can post in many ways, it is a bit simplistic using a console app and letting windows task scheduler handle the invoking, but then simplicity is King they say, allegedly :) i'd base off a .net background worker service like suggested in the thread or if available an Azure function.

So "can I post/ recieve pushed data", yea surely you can, depending on your experience you may want to favour WCF, SOAP, REST to name a few receiving/ client options that you can find a virtual shovel load of examples of doing on the web, depending on what your business partner knows, and if you get to choose, choose REST. But it's really not significant, which brings more smile to business partner, you could give them access to a simple FTP service and handle files as a batch thing or use the FileSystemWatcher to trigger when they upload.

so how does the other party push data? well that depends on what they support, are you sure they don't have an endpoint that you get to query? depends on who gets to have the initiative and mostly we'd build systems where somebody has do do something and we have to do nothing except what they ask of our API.

So we end up you'll have to beyond building your planned stuff, read though and replicate this one then modify a POST address to accept what payload they off, if they can play HTTP REST: https://learn.microsoft.com/en-us/aspnet/core/tutorials/first-web-api?view=aspnetcore-5.0&tabs=visual-studio

T. Nielsen
  • 835
  • 5
  • 18