2

I have build an API where an external server can post JSON data to our ColdFusion Server. The ColdFusion Server then processes this data (loops over arrays and inserts data into a database etc), and then responds to the external server that everything went fine or not.

But I would rather run the process in a CFTHREAD due the load will become more heavy in the future. How can I do this kind of asynchronous action, and still respond to the server that made the request?

I've looked into Event Gateways, but that does not seem to be the way to go. I've also thought about letting the external server do recurring calls, like "are you done yet", "are you done yet", but that does not feel like an ultimate solution.

I can get the guys managing the external server, to call our ColdFusion server the way we want, so at least I'm not bound by that.

Here is a simplified version of my index.cfm that receives the call:

<cftry>
  <cfinclude template="udf.cfm" />
  <cfset _run()>
  <cfcatch type="any">
    <cfcontent type="application/json; charset=utf-8" reset="yes">
    <cfoutput>{"success":0,"message":"#cfcatch.Message# - #cfcatch.Detail#"}</cfoutput>
  </cfcatch>
</cftry>

And here is my udf.cfm, that processes the call (again a simplified version of it):

<cffunction name="_run" output="yes" returntype="void">
    <cfset local.success=true>
    <cfset local.msg="">
    <cftry>
        <!---get request data--->
        <cfset local.requestData=GetHttpRequestData()>
        <!---validate post--->
        <cfif NOT StructKeyExists(local.requestData,"headers")>
            <!---headers is missing--->
            <cfset _returnJson(msg="Headers is missing")>
        </cfif>
        <!---and a bunch of other validations...--->
        <!---get body--->
        <cfset local.isBinaryBody=IsBinary(local.requestData.content)>
        <cfset local.bodyAsString=local.isBinaryBody ? ToString(local.requestData.content) : local.requestData.content>
        <cfset local.body=DeserializeJson(local.bodyAsString)>
        <cftransaction action="begin">
            <cftry>
                <!---prepare for database inserts by cleansing tables etc--->
                <!---loop data array, can be tens of thousands of rows--->
                <cfloop array="#local.body.items#" index="local.item">
                    <!---do some inserts/updates into MySQL Database--->
                </cfloop>
                <cfcatch type="any">
                    <!---error--->
                    <cfset local.msg=_parseCatchError(catch=cfcatch)>
                    <cfset local.success=false>
                </cfcatch>
            </cftry>
            <!---rollback or commit transaction depending on success flag--->
            <cftransaction action="#local.success ? 'commit' : 'rollback'#" />
        </cftransaction>
        <cfcatch type="any">
            <!---error--->
            <cfset local.msg=_parseCatchError(catch=cfcatch)>
            <cfset local.success=false>
        </cfcatch>
    </cftry>
    <!---return JSON--->
    <cfset _returnJson(msg=local.msg,success=local.success)>
</cffunction>
<cffunction name="_returnJson" output="yes" returntype="void">
    <cfargument name="msg" type="string" required="yes">
    <cfargument name="success" type="boolean" default="false">
    <cfscript>getPageContext().getOut().clearBuffer();</cfscript>
    <cfcontent type="application/json; charset=utf-8" reset="yes">
    <cfoutput>#SerializeJson(arguments)#</cfoutput>
    <cfabort>
</cffunction>
<cffunction name="_parseCatchError" output="no" returntype="string">
    <cfargument name="catch" type="any" required="yes">
    <cfset local.error="#Trim(arguments.catch.message)# - #Trim(arguments.catch.detail)#">
    <cfif StructKeyExists(arguments.catch,"tagContext")>
        <!---grab line info--->
        <cfset local.tagContext=arguments.catch.tagContext[1]>
        <cfset local.error=local.error&" - #Trim(local.tagContext.template)# line #Trim(local.tagContext.line)#">
    </cfif>
    <cfreturn local.error>
</cffunction>
Jörgen Lundgren
  • 273
  • 1
  • 13
  • If you are expecting a lot of activity, `` is more of a problem substitution than a solution. Too many simultaneous threads are so problematic that handling that situation is available in the CF Monitor. – Dan Bracuk Oct 30 '17 at 17:47
  • Ok, perhaps I better focus then on executing my processes in the most effective way I can on request. And also not allowing too large chunks of data being received, but rather split it down to handle single objects in single requests rather than array of objects in one request. – Jörgen Lundgren Oct 30 '17 at 18:28
  • I would need to see some code before going down that road. I have worked on things where bundling was the proper approach. DB Servers tend to be more scalable than CF. But then again, all this is way too much speculation. – James A Mohler Oct 30 '17 at 19:35
  • James A Mohler, I've updated my question with some code. It would be to vast to include all of it, so I simplified it. If you're more interested in the database stuff, please let me know. – Jörgen Lundgren Oct 31 '17 at 08:09
  • This article may be relevant https://www.bennadel.com/blog/3170-exploring-the-possibility-of-parallelizing-queries-in-coldfusion-using-cfthread.htm . – Nebu Nov 02 '17 at 20:05
  • Could be interesting to know that we have abandon the idea with posting JSON and instead we are going to let their database (MSSQL) pretty much dump some tables to a empty database (MySQL) on the web server, then we can run our cfthreads (or whatever we want) when we process that data and inserts it the way we want into the main database on our server. – Jörgen Lundgren Nov 03 '17 at 07:33

2 Answers2

1

Sounds like you just need a callback supplied by the caller.

  1. accept request, including an external url for your code to hit once complete
  2. start thread, do not join
  3. return info to caller that request is being processed, ending the primary request
  4. thread keeps running and at end hit the callback url, supplying any additional info about success or failure

A few other options are available here as well. First of all you could just accept and log the data needed for the request in a queue and then hit them at a later time. Also, as you stated, you can provide a status endpoint for the caller to check on progress.

Dan Roberts
  • 4,664
  • 3
  • 34
  • 43
  • I think you are right. Using a callback to a provided URL would probably be the best solution to my original question, if both servers are web servers. In my case, the caller is not a web server. It’s beyond my knowledge if it still would be possible to make a callback to that server though. – Jörgen Lundgren Nov 03 '17 at 21:21
1

Keep in mind that threads in coldfusion are limited by the server version - i think that the standard edition support up to 10 threads - others goes in queue. Supplying a "callback" to the thread will do the job. Also scheduled tasks can be leveraged in after the sql dumps are finished.

axltop
  • 11
  • 2