8

I have been struggling with a SQL Server CLR stored procedure.

Background:

We are using SQL Server 2014 and a CLR stored procedure has been implemented which calls a customer's web service.

The threading was initially used not to slow the main thread of SQL Server CLR.

Although, now, I know that using threading under CLR is no best idea, it has been working correctly for 6 years (since SQL Server 2008). It has been migrated to SQL Server 2014 recently.

The problem

On my development machine, same as on test system we have no problem with the solution.

On the customer system, the thread, which calls the web service, is never executed for some reason.

I can see from the log files that everything is working correctly till the thread execution.

There is no specific error, nothing.

We have been trying to change the permissions, but without a success. Therefore I think its not a permission issue.

Questions

  1. Does anyone know how to change the behavior? We couldn't find any configuration which might does the trick.

  2. Would it be good idea to remove the threading completely, and having the calling of web services directly on SQL Server main thread?

Thank you for any advice, Petr

Solomon Rutzky
  • 46,688
  • 9
  • 128
  • 171
Petr B.
  • 123
  • 1
  • 7
  • Add logging to the thread to see if it starts at all and where it stops making progress. Do you have any .NET level error handling that might suppress errors? – usr Jan 31 '16 at 18:19
  • Thats what I did already. The thread does not start at all. – Petr B. Jan 31 '16 at 20:27

1 Answers1

5

Not sure about Question #1, though it might not matter given the recommendation for Question #2. Still, one difference between SQL Server 2008 (where it is working) and SQL Server 2014 (where it is not working) is the CLR version that SQL Server is linked to. SQL Server 2005 / 2008 / 2008 R2 are linked to CLR v2.0 while SQL Server 2012 and newer are linked to CLR v 4.0. Since you are not seeing the error and your client is, I would make sure that their system has been updated to the same .NET Framework version that you are running.

For Question #2, I would recommend removing the multi-threading. That has too much potential for problems, and requires the Assembly to be UNSAFE. If you remove the threading, you can set the Assembly to EXTERNAL_ACCESS.

If you want to reduce contention, then assuming the Web Service calls are to the same URI, then you need to increase the number of allowed concurrent web requests. That can be done by setting the ServicePointManager.DefaultConnectionLimit Property. The default value is 2. Which means, any additional requests will wait and wait until one of the current 2 is closed.

Also, be sure to properly Dispose of the WebRequest.


The concern about making external calls (i.e. the Web Service) that can potentially not complete quickly is that SQL Server uses Cooperative Multitasking wherein each thread is responsible for "yielding" control back to the Scheduler (effectively pausing it) at various points so that the Scheduler can shuffle things around and run other things that are currently "sleeping". This concern with regards to SQLCLR code can typically be mitigated by doing at least one of the following:

  • Perform data access / querying the instance
  • Calling thread.sleep(0);

However, an external call is not doing data access, and you cannot easily call thread.sleep(0) while waiting for the WebResponse to complete. Yes, you can call the WebService on a separate thread and while waiting for it to finish, assuming you are just looping and checking, the sleep(x) will allow for the yield.

But is doing the Web Service call asynchronously necessary? It certainly has the downside of requiring the Assembly to be marked as WITH PERMISSION_SET = UNSAFE. It greatly depends on how long the call usually takes, and how frequently it is being called. The more frequent the call, the more likely it is that any delays are, at least in part, caused by the low default value for how many concurrent connections are allowed per each URI. This relates to the recommendation I made at the top.

But if you want to see how SQL Server actually works, this should be fairly easy to test. On my laptop, I went to the Server "Properties" in Object Explorer, went to "Processors", unchecked the "automatically set processor affinity..." option, selected only a single CPU under "Processor Affinity" in the tree view in the middle of the dialog, clicked "OK", and then restarted the service. I then set up a web page that did nothing but call "sleep" for 60 seconds. I have a SQLCLR TVF that calls web pages so I ran that concurrently in two different tabs / sessions. In a 3rd tab / session, I ran:

SELECT SUM(so1.[schema_id]), so1.[type_desc], so2.[type_desc]
FROM sys.objects so1
CROSS JOIN sys.objects so2
CROSS JOIN sys.objects so3
CROSS JOIN sys.objects so4
CROSS JOIN sys.objects so5
WHERE so3.[create_date] <> so4.[modify_date]
GROUP BY so1.[type_desc], so2.[type_desc], so5.[name]
ORDER BY so2.[type_desc], so5.[name] DESC;

And finally, in a 4th tab, after kicking off the first 3, I ran the following to monitor the system:

SELECT * FROM sys.dm_os_schedulers WHERE [scheduler_id] = 0;

SELECT *
FROM sys.dm_exec_requests
WHERE [scheduler_id] = 0
AND [status] <> N'background'
ORDER BY [status] DESC, session_id;

The status for the 2 sessions running the SQLCLR function was always "running" and the status for the session running that ugly query in tab 3 was always "runnable". But just to be sure, running that ugly query again, when neither of the SQLCLR functions was executing, took the same 1 minute and 14 seconds that it did when running concurrently with the 2 sessions running the SQLCLR call to the web page that was sleeping for 60 seconds.

Please do not infer that there is no cost to running the SQLCLR code to make the web calls. Since those threads were busy the whole time, if the system was busy then it would have reduced the ability for SQL Server to allocate those threads to complete other queries faster. But it does seem safe to conclude that, at least on systems with low to moderate load, the benefit gained by adding the threading doesn't seem to be worth the cost of increased complexity (especially since now there is a not-yet-reproducable problem to debug).

Solomon Rutzky
  • 46,688
  • 9
  • 128
  • 171
  • My worries are about removing the threading and leaving all the logic within the main thread of MSSQL CLR. Which I believe has more important things to do than calling some web services. If I understand correctly, what you suggest has a different purpose. Or am I wrong? – Petr B. Jan 31 '16 at 20:31
  • @PetrB. What "main" thread? You have a set of schedulers, usually 1 per logical CPU. You can see them using `SELECT * FROM sys.dm_os_schedulers;`. What I am suggesting is to remove the main bottleneck to your Web Service calls from completing sooner. It depends on how long the call should take and how frequently those calls are made, but only allowing 2 active `WebRequests` at a time can easily make those calls take longer given that all Sessions are sharing that AppDomain. Be sure to Dispose of the `WebRequests` and increase the `DefaultConnectionLimit`. – Solomon Rutzky Feb 01 '16 at 07:25
  • @PetrB. After that, you can easily test this. Just create a simple WebService that does nothing but `Thread.Sleep(x)` and pass in the value for `x`. Execute that code in one tab in SSMS, passing in a high value for `x`, and then open more tabs and run queries. Check `SELECT * FROM sys.dm_exec_requests` and pay attention to the `scheduler_id` field. Find the `scheduler_id` for the `session_id` running the Web Service call, then see if that `scheduler_id` is running anything else. Check the `status` field to see if any other `session_id` ever shows `running` for that same `scheduler_id`. – Solomon Rutzky Feb 01 '16 at 07:32
  • @strutzky Do I understand it correctly, that each time the CLR SP is executed it runs on its own thread anyway? Therefore it does not make sense to use threading at all? I think I read somewhere that these stored procedures are running under "main" thread of MSSQL. – Petr B. Feb 01 '16 at 08:37
  • @PetrB. Where did you read about stored procedures, or maybe just SQLCLR, running under a "main" thread? And if you aren't sure, then why do you trust it? I just updated my answer with more details. – Solomon Rutzky Feb 01 '16 at 18:46
  • 3
    Also make sure the web request is wrapped in [`Thread.BeginThreadAffinity()`](https://msdn.microsoft.com/en-us/library/system.threading.thread.beginthreadaffinity(v=vs.110).aspx) / [`Thread.EndThreadAfinity()`](https://msdn.microsoft.com/en-us/library/system.threading.thread.endthreadaffinity(v=vs.110).aspx) to avoid [CLR quantum punishment](http://blogs.msdn.com/b/sqlclr/archive/2006/04/14/576716.aspx). – Remus Rusanu Feb 01 '16 at 19:10
  • @RemusRusanu Thanks for mentioning that. Is it something to worry about only if you notice `forced_yield_count` being > 0? I ask because `BeginThreadAffinity` and `EndThreadAffinity` require the Assembly to be `UNSAFE`. Beyond avoiding "CLR Quantum Punishment", is there a reason to mark the thread affinity? Could something go wrong in the process if it is moved between threads? I would assume so given that they have this feature to prevent that move, but I have never come across it, I don't think. I want to add this info to my answer but want to understand it better first. – Solomon Rutzky Feb 01 '16 at 21:16
  • @RemusRusanu Also, is this still an issue on more recent versions of SQL Server? Or has it decreased with changes made in 2008 and 2012? I found this ( https://www.spotlightessentials.com/waitopedia/waits/SQLCLR_QUANTUM_PUNISHMENT ) but not sure how reliable the numbers are. And, how does one go about seeing how much quantum is being used and what the limit is? I did see some additional quantum fields in `sys.dm_os_workers`. I also see `quantum_length_us` in [sys.dm_os_schedulers](https://msdn.microsoft.com/en-us/library/ms177526.aspx) but not how usable that value is. – Solomon Rutzky Feb 01 '16 at 21:21
  • It absolutely still applies to newer releases. You can't move the processing between threads, how would that happen? You do not need to know the quantas. You must declare your 'thread affinity' whenever 1) your own CLR code does lenghty computations (eg. iterates a list of unknown size, processes an unknown size HTML response etc) or 2) the CLR code is calling some code that has the potential to block without SQL seeing it as 'blocked'. – Remus Rusanu Feb 02 '16 at 05:40
  • @RemusRusanu I asked about it still applying because the article you referenced is one of the only articles to mention it and it was from 2006 (a few others copy from it, and your article on CodePlex mentions it). But good to know that it is still relevant. And regarding moving a process between threads, I got that from the `BeginThreadAffinity()` page that you linked to (in Remarks): "A host that provides its own thread management can move an executing task from one physical operating system thread to another at any time." – Solomon Rutzky Feb 03 '16 at 20:17
  • any update or hack on this issue? i am facing the same issue, and i need to run my clr sp outside the main thread. – Edwin O. Sep 27 '16 at 18:26
  • 1
    @Edwin How exactly are you doing the threading? Why are you doing it? I might need to see some code in order to attempt to reproduce the issue. – Solomon Rutzky Sep 27 '16 at 21:12
  • i have created another thread here, with detail explanation of the query, please attend to me there. thanks! http://stackoverflow.com/questions/39735746/sql-clr-awaitable-not-getting-executed – Edwin O. Sep 27 '16 at 23:20