How to limit I/O operations in .NET application?

Question

I'm developing an application (.NET 4.0, C#) that:
1. Scans file system.
2. Opens and reads some files.

The app will work in background and should have low impact on the disk usage. It shouldn't bother users if they are doing their usual tasks and the disk usage is high. And vice versa, the app can go faster if nobody is using the disk.
The main issue is I don't know real amount and size of I/O operations because of using API (mapi32.dll) to read files. If I ask API to do something I don't know how many bytes it reads to handle my response.

So the question is how to monitor and manage the disk usage? Including file system scanning and files reading...

Check performance counters that are used by standard Performance Monitor tool? Or any other ways?

Will the service have Administrative privileges or run as a normal user account? (Performance Counters for Physical disk require admin privilege). Depending on answer, there may be another way. — Neil Fenwick, Oct 25 '10 at 11:06

Neil Fenwick · Accepted Answer · 2013-12-02T13:59:11.563

Using the System.Diagnostics.PerformanceCounter class, attach to the PhysicalDisk counter related to the drive that you are indexing.

Below is some code to illustrate, although its currently hard coded to the "C:" drive. You will want to change "C:" to whichever drive your process is scanning. (This is rough sample code only to illustrate the existence of performance counters - don't take it as providing accurate information - should always be used as a guide only. Change for your own purpose)

Observe the % Idle Time counter which indicates how often the drive is doing anything. 0% idle means the disk is busy, but does not necessarily mean that it is flat-out and cannot transfer more data.

Combine the % Idle Time with Current Disk Queue Length and this will tell you if the drive is getting so busy that it cannot service all the requests for data. As a general guideline, anything over 0 means the drive is probably flat-out busy and anything over 2 means the drive is completely saturated. These rules apply to both SSD and HDD fairly well.

Also, any value that you read is an instantaneous value at a point in time. You should do a running average over a few results, e.g. take a reading every 100ms and average 5 readings before using the information from the result to make a decision (i.e., waiting until the counters settle before making your next IO request).

internal DiskUsageMonitor(string driveName)
{

    // Get a list of the counters and look for "C:"

    var perfCategory = new PerformanceCounterCategory("PhysicalDisk");
    string[] instanceNames = perfCategory.GetInstanceNames();

    foreach (string name in instanceNames)
    {
        if (name.IndexOf("C:") > 0)
        {
            if (string.IsNullOrEmpty(driveName))
               driveName = name;
        }
    }


    _readBytesCounter = new PerformanceCounter("PhysicalDisk", 
                                               "Disk Read Bytes/sec", 
                                               driveName);

    _writeBytesCounter = new PerformanceCounter("PhysicalDisk", 
                                                "Disk Write Bytes/sec", 
                                                driveName);

    _diskQueueCounter = new PerformanceCounter("PhysicalDisk", 
                                               "Current Disk Queue Length", 
                                               driveName);

    _idleCounter = new PerformanceCounter("PhysicalDisk",
                                          "% Idle Time", 
                                          driveName);
    InitTimer();
}

internal event DiskUsageResultHander DiskUsageResult;

private void InitTimer()
{
    StopTimer();
    _perfTimer = new Timer(_updateResolutionMillisecs);
    _perfTimer.Elapsed += PerfTimerElapsed;
    _perfTimer.Start();
}

private void PerfTimerElapsed(object sender, ElapsedEventArgs e)
{
    float diskReads = _readBytesCounter.NextValue();
    float diskWrites = _writeBytesCounter.NextValue();
    float diskQueue = _diskQueueCounter.NextValue();
    float idlePercent = _idleCounter.NextValue();

    if (idlePercent > 100)
    {
        idlePercent = 100;
    }

    if (DiskUsageResult != null)
    {
        var stats = new DiskUsageStats
                        {
                                DriveName = _readBytesCounter.InstanceName,
                                DiskQueueLength = (int)diskQueue,
                                ReadBytesPerSec = (int)diskReads,
                                WriteBytesPerSec = (int)diskWrites,
                                DiskUsagePercent = 100 - (int)idlePercent
                        };
        DiskUsageResult(stats);
    }
}

Great answer! I meant the same counters but missed related classes from the System.Diagnostics namespace. Very useful example. Now I see the solution better. This approach meets my expectations. Many thanks!!! — Anatoly, Oct 25 '10 at 13:28

score 3 · Answer 2 · answered Oct 25 '10 at 12:26

A long term ago Microsoft Research published a paper on this (sorry I can’t remember the url).
From what I recall:

The program started off doing very few "work items".
They measured how long it took for each of their "work item".
After running for a bit, they could work out how fast an "work item" was with no load on the system.
From then on, if the "work item" were fast (e.g. no other programmers making requests), they made more requests, otherwise they backed-off

The basic ideal is:

“if they are slowing me down, then I must be slowing them down, so do less work if I am being slowed down”

Wim Coenen · Answer 3 · 2016-01-07T14:55:01.800

Something to ponder: what if there are other processes which follow the same (or a similar) strategy? Which one would run during the "idle time"? Would the other processes get a chance to make use of the idle time at all?

Obviously this can't be done correctly unless there is some well-known OS mechanism for fairly dividing resources during idle time. In windows, this is done by calling SetPriorityClass.

This document about I/O prioritization in Vista seems to imply that IDLE_PRIORITY_CLASS will not really lower the priority of I/O requests (though it will reduce the scheduling priority for the process). Vista added new PROCESS_MODE_BACKGROUND_BEGIN and PROCESS_MODE_BACKGROUND_END values for that.

In C#, you can normally set the process priority with the Process.PriorityClass property. The new values for Vista are not available though, so you'll have to call the Windows API function directly. You can do that like this:

[DllImport("kernel32.dll", CharSet=CharSet.Auto, SetLastError=true)]
public static extern bool SetPriorityClass(IntPtr handle, uint priorityClass);

const uint PROCESS_MODE_BACKGROUND_BEGIN = 0x00100000;

static void SetBackgroundMode()
{
   if (!SetPriorityClass(new IntPtr(-1), PROCESS_MODE_BACKGROUND_BEGIN))
   {
      // handle error...
   }
}

I did not test the code above. Don't forget that it can only work on Vista or better. You'll have to use Environment.OSVersion to check for earlier operating systems and implement a fall-back strategy.

`SetPriorityClass` expects a handle, `Process.GetCurrentProcess().Id` is a process ID, so this won't work. Luckily you can simply pass `new IntPtr(-1)` to `SetPriorityClass`, since that's a pseudohandle for the current process. With these changes it works. Also you should check the return value of `SetPriorityClass` to see if it actually worked. — CodesInChaos, Oct 12 '11 at 20:13

score 1 · Answer 4 · edited May 23 '17 at 12:31

1

See this question and this also for related queries. I would suggest for a simple solution just querying for the current disk & CPU usage % every so often, and only continue with the current task when they are under a defined threshold. Just make sure your work is easily broken into tasks, and that each task can be easily & efficiently start/stopped.

edited May 23 '17 at 12:31

Community

1
1

answered Oct 25 '10 at 11:05

mike

3,146
5
32
46

Thanks for the second link. There are counters I meant but missed managed wrappers. – Anatoly Oct 25 '10 at 13:17

score 0 · Answer 5 · answered Oct 25 '10 at 10:53

0

Check if the screensaver is running ? Good indication that the user is away from the keyboard

answered Oct 25 '10 at 10:53

Fredrik Leijon

2,792
18
20

The app will be deployed as Win Service – Anatoly Oct 25 '10 at 10:56
Therewith, multiple users can be logged in at the same time. And as for me I usually turn off the screensaver at all ;) So not sure it's a good idea. – Anatoly Oct 25 '10 at 11:02
Btw, any other services can use the disk even nobody is logged in. I can't affect their performance. – Anatoly Oct 25 '10 at 11:06

How to limit I/O operations in .NET application?

5 Answers5

Linked

Related