It is voodoo programming. Usually inspired by a programmer seeing that File.Delete() doesn't always actually delete the file. A pretty inevitable side-effect of running programs on a multi-tasking operating system, the kind that also lets other processes open files.
And there's a category of shrink-wrapped malware that programmers voluntarily install on their machines that can cause a delay. Anti-malware is on the top of that list, search indexers and cloud storage utilities are next.
They use the FileSharing.Delete option to open the file to try to minimize their impact. Tends to work okay, until you do things like trying to rewrite the file immediately after deleting it. That doesn't work, the file is in "delete pending" mode and you get slapped with an UnauthorizedAccessException.
The victim programmer then tends to notice that "waiting for a bit" helps to avoid the exception. Giving the other process enough time to finish whatever it is doing with the file and close the file handle. Once the last handle is closed, the file actually disappears from the file system and recreating it won't fail.
Sleeping for 20 millisecond is completely arbitrary and pretty unlikely to be enough. You can never know how long to wait, other than by repeatedly trying. The real problem is that this strategy is just a very broken way to recreate files. Deleting first, then creating and writing the file is very dangerous. As noted, deleting works fine but creating it is perilous. When this goes wrong, the user is left without a copy of the file. Complete data loss. Never a very desirable feature of a program :)
Don't use voodoo. The proper way to do it is to create a new file, then use File.Replace() to swap it with the old file, then delete the old file. Never any data loss, the malware can do little to screw this up. Deleting a file by moving it to the recycle bin is a fine approach as well, as long as it doesn't happen too often, FileSystem.DeleteFile() in everybody's favorite namespace.