Trying to batch remove non visible special characters from filenames

Question

I have 100+ filenames that are created from a program that have hidden special characters. In windows explorer, the filenames look correct, but copying and pasting the filename into a program such as notepad++ pastes with ?'s on either end. Ie, ?filename?. Renaming the filename manually by right clicking, deleting the filename, and retyping the filename fixes the problem. In order to see the extra characters, I have to switch the encoding in notepad++ from UTF-8 to ANSI. With help, I've identified the trailing '?' as id 65279, or a BOM. What is this char? 65279 ''

I need to load the files back into the program, but due to the hidden special characters, the program isn't seeing them to read in properly.

Is there a way to use PowerShell to scrub the files? Ideally, only the hidden special characters are removed, and the rest of the file name (including the underscores) is left alone. Filename collisions should not be an issue with the current situation, but an automatic overwrite would be a good solution if there was an exception. The output filenames are generated by a java script containing the following:

var objName = f[myCounter].contents.replace(/ /g,"_").toLowerCase();
app.pngExportPreferences.pageString = curPage.name;
var myFilePath = myDoc.filePath + "/" + objName + ".png"; //export to a folder of the current document
var myFile = new File(myFilePath);
myDoc.exportFile(ExportFormat.PNG_FORMAT, myFile, false);

In case the problem is easier to solve there. I am very new to PowerShell and javascript.

I've tried a few PowerShell scripts I've found, including:

dir -Recurse | ?{$_.Name -match $re}  | %{ren -literalpath $_.FullName -newname (join-path (get-item $_.PSPArentPath) $($_.Name -replace $re,""))}

gci *.png | Rename-Item -NewName {$_ -replace '_*(\[.*?\]|\(.*?\))_*' -replace '_+', ' '}

They didn't remove the hidden special characters.

Answer 1

To replicate a filename with the problem:

echo hi > ([char]65279 + 'hithere' + [char]65279 + '.txt')

Try this one. If it looks good, take off the -whatif after rename-item, so it actually takes effect.

dir | foreach {
  $name = $_.name
  $chars = [char[]]$name | where { $_ -in [char]' '..[char]'~' } # printable ascii
  $newname = -join $chars   # make a string again
  # $newname = $name -replace '[^ -~]'   # alternative
  if ($newname.length -lt $name.length) { # ascii name is smaller  
    $_ | rename-item -newname $newname -whatif
  }
}

Ref: http://facweb.cs.depaul.edu/sjost/it212/documents/ascii-pr.htm

Answer 2

The following script could help. Based on Character classes in regular expressions.

Regex updated to '\p{IsGeneralPunctuation}|\ufeff' after you identified the problematic character as U+FEFF Zero Width No-Break Space.
Should work for most file names, even for non-ascii ones (see Naming Conventions).

Get-ChildItem -Recurse -File |
    ForEach-Object {
        $strange = $_.Name
        $string  = $strange -creplace '\p{IsGeneralPunctuation}|\ufeff'
        if ( $strange.Length -ne $string.Length ) {
            'strange {0,3} {1}' -f $strange.Length, $strange
            'string  {0,3} {1}' -f $string.Length,  $string
            $_ | Rename-Item -NewName $string -WhatIf
        }
    }