0

I have a folder contains a lots of files created over the years. Lets call it folder A, in Folder A i have files created on 2012,2013,2014 and so on, files will continuously created in this folder. There is no structure under Folder A, just bunch of files.

Task1:

Now I have to make a copy of those files based on month and year, I have a folder B, in this Folder B, First I d like to have a level by year, then under each year, I will have folders by month, so all the files from Folder A will be copied into each location in folder B based on their date of Creation.

--B
---2013
--------Jan
--------Feb
--------Mar
---2014
--------Jan
--------Feb
--------Mar
.......

Task2:

This script will be scheduled 3 times per day, each time when it runs, it has to compare "last modify date" between source and destination, if "last modify date" on source is newer than destination, then copy the modified file to destination with "original name + modified date" as new file name. Skip if the files remain untouched.

Root Loop
  • 3,004
  • 9
  • 46
  • 72
  • Is folder A unstructured? Basically just a bunch of files in one folder. – Patrick Mcvay Apr 22 '19 at 17:17
  • Yes, no structure under A, just bunch of files. – Root Loop Apr 22 '19 at 17:24
  • Your ordering of the month looks nice, but with mostly ntfs as the file system it will be in fact `Apr Aug Dec Feb Jan Jul Jun Mar May Nov Oct Sep` –  Apr 22 '19 at 21:01
  • ordering of the month does not matter. my point is creating subfolders based on month. – Root Loop Apr 23 '19 at 01:46
  • Are the 3 tasks per day scheduled with an equal time difference of 8 hours? –  Apr 23 '19 at 05:27
  • 1
    As far as your Task 2, I agree with @LotPings that is a new question entirely. I don't know if PowerShell is really the direction you should be going with that. I think the most efficient way would be a background application making use of the System.IO.FileSystemWatcher that would just copy your files as they are updated. – Patrick Mcvay Apr 23 '19 at 13:36
  • 1
    I see a problem with your Task2 approach, once a copy with attached date from `LastWriteTime` is done you need to also check these new files (or restrict the time frame after last run), otherwise you create new copies on each run. As already mentioned this is a **NEW** question. BTW If an answer solves your question or you find it helpful you should consider to [accept the answer](http://stackoverflow.com/help/accepted-answer) and/or [vote up](https://stackoverflow.com/help/why-vote) –  Apr 23 '19 at 15:44
  • Agree, next time I will do so. I am studying codes below, will accept the answer soon.Thank you for the help – Root Loop Apr 23 '19 at 17:58

4 Answers4

1

The most effecient method is IMO

  • to create a calculated property from CreationDate year and month
  • Group by that property
  • create target folder if non existent
  • move files of that group
## Q:\Test\2019\04\22\SO_55798207.ps1

$Source = "X:\FolderA"
$Target = "Y:\FolderB"

Push-Location $Source

Get-ChildItem $Source -File |
  Select-Object *,@{n='YearMonth';e={$_.CreationTime.ToString('yyyy\\MMM')}} |
    Group-Object YearMonth | ForEach-Object {
      $NewFolder = Join-Path $Target $_.Name
      If (!(Test-Path $NewFolder)){New-Item $NewFolder -ItemType Directory|Out-Null}
      $_.Group | Copy-Item -Destination $NewFolder
    }
Pop-Location
  • This solution works, It took too long, the test run for 25G of data took me over 10 mins. is there any other way to be more efficient? I edited my original post with extra task, hope you can help as well. thx – Root Loop Apr 23 '19 at 03:19
  • 1
    Well, I don't know what you expect, the initial copy of 25G of data and the grouping takes some time. With additional comparing of LastWriteTime the code will get even longer; the time succesive runs will take depends on the number of files to compare and copy. Your task 2 is a **NEW** question and should be posted as such. In general the `Group-Object` in my code can be replaced by a `ForEach-Object` treating every single file with the needed comparison of LastWrieTime in an `IF`. –  Apr 23 '19 at 05:25
0

This should fullfil your request:

$Files = Get-ChildItem -Path "C:\FolderA"

Foreach ($File in $Files) {

   $File | Copy-Item -Destination "C:\FolderB\$( $File.LastWriteTime.Year )\$( $File.LastWriteTime.ToString("MMM") )"

}
Walhalla
  • 396
  • 2
  • 5
  • 16
  • This is simple to understand, but wont work....`Copy-Item : Could not find a part of the path`, Copy-Item expects the destination to be a file or directory that already exists – Root Loop Apr 23 '19 at 02:08
0

Ok, this has been reworked. It is the most efficient way I can think to do this with this technology.

$indexFilePath = "C:\Folder B\FileUpdateTracker.Json"

if ((Test-Path -Path $indexFilePath) -eq $false)
{
    [psobject]@{
                LastUpdate = [datetime]::MinValue.Date;
               } | ConvertTo-Json | Out-File -FilePath $indexFilePath -NoClobber
}

$oldFileIndex = Get-Content -Path $indexFilePath | ConvertFrom-Json

$testFiles = @(Get-ChildItem -Path "C:\Folder A" | Where { $_.LastWriteTime.Date -ge $oldFileIndex.LastUpdate })


if ($testFiles.Count -eq 0)
{
    return
}

$fileIndex = [psobject]@{
                            LastUpdate = (Get-Date).Date;
                        }

foreach ($file in $testFiles)
{
    $year = $file.CreationTime.Year
    $month = $file.CreationTime.ToString("MMM")

    $archivePath = "C:\Folder B\$year\$month"

    mkdir -Path $archivePath -ErrorAction SilentlyContinue

    $copiedFileName = $null

    if ((Test-Path "$archivePath\$($file.Name)") -and $file.LastWriteTime.Date -ge $oldFileIndex.LastUpdate)
    {
        $copiedFileName = "$archivePath\$($file.Name.Replace($file.Extension, "_$($file.LastWriteTime.ToString("MM-dd-yy_hh-mm"))$($file.Extension)"))"
    }else{
        $copiedFileName = "$archivePath\$($file.Name)"
    }

    Copy-Item -Path ($file.FullName) -Destination $copiedFileName
}


$fileIndex | ConvertTo-Json | Out-File -FilePath $indexFilePath -Force
Patrick Mcvay
  • 2,221
  • 1
  • 11
  • 22
  • 1
    no such cmdlet `new-directory`, it is `new-item -itemtype directory`. I dont think `out-file` can do copy job, all files that "out-file"ed to destination were 1kb only. I am trying to copy media files, not logs. but this script did create folder structures – Root Loop Apr 23 '19 at 02:47
  • well, folder structure created, but some of the "year" folder can not be opened due to no permission? weird. – Root Loop Apr 23 '19 at 03:00
  • What version of PowerShell are you using? – Patrick Mcvay Apr 23 '19 at 13:29
  • The only reason I was doing Out-File was for the -NoClobber switch – Patrick Mcvay Apr 23 '19 at 13:48
  • Sorry about the New-Directory, apparently that is a custom alias, but mkdir should do the job. – Patrick Mcvay Apr 23 '19 at 13:53
  • Ok, I have completely reworked this to address your specific issue. The initial copy will take a while, depending on the amount of data. – Patrick Mcvay Apr 23 '19 at 15:21
  • @RootLoop by all means change the variable names on this. As you can probably tell, I was thinking about going a different direction with this, but never changed the variable names. As far as that no permission thing, you may want to check the directory that you are copying into (ie. Folder B's parent directory), that is more than likely where the permission issue is. – Patrick Mcvay Apr 23 '19 at 15:42
  • I am studying this codes, have a good feeling about it. will update soon. – Root Loop Apr 23 '19 at 17:55
  • OK, first run was successful.then I modified some files, run it again, nothing happened. checked that json file, it only contains one line `LastUpdate: "\/Date (1555992000000)"`. this file supposed to contain the records for all files right? – Root Loop Apr 23 '19 at 18:57
  • Sorry for the late reply. No, it will only contain the last update datetime. – Patrick Mcvay May 07 '19 at 15:05
  • Can you check to see what the value is on the LastWriteTime for the files that were modified? – Patrick Mcvay May 07 '19 at 15:09
0

Another way is to use BitsTransfer.

$dirA = "C:\Test\A"
$dirB = "C:\Test\B"

$params = Get-ChildItem $dirA -File | foreach {
    $destFile = [IO.FileInfo]("{0}\{1:yyyy}\{1:MMM}\{2}_{3:yyyyMMddhhmmss}{4}" -f $dirB,$_.CreationTime,$_.BaseName,$_.LastWriteTime,$_.Extension)
    if($destFile.Exists) { return }
    $destFile.Directory.Create()
    @{ Src = $_.FullName; Dest = $destFile.FullName }
}

if(!$params) { return }
Start-BitsTransfer -Source $params.Src -Destination $params.Dest -DisplayName "Backup"
rokumaru
  • 1,244
  • 1
  • 8
  • 11