1

I would like to copy files to a specific folder based on a certain part in their name. Below you will find my folder structure and where the files are. In both the D0 and D1 folders you will find files that are named like this structure: 20210308_DML_D0_Temp_s1_t1.txt or 20210308_DML_D1_weather_s3_t6.txt with D0/D1 in which folder it is situated, Temp/weather whether it is temperature or weather file, s1/s3 is the location and t1/t6 is the timepoint. The first thing that I wanted to do is to loop over the txt files in both D0 and D1 files and move the files that have Temp in their name to the temperature subfolder and files that have weather in their name to the weather subfolder in both D0 and D1 folders

main Directory
|
|___ weather_day
        ├── D0
           ├── temperature
        │  └── weather
           |__ 20210308_DML_D0_Temp_s1_t1.txt
           |__ 20210308_DML_D1_weather_s3_t6.txt
        └── D1
           ├── temperature
           └── weather
           |__ 20210308_DML_D0_Temp_s1_t1.txt
           |__ 20210308_DML_D1_weather_s3_t6.txt

I tried to do it with a for loop such as:

wd = getwd() #set working directory to subfolder
pathway = paste0(wd,"/weather_day/")

for (i in pathway){
    file.copy(i,"temperature)
    file.copy(i,"weather")
}

In the end I want it like this that the txt files are in the folder according whether they have temperature or weather in their name:

main Directory
    |
    |___ weather_day
            ├── D0
               ├── temperature
                        |__20210308_DML_D0_Temp_s1_t1.txt
               └── weather
                        |__ 20210308_DML_D0_weather_s3_t6.txt
            ├── D1
               ├── temperature
                        |__20210308_DML_D1_Temp_s1_t1.txt
               └── weather
                        |__20210308_DML_D1_weather_s3_t6.txt

However, it does not work for me. I think I have to use file.copy, but how can I use this function to move the file based on a certain name pattern of the file and can I use a for loop in a for loop to read over the folders D0 and D1 and then the txt files in these folders?

alexdegrote1995
  • 95
  • 8

2 Answers2

1

Edited to include more filenames, pre-conditions (no dir structure), and post-conditions. (Plus move instead of copy.)

files <- c("20210308_DML_D0_Temp_s1_t1.txt", "20210308_DML_D0_weather_s3_t6.txt",
           "20210308_DML_D1_Temp_s1_t1.txt", "20210308_DML_D1_weather_s3_t6.txt")
# write some temp (empty) files for copying
for (f in files) writeLines(character(0), f)

parts <- strcapture(".*_(D[01])_([Tt]emp|[Ww]eather)_.*", files, list(d="", tw=""))
parts
#    d      tw
# 1 D0    Temp
# 2 D0 weather
# 3 D1    Temp
# 4 D1 weather

dirs <- do.call(file.path, parts[complete.cases(parts),])
dirs
# [1] "D0/Temp"    "D0/weather" "D1/Temp"    "D1/weather"

### pre-condition, only files, no dir-structure
list.files(".", pattern = "D[0-9]", full.names = TRUE, recursive = TRUE)
# [1] "./20210308_DML_D0_Temp_s1_t1.txt"    "./20210308_DML_D0_weather_s3_t6.txt" "./20210308_DML_D1_Temp_s1_t1.txt"   
# [4] "./20210308_DML_D1_weather_s3_t6.txt"

### create dirs, move files
Vectorize(dir.create)(unique(dirs), recursive = TRUE) # creates both D0 and D0/Temp, ...
#    D0/Temp D0/weather    D1/Temp D1/weather 
#       TRUE       TRUE       TRUE       TRUE 
file.rename(files, file.path(dirs, files))
# [1] TRUE TRUE TRUE TRUE

### post-condition, files in the correct locations
list.files(".", pattern = "D[0-9]", full.names = TRUE, recursive = TRUE)
# [1] "./D0/Temp/20210308_DML_D0_Temp_s1_t1.txt"       "./D0/weather/20210308_DML_D0_weather_s3_t6.txt"
# [3] "./D1/Temp/20210308_DML_D1_Temp_s1_t1.txt"       "./D1/weather/20210308_DML_D1_weather_s3_t6.txt"
r2evans
  • 141,215
  • 6
  • 77
  • 149
  • Thank you for your suggestion. However, if I understand your script correctly, you will only use the two example files to go move to the map. Those two were only examples. I have more files than those two with a similar structure in the folder. Is there a way to loop over all the files and move them to the correct folder in both D0 and D1? – alexdegrote1995 Mar 29 '21 at 17:49
  • Your interpretation is not quite correct, this works on 2 or 2000 (or more) without a concern. The only limit is that the parse-able format of the filenames needs to be consistent. Try my call with the full vector of names and let me know if you have more questions. – r2evans Mar 29 '21 at 17:54
  • 1
    Okay, interesting. And where in the code is stated what the pathway is where you want to look for the files? Is that the do.call(file.path) function? Because when I run this code, i get an error: Error in dir.create(unique(dirs), recursive = TRUE) : invalid 'path' argument – alexdegrote1995 Mar 29 '21 at 18:16
  • My bad completely ... `dir.create` needs to be vectorized ... stby 10 secs, see my edit, sorry about that. – r2evans Mar 29 '21 at 18:40
  • With your code this is the output: [1] "D0/Temp" "D1/weather" D0/Temp D1/weather TRUE TRUE [1] "D0/Temp/20210308_DML_D0_Temp_s1_t1.txt" [2] "D1/weather/20210308_DML_D1_weather_s3_t6.txt" [1] FALSE FALSE It seems that it creates two new folders in the main directory: D0 and D1. In D0 there is a temp folder and in the D1 folder there is a weather folder created. I updated my question above with the folder structure what the current structure is and what I actually want. In my case the folders temperature and weather are already created. – alexdegrote1995 Mar 30 '21 at 16:53
  • Maybe that gives you some more indication what I actually want to do – alexdegrote1995 Mar 30 '21 at 16:55
  • Not sure what you were seeing that is not correct. I've edited my answer to *demonstrate* an actual walk-through creating the directories and copying the files into place. – r2evans Mar 30 '21 at 17:11
  • Hi, I'm still having a problem with the code. It seems that it only copies the files that are listed in the value files and not the other files with the same pattern. I tried to solve this with the a list.files function, but then it overwrites the files to files with the same name, but size 0 instead of copying them. How can I change this? – alexdegrote1995 Apr 17 '21 at 11:11
1

You didn't provide very much information to go off of. If I understand what you're asking, this should work.

library(tidyverse)

# collect a list of files with their paths
collector = list.files(paste0(getwd(), "/weather_day"), 
                       full.names = T, # capture the file names along with the full path
                       recursive = T)  # look in subfolders 

# establish the new 'weather' path
weather = paste0(getwd(), "/weather/")

# establish the new 'temp' path
temp = paste0(getwd(), "/temp/")

collector = data.frame("files" = collector) %>%    # original path
  mutate(files2 = ifelse(str_detect(str_extract(files, 
                                                "([^\\/]+$)"),
                                    "weath"),  # if weather, make a new path
                         paste0(weather, 
                                str_extract(files,
                                            "([^\\/]+$)")
                         ), # end paste0/ if true
                         ifelse(str_detect(str_extract(files,
                                                       "([^\\/]+$)"),
                                           "temp"), # if temp, make a new path
                                paste0(temp, 
                                       str_extract(files,
                                                   "([^\\/]+$)")
                                ), # end paste0/ if true
                                files)    # if not weather or temp, no change
  ) # end if
  ) # end mutate

dir.create(weather)    # create directories
dir.create(temp)

# move the files
file.rename(from = collector[,1],
            to = collector[,2])

# validate the change
list.files(weather) # see what's different
list.files(temp)    # see what's different

Based on what @alexdegrote1995 added, how about this:

# collect a list of files with their paths
collector = list.files(paste0(getwd(), "/weather_day"), 
                       full.names = T, # capture the file names along with the full path
                       recursive = T)  # look in subfolders 

# establish the new 'weather' path
weather = paste0(getwd(), "/D0/weather/")

# establish the new 'temp' path
temp = paste0(getwd(), "/D0/temperature/")

collector = data.frame("files" = collector) %>% 
  mutate(files2 = ifelse(str_detect(str_extract(files, 
                                                "([^\\/]+$)"),
                                    "weath"),
                         paste0(weather, 
                                str_extract(files,
                                            "([^\\/]+$)")
                         ), # end paste0/ if true
                         ifelse(str_detect(str_extract(files,
                                                       "([^\\/]+$)"),
                                           "temp"),
                                paste0(temp,
                                       str_extract(files,
                                                   "([^\\/]+$)")
                                ), # end paste0/ if true
                                files)    # if not weather or temp, don't change
  ), # end if
  filesD1 = paste0(gsub(pattern="D0",          # make a third column for the D1 folder
                        replacement="D1",
                        x =files2,))) # end mutate


file.rename(from = collector[,1],  # move files to the D0 folder
            to = collector[,2])

file.copy(from = collector[,2],    # add copy to the D1 folder
          to = collector[,3])
Kat
  • 15,669
  • 3
  • 18
  • 51
  • Thank you for your suggestion. I updated my question a little bit with the folder structure that I want and where the files has to be put in. The folder weather and temperature are already created. I want to loop over the txt files and depending on whether they have weather or temp in their name place them in the corresponding folder, weather or temperature respectively in both D0 and D1. – alexdegrote1995 Mar 29 '21 at 17:52
  • See if the changes I made are what you're looking for @alexdegrote1995 – Kat Mar 30 '21 at 01:29
  • With your code I get an error: Error in file.rename(collector[, 1], to = collector[, 2]) : invalid 'from' argument. What is the problem here? – alexdegrote1995 Mar 30 '21 at 16:51
  • @alexdegrote1995 I ran the code again and I am not getting an error. Can you tell me the results of `dim(collector)`? You should have 3 columns and it should be a data frame. Are there any other warnings given? – Kat Apr 03 '21 at 06:06
  • I get indeed a dataframe with three columns which are called files, files2 and filesD1. The dataframe that I get has four rows. I think there is something wrong with the second and third row, because in the 1st and 3rd row I get the number of the row in it (1 in row 1 and 3 in row three. In the second and fourth row I get a string with the location of the "20210308_DML_D0_weather_s3_t6.txt" file. – alexdegrote1995 Apr 04 '21 at 08:01
  • This is a great way to move all of the files. I just used it on 83 text files. However, if you are currently have R set to that directory as the working dir `list.files(paste0(getwd(), "/weather_day")),` returns an empty vector because it is looking for another folder. So it needs to be left as `list.files()`. – alittleloopy Oct 09 '21 at 15:48