0

I am trying to paste 3 CSVs together and it seems to work for the first 2 and then the 3rd pastes overtop of the 1st CSV.

csv.csv

1,Thu-Aug-19-21:54:38-2021,125094,car

tsv.csv

2,4856,PC7C042B7

fixed_width_data.csv

PTE,VC96

What I've tried:

paste -d ',' csv.csv tsv.csv fixed_width_data.csv > ed.csv

I get this:

,PTE,VC96-19-21:54:38-2021,125094,car,2,4856,PC7C042B7

Can anyone tell me what I'm doing wrong? It is pasting the 3rd file over top of the data from the first.

woogiesauce
  • 11
  • 1
  • 2
  • looks like one of your files (`tsv.csv`?) may have windows/dos line endings; run each of your files through `file ` or `od -c `; if a file has windows/dos line endings then `file` will report `... with CRLF line terminators` while `od -c` will show a `\r` character at the end of the line; quick-fix: `dos2unix ` will remove the `\r` character; you can also run `dos2unix *.csv` to check and fix all of your `*.csv` files (`dos2unix` does not modify the contents of a file that does not contain `\r` characters) – markp-fuso Apr 16 '22 at 15:22
  • It doesn't matter what I name those files, the same thing keeps happening. Is it possible that it is because the original files that the data was cut from only had 7 fields to begin with. And now when it reaches the end of the 7 fields it starts to paste at the beginning again? – woogiesauce Apr 16 '22 at 15:27
  • by *`windows/dos line endings`* I'm talking about the ***contents*** of the file, I'm ***not*** talking about the name of the file; run `od -c filename` to see what I'm talking about (do you see a `\r` character in the `od -c` output for any of the files?) – markp-fuso Apr 16 '22 at 15:28
  • Yes. The tsv.csv has \r when I run that. That is way over my head though. I am very new to all of this. – woogiesauce Apr 16 '22 at 15:42
  • right, so you want to get rid of the `\r` characters; easiest method: `dos2unix tsv.csv`; running `od -c tsv.csv` again should show the `\r` character is gone at which point your original `paste` command should work; if you get an error message `dos2unix: command not found` then we'll need to look at another method to remove the `\r` characters ... – markp-fuso Apr 16 '22 at 15:45
  • That file was originally a tab delimited file. I cut and transformed it with:cut -f5-7 tollplaza-data.tsv > tsv_data.csv; tr -s '[:blank:]' ',' < tsv_data.csv > tsv.csv – woogiesauce Apr 16 '22 at 15:49
  • I got command not found. I'm using linux. not sure if that matters – woogiesauce Apr 16 '22 at 15:51
  • this is a very common issue with lots of ways to address said issue; `unix2dos` is the easiest but there are many other methods; see the myriad of answers to [this](https://stackoverflow.com/q/11680815) and [this](https://stackoverflow.com/q/2613800); the command you used to create the file doesn't matter as much as a) the contents of the source file (ie, does it contain `\r` characters) or b) the OS in which you ran the command (eg, windows-based commands may append `\r` on the end of each line) – markp-fuso Apr 16 '22 at 15:51
  • I would suggest you spend some time perusing the 2 links I provided as they should give you a good bit of info on this issue; in the meantime `tr` is typically installed in base linux distros so try `tr -d '\r' < tsv.csv > tsv.new.csv`; those links include other options like `sed`, `awk`, `perl`, etc – markp-fuso Apr 16 '22 at 15:53
  • This is what the original file outputs: 1 Thu Aug 19 21:54:38 2021 125094 car 2 4856 PC7C042B7 – woogiesauce Apr 16 '22 at 15:54
  • run `od -c original_file` and see if there is a `\r` at the end of each line; regardless of what's in the source, or how you processed the source, you *know* the current file contains `\r` characters, and you've got the suggestions/info/links on how to remove the `\r` characters from `tsv.csv` – markp-fuso Apr 16 '22 at 15:56
  • The tr -d '\r' worked. Thank you! I guess that's a loittle frustrating because we did not learn that in our class and they just assume we're supposed to know that info. Thanks again! – woogiesauce Apr 16 '22 at 16:19

0 Answers0