2

Due to my lack of experience with script language, I have referred to several sources: link seemed resolved with ls *.extension | xargs -n 1 tail -n +2

This didn't for me, even after adding > merged.txt at the end, nor the following:

for f in *.txt
do
    tail -n +2 $f >> /path/to/some/dir/with/files/file_name
done

I also tried sed -e'1d' $FILE in replacement of the tail command. Didn't work.

tail -n +2 file_name.extension, cat LIN_1994-11_0100.txt | tail -n +2, awk 'FNR != 1' *.extension has no effect to the file.

I am uncertain if this has anything to do with the current issue.

If anyone could find the reason for this problem or way out of it..would be majorly grateful. Thanks for all the input so far as well.

dia
  • 431
  • 2
  • 7
  • 22
  • Are any of your commands are working for an individual file? – NVRM Jan 16 '18 at 17:13
  • If the above commands are functional for other users, then I'm afraid not. Didn't produce any change.. – dia Jan 16 '18 at 17:15
  • `tail -n +2 file` doesn't delete first line? – PesaThe Jan 16 '18 at 17:19
  • I now know why I got tail, illegal offset error (spacing btw n and number) but still not sure why the rest commands are unworking.....what am I doing wrong.. could anyone try with the file I'm using? – dia Jan 16 '18 at 17:19
  • So is does `tail -n +2 file` work on one file? – PesaThe Jan 16 '18 at 17:21
  • it doesn't print any error, but makes no change to the file. – dia Jan 16 '18 at 17:23
  • 1
    if you mean `/path/to/some/dir/with/files` when you write `(directory_path)`, you should say so. `(dir_path)/file` will not work. Can't see why your `tail` solution isn't working (seems like it should). If your files are from MS-Windows, then test one with `cat -vet file | head -10`. If you see `^M$` at the end of each line, the cleanup your whole set of files with `dos2unix files*` and the retest your code. Good luck. – shellter Jan 16 '18 at 17:24
  • 1
    It's not supposed to change any files. It produces output on stdout. – melpomene Jan 16 '18 at 17:24
  • Just try this: `tail -qn +2 *.txt >> merged`. – PesaThe Jan 16 '18 at 17:25
  • I think O.P. means that `merged.txt` is "still" empty. Good luck to all. – shellter Jan 16 '18 at 17:25
  • @melpomene whoopsie. You are right. Fixed. – PesaThe Jan 16 '18 at 17:26
  • Do you mean try `tail -q n +2 *.txt >> merged` or `tail -q n +2 *.txt >> directory/filename`? I don't know all the parameters very well, but I haven't seen -q and n together before.. – dia Jan 16 '18 at 17:29
  • You can group options like that :) `tail -q -n +2` = `tail -qn +2`. – PesaThe Jan 16 '18 at 17:30
  • Didn't tried the above, should works. I would try to echo instead echo $(tail -qn +2 *.txt) >> merged (Untested) – NVRM Jan 16 '18 at 17:38
  • sorry guys, but none worked. :( [this](https://unix.stackexchange.com/questions/154861/why-does-tail-f-tail-fail-to-produce-any-output) tells me something but I'm not using -f parameter, so unsure if it relates to my issue yet. – dia Jan 16 '18 at 17:41
  • I tried from my home the above echo commands, and it create a file named merged containing some melimelo text from many of my txt files. It should not be far away. You need to echo the command result stdout in the file. – NVRM Jan 16 '18 at 17:44
  • You are not using piping with the command showns. (Piping use the char | ) – NVRM Jan 16 '18 at 17:46
  • I tried echo for the base file (copied the first file with header) in another directory -> made no change; and for the same directory without .txt extension -> got an empty file – dia Jan 16 '18 at 17:49
  • Actually, this `cat myfile.txt | tail -n +2` didn't make any change either. I don't know about piping for other commands. When I don't know, I just apply them as they are advised. – dia Jan 16 '18 at 17:51
  • Just do: echo $(find *txt) >> merged You should get a list of all your txt files. It must works first – NVRM Jan 16 '18 at 17:53
  • That gave me a list of files in txt extension as you intended. It produced nothing when I used *.txt instead of *txt.. which is fine since I did merge all the files using `cat * > merged_file.txt` – dia Jan 16 '18 at 17:55
  • 1
    Are you entirely sure `tail -q -n +2 *.txt >> merged` doesn't work? – PesaThe Jan 16 '18 at 17:56
  • The thing is, my terminal gets stuck when I type that command. It processes for a time, and then produces nothing. The same is with echo `$(tail -qn +2 *.txt) >> merged`. Which is why I wondered if it is an issue with buffer? But @Cryptopat where should I pipe? – dia Jan 16 '18 at 18:05
  • Forget piping for a time. First try to just concatenate, it must works! echo $(cat *.txt) >> merged – NVRM Jan 16 '18 at 18:12
  • 1
    the terminal is nearly stuck;; and my laptop is making a loud noise compare to when I used `cat * > merged_file.txt` command. Still i can see the progress - the file is collecting data from text files. And it collected all! with the headers included haha – dia Jan 16 '18 at 18:22
  • Yep, good start. Just play around with tail, it will comes. You are not in a good dev environment for learning and practice. You should train with some dummy files from a lighter folder. As soon as your command works for a couple of files, it will also for many hundreds. Good luck – NVRM Jan 16 '18 at 18:32
  • 1
    Again, your original code ideas should work. Did you test your files with `cat -vet file | head -10` per my comments a while ago? Note that you'll need to truncate `merged.txt` before starting the loop, or the append operation will alway be putting your new attempt to solve the problem **at the end of the file**, where you will not see it until you deliberately scroll done. Before your `for` loop, add `true> merged.txt` to truncate the file, so you're always starting for a know point (an empty file). Good luck. – shellter Jan 16 '18 at 19:09
  • `cat -vet file | head -10` gave me a very massive and messy..output. I checked that these are to display 'non-printing characters' and I'm not sure how to understand this. Is this why the `tail` command is malfunctioning? Of course I scrolled down to check the data were merged every time. I don't know what `true> merged.txt` means, but let me try after the tail/awk/sed or whichever works for at least a single file..thanks – dia Jan 16 '18 at 19:18
  • 2
    If `cat -vet file | head -10` gives a massive output, there is something fundamentally wrong because that command only prints 10 lines. So your lines are longer than you think. So the files either have Windows line-endings, or are binary, or otherwise non text-like. Please try running the `file` command on a few and see what type of files it thinks you have. So in concrete terms, please run `file OneOfYourFiles.txt` – Mark Setchell Jan 16 '18 at 23:24
  • It printed `ASCII text, with CR line terminators`! Thank you for a little bit of explaining. I couldn't entirely understand the directions from @shellter....sorry. So it seems I do have ^M at the end, not $ though...do I still need to use `dos2unix files*` ? Or mac2unix like [here](https://stackoverflow.com/questions/14080306/remove-cr-line-terminators)? Btw these data are retrieved from ftp server, and I'm on mac. – dia Jan 17 '18 at 00:00
  • `mac2unix` on a single file finally gave me different outputs for cat -vet file | head -10 plus tail command is now working!!! Special thanks to @MarkSetchell and shellter – dia Jan 17 '18 at 00:20
  • for future Qs you'll want to include a tag for [OS-X] or which ever has the greatest number of followers. Glad you got your code to work. – shellter Jan 17 '18 at 03:41

2 Answers2

0

How about

awk 'FNR==1 {next}; 1' *.txt > all_data.txt
glenn jackman
  • 238,783
  • 38
  • 220
  • 352
  • This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - [From Review](/review/low-quality-posts/18536073) – Jodocus Jan 16 '18 at 21:19
  • @Jodocus Why do you think it doesn't? – PesaThe Jan 16 '18 at 22:03
0

You haven't mentioned what shell you're using a POSIX shell (or bash et al), the following might do:

$ for f in *.txt; do { read x; cat; } < "$f"; done
ghoti
  • 45,319
  • 8
  • 65
  • 104