0

This is my first question here.

I tried to load a data file with Python. The file demo.txt is similar as below.

12,23,34.5,56,

78,29,33,

44,55,66,78,59,100

(the number of the lines in the files are different and the number of column in each line may be different. I need to work on many data files)

numpy.loadtxt("demo.txt",delimiter=",")

gives the error message "could not convert string to float:".

To fix this problem, I try to use the command 

sed -i -e 's/,\n/,/g' demo.txt

to remove the line breaks at the end of each line to combine all lines into a single line. But it failed.

However, in the VIM, it is OK to use ":s/,\n/,/g" to remove the line breaks.

Thus, my questions are

  1. is it possible to load the data file in python without modifying the files?

  2. if not, how can I use a command like "sed" (as I need to put this command into my script to handle a bunch of files, a shell command like "sed" is necessary) to remove the line breaks at the end of each line to combine all lines into one single line? Without the line breaks at all lines, I can read the data with numpy.loadtxt easily.

Best regards,

Yiping

Yiping
  • 3
  • 1
  • 3
  • 1
    This is similar to the below: https://stackoverflow.com/questions/16729210/numpy-loadtxt-valueerror-could-not-convert-string-to-float – haraprasadj Feb 06 '18 at 10:21
  • Possible duplicate of [numpy.loadtxt, ValueError: could not convert string to float](https://stackoverflow.com/questions/16729210/numpy-loadtxt-valueerror-could-not-convert-string-to-float) – phd Feb 06 '18 at 13:47
  • the number of the columns of the lines are different. Thus, the workaround in the other questions is not working here. – Yiping Feb 06 '18 at 15:47
  • You want to remove newlines? Do you mean that you want to combine all of the lines into a single line? Or do you just want to remove the trailing commas? – Beta Feb 06 '18 at 18:22
  • Sorry that I didn't make it clear. Yes, combing all lines into a single line -- this is what I want. Then it will be easy to be handled by numpy.loadtxt. – Yiping Feb 07 '18 at 03:01

1 Answers1

0

Remove all newlines from a file with tr -d '\n':

$ echo -e "some\nfile\nwith\n\newlines" > file_with_newlines
$ cat file_with_newlines 
some
file
with

ewlines
$ cat file_with_newlines | tr -d '\n' > file_without_newlines
$ cat file_without_newlines 
somefilewithewlines$

I don't know if this will actually help you with your numpy problem, but it will remove all the (UNIX) newlines from a file.

JawguyChooser
  • 1,816
  • 1
  • 18
  • 32