6

I have a bit trouble with some data stored in a text file on hand for regression analysis using Python.

The data are stored in the format that look like this:

2104,3,399900 1600,3,329900 2400,3,369000 ....

I need to do some analysis like finding mean by this: (2104+1600+...)/number of data

I think the appropriate steps is to store the data into array. But I have no idea how to store it. I think of two ways to do so. The first one is to set 3 array that stores like 

a=[2104 1600 2400 ...] b=[3 3 3 ...] c=[399900 329900 36000 ...]

The second way is to store in 

a=[2104 3 399900], b=[1600 3 329900] and so on.

Which one is better?

Also, how to write code that allows the data can be stored into array? I think of like this:

with open("file.txt", "r") as ins:
array = []
elt.strip(',."\'?!*:') for line in ins:
array.append(line)

Is that correct?

Sнаđошƒаӽ
  • 16,753
  • 12
  • 73
  • 90
poonck1
  • 71
  • 1
  • 1
  • 6
  • 4
    please properly indent your code example. Best way to do so is to copy and paste your code with proper formatting then select it all and press ctrl+K – R Nar Feb 22 '17 at 14:15
  • 1
    why not store it in one list ... `[2104,3,399900....]` – depperm Feb 22 '17 at 14:15
  • 1
    Does the file have spaces separating the triplets or newlines? – armatita Feb 22 '17 at 14:15
  • as @depperm if you are wanting to find the mean of all the data, one list would probably be better – WhatsThePoint Feb 22 '17 at 14:16
  • @depperm, how to obtain the the value that I want to have calculation? – poonck1 Feb 22 '17 at 14:25
  • @armatita, it has soace between triplets but it doesn't seem to have new lines for each triplets, should I break each triplets into new line first? – poonck1 Feb 22 '17 at 14:27
  • unclear question but if `a` is one list then `ans=sum([a[i] for i in range(0,len(a),3)])/(len(a)/3)` – depperm Feb 22 '17 at 14:30
  • Not at all. Its just easier to parse the file if the separation is always the same. The comma was clear from your question (between elements of the same triple), but not between triplets. I'll make my suggestion in a bit. – armatita Feb 22 '17 at 14:52

4 Answers4

2

Instead of having multiple arrays a, b, c... you could store your data as an array of arrays (a 2 dimensional array). For example:

[[2104,3,399900],
 [1600,3,329900],
 [2400,3,369000]...]

This way you don't have to deal with dynamically naming your arrays. How you store your data, i.e. 3 * array of length n or n * array of length 3 is up to you. I would prefer the second way. To read the data into your array you should then use the split() function, which will split your input into an array. So in your case:

with open("file.txt", "r") as ins:
    tmp = ins.read().split(" ")
    array = [i.split(",") for i in tmp]

>>> array
[['2104', '3', '399900'], ['1600', '3', '329900'], ['2400', '3', '369000']]

Edit: To find the mean e.g. for the first element in each list you could do the following:

arraymean = sum([int(i[0]) for i in array]) / len(array)

Where the 0 in i[0] specifies the first element in each list. Note that this code uses list comprehension, which you can learn more about in this post if you want to.

Also this code stores the values in the array as strings, hence the cast to int in the part to get the mean. If you want to store the data as int directly just edit the part in the file reading section:

array = [[int(j) for j in i.split(",")] for i in tmp]
Community
  • 1
  • 1
Leon Z.
  • 542
  • 1
  • 6
  • 11
  • So how to extract data for further analysis in the next step? Like (2104+1600+2400) for finding the mean? – poonck1 Feb 22 '17 at 14:47
  • I use the len(array) function, but it gives out the count to be 1. I know I have 47 set of data, how can I give this answer? – poonck1 Feb 25 '17 at 14:37
  • @poonck1 Could you please provide your whole code by updating the question or asking a new one? Or else it is difficult to know why your code is behaving this way. – Leon Z. Feb 26 '17 at 11:52
2

Using pandas and numpy you can get the data into an array as follows:

In [37]: data = "2104,3,399900 1600,3,329900 2400,3,369000"

In [38]: d = pd.read_csv(StringIO.StringIO(data), sep=',| ', header=None, index_col=None, engine="python")

In [39]: d.values.reshape(3, d.shape[1]/3)
Out[39]: 
array([[  2104,      3, 399900],
       [  1600,      3, 329900],
       [  2400,      3, 369000]])
languitar
  • 6,554
  • 2
  • 37
  • 62
2

You could use :

with open('data.txt') as data:
    substrings = data.read().split()
    values = [map(int, substring.split(',')) for substring in substrings]
    average = sum([a for a, b, c in values]) / float(len(values))
    print average

With this data.txt, :

2104,3,399900 1600,3,329900 2400,3,369000
2105,3,399900 1601,3,329900 2401,3,369000

It outputs :

2035.16666667
Eric Duminil
  • 52,989
  • 9
  • 71
  • 124
0

This a quick solution without error checking (using a list comprehension technique, PEP202). But if your file has a consistent format you can do the following:

import numpy as np

a = np.array([np.array(i.split(",")).astype("float") for i in open("example.txt").read().split(" ")])

Should you print it:

print(a)
print("Mean of column 0: ", np.mean(a[:, 0]))

You'll obtain the following:

[[  2.10400000e+03   3.00000000e+00   3.99900000e+05]
 [  1.60000000e+03   3.00000000e+00   3.29900000e+05]
 [  2.40000000e+03   3.00000000e+00   3.69000000e+05]]
Mean of column 0:  2034.66666667

Notice how, in the code snippet, specified the "," as separator inside triplet, and the space " " as separator between triplets. This is the exact contents of the file I used as an example:

2104,3,399900 1600,3,329900 2400,3,369000
armatita
  • 12,825
  • 8
  • 48
  • 49