Having problems with NumPy matrices

Question

I'm very familiar with MATLAB, but I'm having trouble learning Python.

The problem is creating an array X(idx,:) where I increment idx with each line I read and assign X(idx,:) to a row of data. The data, and a sample Python program for as far as I've got, follows.

File: "sample_data"

TRE-G3T- Triumph-        0.000 10/01/2013 227001.30 1760.00000 10/01/2013 227016.30   41 31 27.35998  -70 40 19.00843      -28.130        0.707   882922.244  2652775.212      -65.517      -24.677      -13.470        0.020        0.022        0.041        0.051      -13.469       -0.771   0.0109 2   1.80  7        0.005 S              0.032    -0.024     0.001  -0.256  -0.162   0.554
TRE-G3T- Triumph-        0.000 10/01/2013 227001.40 1760.00000 10/01/2013 227016.40   41 31 27.35993  -70 40 19.00854      -28.123        0.714   882922.235  2652775.207      -65.519      -24.679      -13.464        0.020        0.022        0.041        0.051      -13.463       -0.650   0.0136 2   1.80  7        0.006 S              0.005    -0.039     0.057   0.066   0.301  -0.602
TRE-G3T- Triumph-        0.000 10/01/2013 227001.50 1760.00000 10/01/2013 227016.50   41 31 27.35997  -70 40 19.00848      -28.128        0.710   882922.240  2652775.211      -65.518      -24.678      -13.468        0.020        0.022        0.041        0.051      -13.467       -0.631   0.0103 2   1.80  7        0.007 S              0.012    -0.011    -0.005   0.006  -0.094   0.606
TRE-G3T- Triumph-        0.000 10/01/2013 227001.60 1760.00000 10/01/2013 227016.60   41 31 27.36002  -70 40 19.00858      -28.122        0.715   882922.232  2652775.216      -65.520      -24.676      -13.462        0.020        0.022        0.041        0.051      -13.462       -0.829   0.0109 2   1.80  7        0.007 S              0.014    -0.021     0.056  -0.021   0.214  -0.435
TRE-G3T- Triumph-        0.000 10/01/2013 227001.70 1760.00000 10/01/2013 227016.70   41 31 27.36005  -70 40 19.00849      -28.125        0.712   882922.239  2652775.219      -65.518      -24.675      -13.465        0.020        0.022        0.041        0.051      -13.465       -1.040   0.0106 2   1.80  7        0.006 S              0.011     0.000     0.011   0.110   0.264  -0.284

Python program:

#!/usr/local/bin/python 


import numpy as np
import matplotlib.pyplot as plt


def dms2deg(deg,min,sec):
# jad - 20131103
        sgn = float(deg) / abs(float(deg))
        return sgn * (abs(float(deg)) + (float(min) + (float(sec)/60) ) / 60 )


def decdeg2dms(dd):
# http://stackoverflow.com/questions/2579535/how-to-convert-dd-to-dms-in-python
   is_positive = dd >= 0
   dd = abs(dd)
   minutes,seconds = divmod(dd*3600,60)
   degrees,minutes = divmod(minutes,60)
   degrees = degrees if is_positive else -degrees
   return (degrees,minutes,seconds)





#  indices into GrafNav output record
idx0 =  7   #  index of GPS seconds
idx1 =  8   #  beginning index for lat / lon
idx2 = 28   #  solution quality

f = open('sample_data','r')


ctr2 = -1

X=[]
X=np.array(X)



for line in f:

    var = line.split()
    l=len(var)

    if l > 35:                 #  data has more than 35 columns
        GPS_sec = var[idx0]

        lat_deg = var[idx1+0]
        lat_min = var[idx1+1]
        lat_sec = var[idx1+2]

        lon_deg = var[idx1+3]
        lon_min = var[idx1+4]
        lon_sec = var[idx1+5]

        h_ell = var[idx1+6]

        latd = dms2deg(lat_deg,lat_min,lat_sec)
        lond = dms2deg(lon_deg,lon_min,lon_sec)


        Q = var[idx2]
        stdev = var[idx2-4]
        h_sep = var[idx2-2]
        amb_drift = var[idx2-1]
        nsat = var[idx2+2]

        ctr2 += 1
             X=np.array([float(GPS_sec),float(latd),float(lond),float(h_ell),int(Q),int(nsat),float(stdev),float(h_sep),float(amb_drift)])
        print GPS_sec,  latd,  lond, h_ell,  Q, nsat, stdev, h_sep, amb_drift

f.close()

What exactly are you trying to do...return a lists that are equivalent to the full columns you want or return each row only consisting of the 3 columns? — Ryan Saxe, Nov 17 '13 at 00:46

score 1 · Answer 1 · answered Nov 17 '13 at 02:03

1

You'll get much better answers if you can focus your question on what you're really asking. Your code has lot of stuff that's irrelevant to the question that's hard to dig though to get to the real meat of the stuff. I think you want something like this:

X = []
for line in f:
    ....
    row = np.array([GPS_sec, latd, lond, h_ell, Q, nsat, stdev, h_sep, amb_drift],
                   dtype=float)
    X.append(row)

X = np.array(X)

What this does is hold the rows of your final array in a list and then convert it to an array after the loop. Also you should know that ndarrays can only have one dtype, I believe that's the same as matlab, so it's not clear what you're trying to accomplish by doing np.array([float(a), int(b)]). Hope this helps.

answered Nov 17 '13 at 02:03

Bi Rico

25,283
3
52
75

Thanks for your responses and comments on how better to post. I'm new at this site, so please forgive any trnasgressions. Basically what I'm trying to do could be done in MATLAB with, for example – user2817154 Nov 18 '13 at 20:42
I came from Matlab too so I know that some things in python/numpy are different and/or unintuitive. [This](http://wiki.scipy.org/NumPy_for_Matlab_Users) site helped me a lot, you might have already seen it. Also, [SSCCE](http://sscce.org/) is a horrible acronym, but it makes for good questions (and clear answers). – Bi Rico Nov 18 '13 at 22:13
What I'm trying to do is build a 2-D matrix that could be done in MATLAB with, for example x(1,:) = [1,2,3,4]; x(2,:) = [5,6,7,8]; ... ... And then make various plots, e.g plot(x(:,1),x(:,3)) or plot(x(:,1), x(:,4)) I'm completely bewildered by the syntax in PYTHON, from getting the data into the matrix to slicing it correctly to make a plot. jim – user2817154 Nov 19 '13 at 13:44

score 1 · Answer 2 · answered Nov 17 '13 at 03:07

It appears you just want to convert your text data to numpy array. As @BiRico pointed out, you can't have more than one data type dtypein one array, if you must do that you have to use a strutured array, or better yet, pandas dataframe.

Also numpy has build IO functions genfromtxt(), which is quite fast. Use it instead of hard code one:

>>> dtypeLS=[('Var0', 'S10'),
    ('Var1', 'S10'),     
    ('Var2', 'f8'),      
    ('Var3', '<M8[D]'),  
    ('Var4', 'f8'),      
    ('Var5', 'f8'),      
    ('Var6', '<M8[D]'),  
    ('Var7', 'f8'),      
    ('Var8', 'f8'),      
    ('Var9', 'f8'),      
    ('Var10', 'f8'),     
    ('Var11', 'f8'),     
    ('Var12', 'f8'),     
    ('Var13', 'f8'),     
    ('Var14', 'f8'),     
    ('Var15', 'f8'),     
    ('Var16', 'f8'),     
    ('Var17', 'f8'),     
    ('Var18', 'f8'),     
    ('Var19', 'f8'),     
    ('Var20', 'f8'),     
    ('Var21', 'f8'),     
    ('Var22', 'f8'),     
    ('Var23', 'f8'),     
    ('Var24', 'f8'),     
    ('Var25', 'f8'),     
    ('Var26', 'f8'),     
    ('Var27', 'f8'),     
    ('Var28', 'f8'),     
    ('Var29', 'f8'),     
    ('Var30', 'f8'),     
    ('Var31', 'f8'),     
    ('Var32', 'S10'),    
    ('Var33', 'f8'),     
    ('Var34', 'f8'),     
    ('Var35', 'f8'),     
    ('Var36', 'f8'),     
    ('Var37', 'f8'),     
    ('Var38', 'f8')]
>>> a=genfromtxt('temp.txt', dtype=dtypeLS)
>>> a[0]
('TRE-G3T-', 'Triumph-', 0.0, datetime.date(2013, 10, 1), 227001.3, 1760.0, datetime.date(2013, 10, 1), 227016.3, 41.0, 31.0, 27.35998, -70.0, 40.0, 19.00843, -28.13, 0.707, 882922.244, 2652775.212, -65.517, -24.677, -13.47, 0.02, 0.022, 0.041, 0.051, -13.469, -0.771, 0.0109, 2.0, 1.8, 7.0, 0.005, 'S', 0.032, -0.024, 0.001, -0.256, -0.162, 0.554)
>>> a['Var11']
array([-70., -70., -70., -70., -70.])
>>> a['Var12']
array([ 40.,  40.,  40.,  40.,  40.])
>>> a['Var13']
array([ 19.00843,  19.00854,  19.00848,  19.00858,  19.00849])
>>> np.sign(a['Var11'])*(np.abs(a['Var11'])+a['Var12']/60+a['Var13']/3600)
array([-70.67194679, -70.67194682, -70.6719468 , -70.67194683, -70.6719468 ])

Finally, just like in Matlab, always vectorize if you want things to run fast. See the dd-to-dms converting code in the last line.

Also, I have to change your day format from 10-01-2013 to 2013-10-01 to use datetime dtype.

Having problems with NumPy matrices

2 Answers2