4

EDIT: To save people the trouble of scrolling, the problem stemmed from the "decode" operation needing an output variable; my script failed to do this. I thought that the "for" loop would modify the variable in-situ, but this isn't the case.

To make a long story short, I have some netCDF files from which I am generating a series of maps. The script for that works fine, but I am having major issues getting the title to display correctly. I get the variable from the netCDF file that will act as my title (a simple timestamp, basically). First I tried making it a Python variable, then using it as the plot title.

Unfortunately, I have learned that it is what is known as a "bytes" string. Which means that the title has a bunch of lowercase 'b's in front of them. Not just one at the start. ie:

b'T' b'i' b't' b'l' b'e'

This is because the netCDF variable is a masked array. I managed to get some workable code to convert that array into a list, then into a string, and everything seemed like it would work. The linchpin of the entire thing, however, is the "bytes.decode()" operation.

As far as I understand it, this operation takes in bytes objects, and then returns them as plain strings. Afaik, these are in utf-8, and I checked the type going in and found they were all classed as "bytes". Yet, when I attempt to use decode, it tells me that the objects are not bytes, literally moments after it told me they were? See code below and the output/error.

Code:

#check the type, shape, and data of times
print(type(times))
print(times.shape)
print(times.data)

#change the times masked array to a list
timeslist = times.tolist(fill_value=-9999)

#check to see if elements of the list are bytes
    for x in timeslist:
    print(type(x))

#new list for decoded chars
fixedtimeslist = []

#decode the bytes list      
for x in timeslist:
    bytes.decode('utf-8')
    fixedtimeslist.append(x)

Output/Error:

<class 'numpy.ma.core.MaskedArray'>
(19,)
[b'2' b'0' b'1' b'2' b'-' b'1' b'0' b'-' b'0' b'4' b'_' b'0' b'3' b':' b'0' b'0' b':' b'0' b'0']
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
Traceback (most recent call last):
  File "Wind10.py", line 82, in <module>
    bytes.decode('utf-8')
TypeError: descriptor 'decode' requires a 'bytes' object but received a 'str'

EDIT: A few people have asked, and yes I have tried doing this using "x.decode" an iteration ago. When I do that instead, and re-check the type, it remains as bytes.

Code:

#decode the bytes list
for x in timeslist:
    x.decode('utf-8')
    fixedtimeslist.append(x)

#recheck to see if decode worked
for x in fixedtimeslist:
    print(type(x))

Output:

(19,)
[b'2' b'0' b'1' b'2' b'-' b'1' b'0' b'-' b'0' b'4' b'_' b'0' b'3' b':' b'0' b'0' b':' b'0' b'0']
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>
<class 'bytes'>

So I'm kind of at a loss for what to do with this problem. I don't know if I'm just not understanding something in the semantics or I found a bug or what.

I realize that questions similar to this have been asked, and I have seen them, and tried to emulate their solutions with no success. This is the 4th or 5th program iteration I've tried. Either the decode seems to do nothing at all (ie: the string still has the b'' part), or I get this error.

If it matters, I'm using Python 3.6 miniconda on CentOS 6.8 I think.

Any and all help is appreciated! I apologize if this is trivial; I'm no computer scientist.

Brock
  • 53
  • 1
  • 6
  • It looks like you have a typo, shouldn't `bytes.decode('utf-8')` be `x = x.decode('utf-8')`? You're trying to decode each element in the list and take the result of that to append to a list, right? – Thtu Aug 09 '18 at 23:43
  • I've tried that as well, and it doesn't seem to do anything. No error, and upon re-checking the types, it still says they're bytes. – Brock Aug 09 '18 at 23:45
  • Decoding something from bytes returns string. RETURNS. you have to put it into variables. Decodig Does NOT modify the initial variable. In your code you put initial variable into list, which of course, only contain bytes again. – dgan Aug 09 '18 at 23:48
  • Change this particular piece of code `bytes.decode('utf-8')` to exactly `x = x.decode('utf-8')` (which it does not sound like you've tried at all. As @dgan alluded to, you need to actually DO something with the result of decoding your bytestring. x.decode('utf-8') just creates an unnamed variable that gets tossed. Again, change this line `bytes.decode('utf-8')` to *exactly* `x = x.decode('utf-8')`. It does not seem like you've tried this at all. – Thtu Aug 09 '18 at 23:53
  • Wait I thought when I had the for loop: " for x in timeslist: x.decode('utf-8') fixedtimeslist.append(x)" I was appending the "fixed" value? – Brock Aug 09 '18 at 23:58
  • Tried that, and it did the trick. I see; so I just had to stash what the decode did somewhere. I thought it would work through the "for" loop in order of appearance, as you said, "modifying" the x. I guess it doesn't. As I thought, it was something trivial. Thanks a ton! e-cookies for you folks. – Brock Aug 10 '18 at 00:05
  • `bytes.decode` does not modify anything in-place. You can check whether a method/function/operator does something inplace by binding another name to the same variable, running the method/function/operator in question and then comparing the two variables. e.g. Not in place: `a = b = [1,2,3]; a = a + [4]; assert not (a is b);` In place: `a = b = [1, 2, 3]; a += [4]; assert a is b` (the `+` operator is *not* inplace, and the `+= *is* inplace.) – Thtu Aug 10 '18 at 00:12
  • I think I understand. In the first example, just because a = b =[1,2,3] was first, doesn't mean that the addition that follows will also apply to b; You have to explicitly tell it to do so? – Brock Aug 10 '18 at 00:29
  • Yes, for operations/functions/methods that do not act on an object in place, you need to rebind their results back to a variable to use that result. For operations that DO act in place, they return nothing so they'll "act through" or "modify" the object directly. `bytes.decode` does NOT act inplace, it returns a new object. – Thtu Aug 10 '18 at 01:08

2 Answers2

1

You have to decode x into bytes and append the returning value to the fixedtimeslist list.

for x in timeslist:
    fixedtimeslist.append(x.decode('utf-8'))
blhsing
  • 91,368
  • 6
  • 71
  • 106
  • 1
    That works similarly to what others suggested above. I thought it would simply work through the "for" loop, changing the x as it went, but this is not the case I guess. Thanks for your time. – Brock Aug 10 '18 at 00:07
  • fixedtimeslist = [x.decode('utf-8') for x in timeslist] # via list comprehension – Steve L Dec 09 '22 at 23:08
0

I think you mean x.decode('utf-8'). I haven't any experience with bytes objects but I believe this is what you want

N Chauhan
  • 3,407
  • 2
  • 7
  • 21
  • I've tried that as well, and it doesn't seem to do anything. No error, and upon re-checking the types, it still says they're bytes. – Brock Aug 09 '18 at 23:45