2

I have a csv file being read into python, I then save the reader as an array (I guess).

I then compare the csv file results against some Oracle db results:

readerSetSAP = []
readerSAP = csv.reader(StringIO.StringIO(request.POST['sap'].value),dialect=csv.excel)
readerSetSAP.extend(readerSAP)

empsTbl = meta.Session.query(model.Person).all();

Then use a nested loop to compare:

 if i.userid != currEmp[0].strip():
                        updated = True
                        print "userid update"

The problem is, I often have the warning:

eWarning: Unicode unequal comparison failed to convert both arguments to Unicode - interpreting them as being unequal

So my question is:

What is the most robust way of comparing strings of this type in Python?

RonnyKnoxville
  • 6,166
  • 10
  • 46
  • 75
  • 1
    Can you give an example where that warning appears? My first suggestion is that you have to be sure if both arguments are equally encoded. So you need to know how the input data is encoded to convert it to unicode properly. – Diego Navarro Jan 19 '12 at 11:45
  • You are leaving out some relevant clues. What module are you using to read the Oracle database -- what does "meta.Session.query"refer to? How does your loop code set I and currEmp? What is the type of the DB field supplying currEmp[0]? – Jim DeLaHunt Jan 19 '12 at 11:47
  • Check this answer on how to understand unicode in Python: http://stackoverflow.com/a/1818326/244672 Hope this helps. – Miguel Rodrigues Jan 19 '12 at 11:48

1 Answers1

5

Your problem here is not 'a robust way " to compare strings. A robust way to compare strigns in Python is the equality operator == - Your problem is that your data is being covnerted to Unicode somewhere, without you being aware of that.

You, and everyone else who writes code, should be aware that text is not ASCII - not in a post 1990 world. Even if all of your application is restricted to English only, and should never run in an internatiol environment, you are bound to find some non-ASCII characters in peoples names, or in words like "resumé".

Here is a Python console example of when the problem might happen:

>>> "maçã" == u"maçã"
__main__:1: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
False

Python's CSV module do no authomatic conversion, and works with byte strigns (that is - strigns aready converted to some encoding) - which means that that result you are fetching from the DB is in Unicode. Probably your connection is using some default.

To solve that, assuming the data in your database is correctly formatted (and you did not already lost character information during the insertion), is to decode the string read from the CSV file, using an explicit encoding - so that both are in unicode (Python's internal encoding agnostic) string format -

>>> "maçã".decode("utf-8") == u"maçã"
True

So, you do use the "decode" method on the string read form the CSV file in order to have a proepr conversion, before comparing it. If you are on Windows, use the "cp1251" for decoding., In any other mainstream (application) O.S. it should be "utf-8".

I'd advise reading of this piece - it is rather useful: http://www.joelonsoftware.com/articles/Unicode.html

jsbueno
  • 99,910
  • 10
  • 151
  • 209