cleaning a string from escape character with python

Question

I get Data from a webapp in JSON that includes various python escape characters including "\n" and "\r"

I build a little function to clean the data from problematic chars and spaces before feeding it to the sql. (The problematic chars are problematic to another app using the sql).

the current function is:

bad_tokens = [",",";",".","!","'",".","-",'"',"@",r"\n",r"\r"]

from types import StringType, UnicodeType

def sql_text(sqltext, trim = None):
    '''
    helper function to clean text inserted to sql from Priority problematic characters specified bad_tokens

    '''
    thistype = type(sqltext)
    if thistype not in (StringType, UnicodeType):
        return sqltext

    sqltext = sqltext.strip() #priority can't handle string starting with space
    for token in bad_tokens:
        sqltext = sqltext.replace(token,"")
    sqltext = " ".join([i for i in sqltext.split(" ") if i != ""]) #priority can't handle string containing double spaces

    if trim:
        sqltext = sqltext[0:trim]
    return sqltext

This approach works fine for regular chars but doesn't seem to clean the \n and \r escape symbols. adding r (as raw string) to the escape symbols doesn't help either.

thanks for the help

EDIT: I'm using an orm (sqlalchemy), so I don't access DBApi directly, and while sqlalchemy does a lot of escaping automatically since sql treats those chars as legal so does sqlalchemy. back to square on - I need to clean the string correctly.

Why not use the sql parameter functionality of your database client library instead? You are reinventing the wheel here. See http://wiki.python.org/moin/DbApiFaq — Martijn Pieters, Feb 14 '13 at 18:02
...or see [this](http://stackoverflow.com/questions/8115261/how-to-remove-all-the-escape-sequences-from-a-list-of-strings) question — Fredrik Pihl, Feb 14 '13 at 18:02
@MartijnPieters I'm using sqlalchemy already but \r and \n aren't escaped by it at least as I can tell from the results — alonisser, Feb 14 '13 at 21:40
@alonisser: SQLAlchemy escapes *everything* that needs escaping, most certainly including `\r` and `\n`. I think you are barking up the wrong tree here. — Martijn Pieters, Feb 14 '13 at 21:52
@MartijnPieters my problem is that sqlalchemy doesn't clean something that doesn't bother sql by itself but the other program (not mine, .net, can't fix it) using the mssql server is bothered by them, especially the \n inside one of the text fields drive it crazy.. — alonisser, Feb 14 '13 at 21:55
r'\n' is *two* characters, a backslash plus `n`. If you need to replace actual newlines, you do need to look for `'\n'` (no `r`). Can you give an example piece of input (use `repr()` so we can see if you have actual newlines in there). Replacing those bad characters can certainly be done simpler. — Martijn Pieters, Feb 15 '13 at 10:04

score -1 · Answer 1 · answered Feb 14 '13 at 18:06

-1

import re

newbuff = re.sub("\n|\r| |moreoptions","",yourbuff)

answered Feb 14 '13 at 18:06

pyInTheSky

1,459
1
9
24

cleaning a string from escape character with python

1 Answers1