0

i built application in django that gives the user a table of employees, it works okay to a point when i want to filter this table - basically I'm sending HTTP GET request with a name or part of a name as one of the parameters.

Problem is, I'm czech and we use weird characters in our names like Ř/ř or Ě/ě, i cannot filter these people because there's an encoding problem.

UnicodeEncodeError: 'latin-1' codec can't encode character '\u0161' in position 324: ordinal not in range(256)

My understanding is that django is trying to use latin-1 which doesn't contain those characters, so it cannot handle them.

How would i change that? I'm lost, please help me, thank you.

This is my view function i use:

def track1(request):
    year = int(request.COOKIES.get('year', str(date.today().year)))
    month = int(request.COOKIES.get('month', str(date.today().month - 1)))
    name = request.COOKIES.get('name', '')
    project = request.COOKIES.get('project', '')
    if month == 0:
        month = 12

    sql_query = """select p.name project,
            p.id projectid,
            i.title issue,
            i.id issueid,
            u.id userid,
            u.name, 
            replace(ROUND(t.time_spent/3600.0, 1)::text, '.', ',') as spent,
            TO_CHAR(t.spent_at + interval '2h', 'dd.mm.yyyy HH24:MI:SS') date_spent, substring(n.note for 300) note
            from issues i
            left join projects p on p.id = i.project_id
            left join timelogs t on t.issue_id = i.id
            left join users u on u.id = t.user_id
            left join notes n on n.id = t.note_id"""

    if(request.method=="GET"):
        form = UserSpentOnProjectForm(request.GET)

        if form.is_valid():
            year = form.cleaned_data['year']
            month = form.cleaned_data['month']
            name = form.cleaned_data['name']
            project = form.cleaned_data['project']


    user_spent_on_project = Userspentonprojects.objects.raw(sql_query +
        ''' where (t.spent_at + interval '2h') between '%s-%s-01' and '%s-%s-%s 23:59:59'
            order by 8''', [year, month, year, month, calendar.monthrange(year=year, month=month)[1]]
    )
    if name is not "" and project is "":
        name = '%' + name + '%'

        user_spent_on_project = Userspentonprojects.objects.raw(sql_query +
            ''' where (t.spent_at + interval '2h') between '%s-%s-01' and '%s-%s-%s 23:59:59' and u.name LIKE %s
                order by 7, 1, 3''', [year, month, year, month, calendar.monthrange(year=year, month=month)[1], name]
        )
    if project is not "" and name is "":
        project = '%' + project + '%'
        user_spent_on_project = Userspentonprojects.objects.raw(sql_query +
            ''' where (t.spent_at + interval '2h') between '%s-%s-01' and '%s-%s-%s 23:59:59' and p.name LIKE %s
                order by 7, 1, 3''', [year, month, year, month, calendar.monthrange(year=year, month=month)[1], project]
        )
    if project and name is not "":
        project = '%' + project + '%'
        name = '%' + name + '%'
        user_spent_on_project = Userspentonprojects.objects.raw(sql_query +
            ''' where (t.spent_at + interval '2h') between '%s-%s-01' and '%s-%s-%s 23:59:59' and p.name LIKE %s and u.name LIKE %s
                order by 7, 1, 3''', [year, month, year, month, calendar.monthrange(year=year, month=month)[1], project, name]
        )

    form = UserSpentOnProjectForm(initial={
        'year': year,
        'month': month,
        'name': name.strip("%"),
        'project': project.strip("%")
    })

    spent_sum = 0
    for record in user_spent_on_project:
        spent_sum = spent_sum + float(record.spent.replace(",", "."))

    context = {
        'user_spent_on_project' : user_spent_on_project,
        'form' : form,
        'name': name.strip("%"),
        'project': project.strip("%"),
        'month': month,
        'year': year,
        'spent_sum' : round(spent_sum, 2)
    }

    response = render(request, 'trackApp/track1.html', context=context)
    response.set_cookie('year', year)
    response.set_cookie('month', month)
    response.set_cookie('project', project.strip("%"))
    response.set_cookie('name', name.strip("%"))
    return response

My db config in settings.py looks like this:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        # NASTAVENI PRO LOKALNI TESTOVANI
        'NAME': 'dumptables',
        'USER': 'postgres',
        'PASSWORD': 'admin',
        'HOST': 'localhost',
        'PORT': 5432,
    }
}

Traceback:

[20/Oct/2022 14:07:03] "GET /tracker/us HTTP/1.1" 200 14078
Traceback (most recent call last):
  File "C:\Users\myname\AppData\Local\Programs\Python\Python310\lib\wsgiref\handlers.py", line 138, in run
    self.finish_response()
  File "C:\Users\myname\AppData\Local\Programs\Python\Python310\lib\wsgiref\handlers.py", line 184, in finish_response
    self.write(data)
  File "C:\Users\myname\AppData\Local\Programs\Python\Python310\lib\wsgiref\handlers.py", line 288, in write
    self.send_headers()
  File "C:\Users\myname\AppData\Local\Programs\Python\Python310\lib\wsgiref\handlers.py", line 347, in send_headers
    self._write(bytes(self.headers))
  File "C:\Users\myname\AppData\Local\Programs\Python\Python310\lib\wsgiref\headers.py", line 142, in __bytes__
    return str(self).encode('iso-8859-1')
UnicodeEncodeError: 'latin-1' codec can't encode character '\u0161' in position 290: ordinal not in range(256)
Bergerino
  • 23
  • 5
  • That won't be in the code, that'll be in settings somewhere, or in the database setup. As you probably know, you want to use either latin-2 or utf-8 (or possibly utf8mb4)... – Jiří Baum Oct 20 '22 at 11:29
  • Yess, one of those.. Could the problem be in my db config in settings.py? I have updated my post and added the configuration lines – Bergerino Oct 20 '22 at 11:42
  • Could also be in the database itself, the encoding configured in its schema – Jiří Baum Oct 20 '22 at 11:56
  • BTW, do you have the full traceback for the exception? That may tell us which part of the system has the problem encoding... – Jiří Baum Oct 20 '22 at 11:57
  • i don't think there's anything wrong with the database, because when displaying unfiltered employees (displaying all of them at once) displays all of the employees even those with bad characters. I have updated the post with the traceback – Bergerino Oct 20 '22 at 12:26
  • Ah, the traceback is in wsgi; it'll be that part of the configuration, around wsgi and the interface with the webserver. – Jiří Baum Oct 20 '22 at 12:44
  • And someone else is pointing out that the problem is cookies; in that case, you just need to encode and decode those somehow – Jiří Baum Oct 20 '22 at 12:46

0 Answers0