UnicodeEncodeError: 'ascii' codec can't encode characters in position

[poswald]

This is a bit of a strange one. I have both django-debug-toolbar and django-rosetta installed. Once I go to the http://localhost:8000/rosetta/ url and pick a language for translation, I get this error. Once this error has come out, it won't be able to render any other pages without error. I am running this with postgresql and manage.py runserver:

Traceback (most recent call last):

    File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/core/servers/basehttp.py", line 279, in run
    self.result = application(self.environ, self.start_response)

    File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/core/servers/basehttp.py", line 651, in __call__
    return self.application(environ, start_response)

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/core/handlers/wsgi.py", line 245, in __call__
    response = middleware_method(request, response)

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/debug_toolbar/middleware.py", line 91, in process_response
    response.content = replace_insensitive(smart_unicode(response.content), u'</body>', smart_unicode(self.debug_toolbars[request].render_toolbar() + u'</body>'))

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/debug_toolbar/toolbar/loader.py", line 72, in render_toolbar
    'BASE_URL': self.request.META.get('SCRIPT_NAME', ''),

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/loader.py", line 108, in render_to_string
    return t.render(context_instance)

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/test/utils.py", line 29, in instrumented_test_render
    return self.nodelist.render(context)

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/__init__.py", line 779, in render
    bits.append(self.render_node(node, context))

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/debug.py", line 71, in render_node
    result = node.render(context)

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/defaulttags.py", line 155, in render
    nodelist.append(node.render(context))

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/defaulttags.py", line 243, in render
    return self.nodelist_true.render(context)

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/__init__.py", line 779, in render
    bits.append(self.render_node(node, context))

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/debug.py", line 71, in render_node
    result = node.render(context)

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/debug.py", line 87, in render
    output = force_unicode(self.filter_expression.resolve(context))

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/__init__.py", line 546, in resolve
    obj = self.var.resolve(context)

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/__init__.py", line 687, in resolve
    value = self._resolve_lookup(context)

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/__init__.py", line 722, in _resolve_lookup
    current = current()

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/debug_toolbar/panels/request_vars.py", line 32, in content
    return render_to_string('debug_toolbar/panels/request_vars.html', context)

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/loader.py", line 108, in render_to_string
    return t.render(context_instance)

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/test/utils.py", line 29, in instrumented_test_render
    return self.nodelist.render(context)

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/__init__.py", line 779, in render
    bits.append(self.render_node(node, context))

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/debug.py", line 71, in render_node
    result = node.render(context)

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/defaulttags.py", line 243, in render
    return self.nodelist_true.render(context)

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/__init__.py", line 779, in render
    bits.append(self.render_node(node, context))

  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/debug.py", line 81, in render_node
    raise wrapped

TemplateSyntaxError: Caught an exception while rendering: ('ascii', u'# SOME DESCRIPTIVE TITLE.\n# Copyright (C) 2010\n# This file is distributed under the same license as the package.\n#

... snip ...

  Original Traceback (most recent call last):
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/debug.py", line 71, in render_node
    result = node.render(context)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/defaulttags.py", line 155, in render
    nodelist.append(node.render(context))
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/debug.py", line 87, in render
    output = force_unicode(self.filter_expression.resolve(context))
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/__init__.py", line 572, in resolve
    new_obj = func(obj, *arg_vals)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/defaultfilters.py", line 37, in _dec
    args[0] = force_unicode(args[0])
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/utils/encoding.py", line 74, in force_unicode
    s = unicode(str(s), encoding, errors)
  UnicodeEncodeError: 'ascii' codec can't encode characters in position 832-835: ordinal not in range(128)

The trace shows django-debug-toolbar in the trace so I am opening the issue here. Disabling the debug-toolbar code makes it work again.

Please let me know if you need more information than this to figure it out. Also, I don't really understand how the debug toolbar works with other plugins so if you think this is actually a bug in the other middleware, let me know and I will open it there.

[nickers]

I thinks that these errors are quite similiar(in fact 2 of them are the same error to me):

http://github.com/robhudson/django-debug-toolbar/issues#issue/43
http://github.com/robhudson/django-debug-toolbar/issues/unreads#issue/67
http://github.com/robhudson/django-debug-toolbar/issues/unreads#issue/18

All is about inproper handling non-ascii symbols. I also had problem with rosetta, but this worked for me:

def content(self):
    context = {
        'get': [(unicode(k), unicode(self.request.GET.getlist(k))) for k in self.request.GET],
        'post': [(unicode(k), unicode(self.request.POST.getlist(k))) for k in self.request.POST],
        'cookies': [(unicode(k), unicode(self.request.COOKIES.get(k))) for k in self.request.COOKIES],
        'view_func': '%s.%s' % (self.view_func.__module__, self.view_func.__name__),
        'view_args': self.view_args,
        'view_kwargs': self.view_kwargs
    }
    if hasattr(self.request, 'session'):
        context['session'] = [(unicode(k), unicode(self.request.session.get(k))) for k in self.request.session.iterkeys()]

    return render_to_string('debug_toolbar/panels/request_vars.html', context)

Only changes I made was adding unicode(), so all objects are converted to unicode strings, not to ascii string(str()) is default used here(as i understand docs).

Here the problem is with characters from POST data (translated strings :) )

[nickers]

I do not know if this is proper solution, but if not it may help you finding problem.

[poswald]

Interesting stuff. I still don't really understand what could be wrong with the way that debug-toolbar is written right now.

http://docs.djangoproject.com/en/dev/ref/unicode/#form-submission

"Django adopts a "lazy" approach to decoding form data. The data in an HttpRequest object is only decoded when you access it. In fact, most of the data is not decoded at all. Only the HttpRequest.GET and HttpRequest.POST data structures have any decoding applied to them. Those two fields will return their members as Unicode data. All other attributes and methods of HttpRequest return data exactly as it was submitted by the client."

It's crashing on render_to_string shortcut called with the request_vars.html template. It is crashing on the content of the .po translation file that was loaded from the filesystem. Normally, the rosetta toolbar loads this file and renders it itself. In this case, the debug-toolbar is getting in there and trying to render it as well.

As far as I can tell, debug-toolbar is simply passing these through to the django template engine which is assuming they are ascii. I don't see why as I thought it would assume utf-8.

I can see that it might be a good idea to coerce the strings into unicode strings as they are pulled from the request to the context. Perhaps it's best to use the smart_str function?

[jacobsvante]

I guess you were trying to link to this ssbarnea?
http://code.google.com/p/django-rosetta/issues/detail?id=71

Both modules are really important to me so I'm crossing my fingers for a fix =)

[philotas]

nickers fix worked for me.

[fcamel]

I encountered a similar bug when using v0.8.3:

My SQL contains Chinese words and it works without debug-toolbar, but it fails with it.

  File "/home/fcamel/python_ve/se/lib/python2.5/site-packages/debug_toolbar/panels/sql.py", line 122, in execute
    'hash': sha_constructor(settings.SECRET_KEY + sql + _params).hexdigest(),
UnicodeEncodeError: 'ascii' codec can't encode characters in position 124-126: ordinal not in range(128)

The problem seems to be that sha_constructor cannot take unicode as input. I'm sure that all my SQLs are unicode objects, so I fixed it by encode the input to string:

class DatabaseStatTracker(util.CursorDebugWrapper):
    """
    Replacement for CursorDebugWrapper which stores additional information
    in `connection.queries`.
    """
    def execute(self, sql, params=()):
        # ...
                 'hash': sha_constructor((settings.SECRET_KEY + sql + _params).encode('utf8')).hexdigest(),

I guess a better fix is to detect the type and then covert it to str if it's unicode.

[jacobsvante]

Try the latest rosetta trunk. Fixed it for me. http://code.google.com/p/django-rosetta/issues/detail?id=71

[fcamel]

I cannot reproduce the bug using debug-toolbar v0.8.3 with neither Rosetta v0.5.6 nor Rosetta trunk (r119). I can view the rosetta pages without any error with debug-toolbar. Follow your link, I saw mbonetti couldn't reproduce, either. Is it still a bug?

[jacobsvante]

I always got this error with debug-toolbar 0.8.3 and Rosetta 0.5.6. Using latest Rosetta trunk fixed it for me.

[jacobsvante]

*0.8.4

[poswald]

I tried updating to the latest of django-rosetta (0.5.6) and django-debug-toolbar (0.8.4). I ran makemessages and compilemessages. I can load the root url with both enabled. Once I pick a sublanguage though I still get the following stack trace:

Traceback (most recent call last):
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/core/servers/basehttp.py", line 280, in run
    self.result = application(self.environ, self.start_response)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/core/servers/basehttp.py", line 674, in __call__
    return self.application(environ, start_response)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/core/handlers/wsgi.py", line 245, in __call__
    response = middleware_method(request, response)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/debug_toolbar/middleware.py", line 106, in process_response
    smart_unicode(self.debug_toolbars[request].render_toolbar() + self.tag))
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/debug_toolbar/toolbar/loader.py", line 78, in render_toolbar
    return render_to_string('debug_toolbar/base.html', context)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/loader.py", line 186, in render_to_string
    return t.render(context_instance)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/__init__.py", line 173, in render
    return self._render(context)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/test/utils.py", line 54, in instrumented_test_render
    return self.nodelist.render(context)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/__init__.py", line 796, in render
    bits.append(self.render_node(node, context))
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/debug.py", line 72, in render_node
    result = node.render(context)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/defaulttags.py", line 174, in render
    nodelist.append(node.render(context))
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/defaulttags.py", line 258, in render
    return self.nodelist_true.render(context)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/__init__.py", line 796, in render
    bits.append(self.render_node(node, context))
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/debug.py", line 72, in render_node
    result = node.render(context)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/debug.py", line 89, in render
    output = self.filter_expression.resolve(context)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/__init__.py", line 553, in resolve
    obj = self.var.resolve(context)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/__init__.py", line 696, in resolve
    value = self._resolve_lookup(context)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/__init__.py", line 731, in _resolve_lookup
    current = current()
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/debug_toolbar/panels/request_vars.py", line 44, in content
    return render_to_string('debug_toolbar/panels/request_vars.html', context)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/loader.py", line 186, in render_to_string
    return t.render(context_instance)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/__init__.py", line 173, in render
    return self._render(context)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/test/utils.py", line 54, in instrumented_test_render
    return self.nodelist.render(context)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/__init__.py", line 796, in render
    bits.append(self.render_node(node, context))
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/debug.py", line 72, in render_node
    result = node.render(context)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/defaulttags.py", line 258, in render
    return self.nodelist_true.render(context)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/__init__.py", line 796, in render
    bits.append(self.render_node(node, context))
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/debug.py", line 72, in render_node
    result = node.render(context)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/defaulttags.py", line 174, in render
    nodelist.append(node.render(context))
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/debug.py", line 89, in render
    output = self.filter_expression.resolve(context)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/__init__.py", line 579, in resolve
    new_obj = func(obj, *arg_vals)
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/template/defaultfilters.py", line 34, in _dec
    args[0] = force_unicode(args[0])
  File "/Users/poswald/.virtualenvs/hats/lib/python2.6/site-packages/django/utils/encoding.py", line 69, in force_unicode
    s = unicode(str(s), encoding, errors)
TemplateSyntaxError: Caught UnicodeEncodeError while rendering: ('ascii', u'# SOME DESCRIPTIVE TITLE.\n# Copyright (C) 2010\n# This file is distributed under the same license as the package.\n# Paul Oswald <pauloswald@gmail.com>, 2010.\n# \nmsgid ""\nmsgstr ""\n"Project-Id-Version: PACKAGE VERSION\\n"\n"Report-Msgid-Bugs-To: \\n"\n"POT-Creation-Date: 2011-02-01 13:39+0900\\n"\n"PO-Revision-Date:

 ... snipped out an entire file dump here ...

msgid "Privacy"\n#~ msgstr "\u30d7\u30e9\u30a4\u30d0\u30b7\u30fc"\n', 2903, 2904, 'ordinal not in range(128)')

So it looks like there is still something that is not right. Disabling the debug toolbar allows me to view the rosetta url properly. I'm still not sure I understand how these two projects interact well enough.

[poswald]

I'm sorry I didn't realize you said using trunk fixed it. Let me try that instead of 0.8.4

[fcamel]

Sorry, I haven't used L10N / I18N before. Could somebody provide a more detailed steps to reproduce the bug? I enabled rosettal with 0.5.9 (r119) and debug-toolbar 0.8.4 (8cb8aba) and can view the index page correctly. See http://img840.imageshack.us/i/rosettaanddebugtoolbar.png/

[poswald]

I can get to the index page. It's viewing a language po file that causes the issue I think. Try this:

*Set up a project with a translation
*run makemessages and compilemessages
*translate a string to one with multibyte chars like "日本"
*view the language translation overview page
*if that doesn't cause it try recompiling messages.

I'll try to reproduce it on a fresh project when I get a chance. I suspect it's a Rosetta bug.

[aaugustin]

If you look at the stack trace below return render_to_string('debug_toolbar/panels/request_vars.html', context), it's fully contained in Django.

There have been significant improvements in this area since this ticket was opened 4 years ago and the bug may have been fixed in django.utils.encoding.force_unicode.

I'm going to leave this ticket open for the time being but I haven't attempted to reproduce it. If someone has the time to confirm whether it still exists or not, that'd help.