Detecting duplicate queries breaks for non-hashable query params

[tilboerner]

Since commit 844353e, SQL query params are used to detect duplicate queries. A duplicate_key is created and then used as a key to a dict:

https://github.com/jazzband/django-debug-toolbar/blob/05ffccfb68fac64eeba9894707f9b6771e462648/debug_toolbar/panels/sql/panel.py#L151-L153

https://github.com/jazzband/django-debug-toolbar/blob/844353e4cb9855d76e4fa8b0f48de140d2588204/debug_toolbar/panels/sql/panel.py#L177

https://github.com/jazzband/django-debug-toolbar/blob/844353e4cb9855d76e4fa8b0f48de140d2588204/debug_toolbar/panels/sql/panel.py#L248

---

This fails when raw_params contains non-hashable items like lists. I saw it happen for the following query using a Postgres ArrayField, but it should also be triggered by a simple .filter(pk__in=[1, 2, 3]):

cls.objects.filter(
            Q(to_emails__contains=[customer.email]) |
            Q(cc_emails__contains=[customer.email]) |
            Q(bcc_emails__contains=[customer.email])
        )

Traceback (most recent call last):
  File "/.../lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/.../lib/python3.6/site-packages/django/utils/deprecation.py", line 97, in __call__
    response = self.process_response(request, response)
  File "/.../lib/python3.6/site-packages/debug_toolbar/middleware.py", line 126, in process_response
    panel.generate_stats(request, response)
  File "/.../lib/python3.6/site-packages/debug_toolbar/panels/sql/panel.py", line 178, in generate_stats
    query_duplicates[alias][duplicate_key(query)] += 1
TypeError: unhashable type: 'list'

(Edit: passing lists to __in works fine.)

[matthiask]

What are the contents of query in your case?

[tilboerner]

I'm not in front of a computer currently, but in this case, query['raw_params'] contains three lists with one string in them each.

Do you need to know more? I can get you some reprs once I'm plugged in again.

[Ashlett]

@matthiask

{'duration': 1.3995170593261719,
 'is_select': True,
 'is_slow': False,
 'params': '["[\'c36321localbackend@thermondo-test.de\']", '
           '"[\'c36321localbackend@thermondo-test.de\']", '
           '"[\'c36321localbackend@thermondo-test.de\']"]',
 'raw_params': (['c36321localbackend@thermondo-test.de'],
                ['c36321localbackend@thermondo-test.de'],
                ['c36321localbackend@thermondo-test.de']),
 'raw_sql': 'SELECT "emails_emailtask"."id", "emails_emailtask"."created", '
            '"emails_emailtask"."modified", "emails_emailtask"."from_email", '
            '"emails_emailtask"."from_name", "emails_emailtask"."to_emails", '
            '"emails_emailtask"."cc_emails", "emails_emailtask"."bcc_emails", '
            '"emails_emailtask"."subject", "emails_emailtask"."text_body", '
            '"emails_emailtask"."html_body", '
            '"emails_emailtask"."created_by_id", '
            '"emails_emailtask"."content_type_id", '
            '"emails_emailtask"."object_id", "emails_emailtask"."sf_id", '
            '"emails_emailtask"."related_to_sf_id" FROM "emails_emailtask" '
            'WHERE ("emails_emailtask"."to_emails" @> %s::varchar(254)[] OR '
            '"emails_emailtask"."cc_emails" @> %s::varchar(254)[] OR '
            '"emails_emailtask"."bcc_emails" @> %s::varchar(254)[]) ORDER BY '
            '"emails_emailtask"."modified" DESC, "emails_emailtask"."created" '
            'DESC',
 'sql': 'SELECT "emails_emailtask"."id", "emails_emailtask"."created", '
        '"emails_emailtask"."modified", "emails_emailtask"."from_email", '
        '"emails_emailtask"."from_name", "emails_emailtask"."to_emails", '
        '"emails_emailtask"."cc_emails", "emails_emailtask"."bcc_emails", '
        '"emails_emailtask"."subject", "emails_emailtask"."text_body", '
        '"emails_emailtask"."html_body", "emails_emailtask"."created_by_id", '
        '"emails_emailtask"."content_type_id", "emails_emailtask"."object_id", '
        '"emails_emailtask"."sf_id", "emails_emailtask"."related_to_sf_id" '
        'FROM "emails_emailtask" WHERE ("emails_emailtask"."to_emails" @> '
        "ARRAY['c36321localbackend@thermondo-test.de']::varchar(254)[] OR "
        '"emails_emailtask"."cc_emails" @> '
        "ARRAY['c36321localbackend@thermondo-test.de']::varchar(254)[] OR "
        '"emails_emailtask"."bcc_emails" @> '
        "ARRAY['c36321localbackend@thermondo-test.de']::varchar(254)[]) ORDER "
        'BY "emails_emailtask"."modified" DESC, "emails_emailtask"."created" '
        'DESC',
 'stacktrace': [('/home/alicja/.virtualenvs/backend/lib/python3.6/site-packages/django/contrib/staticfiles/handlers.py',
                 66,
                 '__call__',
                 'return self.application(environ, start_response)'),
                ('/home/alicja/.virtualenvs/backend/lib/python3.6/site-packages/whitenoise/middleware.py',
                 49,
                 '__call__',
                 'response = self.get_response(request)'),
                ('/home/alicja/.virtualenvs/backend/lib/python3.6/site-packages/s3file/middleware.py',
                 17,
                 '__call__',
                 'return self.get_response(request)'),
                ('/home/alicja/projects/thermondo-backend/common/middleware.py',
                 77,
                 '__call__',
                 'response = self.get_response(request)'),
                ('/home/alicja/.virtualenvs/backend/lib/python3.6/site-packages/django/contrib/admin/options.py',
                 575,
                 'wrapper',
                 'return self.admin_site.admin_view(view)(*args, **kwargs)'),
                ('/home/alicja/.virtualenvs/backend/lib/python3.6/site-packages/django/views/decorators/cache.py',
                 44,
                 '_wrapped_view_func',
                 'response = view_func(request, *args, **kwargs)'),
                ('/home/alicja/.virtualenvs/backend/lib/python3.6/site-packages/django/contrib/admin/sites.py',
                 223,
                 'inner',
                 'return view(request, *ar... <trimmed 5649 bytes string>

[jonathan-golorry]

I'm also seeing this issue when I run Model.objects.filter(arrayfield__contains=[1234567890])

[matthiask]

Alright. I fought a bit with Travis CI to add a PostgreSQL-specific build so that someone can add a test and fix this issue: 57053f7

[matthiask]

The additional pformat() call seems to fix the issue, but 1. I don't know how to construct a test case for this (since the model and field can only be loaded when running on postgresql) and 2. running the SQL or explaining it breaks (resp. generates a 500 error in the backend) but this is probably fine for now since it didn't work before either.

Anyone has some feedback and/or help to offer?

[matthiask]

Still no unit test, but seems to work fine for me. Released as 1.10.1 (with a few other bugfixes and improvements)