django-debug-toolbar is crashing postgresql by not closing connections

[elapouya]

I am using Django 4.2.2, django-debug-toolbar 4.1.0, Python 3.11 and Postgresql 15.2
While developing with DEBUG=True and django-debug-toolbar activated in settings.py, after about 100 pages displayed, Postgresql is crashing with the messages.

FATAL: remaining connection slots are reserved for non-replication superuser connections

My max connexion is set at 100 on my Postgresql server. Nevertheless despite I am alone to request pages to Django runserver (there is no concurrency), the opened postgresql connections count is always rising up to the max, then the above message appears.

To check open connections I do :

SELECT count(*) AS total_connections FROM pg_stat_activity;

If I use chrome to consult pages slowly or just a bench with ab -n 200 : I have the problem anyway.
If I restart runserver, I have again the right to read about 100 pages, but after it crashes.
The only solution I found is to remove django-debug-toolbar from settings.py : then total_connections is stable.

I guess django-debug-toolbar is not closing DB connections in some way : could you give me a workaround because your package is an absolute must.

[tim-schilling]

Thank you for opening an issue. What version of psycopg are you using? Are you able to investigate where in the toolbar it might be leaving a connection open? I suspect it's mogrify, but that's only a hunch. Another thing that would help immensely is creating a reproducible example.

…

On Thu, Jun 22, 2023, 8:54 AM Eric Lapouyade ***@***.***> wrote: I am using Django 4.2.2, django-debug-toolbar 4.1.0, Python 3.11 and Postgresql 15.2 While developing with DEBUG=True and django-debug-toolbar activated in settings.py, after about 100 pages displayed, Postgresql is crashing with the messages. FATAL: remaining connection slots are reserved for non-replication superuser connections My max connexion is set at 100 on my Postgresql server. Nevertheless despite I am alone to request pages to Django runserver (there is no concurrency), the opened postgresql connections count is always rising up to the max, then the above message appears. To check open connections I do : SELECT count(*) AS total_connections FROM pg_stat_activity; If I use chrome to consult pages slowly or just a bench with ab -n 200 : I have the problem anyway. If I restart runserver, I have again the right to read about 100 pages, but after it crashes. The only solution I found is to remove django-debug-toolbar from settings.py : then total_connections is stable. I guess django-debug-toolbar is not closing DB connections in some way : could you give me a workaround because your package is an absolute must. — Reply to this email directly, view it on GitHub <#1799>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJYZP6BYMNCH5ZBHYWO5Z3XMRFBBANCNFSM6AAAAAAZQH54FQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[elapouya]

I checked I am using the very latest psycopg package on pypi. I will try to build a minimal project to reproduce the problem without all packages I use.
I did not have investigated yet into the django-debug-toolbar code, I will try ...

[elapouya]

I made a minimal Django project and I reproduced the problem and I found some clues :

It comes from the StaticFilesPanel.generate_stats() : It uses ThreadCollector to store some data in RAM that persist accross all requests. It uses threading.current_thread() as an index to a dictionary to store data.
Unfortunately, I noticed by printing id(thread) that despite runserver is mono-threaded, the thread object ID is changing almost all the time after a while. I tried gunicorn : it does not have such a problem : the thread ID is always the same if I start only one worker and postgresql is not crashing.

I tried to force thread = 1 like that :

class ThreadCollector:
    def __init__(self):
        if threading is None:
            raise NotImplementedError(
                "threading module is not available, "
                "this panel cannot be used without it"
            )
        self.collections = {}  # a dictionary that maps threads to collections

    def get_collection(self, thread=None):
        """
        Returns a list of collected items for the provided thread, of if none
        is provided, returns a list for the current thread.
        """
        thread = 1
        if thread is None:
            thread = threading.current_thread()
        if thread not in self.collections:
            self.collections[thread] = []
        return self.collections[thread]

    def clear_collection(self, thread=None):
        thread = 1
        if thread is None:
            thread = threading.current_thread()
        if thread in self.collections:
            del self.collections[thread]

    def collect(self, item, thread=None):
        thread = 1
        self.get_collection(thread).append(item)

This is much better, but not perfect, I have still 30 on 200 requests that makes postgresql crash.

[elapouya]

The relation between ThreadCollector and Postgresql is that - I guess - some data indirectly references a DB connection somewhere in staticfiles panel, as thread ID is changing, cleaning is not done in self.collections and the connection object is never processed by the garbage collector.

I made a lot of dichotomy, this is when I by-passed StaticFilesPanel.generate_stats() that the problem totally disappeared.

[tim-schilling]

Can you share the reproducible django project?

[elapouya]

Yes :

git clone https://github.com/elapouya/ddtb-debug.git

[tim-schilling]

@elapouya thank you for creating that repository. It made it super easy for me to take a look at.