I'd change the title to something that makes clear the report is about Django >= 1.6 persistent DB connections feature.

I'm going to try to add some extra info with an excerpt of an internal issue in which we are exploring & testing changing CONN_MAX_AGE from its current default value 0. We use the gevent worker:

Django handles DB connections like this:

• If CONN_MAX_AGE = 0 then connections are opened when needed & closed at the end of every request. This is implemented with the request_finished signal.
• If CONN_MAX_AGE != 0 (n or None) they are handled per [1]process, every worker process started by the web server can have one connection which it uses for the requests it's in charge of: It reuses it (or opens it if it's closed) when needed BUT doesn't necessarily closes it at the end of requests. They are recycled every n seconds (by closing them if the TTL has been reached, this is implemented with both request_started request_finished signals) or never (None).

If the web server uses a concurrency model based on greenlets (eventlet, gevent workers) then connections are still opened when needed on every request and...:

• In the default case of CONN_MAX_AGE = 0 there aren't problems as they are closed at the end of the request.
• Now, if we set CONN_MAX_AGE != 0 then what happens is that connections aren't closed at the end of requests. So they start to accumulate.

Some options are:

1. Make connections persistent with CONN_MAX_AGE != 0 but switch the gunicorn worker to the 'sync' one (process-based)
2. Set CONN_MAX_AGE = 0 and switch to a DB connection pooling solution:
   • External like pgBouncer, or
   • Something like https://github.com/jneight/django-db-geventpool which activates psycopg greenlet support with psycogreen.

[1] This is actually per thread, but in actual web server deployment scenarios with e.g. Linux + apache or gunicorn, multithreading model is rarely used. Sync, multi-process (which means a one-to-one process-thread mapping) or async greenlets-based concurrency models are the usual choices.

If the above analysis is correct, I'm not sure it's in gunicorn scope to 'fix' this. Don't think it's in Django scope either (i.e., [try to] detect what concurrency model does the web server it has been deployed to uses and adapt its handling of these persistent DB connections accordingly.)

@ramiro I agree with your analysis.

I think, given the analysis, that this will not be fixed. If multiple, concurrent connections are made by an async worker the application needs to use a thread safe connection pool to get safe, correct behavior.

Re: "Now, if we set CONN_MAX_AGE != 0 then what happens is that connections aren't closed at the end of requests. So they start to accumulate."

Why is that an issue with async workers? As soon as an async worker completes a request it still closes the connection? Why would connections accumulate then? By accumulate you mean connections are getting more and more (and finally exceeding the DB's connection limit)?

I can understand that async workers produce more connections (because a single worker can handle more than 1 request at a time, hence opening more db connections) but once a certain number of connections is reached, it should converge.

Additionally can someone explain why/how https://github.com/jneight/django-db-geventpool would help in this case?

Could somebody (perhaps @ramiro or @tilgovi) please confirm that https://github.com/jneight/django-db-geventpool should actually help in this case? I think I ran into this problem but it kept occurring after installing django-db-geventpool (with the import from eventlet included, and the import from gevent not included). I was getting a log of "FATAL: sorry, too many clients already" which indicates that I opened up more than the allowed limit of connections to the database. CPU was also maxing out so perhaps that's why I kept getting the same error after installing django-db-geventpool...?

Following up from my last post, it seems possible that the CPU max out was (at least partially) caused by the issue @Starefossen raised above. http://stackoverflow.com/a/43799635/805141 indicates this is a possibility.

Also, shouldn't this issue be documented somewhere?

The lack of issue resolution that I experienced may have been caused by jneight/django-db-geventpool#21

Re: "Now, if we set CONN_MAX_AGE != 0 then what happens is that connections aren't closed at the end of requests. So they start to accumulate."

Why is that an issue with async workers? As soon as an async worker completes a request it still closes the connection? Why would connections accumulate then? By accumulate you mean connections are getting more and more (and finally exceeding the DB's connection limit)?

I can understand that async workers produce more connections (because a single worker can handle more than 1 request at a time, hence opening more db connections) but once a certain number of connections is reached, it should converge.

Additionally can someone explain why/how https://github.com/jneight/django-db-geventpool would help in this case?

I think it is because the established connections will only be closed when requests starts or finishes if conn_max_age is not 0. So if async workers die for some reason such as timeout, they won't receive new requests and then the db connections can not be closed for ever (they can only be closed in reqeusts starts or finishes).

As for the django-db-geventpool, it is used when conn_max_age is 0. When it is 0, every new request will create new db connection. The db pool can improve the performance.

Correct me if I'm wrong. :D

In current design, connections will be kept alive to handle HTTP keepalive before releasing. New coming requests are handled in a new greenlet.

The Django documentation states:

Caveats¶

Since each thread maintains its own connection, your database must support at least as many > simultaneous connections as you have worker threads.

I guess the way django persistent connections work is to associate them to the thread id. If threads are patched in the gevent or eventlet workers, it is probably creating a new connection / greenlets which is limited by the number of maximum connections which may explains what you observe. If it' that then maybe django could be patched to actually use the real thread on which it run? Or by limiting the number of concurrent connections.

I made a patch to allow db pooling with async workers.
https://pypi.org/project/django-db-pooling/

Read this thread but I am lost. Does anyone know how to get DB connections properly reused when running with uvicorn?

gunicorn Application.asgi:application --max-requests 4000 -w 4 -k uvicorn.workers.UvicornWorker

I have the exact same issue of each request resulting in a new DB connection that never clears if I set CONN_MAX_AGE != 0

@patrik7 did you figure this out ?

@patrik7 did you figure this out ?

Nope, I resigned - I am still opening a new connection for every single HTTP request