As this part said, we can log all queries in Django with DEBUG=True.
DEBUG option is a global configure item, if we set it True, it will also affect other codes. So it is hard to active it in production environment.
How to get all SQL queries or we want in production? MySQL or database middleware can log these. But for all kinds of limitation, it's better to log it by ourselves.
Here are some methods to log it without active DEBUG option.
Logging Methods
When we set DEBUG with True, we have two places to get queries.
One is in log file, the output like this:1
2DEBUG 2020-02-21 10:35:12,214 33337 [utils.py:91] (0.036) SELECT @@SQL_AUTO_IS_NULL; args=None
DEBUG 2020-02-21 10:35:12,251 33337 [utils.py:91] (0.036) SELECT VERSION(); args=None
Another one is using Django database connection's 'queries' property(code1, doc1). It stores all queries in one request, and is cleared automatically in here. So we can add a middleware to traverse 'queries' to log all SQLs.
Above two methods can be enabled by set DEBUG with True.
The following part will discuss how to enable database logging function without set DEBUG.
Custom Methods
Enable database logging function.
From codes, we find database logging/recording SQL functions are controlled by 'queries_logged' property. And it is determined by settings.DEBUG and self.force_debug_cursor, the last one is hard-coded without function argument, setting option to set it. Even in subclass, the concrete database backends.
1 | # https://github.com/django/django/blob/3.0.3/django/db/backends/base/base.py#L150 |
Can we set self.force_debug_cursor with True? Yes, we can.
self.force_debug_cursorisBaseDatabaseWrapperobject's field, not class's.BaseDatabaseWrapperis base class of all backends' DatabaseWrapper, which are the database connection's class we use in code.- Every thread has its own DatabaseWrapper(database connection) object, not a singleton. The following is related code:
1 | class ConnectionHandler: |
ConnectionHandler is the class of django.db.connections. self._connections is a thread local variable. So we can set force_debug_cursor to True in middleware, for each threading's connection object. Like this:
1 | from django.db import connections |
Or if there are other thread scope initialization points, we can also do in there. If you know, please tell me.
Write custom database backend.
Since there are no choice to set force_debug_cursor, how about add a setting method?
Yes, we can. Write a custom database backend:
1 | # $ tree custom_backend/ |
In settings, use this module to access MySQL service:
1 | DATABASES = { |
Same with other database backends.
Hook exist backend.
With previous two methods, we can enable logging SQL into log files and connection.queries variable. This method does not aim to enable someone, but to log by ourselves.
From documents and code, we can find in backend internal it uses cursor object to do fetch operation. code1 code2 doc1 doc2
We can also find Django implements logging function by different wrapper classes for cursor:1
2
3
4
5
6
7
8
9
10
11
12# https://github.com/django/django/blob/3.0.3/django/db/backends/base/base.py#L224
def _prepare_cursor(self, cursor):
"""
Validate the connection is usable and perform database cursor wrapping.
"""
self.validate_thread_sharing()
if self.queries_logged:
wrapped_cursor = self.make_debug_cursor(cursor)
else:
wrapped_cursor = self.make_cursor(cursor)
return wrapped_cursor
So we can implement our own cursor wrapper class to record anything as we want. And next, how to let Django use our custom cursor wrapper class.
We can also implement a custom database backend and overwrite cursor creator function, like MySQL backend does:
1 | # https://github.com/django/django/blob/3.0.3/django/db/backends/mysql/base.py#L252 |
Another one is to hook django.db.backends.utils.CursorWrapper class in process scope. Here is a example from django-extensions: link