Fixed #27452 -- Added serial fields to contrib.postgres. #18123

csirmazbendeguz · 2024-05-03T05:59:08Z

Trac ticket number

ticket-27452

Branch description

Added SmallSerialField, SerialField, BigSerialField to contrib.postgres.

Previous PR

Checklist

This PR targets the main branch.
The commit message is written in past tense, mentions the ticket number, and ends with a period.
I have checked the "Has patch" ticket flag in the Trac system.
I have added or updated relevant tests.
I have added or updated relevant docs, including release notes if applicable.
I have attached screenshots in both light and dark modes for any UI changes.

LilyFoote

This looks like a great start!

In addition to the specific comments I added, we'll need some migrations tests to cover adding/removing serial fields and converting between serial and auto fields.

django/contrib/postgres/fields/serial.py

django/db/backends/postgresql/operations.py

django/db/backends/postgresql/schema.py

LilyFoote · 2024-05-04T13:36:46Z

tests/schema/tests.py

+    @unittest.skipUnless(connection.vendor == "postgresql", "PostgreSQL specific")
+    @isolate_apps("schema")
+    @tag("serial")
+    def test_serial_to_integer(self):
+        from django.contrib.postgres.fields import SerialField
+
+        self.serial_to_integer_test(SerialField, IntegerField)
+
+    @unittest.skipUnless(connection.vendor == "postgresql", "PostgreSQL specific")
+    @isolate_apps("schema")
+    @tag("serial")
+    def test_serial_to_small_integer(self):
+        from django.contrib.postgres.fields import SerialField
+
+        self.serial_to_integer_test(SerialField, SmallIntegerField)
+
+    @unittest.skipUnless(connection.vendor == "postgresql", "PostgreSQL specific")
+    @isolate_apps("schema")
+    @tag("serial")
+    def test_serial_to_big_integer(self):
+        from django.contrib.postgres.fields import SerialField
+
+        self.serial_to_integer_test(SerialField, BigIntegerField)
+
+    @unittest.skipUnless(connection.vendor == "postgresql", "PostgreSQL specific")
+    @isolate_apps("schema")
+    @tag("serial")
+    def test_small_serial_to_integer(self):
+        from django.contrib.postgres.fields import SmallSerialField
+
+        self.serial_to_integer_test(SmallSerialField, IntegerField)
+
+    @unittest.skipUnless(connection.vendor == "postgresql", "PostgreSQL specific")
+    @isolate_apps("schema")
+    @tag("serial")
+    def test_small_serial_to_small_integer(self):
+        from django.contrib.postgres.fields import SmallSerialField
+
+        self.serial_to_integer_test(SmallSerialField, SmallIntegerField)
+
+    @unittest.skipUnless(connection.vendor == "postgresql", "PostgreSQL specific")
+    @isolate_apps("schema")
+    @tag("serial")
+    def test_small_serial_to_big_integer(self):
+        from django.contrib.postgres.fields import SmallSerialField
+
+        self.serial_to_integer_test(SmallSerialField, BigIntegerField)
+
+    @unittest.skipUnless(connection.vendor == "postgresql", "PostgreSQL specific")
+    @isolate_apps("schema")
+    @tag("serial")
+    def test_big_serial_to_big_integer(self):
+        from django.contrib.postgres.fields import BigSerialField
+
+        self.serial_to_integer_test(BigSerialField, BigIntegerField)
+
+    @unittest.skipUnless(connection.vendor == "postgresql", "PostgreSQL specific")
+    @isolate_apps("schema")
+    @tag("serial")
+    def test_big_serial_to_integer(self):
+        from django.contrib.postgres.fields import BigSerialField
+
+        self.serial_to_integer_test(BigSerialField, IntegerField)
+
+    @unittest.skipUnless(connection.vendor == "postgresql", "PostgreSQL specific")
+    @isolate_apps("schema")
+    @tag("serial")
+    def test_big_serial_to_small_integer(self):
+        from django.contrib.postgres.fields import BigSerialField
+
+        self.serial_to_integer_test(BigSerialField, SmallIntegerField)


This feels like a good use case for subTest to me.

Yes, the issue is we would either need to generate the test models with a dynamic name, or clean up the test models after the subtests - IMO it's more trouble than worth it.

tests/schema/tests.py

csirmazbendeguz · 2024-05-05T11:09:05Z

This looks like a great start!

In addition to the specific comments I added, we'll need some migrations tests to cover adding/removing serial fields and converting between serial and auto fields.

@LilyFoote , thanks for the review! Could you point me to some tests I could copy to get started? I'm not sure how to test the migrations other than with the tests/schema tests.

LilyFoote · 2024-05-05T12:13:36Z

For database defaults I added tests in tests/migrations/test_autodetector.py and tests/migrations/test_operations.py.

sarahboyce

Thank you for the updates @csirmazbendeguz
I think there are a few tests/asserts we need to add 👍

django/contrib/postgres/fields/serial.py

django/core/management/commands/inspectdb.py

django/db/backends/postgresql/operations.py

csirmazbendeguz · 2024-05-27T16:49:41Z

Thanks @sarahboyce , fair enough, I made some adjustments.

InvalidInterrupt

I haven't really read many of the tests yet; not much has caught my eye that isn't already being discussed.

InvalidInterrupt · 2024-05-28T03:25:04Z

django/contrib/postgres/fields/serial.py

+    def _check_default(self):
+        if self.default is NOT_PROVIDED:
+            return []
+        return [
+            checks.Error(
+                f"{self.__class__.__name__} does not accept default values.",
+                obj=self,
+                id="fields.E014",
+            ),
+        ]


While I would expect most users to operate this way; it feels overly prescriptive to me.

Suppose for some reason I only wanted a subset of objects to be part of the sequence, and the rest to default to 0, I might want to set default on the field and pass DatabaseDefault() only when I want to use the sequence.

Thanks @InvalidInterrupt !

I don't think this is an issue, it's possible to create an object without using the sequence:

class Dummy(Model): serial = SerialField() Dummy.objects.create(serial=0).serial # 0 Dummy.objects.create(serial=0).serial # 0 Dummy.objects.create(serial=0).serial # 0 Dummy.objects.create().serial # 1 Dummy.objects.create().serial # 2 Dummy.objects.create().serial # 3 Dummy.objects.create(serial=1).serial # 1 Dummy.objects.create().serial # 4

Is this what you're suggesting?

class Dummy(Model): serial = SerialField(default=0) Dummy.objects.create().serial # 0 Dummy.objects.create().serial # 0 Dummy.objects.create().serial # 0 Dummy.objects.create(serial=DatabaseDefault()).serial # 1 Dummy.objects.create(serial=DatabaseDefault()).serial # 2 Dummy.objects.create(serial=DatabaseDefault()).serial # 3 Dummy.objects.create(serial=1).serial # 1 Dummy.objects.create(serial=DatabaseDefault()).serial # 4

I think the first one is better because it's consistent with how AutoField works.

I've been testing on a little project, if we add a SerialField to an existing model I cannot make it nullable or assign a default and so I have to give a temporary default in order to create a migration and add the field.

However, whatever I give here is ignored and the field is populated everywhere with as a serial starting from 1
I feel like it shouldn't prompt us in that case (forcing my input and then ignoring it to me is a bug).
Also considering this, a default option might be nice 🤔 (cc @LilyFoote)

I see 🤔 Thanks for catching that @sarahboyce ! I tested it too but I didn't notice the the input was ignored. 😱 I'll investigate.

Yes, default makes sense in the context of migrations.

Good catch! I think the minimal change feature-wise would be to adapt the migration generation code to know that SerialField can be created without an explicit default.

For a user-chosen default my question would be "why a SerialField and not an IntegerField?".

Thanks for explaining @InvalidInterrupt !

I think the only place this could be useful is in migrations. So when adding a serial field, it could be pre-filled with a default number.

I would find this API confusing otherwise, users may confuse it with the start parameter for example.
I don't think it's too restrictive since users can still opt out of using the sequence by providing an explicit value.

That said, it's always easier to remove a restriction than to add one.

In my opinion, the use-case of migrations is an edge-case and it can be solved by first creating an IntegerField then changing it to a SerialField, so it wouldn't justify enabling this.

If someone can provide a good use-case this restriction could be removed in the future, but for now I have to disagree.

My description of the features potential use may have been confusing; but using default in this manner would be just like (every?) other field type, so I'm sceptical it would be confused for something like start outside of the context of this thread. In regards to API design, as it stands SerialField will be the exception. Eliminating that exception would probably also fix the migrations without adding a kludge to the migration code for this.

I've just finished mocking up the removal of this restriction to confirm that it works as expected and doesn't break anything else. I'm going to finish cleaning up the code and then push it to my fork for reference as a PoC (if anyone is curious: https://github.com/InvalidInterrupt/django/tree/ticket_27452).

That being said, I don't think this is a big issue; and it shouldn't hold up this PR. If I'm not persuading anyone that we should remove this guardrail and trust the Django users, then we should just move along for now (Sarah's migration issue notwithstanding).

Also, even AutoField allows you to set a default, even though that means that you'll only be able to save a single instance of the model without specifying an explicit value for the PK

I checked and AutoField allows a default too, it will cause integrity errors of course but it allows it. So yes you are raising a valid point, this would be an exception, thank you for pointing that out. It may be worth enabling for consistency's sake. @LilyFoote , @sarahboyce what do you think?

Actually, there's a problem with migrating a SerialField with a default value. It's impossible to do this in Postgres, since serial already implies the field has a default sequence. As far as I can tell, there's no trivial way to add a serial field to a table with an initial default value other than the sequence.

So I think the simplest option would be to disable default after all. Or to not use the built-in serial type when creating a field.

docs/ref/contrib/postgres/fields.txt

csirmazbendeguz · 2024-05-28T07:39:16Z

Thanks for the review @InvalidInterrupt , it's much appreciated! 🙏

sarahboyce · 2024-05-28T08:56:57Z

django/core/management/commands/inspectdb.py

@@ -75,6 +76,7 @@ def handle_inspection(self, options):
                "field names."
            )
            yield "from %s import models" % self.db_module
+


@csirmazbendeguz these are minor clean up changes that will happen on the PR either by you or by a merger

we want to limit the changed lines to keep the diff really tight to what's important, occasionally you have added a new line or removed a line, I will probably revert some of this (you can too if you like). This is considered formatting changes

you have added @tag("serial") for your tests, this is not how we organise tests generally and needs to be removed

the commits need to be squashed. This will likely be 1 commit (or if we can think of a nice way of splitting them we will)

I added this for informational purposes 👍

I forgot to say that I have tested this with a local project and tried out a number of things to try and see if I can find any issues (had a look in the admin, reversed migrations, used them in GeneratedFields) and this comment is the only thing I'm concerned about. It looks really good ⭐

django/db/migrations/autodetector.py

csirmazbendeguz · 2024-05-29T03:58:52Z

I realized instead of adding new system checks, I could just raise a ValueError in __init__. This is how GeneratedField does it. WDYT?

I think it's better, because:

it's consistent with how it's done elsewhere
it's stricter, the code will fail immediately, doesn't need to wait for checks to run
it doesn't require adding new checks

tests/inspectdb/tests.py

django/db/backends/postgresql/schema.py

django/db/migrations/autodetector.py

csirmazbendeguz · 2024-05-29T16:01:41Z

So I would like to summarize what I figured out today for both myself and others to see.

We have two options: either enable serial fields to define a default value, or not. As @InvalidInterrupt pointed out, it would be preferrable to enable this for consistency's sake.

If default is not allowed, then:

Adding a serial field to a table shouldn't trigger a prompt for a default value. The new field will be populated by the sequence.
Changing a nullable integer field to a serial field will trigger an error if the table contains any null values. I am looking into if we could also populate the null values with the sequence in this case.

If default is allowed, then:

Adding a serial field to a table should trigger a prompt for a default value. The new field should be populated by the default value. QUESTION: what if I want to populate the field with a sequence by default? I think if we need to pick one, it's more sensible to populate the serial field with a sequence instead of a single default value.
We can't use Postgres's built-in serial type when creating a serial field. This is because it's impossible to create a serial field with a default value, since serial fields already have a sequence by default. We could create a regular integer field instead and attach a sequence to it in multiple steps.

csirmazbendeguz · 2024-05-29T17:46:41Z

some notes so I don't forget:
when changing a nullable integer to a serial field, the issue is with the order the queries are executed (fragment vs other_actions).
it would be nice if _alter_column_type_sql could return a list of actions instead of a single action (fragment). this is because setting up a serial field needs multiple queries.

EDIT:
it's not necessary, queries can be executed directly in _alter_column_type_sql instead.

csirmazbendeguz · 2024-05-30T07:04:31Z

So I decided to stop fighting against what's already in place and try to conform to the existing code. I managed to achieve the following:

no default is allowed, it's always set to DatabaseDefault
added the SerialFieldSequence class for serial fields db_default
adding a serial field to an existing table doesn't trigger a prompt
adding a serial field to an existing table will populate it with a sequence starting from 1
changing a nullable integer to a serial field will populate the null values with a sequence starting from max(field)

charettes · 2024-06-01T14:30:00Z

tests/schema/tests.py

+    def test_serial_to_integer(self):
+        from django.contrib.postgres.fields import SerialField
+
+        self.serial_to_integer_test(SerialField, IntegerField)


I kind of wish we tested all these in something like postgres_tests/test_schema but given this is a pre-existing pattern here import testing against django.contrib.postgres this doesn't have to be done in this PR.

charettes · 2024-06-01T14:40:49Z

django/contrib/postgres/fields/serial.py

+        table = self.field.model._meta.db_table
+        column = self.field.column
+        return f"nextval(pg_get_serial_sequence('{table}', '{column}'))", []


This can be an SQL injection vector and will crash if any table or column name includes a quote.

Suggested change

table = self.field.model._meta.db_table

column = self.field.column

return f"nextval(pg_get_serial_sequence('{table}', '{column}'))", []

table = self.field.model._meta.db_table

column = self.field.column

return "nextval(pg_get_serial_sequence(%s, %s))", [table, column]

I would personally use Func instead to avoid this error prone boilerplate

class NextSerialSequence(Func): template = "nextval(pg_get_serial_sequence(%(expressions)s))"

And override SerialFieldMixin._db_default_expression to return

@cached_property def _db_default_expression(self): return NextSerialSequence( Value(self.model._meta.db_table), Value(self.column), output_field=self, )

Which should also avoid the db_default injection in __init__ and removal in deconstruct.

Thanks @charettes , fair enough. We should really set db_default to something though, overriding _db_default_expression is not enough because this variable is checked during migrations.

charettes · 2024-06-01T14:42:13Z

django/contrib/postgres/fields/serial.py

+    def get_prep_value(self, value):
+        value = super().get_prep_value(value)
+        if value is None:
+            value = self.get_default()
+        return value


I'm surprised this is required given we set default = DatabaseDefault()?

It's for supporting None arguments:

SerialModel.objects.create(serial=None)

Since it's possible to provide None to auto fields, it should also be possible to provide it for serial fields.

If that's not necessary, we can remove it.

charettes · 2024-06-02T04:12:02Z

django/db/backends/postgresql/schema.py

+    sql_create_sequence = (
+        "CREATE SEQUENCE IF NOT EXISTS %(sequence)s "
+        "AS %(data_type)s "
+        "OWNED BY %(table)s.%(column)s"
+    )
+    sql_reset_sequence = (
+        "SELECT setval(pg_get_serial_sequence(%s, %s), "
+        "coalesce(max(%s), 1), max(%s) IS NOT NULL) FROM %s;"
+    )
+    sql_rename_sequence = "ALTER SEQUENCE %s RENAME TO %s;"


Thinking more about it isn't there something fundamentally wrong here about not using the native serial types?

Why are we going the extra length of manually emulating through schema definition and explicitly using nextval(pg_get_serial_sequence(...)) on INSERT when simply using serial and friends does it by itself?

I could be missing something here but it seems that most of the complexity of this patch stems from trying to emulate what Postgres does very well already by itself?

@charettes , it's because Postgres doesn't support it very well. serial is not a true type, and it's not possible to alter a field's type to serial. we have to emulate it because that's the only way it can work.

Could you elaborate on how Postgres doesn't support serial very well? It has been used to power AutoField for over a decade before we switched to IDENTITY so I'm surprised to learn that it poses problems.

From my understanding the main reason for the manual sequence management and db_default assignment is to support altering back and forth between integer, auto, and serial fields. Is that really going to be a common enough use case that it warrants all the complexity it requires?

I think it's a question worth asking because there are no mentions of supporting this use case in the accepted ticket-27452 and I personally don't think that it will be common enough operation to warrant that much extra complexity to the schema editor.

Don't get me wrong, it is impressive that you got it working and I'm sure you learned a ton doing so. I'm just doubtful that the code complexity tradeoff is worth it (1.4k additional line) considering that this could be simply implemented using mixins including db_returning = True and using get_db_type = "serial" at first.

You can use serial when creating a table or when adding a field, but you can't use it for anything else, e.g. you can't alter a column's type to serial and you can't cast to serial. But do let me know if I'm wrong, it didn't work when I tested it.

I agree it adds some complexity, but the majority of that 1.4k lines is just tests, the schema editor received 160 additions and 44 removals only. I assumed the functionality to transition between integer fields <-> serial fields <-> auto fields would be a requirement because of the similarities between these fields. If all we wanted to support was creating serial fields, I'm not sure this would need any changes to core, it could be a package.

csirmazbendeguz force-pushed the ticket_27452 branch from 65d6140 to fd0e9b4 Compare May 3, 2024 06:09

ngnpope self-requested a review May 3, 2024 06:37

csirmazbendeguz force-pushed the ticket_27452 branch 4 times, most recently from e892da4 to 97cbc23 Compare May 3, 2024 08:31

csirmazbendeguz marked this pull request as draft May 3, 2024 10:21

csirmazbendeguz force-pushed the ticket_27452 branch 12 times, most recently from fccac26 to 2b48193 Compare May 4, 2024 13:05

LilyFoote reviewed May 4, 2024

View reviewed changes

csirmazbendeguz force-pushed the ticket_27452 branch 2 times, most recently from e7fdd21 to 8052e52 Compare May 5, 2024 10:37

csirmazbendeguz force-pushed the ticket_27452 branch from 8052e52 to 42c53c4 Compare May 5, 2024 11:16

csirmazbendeguz force-pushed the ticket_27452 branch 5 times, most recently from 569c960 to 0ebc30b Compare May 6, 2024 06:40

sarahboyce reviewed May 27, 2024

View reviewed changes

InvalidInterrupt reviewed May 28, 2024

View reviewed changes

sarahboyce reviewed May 28, 2024

View reviewed changes

django/db/migrations/autodetector.py Outdated Show resolved Hide resolved

Fixed #27452 -- Added serial fields to django.contrib.postgres

f521fdd

csirmazbendeguz force-pushed the ticket_27452 branch from 5212ef3 to f521fdd Compare May 29, 2024 03:26

csirmazbendeguz added 6 commits May 29, 2024 12:09

Fixed #27452 - Override has_default

9fd9b25

Fixed #27452 - Use value errors instead of system check

1fb8859

Fixed #27452 - Remove unused import

ed4f6a7

Fixed #27452 - Remove unused import

2afeac9

Fixed #27452 - Enable default

cdafbcb

Fixed #27452 - Disable default

ab0f122

sarahboyce reviewed May 29, 2024

View reviewed changes

tests/inspectdb/tests.py Outdated Show resolved Hide resolved

django/db/backends/postgresql/schema.py Outdated Show resolved Hide resolved

django/db/migrations/autodetector.py Outdated Show resolved Hide resolved

Fixed #27452 - Fix

75c3d90

Fixed #27452 - Fix migrations

1daaeb3

csirmazbendeguz added 4 commits May 30, 2024 15:40

Fixed #27452 - Reset sequence before populating nulls

bfedd52

Fixed #27452 - Move tests

d01fa3d

Fixed #27452 - Don't skip

ce776e8

Fixed #27452 - Remove unused code

561e5f3

charettes reviewed Jun 1, 2024

View reviewed changes

csirmazbendeguz added 3 commits June 2, 2024 10:40

Fixed django#373 - Fix parameterization

62291b7

Fixed django#373 - Fix error

56924c3

Fixed django#373 - Trigger build

692b2fe

charettes reviewed Jun 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed #27452 -- Added serial fields to contrib.postgres. #18123

Fixed #27452 -- Added serial fields to contrib.postgres. #18123

csirmazbendeguz commented May 3, 2024 •

edited

LilyFoote left a comment

LilyFoote May 4, 2024

csirmazbendeguz May 5, 2024

csirmazbendeguz commented May 5, 2024

LilyFoote commented May 5, 2024

sarahboyce left a comment

csirmazbendeguz commented May 27, 2024

InvalidInterrupt left a comment

InvalidInterrupt May 28, 2024

csirmazbendeguz May 28, 2024

sarahboyce May 28, 2024

csirmazbendeguz May 28, 2024

LilyFoote May 28, 2024 •

edited

csirmazbendeguz May 29, 2024

InvalidInterrupt May 29, 2024 •

edited

InvalidInterrupt May 29, 2024

csirmazbendeguz May 29, 2024

csirmazbendeguz May 29, 2024 •

edited

csirmazbendeguz commented May 28, 2024

sarahboyce May 28, 2024

sarahboyce May 28, 2024

csirmazbendeguz commented May 29, 2024 •

edited

csirmazbendeguz commented May 29, 2024 •

edited

csirmazbendeguz commented May 29, 2024 •

edited

csirmazbendeguz commented May 30, 2024 •

edited

charettes Jun 1, 2024

charettes Jun 1, 2024

csirmazbendeguz Jun 2, 2024

charettes Jun 1, 2024

csirmazbendeguz Jun 2, 2024

charettes Jun 2, 2024 •

edited

csirmazbendeguz Jun 2, 2024

charettes Jun 2, 2024

csirmazbendeguz Jun 2, 2024

Fixed #27452 -- Added serial fields to contrib.postgres. #18123

Are you sure you want to change the base?

Fixed #27452 -- Added serial fields to contrib.postgres. #18123

Conversation

csirmazbendeguz commented May 3, 2024 • edited

Trac ticket number

Branch description

Checklist

LilyFoote left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

csirmazbendeguz commented May 5, 2024

LilyFoote commented May 5, 2024

sarahboyce left a comment

Choose a reason for hiding this comment

csirmazbendeguz commented May 27, 2024

InvalidInterrupt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LilyFoote May 28, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

InvalidInterrupt May 29, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

csirmazbendeguz May 29, 2024 • edited

Choose a reason for hiding this comment

csirmazbendeguz commented May 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

csirmazbendeguz commented May 29, 2024 • edited

csirmazbendeguz commented May 29, 2024 • edited

csirmazbendeguz commented May 29, 2024 • edited

csirmazbendeguz commented May 30, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

charettes Jun 2, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

csirmazbendeguz commented May 3, 2024 •

edited

LilyFoote May 28, 2024 •

edited

InvalidInterrupt May 29, 2024 •

edited

csirmazbendeguz May 29, 2024 •

edited

csirmazbendeguz commented May 29, 2024 •

edited

csirmazbendeguz commented May 29, 2024 •

edited

csirmazbendeguz commented May 29, 2024 •

edited

csirmazbendeguz commented May 30, 2024 •

edited

charettes Jun 2, 2024 •

edited