Add cookbook recipe for adding a non-nullable column

[ArockiaRajamanickam]

Adds the cookbook recipe requested in the second bullet of #681: how to add a non-nullable column to a table that already has rows, without a server default.

The recipe follows what was agreed in the thread rather than inventing an approach:

• The main recipe is CaselIT's option 2 (add as nullable, backfill with an UPDATE, then alter_column(nullable=False)), which was the one both of you settled on.
• A .. warning:: covers zzzeek's two points: the UPDATE/ALTER can lock the table and take a live site down on a large dataset, and adding a non-nullable column is hard to make work against an application that keeps running during the migration.
• The server_default alternative is mentioned second, with the note that dropping a server default behaves differently across backends.
• Per CaselIT's comment, the create-new-table-and-copy approach is deliberately not suggested; the section links to :ref:batch_migrations`` instead.

The backfill uses the lightweight sa.table() / sa.column() constructs rather than the application model, so the migration keeps working as the model changes.

Scope: this PR only covers the cookbook half of #681. The env.py logging half was declined by zzzeek in the same thread, so it is left alone, and I have not used Fixes: since the issue bundles both requests.

Verification:

• Ran both recipes through a real MigrationContext against SQLite 3.53 and PostgreSQL 16: confirmed the naive one-step add_column(nullable=False) fails on a populated table, that the three-step recipe leaves an enforced NOT NULL column, that a subsequent NULL insert is rejected, that the downgrade drops the column, and that the server_default variant leaves no residual default.
• sphinx -b html builds with no new warnings; the :ref: and :meth: targets and the two SQLAlchemy intersphinx links all resolve.
• Code blocks are black-clean at the project's 79 character line length.

One note on the SQLite cross-reference: the seealso says the nullable=False step may need batch mode on backends with limited ALTER support, rather than stating flatly that SQLite cannot do it. While testing, SQLite 3.53 accepted ALTER TABLE ... ALTER COLUMN ... SET NOT NULL directly, so the absolute phrasing would not have been accurate on current versions. Happy to reword if you would rather it match the framing in batch.rst.

I used an AI assistant while drafting this; the wording and the testing above are mine and I have checked the content against the thread and the docs build.

[MohammedAlkindi]

I executed the recipe rather than just reading it — docs that ship runnable steps deserve a runnable review. Windows 11, CPython 3.13.13, SQLAlchemy 2.x, in-memory SQLite, table pre-populated with two rows.

What checks out, step by step:

• The opening claim reproduces: op.add_column(..., nullable=False) on the populated table is rejected (OperationalError, "Cannot add a NOT NULL column with default value NULL" class of failure).
• Step 1 (nullable add) and step 2 (backfill through the lightweight sa.table()/sa.column() constructs) run exactly as written, and the backfill lands: both pre-existing rows read status='active'.
• After the constraint is applied, a NULL insert is rejected with IntegrityError — the end state is what the recipe promises.

One real gap: step 3 as written fails on SQLite.

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) near "ALTER": syntax error
[SQL: ALTER TABLE account ALTER COLUMN status SET NOT NULL]

SQLite has no ALTER COLUMN, so op.alter_column(...) emits SQL it cannot execute. The same three steps succeed if step 3 goes through batch mode — I verified this end-to-end (recipe completes, backfill correct, NULL insert rejected):

with op.batch_alter_table("account") as batch_op:
    batch_op.alter_column("status", existing_type=sa.String(50), nullable=False)

Since batch mode is this project's own canonical answer to SQLite's ALTER limitations (and works unchanged on the other backends), a short note — or making batch the primary form with a line saying plain alter_column suffices on PostgreSQL/MySQL — would keep the cookbook honest for the one backend where the current text throws. Given how many people trial migrations against SQLite before pointing them at production, I'd guess this exact error is the first thing a fair fraction of readers would hit.

The lock-duration warning at the end is a good inclusion and matches what the operation actually does.

[ArockiaRajamanickam]

Thank you for actually running it, that caught a real problem.

You are right, and I owe you a correction on the detail. When I wrote this I checked ALTER TABLE ... ALTER COLUMN ... SET NOT NULL against SQLite and found it worked, so I only softened the reference to batch mode into a seealso at the end of the section instead of putting it next to the step. Your report made me pin down why we disagree:

• SQLite 3.51.0: Error: in prepare, near "ALTER": syntax error — exactly what you hit
• SQLite 3.53.0 and 3.53.1: accepted, and genuinely applied. The schema becomes status VARCHAR(50) NOT NULL, a subsequent NULL insert raises IntegrityError, and running it against a table that already contains NULLs is refused rather than silently succeeding.

So the syntax arrived somewhere in 3.52/3.53, which is recent enough that whether the recipe works depends on the SQLite your Python happens to be linked against. My machine picks up 3.53, yours does not, and yours is the situation nearly every reader is in today. A seealso at the bottom of the section is the wrong place for something that decides whether the code above it runs at all.

Fixed in 4f553f8. The batch form now appears directly after step 3, with the actual error text so people can match it, plus a line noting the batch form works on the other backends too and can be used unconditionally.

I verified the batch version end to end the way you did: backfill lands on both rows, the resulting schema carries NOT NULL, a NULL insert is rejected with IntegrityError, and the pre-existing rows keep their data. Docs build succeeded with no new warnings, cookbook.html renders the new block, and the code block is black clean at 79 columns.

[MohammedAlkindi]

Confirmed 4f553f8 resolves what I flagged, on a stock SQLite. Ran the updated recipe steps against SQLite 3.50.4 (alembic 1.19.1, SQLAlchemy 2.0.52, Python 3.13, Windows):

• Step 3 as originally written still fails there: (sqlite3.OperationalError) near "ALTER": syntax error — so the error text now quoted in the doc matches what a reader on a pre-3.52 SQLite sees, character for character.
• The new batch block works end to end: the rebuilt schema reads status VARCHAR(50) NOT NULL, both pre-existing rows keep their backfilled values, and a subsequent NULL insert raises IntegrityError: NOT NULL constraint failed: account.status.

3.50.4 is what a current CPython install links against, which supports the placement call: the batch form next to step 3, rather than in a seealso, is what most readers need today. Nothing left open from my review.

[ArockiaRajamanickam]

Thanks for re-running it, and the 3.50.4 number is the useful part.

That is the piece I could not establish from here. My machines link 3.53, so I could only bracket the change between 3.51 (syntax error) and 3.53 (accepted and enforced). Your 3.50.4 being what a current CPython install ships confirms the practical point: for the foreseeable future almost every reader trying this against SQLite is on a version where the plain form throws, which is exactly why it needed to sit next to step 3 rather than in a seealso at the end.

Appreciate you checking the quoted error matches character for character. That was deliberate, so someone who hits it can search the page for what their traceback says.