DEV Community

Kamal Mustafa
Kamal Mustafa

Posted on

Django: Migrate --fake-initial gotchas

Some gotcha we got into while upgrading an old system build since Django-1.0, which then gradually upgraded to 1.4, 1.8 and finally now to 1.11 (the last 1.x version). For notes, database migrations framework only landed in Django-1.7. In this new migrations framework, you have to explicitly generate a migration files. Prior to that, the syncdb command will generate tables (which doesn't exists) on the fly based on your model definition. Subsequent changes to the table, like adding new column has to be handled manually through external means.

Up until 1.8, having explicit migrations files not yet mandatory so the migrate command basically exhibit same behavior as the old syncdb. But since we're updating to 1.11, we have to create the migrations files for all the apps in the system. This initial migrations basically involved creating the tables when we run migrate command and since all the migrations not yet tracked, it will try to apply it which obviously will fail because all the tables already exists.

For that reason, django provide 2 options:-

--fake¶
Tells Django to mark the migrations as having been applied or unapplied, but
without actually running the SQL to change your database schema.

This is intended for advanced users to manipulate the current migration state
directly if they’re manually applying changes; be warned that using --fake runs the
risk of putting the migration state table into a state where manual recovery will
be needed to make migrations run correctly.

--fake-initial¶
Allows Django to skip an app’s initial migration if all database tables with the
names of all models created by all CreateModel operations in that migration already
exist. This option is intended for use when first running migrations against a
database that preexisted the use of migrations. This option does not, however,
check for matching database schema beyond matching table names and so is only safe
to use if you are confident that your existing schema matches what is recorded in
your initial migration.
Enter fullscreen mode Exit fullscreen mode

We decided to use --fake-initial for the the migrate. But we got error table already exists. It doesn't make sense at first, as we can read above it will detect if table already exists and then just "fake apply" the migrations. And we know for sure the table really exists, so why Django still want to create that table ?

I stepped through the migrations code in django/db/migrations/executor.py and put a pdb.set_trace() in this part:-

        # Make sure all create model and add field operations are done
        for operation in migration.operations:
            import pdb;pdb.set_trace()
            if isinstance(operation, migrations.CreateModel):
                model = apps.get_model(migration.app_label, operation.name)
                if model._meta.swapped:
Enter fullscreen mode Exit fullscreen mode

Stepping through this and fortunately the error happened pretty early, I can see it correctly detected the table exists:-

(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(337)detect_soft_applied()
-> if should_skip_detecting_model(migration, model):
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(339)detect_soft_applied()
-> if model._meta.db_table not in existing_table_names:
(Pdb) l
334                         # We have to fetch the model to test with from the
335                         # main app cache, as it's not a direct dependency.
336                         model = global_apps.get_model(model._meta.swapped)
337                     if should_skip_detecting_model(migration, model):
338                         continue
339  ->                 if model._meta.db_table not in existing_table_names:
340                         return False, project_state
341                     found_create_model_migration = True
342                 elif isinstance(operation, migrations.AddField):
343                     model = apps.get_model(migration.app_label, operation.model_name)
344                     if model._meta.swapped:
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(341)detect_soft_applied()
-> found_create_model_migration = True
Enter fullscreen mode Exit fullscreen mode

Until it came to this iteration:-

> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(330)detect_soft_applied()
-> import pdb;pdb.set_trace()
(Pdb) operation.name
'Image'
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(331)detect_soft_applied()
-> if isinstance(operation, migrations.CreateModel):
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(332)detect_soft_applied()
-> model = apps.get_model(migration.app_label, operation.name)
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(333)detect_soft_applied()
-> if model._meta.swapped:
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(337)detect_soft_applied()
-> if should_skip_detecting_model(migration, model):
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(339)detect_soft_applied()
-> if model._meta.db_table not in existing_table_names:
(Pdb) n
> /home/kamal/.local/lib/python3.4/site-packages/django/db/migrations/executor.py(340)detect_soft_applied()
-> return False, project_state
Enter fullscreen mode Exit fullscreen mode

So why it detected the table as not exists ? Inspecting the variable existing_table_names finally gave me a light. The table indeed does not exists !

(Pdb) model._meta.db_table not in existing_table_names
True
(Pdb) model._meta.db_table
'blog_image'
(Pdb) existing_table_names
['auth_group', 'auth_group_permissions', 'auth_permission', 'auth_user', 'auth_user_groups', 'auth_user_user_permissions', 'blog_article', 'django_admin_log', 'django_content_type', 'django_migrations', 'django_session']
Enter fullscreen mode Exit fullscreen mode

So what happened was that it detecting the table as a group, so if there's one table does not exists, then the whole of that particular migrations will be applied, which mean also creating table that already exists. In our case, this happened because when dumping the production database to test this migrations, we exclude some tables with large data to keep the dump small. Which mean the resulting structure not really matching what described in the migrations file, which already being warned in the documentation above, sigh.

Top comments (5)

Collapse
 
simo97 profile image
ADONIS SIMO

I am in the same situation actually, but my main concern is that even when i have all the migrations in the file and i try to makemigrations it keep asking me to create all models even the one existing.
so i followed this : simpleisbetterthancomplex.com/tuto... to restart but nothing.

like migration 0004 has my models, and i when i make migrations the 0005 ask me to CreateModel but those one are alreafy existing.

Collapse
 
k4ml profile image
Kamal Mustafa

By "models" here do you mean "table" and not the model class definition?

Collapse
 
simo97 profile image
ADONIS SIMO

Yes, let's take an example of my situation,

  • migrations from 0001 to 0004 contains my models and changes that are already applied in the db

  • when creating migration 0005 (with makemigrations) it happen to contain CreateModel() of all the existing models even the one already migrated into the DB ...

i am stucked with that since a lot of weeks now

Thread Thread
 
a3n3d3i profile image
Andi

In case you do some manual modifications to your database and want the django migrations to "think it did it" you could fake the specific migration by using the "python manage.py migrate --fake yourAppName 00nn_migration_number" command.
This will mark the migrations as applied without modifying the database tables field.

Collapse
 
eonerekim profile image
Mike

This post just helped me a ton, thanks for sharing!