Newly added column is not getting synced after Realm is migrated

[sskjames]

Hi @mentrena,

Thank you very much for this amazing library. We are using Realm 10.20.1 + SyncKit 1.3.0.
We recently had to change the data type of a column. But since CloudKit doesn't allow changing a column's data type (once the schema is deployed to Production), we added a new column with the correct data type and added migration scripts to handle this change.

This means once the schema is upgraded, the local Realm database will not have the old column.
Under these circumstances, the value in the new column never gets synced to CloudKit.
Even when I change other columns in the same record, those changes were synced but not the newly added column.

However, when I make an explicit change to that column, it is getting synced as expected.
I even tried calling eraseLocalMetadata after Realm objects are migrated, but the problem remains.
Have I missed something? Do I need to do something after Realm objects are migrated? Appreciate your help and insight.

Thanking in advance,
James

[sskjames]

Observed that basically any column (it need not be a new column) that is migrated is not synced. Will there be any Realm notifications during migration? Is that why SyncKit is not tracking these changes?

[sskjames]

We are initializing the synchronizers after performing the Realm migration. Looks like this might be the reason why SyncKit is not receiving any notifications.

How do you all do migration? Kindly throw some light on this.

[mentrena]

Hi @sskjames, I'm afraid there's currently no logic to handle a migration.
Can you paste your migration code here? I would like to add support for migrations and it might help me come up with a solution.

In the meantime, if you're in a rush, I would suggest adding the new column to the model, loading the Realm without a custom migration, and manually assigning the values so the changes will be picked up by SyncKit. Then in the future you could do another migration where you drop the old column. But I understand this might not work for your use case

[sskjames]

Hi @mentrena, thank you very much for your feedback. I managed to find a workaround so that the changes are picked up but yes it would be nice if SyncKit provides an easy way to manage this future.

Here's how our migration code looks like:

 func migrate() {
        let newSchemaVersion = getNewSchemaVersion()
        var migratingSchemaVersion = newSchemaVersion
        
        Realm.Configuration.defaultConfiguration = Realm.Configuration(schemaVersion: newSchemaVersion, migrationBlock: { migration, oldSchemaVersion in
            migratingSchemaVersion = oldSchemaVersion
           
            // here, we just save the values that need to be migrated in UserDefaults so that they can be migrated later
            self.prepareMigration(oldSchemaVersion, migration)
        })
        
        try? Realm.performMigration(for: Realm.Configuration.defaultConfiguration)        
                
        // synchronizers can't be loaded before a migration is performed
        loadSynchronizers()

       // values saved in UserDefaults shall be picked up and transformed here so that SyncKit would receive Realm notifications
        onAfterMigration(migratingSchemaVersion)                        
    }

"prepareMigration" code is just how we normally do Realm migration.

        migration.enumerateObjects(ofType: YourRealmObject.className()) { oldObject, newObject in
            if let oldValue = oldObject?["oldColumnName"] as? Int {
                  // in a typical Realm migration, you would do the migration by setting the value in your newColumn
                 // but if we do like that, SyncKit will not be able to receive notifications as the synchronizers are not yet initialized
            }            
        }

We couldn't load the synchronizers before the migration is performed, as the database couldn't be opened. So in the "prepareMigration" phase, we just save the old object details in UserDefaults currently and then later just apply the changes once the synchronizers are initialized. By doing it this way, SyncKit was able to receive notifications and the migrated changes were synced.

This workflow would be helpful when the user already has the database locally. In the even of the user's data is not available locally but has to be retrieved from CloudKit, we hook into the excellent RecordProcessingDelegate architechture provided by SyncKit to do the migration.

[mentrena]

Hey @sskjames I'm thinking about your scenario and wondering if what you're doing isn't already the best possible solution.
Any kind of migration that removes or renames a field will result in a model that is not backwards compatible. If that model has been synced with CloudKit, then there might be records created with the older version of the model that includes the deleted field, which means that one would forcefully have to implement RecordProcessingDelegate to provide custom handling of those records. So there's no avoiding that.

Do you have any idea of what a better solution would look like? In terms of interface.
Maybe, would something like a function to manually mark fields in an object as updated improve the code? It's not too different from what you're doing though, when you save the changes and then apply them after the migration has finished.