model_construct is now dropping extra fields in aind-metadata-upgrader

[Sun-flow]

In upgrading pins to support aind-data-schema 1.0, it seems the behavior of model_construct has changed.

Changes to pins:
aind-data-schema: 0.38.6 -> 1.0.0
pydantic: < 2.7 -> >=2.7

Now, constructed models are losing the DataDescription.creation_date field, because it no longer exists in the current version of model -- even if it existed in the version of model that was originally created.

This behavior is governed by the field model_config.extra, which is currently set to 'forbid' in AindModel. Unsure why the function of this changed with raising the version of pydantic, but now many upgrades are failing due to missing critical old information from models that are being upgraded.

How should we solve this?

[saskiad]

The change from creation_date and creation_time to just creation_time was back in schema version ~0.14 so I'm unclear why this wasn't an issue with 0.38.6

The upgrader should be merging the date and time fields into a single field.

[Sun-flow]

I agree, I think it is surprising this issue arose. It could be due to upgrading the Pydantic version pin, though.

[saskiad]

I'm not surprised the issue arose - I'm surprised it didn't arise prior to this.

[Sun-flow]

It was an issue with upgrading pydantic, not with upgrading ADS, and we had pinned it to below the breaking change rather than solving it back in May. Check out this old discussion bruno linked for more info: #912

[bruno-f-cruz]

For what is worth here's my previous post describing this behavior along with the corresponding pydantic PR reference:
#912 (comment)

I think the general pattern should be to avoid using model_construct altogether and instead rely on model_validator and field_validator. In this case, a model_validator(..., mode='before') could be used to capture the "creation_data" field and merge its value to "create_time" before any further validation takes place.

Otherwise, inheriting from the BaseModel class and override the classmethod model_construct might also do the trick. It seems that the whole behavior is governed by this line:

https://github.com/pydantic/pydantic/blob/03a90cb0264e5630452a54565ccedcea54a3d93e/pydantic/main.py#L293