refactor: datetime -> Aware for created/last_modified

[dbirman]

This PR refactors the created/last_modified fields to use AwareDatetimeWithDefault to force users to include timezone information or have it added automatically.

Note that there are still a lot of places in the code where people are using datetime.utcnow.isoformat() which returns a naive datetime object without timezone information. Everybody needs to switch to using datetime.now(timezone.utc).isoformat()

[jtyoung84]

We should also add either a validator to coerce the timezone to utc, or a custom serializer so that the string representation is always in UTC. There is a discussion why it is useful to keep the timestamp string representations in DocDB sortable: https://stackoverflow.com/questions/38497442/what-is-the-best-way-to-store-datetime-value-in-documentdb

[dbirman]

We should also add either a validator to coerce the timezone to utc, or a custom serializer so that the string representation is always in UTC. There is a discussion why it is useful to keep the timestamp string representations in DocDB sortable: https://stackoverflow.com/questions/38497442/what-is-the-best-way-to-store-datetime-value-in-documentdb

Do you know how times in docdb get the ms information? Just want to make sure the final output is in the right format.

[jtyoung84]

We should also add either a validator to coerce the timezone to utc, or a custom serializer so that the string representation is always in UTC. There is a discussion why it is useful to keep the timestamp string representations in DocDB sortable: https://stackoverflow.com/questions/38497442/what-is-the-best-way-to-store-datetime-value-in-documentdb

Do you know how times in docdb get the ms information? Just want to make sure the final output is in the right format.

@helen-m-lin Do you know? I'm pretty sure they just take the json we send them. When we write to DocDB, we first run m.model_dump_json() on the pydantic model. It might depend on the version of python unless we add a custom serializer.

[helen-m-lin]

@helen-m-lin Do you know? I'm pretty sure they just take the json we send them. When we write to DocDB, we first run m.model_dump_json() on the pydantic model. It might depend on the version of python

Yeah, the datestamps are being stored as strings in docdb. I'm finding that the serialization is not consistent where some have UTC timezone and some do not. I'll send some examples offline.

[dbirman]

Sounds like we should add a field_serializer to force it to a specific format?

[jtyoung84]

Sounds like we should add a field_serializer to force it to a specific format?

Sounds good. Saskia has an example here:

aind-data-schema/src/aind_data_schema/core/procedures.py

Line 596 in 95e3882

@field_serializer("output_specimen_ids", when_used="json")

[dbirman]

Ah just noticed that we should update the docs to reflect that timezones are in UTC by default, I'll fix that before we merge this nevermind I think we're good

[saskiad]

We don't want to use UTC. we want tz_aware.

[jtyoung84]

We don't want to use UTC. we want tz_aware.

These are for timestamps specific to what's getting stored in DocDB. We want those in a consistent serialized format to make them sortable.

[dbirman]

@jtyoung84 @helen-m-lin is this still waiting on something? I think I cleared out all the requested changes