Experiment Set model for grouping data sets

[miketaormina]

Idea: A JSON model that can be used to link data sets with common features.

This may be outside of the scope of the repository, but I can imagine the usefulness of creating curated groups of data sets and having a human + machine readable record of it.

Possible Uses

• Defining the list of all data assets used in a publication.
• Collecting all data assets generated for grant funded work.
• Grouping individual data sets of the same specimen/subject, but collected with different modalities.
  • Ophys + Zeiss mFISH
  • Neuropixels + SmartSPIM
  • exaSPIM/SmartSPIM + Zeiss mFISH
• Grouping multiple specimens from one or more subjects that are processed or acquired as a batch.
  • BarSeq (not currently modeled), for which a single acquisition is composed of 1-4 slides with 1-4 sections per slide. Splitting these into raw data by specimen would be ideal, but some aspects of the data analysis may incorporate information from all specimens in an acquisition.
• More generally, it keeps the flat structure of the bucket in tact, while allowing a hierarchy/organization of the data that can be used for building dashboards, ingesting into capsules that produce figures, providing single asset URIs for reporting, ensuring that collaborators work on the same set of data when generating downstream scientific work.

Possible downsides

• Bucket organization:
  • Would these be top-level objects consisting of single files?
  • Would it be possible to have a sensible naming convention for them?
• Generating a list of data sets and their modalities is easy, but would it also be easy to include a sensible description to define why they belong in a group?
• Groups of groups: Is it turtles all the way down? E.g., if writing a paper for Ophys + mFISH, is there a list of groups to link the two modalities, and then a group of these groups to collate the entire data set used in the publication?
• How to sensibly version? I could imagine a group getting defined and subsequently more data sets would be added to it, but this needs to be done in a version controlled manner.

---

Overall, I think the concept is a useful one and that it would facilitate the public facing tools that may be built around the open data bucket in the future, but that we would want to be careful about who/how/how often these can be placed into the bucket. Maybe defining the schema and keeping them on the bucket are two separate topics?

I think the model might look something like:

"curators": ["Mike Taormina"],
"name": "AAV Toolbox for Basal Ganglia Cell Types and Circuits",
"description": "Whole mouse brain fluorescence image volumes for screening the specificity and effectiveness of viral genetic tools targeting molecular cell types. . . . .",
"modalities": ["SmartSPIM",],
"species": ["Mus musculus",],
"funding_sources": [LIST OF FUNDING MODELS FOUND IN DATA ASSETS],
"version": "0.0.1",
"curation_date": "2024-11-07_00-00-00",
"data_assets": [
    {
        "modality": "SmartSPIM",
        "source": "Allen Institute for Brain Science",
        "uri": "s3://aind-open-data/SmartSPIM/123456_2024-11-07_00-00-00",
    },
    {. . . },
]

[miketaormina]

Possibly related to discussion #562

Also, I'm in the data club today and think this might be related to and/or duplicated by the DocDB metadata aggregator (?)

[saskiad]

There are several fields in data_description that help for this. The project_name and related_data being the key ones.
Ultimately I think this will happen more via collections in Code Ocean, etc.

But for the use cases you outline,

1. we had considered adding a publications list to the assets, but this will be hard(impossible?) to maintain. I expect we will make unique collections for specific publications. What might be harder is see every publication for a given asset.
2. funding field allows users to identify all assets for a given grant.
3. the use of related_data should allow assets for a given subject to be easily linked together
4. We should talk through this case. For analysis/processing - the processing class should allow us to annotate this. But there might be some other nuance here.

[miketaormina]

Good points. I agree that (1) is basically impossible, and probably undesirable (having a publication as a property of each data object).

I was thinking about the opposite case (which you allude to as "unique collections"), where a JSON stands as it's own object and is a curated list of data objects. There would be no need to alter metadata for individual data sets or try to build a list from the DocDB, which is continually rebuilt (I think).

I could imagine the usefulness, but think that reasonable people might disagree on whether such a thing belongs in this repository or whether the files themselves should live in the data bucket. I think what one would want for publications is a human and machine readable list of data object URIs that is (1) publicly accessible and (2) immutable or version controlled. One could argue that this list belongs in the publication itself, I suppose.

Perhaps the use case here is too limited.

[saskiad]

So the collections are a Code Ocean thing, but I completely agree that the publication (and the accompanying github repo for the publication) should have a list of the assets in the publication.

[tmchartrand]

as Saskia mentioned, we've been working on something along these lines for the use case of processing that involved many assets specifically. see #1167