From d9d48b0e2cd44212b050c1a6cb08d7ec01c4d229 Mon Sep 17 00:00:00 2001 From: Dan Birman Date: Mon, 25 Nov 2024 12:45:47 -0800 Subject: [PATCH] docs: clarifying QC docs (#1158) * docs: clarifying QC docs for reference images * docs: adding alternate workflow documentation * docs: more small fixes * refactor: removing qc portal docs, moving to readme --- docs/source/quality_control.md | 72 +++++----------------------------- 1 file changed, 9 insertions(+), 63 deletions(-) diff --git a/docs/source/quality_control.md b/docs/source/quality_control.md index a561ecb0..45b4e937 100644 --- a/docs/source/quality_control.md +++ b/docs/source/quality_control.md @@ -6,9 +6,9 @@ Quality control is a collection of **evaluations** based on sets of **metrics** `QCEvaluation`s should be generated during pipelines: before raw data upload, during processing, and during analysis by researchers. -Each `QualityControl`, `QCEvaluation`, and `QCMetric` includes a `aind_data_schema.quality_control.State` which is a timestamped object indicating that the Overall QC/Evaluation/Metric passes, fails, or is in a pending state waiting for manual annotation. +The overall `QualityControl`, each `QCEvaluation`, and each `QCMetric` can be evaluated to get a `aind_data_schema.quality_control.State` which indicates whether the Overall QC/Evaluation/Metric passes, fails, or is in a pending state waiting for manual annotation. -The state of an evaluation is set automatically to the lowest of its metric's states. A single failed metric sets an entire evaluation to fail. While a single pending metric (with all other metrics passing) sets an entire evaluation to pending. An optional setting `QCEvaluation.allow_failed_metrics` allows you to ignore failures, which can be useful in situations where an evaluation is not critical for quality control. +The state of an evaluation is set automatically to the lowest of its metric's states. A single failed metric sets an entire evaluation to fail. A single pending metric (with all other metrics passing) sets an entire evaluation to pending. An optional setting `QCEvaluation.allow_failed_metrics` allows you to ignore failures, which can be useful in situations where an evaluation is not critical for quality control. ## Details @@ -30,75 +30,21 @@ Each `QCMetric` is a single value or set of values that can be computed, or obse `QCMetric`s have a `Status`. The `Status` should depend directly on the `QCMetric.value`, either by a simple function: "value>5", or by a qualitative rule: "Field of view includes visual areas". The `QCMetric.description` field should be used to describe the rule used to set the status. Metrics can be evaluated multiple times, in which case the new status should be appended the `QCMetric.status_history`. +**Q: What is a metric reference?** + Metrics should include a `QCMetric.reference`. References are intended to be publicly accessible images, figures, combined figures with multiple panels, or videos that support the metric or provide information necessary for manual annotation of a metric's status. +See the AIND section for specifics about how references are rendered in the QC Portal. + **Q: What are the status options for metrics?** -In our quality control a metric's status is always `PASS`, `PENDING` (waiting for manual annotation), or `FAIL`. `PENDING` should be used when a user must manually annotated the metric's state. +In our quality control a metric's status is always `PASS`, `PENDING` (waiting for manual annotation), or `FAIL`. We enforce this minimal set of states to prevent ambiguity and make it easier to build tools that can interpret the status of a data asset. ## Details for AIND users -### Uploading QC - -#### Preferred workflow - -**Metadata** - -If you are building `QCEvaluation` and `QCMetric` objects prior to raw data upload or in a capsule alongside your processing or analysis, your workflow is: - -``` -from aind_data_schema.core.quality_control import QualityControl - -# Build your QCEvaluations and metrics -evaluations = [QCEvaluation(), ...] - -# Build your QualityControl object -qc = QualityControl(evaluations=evaluations) - -qc.write_standard_file() -``` - -The indexer will pick up this file alongside the other metadata files and handle it appropriately. - -**References** - -Each `QCMetric` you build should have an attached reference. Our preference is that you post these images to [FigURL](https://github.com/flatironinstitute/figurl/blob/main/doc/intro.md) and put the generated URL into the reference. - -We recommend that you write PNG files for images and static multi-panel figures, MP4 files for videos, and Altair charts for interactive figures. - -#### Alternate workflows - -**Metadata** - -We'll post documentation on how to append `QCEvaluations` to pre-existing quality_control.json files, via DocDB using the `aind-data-access-api`, in the future. For now, you can refer to the code snippet in the [`aind-qc-capsule-example`](https://github.com/AllenNeuralDynamics/aind-qc-capsule-example/). - -**References** - -You can also place the references in the data asset itself, to do this include the relative path "qc_images/your_image.png" to that asset inside of the results folder. - -### QC Portal - -The QC Portal is a browser application that allows users to view and interact with the AIND QC metadata and to annotate ``PENDING`` metrics with qualitative evaluations. The portal is maintained by scientific computing, reach out to us with any questions or concerns. - -The portal works by pulling the metadata object from the Document Database (DocDB). When you make changes to metrics or add notes the **submit** button will be enabled, submitting pushes your updates to the DocDB along with a timestamp and your name. - -**Q: When does the state get set for the QCEvaluation and QualityControl objects?** - -The QC portal automatically calls ``QualityControl.evaluate_status()`` whenever you submit changes to the metrics. This first evaluates the individual `QCEvaluation` objects, and then evaluates the overall status. - -**Q: How do reference URLs get pulled into the QC Portal?** - -Each metric is associated with a reference figure (PDF preferred), image (png preferred), or video (mp4 preferred). The QC portal can interpret four ways of setting the reference field: - -- Provide a relative path to a file in this data asset's S3 bucket, i.e. "figures/my_figure.png". The mount/asset name should not be included. -- Provide a url to a FigURL figure -- Provide the name of one of the interactive plots, e.g. "ecephys-drift-map" - -**Q: I saw fancy things like dropdowns in the QC Portal, how do I do that?** - -By default the QC portal displays dictionaries as tables where the values can be edited. We also support a few special cases to allow a bit more flexibility or to constrain the actions that manual annotators can take. Install the [`aind-qcportal-schema`](https://github.com/AllenNeuralDynamics/aind-qcportal-schema/blob/dev/src/aind_qcportal_schema/metric_value.py) package and set the `value` field to the corresponding pydantic object to use these. +Instructions for uploading QC for viewing in the QC portal can be found [here](https://github.com/AllenNeuralDynamics/aind-qc-portal). ### Multi-asset QC @@ -110,4 +56,4 @@ You should follow the preferred/alternate workflows described above. If your mul **Q: I want to be able to store data about each of the evaluated assets in this metric** -Take a look at the `MultiAssetMetric` class in `aind-qc-portal-schema`. It allows you to pass a list of values which will be matched up with the `evaluated_assets` names. You can also include options which will appear as dropdowns or checkboxes. \ No newline at end of file +Take a look at the `MultiAssetMetric` class in `aind-qc-portal-schema`. It allows you to pass a list of values which will be matched up with the `evaluated_assets` names. You can also include options which will appear as dropdowns or checkboxes. \ No newline at end of file