First pass at native NDArray support. #181

cmungall · 2024-02-05T00:21:56Z

See

This introduces first-class array support into LinkML.

A minimal example would be:

    attributes:
      temperature_matrix:
        range: float
        array_info:
          exact_dimensions: 3

The native serialization of this in json/yaml will be a LoLoL. Using linkml-xarrays it will be possible to serialize using hdf5/zarr/etc.

The corresponding nptyping type would be NDArray[Shape["*, *, *"], Float].

(note: modelers will want the ability to use ctypes but this is orthogonal)

Note that this does not force any metadata on the array; we are deferring on the datamodel for what is equivalent to xarray DataArrays, these will be supported via implements for now and first-class incorporation in a future version. This will allow binding between axes are other LinkML arrays.

Minimal metadata can be introduced via naming the axes

    attributes:
      temperature_matrix:
        range: float
        array_info:
          exact_dimensions: 3
          dimensions:
            x:
            y:
            z:

The corresponding nptyping type would be NDArray[Shape["* x, * y, * z"], Float].

The shape can be further constrained; imagine an RGB matrix with coords x, y, and a length 3 r/g/b:

    attributes:
      rgb:
        range: float
        array_info:
          exact_dimensions: 3
          dimensions:
            x:
            y:
            rgb:
              exact_cardinality: 3
              description: r, g, b values
              annotations:
                names: "[red, green, blue]"

corresponds to NDArray[Shape["* x, * y, 3 rgb"]

For now if you do want to bind dimensions to additional metadata this can be done via annotations:

classes:

  TemperatureDataset:
    tree_root: true
    annotations:
      array_data_mapping:
        data: temperatures_in_K
        dims: [x, y, t]
        coords:
          latitude_in_deg: x
          longitude_in_deg: y
          time_in_d: t
    attributes:
      name:
        identifier: true
        range: string
      latitude_in_deg:
        required: true
        range: float
        multivalued: true
        unit:
          ucum_code: deg
        array_info:
          exact_dimensions: 1
      longitude_in_deg:
        required: true
        range: float
        multivalued: true
        unit:
          ucum_code: deg
        array_info:
          exact_dimensions: 1
      time_in_d:
        range: float
        multivalued: true
        implements:
          - linkml:elements
        required: true
        unit:
          ucum_code: d
        array_info:
          exact_dimensions: 1
      temperatures_in_K:
        range: float
        multivalued: true
        required: true
        unit:
          ucum_code: K
        array_info:
          exact_dimensions: 3

See - linkml/linkml#895 - linkml/linkml-arrays#5

tests/input/examples/schema_definition-native-array-rgb.yaml

rly · 2024-02-07T01:44:03Z

tests/input/examples/schema_definition-array-2.yaml

@@ -26,7 +26,7 @@ classes:
    implements:
      - linkml:NDArray
    annotations:
-      dimensions: 1
+      dimensions_info: "1"


Should I reformat these to use:

array_info: exact_dimensions: 1

For this TimestampSeries class, I'm actually not sure it would be allowed to put array_info at the class definition level, but maybe for the other ones?

unfortunately that doesn't quite work because the range of dimensions_info is a collection of dimension_expressions, so these have to be at a different level

e.g. this is invalid yaml:

dimensions_info: 3 x: y: z:

rly · 2024-02-07T14:39:00Z

tests/input/examples/schema_definition-native-array-rgb.yaml

+            x:
+            y:
+            rgb:
+              exact_cardinality: 3
+              description: r, g, b values
+              annotations:
+                names: "[red, green, blue]"


Suggested change

x:

y:

rgb:

exact_cardinality: 3

description: r, g, b values

annotations:

names: "[red, green, blue]"

x:

dimension_index: 0

y:

dimension_index: 1

rgb:

dimension_index: 2

exact_cardinality: 3

description: r, g, b values

annotations:

names: "[red, green, blue]"

It would be good to be explicit here. This also allows some dimensions to be unnamed.

rly · 2024-02-07T15:33:13Z

Notes:

It is possible to have scalar arrays in many array formats (e.g., NumPy, HDF5, Zarr) but not all (e.g., Matlab). They seem primarily used to allow API functions to work seamlessly with both arrays and scalars and as a box to pass scalars to a function by reference. Matlab gets around this by not having special dtypes for scalars and not allowing scalar arrays - they are just 1x1 matrices. I do not know that we need to support scalar arrays. To be conservative, perhaps it would be best to leave out support until there is a need. Refs:
- Interesting comment on why scalar arrays exist in numpy
- Description of scalar dataspaces in HDF5
- Use of 0-D arrays as a data structure in Julia
- Use of 0-D arrays as a data structure in Lisp
- Or put attributes on an HDF5 dataset that is a single value (but you can still do that with a dataset of shape (1, ))
If not specified, minimum_number_dimensions is 1, maximum_number_dimensions is Infinity.
It is possible to specify minimum_number_dimensions without maximum_number_dimensions. We have a use case for a min without a max, but I cannot think of one for a max without a min.
I am not sure there is a use case for minimum_cardinality and maximum_cardinality. Maybe we can omit them and add them later if there is a need? I am concerned about adding features to the metamodel that are useless but need to be supported for a long time to avoid breaking backwards compatibility.

Some useful validation checks:

The number of items in dimensions_info must not exceed maximum_number_of_dimensions
Each dimension_index must not exceed maximum_number_of_dimensions - 1
Each dimension_index must be unique across dimensions_info items
If array_info exists, then multivalued must be True (if we allow scalar arrays, then this is no longer always true)
minimum_number_dimensions <= maximum_number_dimensions
Cannot have both exact_number_dimensions and minimum_number_dimensions or both exact_dimensions and maximum_number_dimensions
minimum_number_dimensions, exact_number_dimensions, maximum_number_dimensions all > 0
minimum_cardinality <= maximum_cardinality
Cannot have both exact_cardinality and minimum_cardinality or both exact_cardinality and maximum_cardinality
minimum_cardinality, exact_cardinality, maximum_cardinality all > 0

sneakers-the-rat

OK I am mostly settled on this, but I have a pitch that i'll make in person about extending the notion of implements and how it could allow for plugins in a minute (and then update this so that it's public later)

sneakers-the-rat · 2024-02-07T18:45:36Z

linkml_model/model/schema/meta.yaml

+  array_info:
+    domain: slot_definition
+    range: array_info_expression
+    inherited: true
+    description: coerces the value of the slot into an array and defines the dimensions of that array
+    status: testing
+
+  dimensions_info:


hopefully not annoying naming question, why not just array and dimensions here?

(we did change it to that)

sneakers-the-rat · 2024-02-07T18:48:21Z

linkml_model/model/schema/meta.yaml

+    inlined: true
+    status: testing
+
+  minimum_dimensions:


how do we model mutual exclusivity and constraints between properties here? eg. one shouldn't use exact_dimensions alongside maximum_dimensions, maximum_dimensions shouldn't be less than minimum_dimensions, etc. Part of why i like expressing ranges/values in a single object like dimensions: 3 or dimensions: {min: 2, max: 3} is that you can model within the scope of that object, but it seems like that can also be done here i'm just not sure how

sneakers-the-rat · 2024-02-07T18:49:21Z

linkml_model/model/schema/meta.yaml

+    range: integer
+    status: testing
+
+  maximum_dimensions:


separate but related question to above - it seems like ranges and exact values will/is a common pattern, thoughts on having a range syntax so we could just have a single property that can take an integer or a 1..2 specification?

sneakers-the-rat · 2024-02-07T18:50:45Z

linkml_model/model/schema/meta.yaml

@@ -1420,6 +1420,39 @@ slots:
      - BasicSubset
      - ObjectOrientedProfile

+  array_info:


are there no minimum properties that need to be specified on an array? eg. having this property without any values is an Any shaped array?

tests/input/examples/schema_definition-native-array-rgb.yaml

Modeling session with Ryan, Ben, Jonny

linkml_model/model/schema/meta.yaml

sneakers-the-rat · 2024-02-08T00:01:50Z

for posterity, archived version of matrix of tradeoffs to be added to consolidated docs later, along with @rly 's notes and examples :)

live: https://wiki.jon-e.net/LinkML_Arrays
archive: https://web.archive.org/web/20240207234701/https://wiki.jon-e.net/LinkML_Arrays

linkml_model/model/schema/meta.yaml

rly · 2024-02-08T01:06:05Z

linkml_model/model/schema/meta.yaml

+  has_extra_dimensions:
+    description: If this is set to true
+    domain: array_expression
+    range: boolean
+    status: testing
+


Suggested change

has_extra_dimensions:

description: If this is set to true

domain: array_expression

range: boolean

status: testing

rly · 2024-02-08T01:06:47Z

linkml_model/model/schema/meta.yaml

+      - exact_number_dimensions
+      - minimum_number_dimensions
+      - maximum_number_dimensions
+      - has_extra_dimensions


Suggested change

- has_extra_dimensions

cmungall added 4 commits February 4, 2024 16:21

First pass at native NDArray support.

f526171

See - linkml/linkml#895 - linkml/linkml-arrays#5

Incorporating hackathon feedback

e4147f6

nptyping example

84664b2

Adding testing annotation

240f129

cmungall requested a review from rly February 7, 2024 00:45

cmungall added 3 commits February 6, 2024 16:47

rename

7cde1a3

Updating example

264b408

rename

c110bc6