[TABLE MODEL] Implement CREATE/SHOW/DROP Function and user-defined scalar function #14223

Cpaulyz · 2024-11-27T04:59:57Z

Description

https://timechor.feishu.cn/docx/GXnNdt3XIok3CQxBvwWcZxrnntd

This pull request introduces several new user-defined functions (UDFs) and corresponding integration tests for the IoTDB project. The changes include the addition of new scalar functions, updates to existing classes, and the creation of new test cases to ensure the functionality and robustness of the UDF management system.

New Scalar Functions:

ScalarFunctionExample: Implements a scalar function that checks if any input value is null and returns a boolean result.

Integration Tests:

IoTDBSQLFunctionManagementIT: Adds multiple test cases to verify the creation, display, and deletion of scalar functions, including handling of invalid function names and URIs.

codecov · 2024-11-27T06:08:10Z

Codecov Report

Attention: Patch coverage is 2.29358% with 213 lines in your changes missing coverage. Please review.

Project coverage is 39.59%. Comparing base (b34b714) to head (30be7e5).

Files with missing lines	Patch %	Lines
...n/execution/config/metadata/ShowFunctionsTask.java	0.00%	58 Missing ⚠️
...otdb/commons/udf/utils/UDFDataTypeTransformer.java	0.00%	38 Missing ⚠️
...olumn/udf/UserDefineScalarFunctionTransformer.java	0.00%	36 Missing ⚠️
...pache/iotdb/commons/udf/access/RecordIterator.java	0.00%	22 Missing ⚠️
...execution/relational/ColumnTransformerBuilder.java	0.00%	14 Missing ⚠️
...ne/plan/relational/metadata/TableMetadataImpl.java	0.00%	14 Missing ⚠️
.../apache/iotdb/commons/udf/utils/TableUDFUtils.java	0.00%	10 Missing ⚠️
...yengine/plan/relational/sql/parser/AstBuilder.java	0.00%	9 Missing ⚠️
.../plan/execution/config/TableConfigTaskVisitor.java	0.00%	6 Missing ⚠️
...otdb/commons/udf/service/UDFManagementService.java	20.00%	4 Missing ⚠️
... and 2 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #14223      +/-   ##
============================================
- Coverage     39.62%   39.59%   -0.04%     
  Complexity       71       71              
============================================
  Files          4242     4245       +3     
  Lines        270437   270647     +210     
  Branches      32792    32813      +21     
============================================
- Hits         107165   107152      -13     
- Misses       163272   163495     +223

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

lancelly · 2024-11-27T07:35:46Z

What a spectacular work 🦾! Fabulous design and implementation, LGTM! ⚡️⚡️🔥

sonarcloud · 2024-11-28T09:12:33Z

Quality Gate failed

Failed conditions
B Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

JackieTien97 · 2024-11-29T09:07:50Z

...i/udf-api/src/main/java/org/apache/iotdb/udf/api/customizer/config/ScalarFunctionConfig.java

+
+public class ScalarFunctionConfig extends UDFConfigurations {
+
+  public ScalarFunctionConfig setOutputDataType(Type outputDataType) {


all public methods are visible to users, so we need to add java doc to explain the functionality.

JackieTien97 · 2024-11-29T09:08:08Z

.../udf-api/src/main/java/org/apache/iotdb/udf/api/customizer/parameter/FunctionParameters.java

+  public List<Type> getChildExpressionDataTypes() {
+    return childExpressionDataTypes;
+  }
+
+  public int getChildExpressionsSize() {
+    return childExpressionDataTypes.size();
+  }
+
+  public Type getDataType(int index) {
+    return childExpressionDataTypes.get(index);
+  }
+
+  public boolean hasSystemAttribute(String attributeKey) {
+    return systemAttributes.containsKey(attributeKey);
+  }
+
+  public Map<String, String> getSystemAttributes() {
+    return systemAttributes;
+  }


all public methods are visible to users, so we need to add java doc to explain the functionality.

JackieTien97 · 2024-11-29T09:09:34Z

iotdb-api/udf-api/src/main/java/org/apache/iotdb/udf/api/relational/ScalarFunction.java

+  void validate(FunctionParameters parameters) throws Exception;
+
+  /**
+   * This method is mainly used to initialize {@link ScalarFunction}. In this method, the user can


Suggested change

* This method is mainly used to initialize {@link ScalarFunction}. In this method, the user can

* This method is mainly used to initialize {@link ScalarFunction} and set the output data type. In this method, the user can

JackieTien97 · 2024-11-29T09:10:41Z

iotdb-api/udf-api/src/main/java/org/apache/iotdb/udf/api/relational/access/Record.java

+  /**
+   * Returns the object value at the specified column in this row.
+   *
+   * @param columnIndex index of the specified column
+   * @return the object value at the specified column in this row
+   */
+  Object getObject(int columnIndex);


Suggested change

/**

* Returns the object value at the specified column in this row.

*

* @param columnIndex index of the specified column

* @return the object value at the specified column in this row

*/

Object getObject(int columnIndex);

JackieTien97 · 2024-11-29T09:15:32Z

...main/java/org/apache/iotdb/db/queryengine/execution/relational/ColumnTransformerBuilder.java

+            UDFDataTypeTransformer.transformUDFDataTypeToReadType(config.getOutputDataType());
+        return new UserDefineScalarFunctionTransformer(
+            returnType, scalarFunction, childrenColumnTransformer);
+      }
    }
    throw new IllegalArgumentException(String.format("Unknown function: %s", functionName));


add current DatanodeId in error msg. You can get that from IoTDBDescriptor.getInstance().getConfig().getDataNodeId()

JackieTien97 · 2024-11-29T10:24:35Z

...n/java/org/apache/iotdb/db/queryengine/plan/execution/config/metadata/ShowFunctionsTask.java

+      } else {
+        if (TableUDFUtils.tryGetScalarFunction(udfInformation.getFunctionName()) != null) {
+          return BINARY_MAP.get(FUNCTION_TYPE_USER_DEFINED_SCALAR_FUNC);
+        } else if (TableUDFUtils.tryGetAggregateFunction(udfInformation.getFunctionName())
+            != null) {
+          return BINARY_MAP.get(FUNCTION_TYPE_USER_DEFINED_AGG_FUNC);
+        } else if (TableUDFUtils.tryGetTableFunction(udfInformation.getFunctionName()) != null) {
+          return BINARY_MAP.get(FUNCTION_TYPE_USER_DEFINED_TABLE_FUNC);
+        }


Each show functions, we need to reflect a UDF class to judge what type it's....
Why not store that in UDFInformation

JackieTien97 · 2024-11-29T10:30:24Z

...rc/main/java/org/apache/iotdb/db/queryengine/plan/relational/metadata/TableMetadataImpl.java

@@ -627,7 +630,26 @@ && isIntegerNumber(argumentTypes.get(2)))) {
        // ignore
    }

-    // TODO scalar UDF function
+    // User-defined scalar function
+    ScalarFunction scalarFunction = TableUDFUtils.tryGetScalarFunction(functionName);


we may add new method instead of calling UDFManagementService.reflect which will print warn log in that.

JackieTien97 · 2024-11-29T10:32:10Z

...rc/main/java/org/apache/iotdb/commons/udf/builtin/relational/TableBuiltinScalarFunction.java

@@ -70,4 +75,14 @@ public enum TableBuiltinScalarFunction {
  public String getFunctionName() {
    return functionName;
  }
+
+  private static final Set<String> NATIVE_FUNCTION_NAMES =


Suggested change

private static final Set<String> NATIVE_FUNCTION_NAMES =

private static final Set<String> BUILT_IN_SCALAR_FUNCTION_NAMES =

JackieTien97 · 2024-11-29T10:32:24Z

...rc/main/java/org/apache/iotdb/commons/udf/builtin/relational/TableBuiltinScalarFunction.java

+              .map(TableBuiltinScalarFunction::getFunctionName)
+              .collect(Collectors.toList()));
+
+  public static Set<String> getNativeFunctionNames() {


Suggested change

public static Set<String> getNativeFunctionNames() {

public static Set<String> getBuiltInScalarFunctionNames() {

JackieTien97 · 2024-11-29T10:34:29Z

iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/udf/utils/TableUDFUtils.java

+public class TableUDFUtils {
+  public static ScalarFunction tryGetScalarFunction(String functionName) {
+    try {
+      return UDFManagementService.getInstance().reflect(functionName, ScalarFunction.class);


may also need to add a method in UDFManagementService, like tryReflect to avoid print warn logs

Cpaulyz added 25 commits November 19, 2024 15:03

refactor

ccb57c8

refactor

ea028ef

save

90d34e1

save

0d971bc

rollback scalar function definition

4490c55

add license

e1dc67e

add license

9983999

fix it

93ad64b

Merge branch 'table_udsf' into udsf

6a68d33

save

499ae65

spotless

007a2d8

save

b8a4fb6

fix builtin

f55f48d

save

7321535

fix it

87dc7ea

fix it

fa3c683

fix it

c215e63

fix it

bc0873d

fix it

656dc4c

fix review

dbe07ca

split drop function plan

f30dda4

fix clear throw NPE

54b24c2

Merge branch 'table_udsf' into udsf

fbe6fde

merge master

0ed1d85

revert useless change

b1912aa

Cpaulyz added 3 commits November 27, 2024 15:51

Save

1e2ce48

add IT

e8ea11c

add license

b8d24a2

Cpaulyz added 3 commits November 28, 2024 15:49

rename beforeStart and add getLocalDate

ebcd1b9

Merge branch 'master' into udsf

5797329

modify getdatatype

30be7e5

JackieTien97 requested changes Nov 29, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TABLE MODEL] Implement CREATE/SHOW/DROP Function and user-defined scalar function #14223

[TABLE MODEL] Implement CREATE/SHOW/DROP Function and user-defined scalar function #14223

Cpaulyz commented Nov 27, 2024 •

edited

Loading

codecov bot commented Nov 27, 2024 •

edited

Loading

lancelly commented Nov 27, 2024

sonarcloud bot commented Nov 28, 2024

JackieTien97 Nov 29, 2024

JackieTien97 Nov 29, 2024

JackieTien97 Nov 29, 2024

JackieTien97 Nov 29, 2024

JackieTien97 Nov 29, 2024

JackieTien97 Nov 29, 2024

JackieTien97 Nov 29, 2024

JackieTien97 Nov 29, 2024

JackieTien97 Nov 29, 2024

JackieTien97 Nov 29, 2024


		public class ScalarFunctionConfig extends UDFConfigurations {

		public ScalarFunctionConfig setOutputDataType(Type outputDataType) {

	* This method is mainly used to initialize {@link ScalarFunction}. In this method, the user can
	* This method is mainly used to initialize {@link ScalarFunction} and set the output data type. In this method, the user can

	private static final Set<String> NATIVE_FUNCTION_NAMES =
	private static final Set<String> BUILT_IN_SCALAR_FUNCTION_NAMES =

	public static Set<String> getNativeFunctionNames() {
	public static Set<String> getBuiltInScalarFunctionNames() {

[TABLE MODEL] Implement CREATE/SHOW/DROP Function and user-defined scalar function #14223

Are you sure you want to change the base?

[TABLE MODEL] Implement CREATE/SHOW/DROP Function and user-defined scalar function #14223

Conversation

Cpaulyz commented Nov 27, 2024 • edited Loading

Description

New Scalar Functions:

Integration Tests:

codecov bot commented Nov 27, 2024 • edited Loading

Codecov Report

lancelly commented Nov 27, 2024

sonarcloud bot commented Nov 28, 2024

Quality Gate failed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Cpaulyz commented Nov 27, 2024 •

edited

Loading

codecov bot commented Nov 27, 2024 •

edited

Loading