Skip to content

Commit

Permalink
[SPARK-42328][SQL] Remove _LEGACY_ERROR_TEMP_1175 from error classes
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

Only occurrence of `_LEGACY_ERROR_TEMP_1175` appears under conversion from Spark data types to Parquet. All supported documented [Spark data types](https://spark.apache.org/docs/latest/sql-ref-datatypes.html) are covered in the [conversion function](https://github.com/apache/spark/blob/3e0808c33f185c13808ce2d547ce9ba0057d31a6/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L517-L745) (`VarcharType` and `CharType` are not present but they are passed down as string before reaching the conversion function), so under normal circumstances user can't force this error.

Convert the error class to `INTERNAL_ERROR`.

### Why are the changes needed?

Remove legacy error classes as part of activity in [SPARK-37935](https://issues.apache.org/jira/browse/SPARK-37935).

### Does this PR introduce _any_ user-facing change?

If the Spark works correctly, user shouldn't be able to run into `INTERNAL_ERROR` by using the public API.

### How was this patch tested?

Added test to `QueryCompilationErrorsSuite` and tested with sbt:
```
project sql
testOnly *QueryCompilationErrorsSuite
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#45183 from nikolamand-db/nikolamand-db/SPARK-42328.

Authored-by: Nikola Mandic <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
  • Loading branch information
nikolamand-db authored and MaxGekk committed Feb 22, 2024
1 parent df4d489 commit 78f7c30
Show file tree
Hide file tree
Showing 3 changed files with 22 additions and 7 deletions.
5 changes: 0 additions & 5 deletions common/utils/src/main/resources/error/error-classes.json
Original file line number Diff line number Diff line change
Expand Up @@ -5118,11 +5118,6 @@
"Unrecognized Parquet type: <field>."
]
},
"_LEGACY_ERROR_TEMP_1175" : {
"message" : [
"Unsupported data type <dataType>."
]
},
"_LEGACY_ERROR_TEMP_1181" : {
"message" : [
"Stream-stream join without equality predicate is not supported."
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1908,8 +1908,9 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat

def cannotConvertDataTypeToParquetTypeError(field: StructField): Throwable = {
new AnalysisException(
errorClass = "_LEGACY_ERROR_TEMP_1175",
messageParameters = Map("dataType" -> field.dataType.catalogString))
errorClass = "INTERNAL_ERROR",
messageParameters = Map("message" ->
s"Cannot convert Spark data type ${toSQLType(field.dataType)} to any Parquet type."))
}

def incompatibleViewSchemaChangeError(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ import org.apache.spark.sql._
import org.apache.spark.sql.api.java.{UDF1, UDF2, UDF23Test}
import org.apache.spark.sql.catalyst.expressions.{Coalesce, Literal, UnsafeRow}
import org.apache.spark.sql.catalyst.parser.ParseException
import org.apache.spark.sql.execution.datasources.parquet.SparkToParquetSchemaConverter
import org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog
import org.apache.spark.sql.expressions.SparkUserDefinedFunction
import org.apache.spark.sql.functions._
Expand Down Expand Up @@ -962,6 +963,24 @@ class QueryCompilationErrorsSuite
"methodName" -> "update",
"className" -> "org.apache.spark.sql.catalyst.expressions.UnsafeRow"))
}

test("INTERNAL_ERROR: Convert unsupported data type from Spark to Parquet") {
val converter = new SparkToParquetSchemaConverter
val dummyDataType = new DataType {
override def defaultSize: Int = 0

override def simpleString: String = "Dummy"

override private[spark] def asNullable = NullType
}
checkError(
exception = intercept[AnalysisException] {
converter.convertField(StructField("test", dummyDataType))
},
errorClass = "INTERNAL_ERROR",
parameters = Map("message" -> "Cannot convert Spark data type \"DUMMY\" to any Parquet type.")
)
}
}

class MyCastToString extends SparkUserDefinedFunction(
Expand Down

0 comments on commit 78f7c30

Please sign in to comment.