[Bug]: the csv resolving bug on CSVToBigQuery template

[OpensourceHU]

Related Template(s)

CSVToBigQuery

Template Version

2024-07-16-00_rc00

What happened?

the csv file resolving will encounter error: "Number of fields in the schema and number of Csv headers do not match."
when csv file fieds has comma in text, for example ,if a we have two field ,field2 is a json string
{field1},{field2}
field1Text,"{""key1"":"value1"",""key2"":"value2""}"
the spiliter will split it to 3 column , which number is not match with the csv header and bq schema, and this row will transform failed.
the problem probably in line 199 of CSVToBigQuery.java
Splitter.on(delimiter.get()).splitToList(context.element()).toArray(new String[0]);
please consider use csv utils package to fix this bad case

Relevant log output

Number of fields in the schema and number of Csv headers do not match.