HCD-481 : Adding reconnection logic added for HDFS2 sink connector #531
Conversation
|
It looks like @SanchayGupta1197 hasn't signed our Contributor License Agreement, yet.
You can read and sign our full Contributor License Agreement here. Once you've signed reply with Appreciation of efforts, clabot |
| conf.getHadoopConfiguration() | ||
| ); | ||
| fileSystem.delete(path, true); | ||
| } catch (IOException ex) { |
There was a problem hiding this comment.
Can we log this error message as well?
There was a problem hiding this comment.
Have added the exception to the log below.
| URI.create(conf.getUrl()), | ||
| conf.getHadoopConfiguration() | ||
| ); | ||
| fileSystem.delete(path, true); |
There was a problem hiding this comment.
Not too familiar with HDFS. Why are we deleting the file for ORC here? Why is schema=null not enough like with AVRO
There was a problem hiding this comment.
The current version of org.apache.hadoop.hive.ql.io.orc.OrcFile.WriterOptions does not supports file overwriting therefore deleting tmp file wheneverAlreadyBeingCreatedException is thrown.
|
|
Problem
The connector was failing for
AvroandORCformat when the connection to the cluster was interrupted.Solution
Fix for Avro
nullin order to invalidate the current temp file.AvroRuntimeExceptionwas caught and thrown as aConnectException.Fix for ORC
nullin order to invalidate the current temp file.AlreadyBeingCreatedExceptionwas handled by deleting the temp file from the cluster.Test Strategy
Testing done:
Issues/Limitation
connect-runtimeand HDFSminicluster.