Implement LivyOperatorAsync
#178
Conversation
Codecov Report
@@ Coverage Diff @@
## main #178 +/- ##
==========================================
+ Coverage 95.76% 95.97% +0.21%
==========================================
Files 39 42 +3
Lines 2101 2336 +235
==========================================
+ Hits 2012 2242 +230
- Misses 89 94 +5
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Can you update the PR title as per http://chris.beams.io/posts/git-commit/ please
sunank200
requested review from
phanikumv,
bharanidharan14 and
rajaths010494
as code owners
Co-authored-by: Kaxil Naik <[email protected]>
…stronomer-providers into livy-operator-async
Co-authored-by: Kaxil Naik <[email protected]>
…stronomer-providers into livy-operator-async
|
|
||
| LIVY_JAVA_FILE = os.environ.get("LIVY_JAVA_FILE", "/spark-examples.jar") | ||
| LIVY_PYTHON_FILE = os.environ.get("LIVY_PYTHON_FILE", "/user/hadoop/pi.py") | ||
|
|
There was a problem hiding this comment.
This DAG still needs work to be self sufficient. We might want to use existing operators and boto3 api to
- spin up the EMR cluster along with Livy, and Spark
- create the SSH tunnel
- execute job using this async livy operator
- terminate EMR cluster
There was a problem hiding this comment.
Yes, I am working on this right now.
| See Tenacity documentation at https://github.com/jd/tenacity | ||
| """ | ||
|
|
||
| def execute(self, context: "Context") -> Any: |
There was a problem hiding this comment.
This should have been None, we dont return anything from this method
Implements: #119