[LIVY-971] Support to get session variables when using JDBC to connect to Livy thrift server.

[jianzhenwu]

… to Livy thrift server.

What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)
Add DescLivySessionOperation to support fetching sessionId, applicationId, sessionState, sessionLogs from Livy thrift service.
Solve the bug that the thrift api gets stuck when creating an interactive session fails.

(Include a link to the associated JIRA and make sure to add a link to this pr on the JIRA as well)
https://issues.apache.org/jira/browse/LIVY-971

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
integration tests

Please review https://livy.incubator.apache.org/community/ before opening a pull request.

[mgaido91]

I think this information is available on the Livy web application. I do not know any SQL engine that allows for a similar command, and we try to follow SQL standard. I think we should use the web app for monitoring the status, so if there is something missing there, I would recommend improving that rather than adding a new (unusual) SQL command.

[jianzhenwu]

hi @mgaido91 thanks for your response. I think this feature has the following two usage scenarios.

1. When other platforms use JDBC to integrate with Livy, they need to save the session ID and application ID.
2. When the connection creation fails, it may be that the spark session failed to start. At this time, they need to check the reason why the spark session failed to start.
   I think it is convenient for users to use one protocol to meet the needs. What do you think?

[mgaido91]

I think in general also in Spark the SQL interface is kept as conform as possible to the ANSI SQL standard and monitoring information is provided through the web API. If there are similar commands in other SQL engines, we can maybe do something similar. But I'd avoid creating something specific to Livy that nobody expects so nobody uses.

[jianzhenwu]

I agree with your points. Thank you for sharing your insights.