Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unpin snakeyaml, add suppressions and licenses #15549

Merged
merged 4 commits into from
Dec 15, 2023

Conversation

janjwerner-confluent
Copy link
Contributor

@janjwerner-confluent janjwerner-confluent commented Dec 13, 2023

Description

This removes the pin of the Snakeyaml introduced in:
#14519
After the updates of io.kubernetes.java-client and io.confluent.kafka-clients, the only uses of the Snakeyaml 1.x are:

  • in test scope, transitive dependency of jackson-dataformat-yaml:jar:2.12.7
  • in compile scope in contrib extension druid-cassandra-storage
  • in compile scope in it-tests.

With the dependency version un-pinned, io.kubernetes.java-client and io.confluent.kafka-clients bring Snakeyaml versions 2.0 and 2.2, consequently allowing to build a Druid distribution without the contrib-extension and free of vulnerable Snakeyaml versions.

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@janjwerner-confluent
Copy link
Contributor Author

The long term solution is to migrate to an updated version of jackson, as updates to 2.12.x branch are unlikely except for critical issues.
see discussion at:
FasterXML/jackson-jaxrs-providers#177

@janjwerner-confluent janjwerner-confluent mentioned this pull request Dec 13, 2023
10 tasks
licenses.yaml Outdated
Comment on lines 2875 to 2886
---

name: org.yaml snakeyaml
license_category: binary
module: extensions/druid-kubernetes-extensions
license_name: Apache License version 2.0
version: 2.0
libraries:
- org.yaml: snakeyaml



Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need this section since you already updated the version in the section above?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed it to protobuf-extensions

(version 1.27)
The contrib extension: druid-cassandra-storage uses version 1.6 in compile
scope
The integration tests use version 1.27 in compile scope.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we know where 1.27 comes from in integration tests? It would be nice if we could upgrade integration tests to 2.x as well.

Copy link
Contributor Author

@janjwerner-confluent janjwerner-confluent Dec 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's transitive dependency of:
com.fasterxml.jackson.dataformat:jackson-dataformat-yaml 2.12.7
so unless we update jackson there, can't upgrade.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's update the version since we pin integration tests to 1.33 and not 1.27

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if this is more hairy, we can do the jackson upgrade as a follow-up

@xvrl
Copy link
Member

xvrl commented Dec 13, 2023

minor nit about version numbers in the comments, otherwise LGTM.

It also, looks intellij-inspections is failling with integration-tests-ex/cases/pom.xml:21 -- Failed to read artifact descriptor for org.yaml:snakeyaml:jar:1.21 #loc

@janjwerner-confluent
Copy link
Contributor Author

I believe I've fixed the versions. Not sure about the failing checks, except for the last one, they all seemed unrelated. gonna investigate a bit further.

@janjwerner-confluent
Copy link
Contributor Author

@xvrl
passed the re-runs.

<notes><![CDATA[
file name: snakeyaml-1.33.jar
file name: snakeyaml-1.27.jar snakeyaml-1.33.jar
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we shouldn't be regressing here and have everything at least on 1.33 like we did before.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just for completeness as snakeyaml is not packaged in the standard distribution
mvn clean install -Pdist -DskipTests
dependency-check-maven is disabled for the contrib extensions and it tests, this suppression stays in place if we re-enable checks on the additional modules)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please see:
#15447
#15026

@xvrl xvrl merged commit fa2c8ed into apache:master Dec 15, 2023
89 checks passed
Pankaj260100 pushed a commit to confluentinc/druid that referenced this pull request Dec 18, 2023
* unpin snakeyaml globally, add suppressions and licenses
* pin snakeyaml in the specific modules that require version 1.x, update licenses and owasp suppression

This removes the pin of the Snakeyaml introduced in:  apache#14519
After the updates of io.kubernetes.java-client and io.confluent.kafka-clients, the only uses of the Snakeyaml 1.x are:
- in test scope, transitive dependency of jackson-dataformat-yaml:jar:2.12.7
- in compile scope in contrib extension druid-cassandra-storage
- in compile scope in it-tests. 

With the dependency version un-pinned, io.kubernetes.java-client and io.confluent.kafka-clients bring Snakeyaml versions 2.0 and 2.2, consequently allowing to build a Druid distribution without the contrib-extension and free of vulnerable Snakeyaml versions.
Pankaj260100 pushed a commit to confluentinc/druid that referenced this pull request Dec 19, 2023
* unpin snakeyaml globally, add suppressions and licenses
* pin snakeyaml in the specific modules that require version 1.x, update licenses and owasp suppression

This removes the pin of the Snakeyaml introduced in:  apache#14519
After the updates of io.kubernetes.java-client and io.confluent.kafka-clients, the only uses of the Snakeyaml 1.x are:
- in test scope, transitive dependency of jackson-dataformat-yaml:jar:2.12.7
- in compile scope in contrib extension druid-cassandra-storage
- in compile scope in it-tests. 

With the dependency version un-pinned, io.kubernetes.java-client and io.confluent.kafka-clients bring Snakeyaml versions 2.0 and 2.2, consequently allowing to build a Druid distribution without the contrib-extension and free of vulnerable Snakeyaml versions.
Pankaj260100 pushed a commit to confluentinc/druid that referenced this pull request Dec 19, 2023
* unpin snakeyaml globally, add suppressions and licenses
* pin snakeyaml in the specific modules that require version 1.x, update licenses and owasp suppression

This removes the pin of the Snakeyaml introduced in:  apache#14519
After the updates of io.kubernetes.java-client and io.confluent.kafka-clients, the only uses of the Snakeyaml 1.x are:
- in test scope, transitive dependency of jackson-dataformat-yaml:jar:2.12.7
- in compile scope in contrib extension druid-cassandra-storage
- in compile scope in it-tests. 

With the dependency version un-pinned, io.kubernetes.java-client and io.confluent.kafka-clients bring Snakeyaml versions 2.0 and 2.2, consequently allowing to build a Druid distribution without the contrib-extension and free of vulnerable Snakeyaml versions.
@janjwerner-confluent janjwerner-confluent deleted the janjwerner-unpin-snakeyaml branch December 26, 2023 13:15
@LakshSingla LakshSingla added this to the 29.0.0 milestone Jan 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants