Possible problem with v2.6.1 with RHEL8 in FIPS mode and using Java 17

[Steve973]

I have followed all of the instructions, and I have looked through a lot of results on google, but there is an error that I am getting that seems like it should not be happening. Here are the details:

I am trying to run gradle (even gradle init) on an EC2 instance where I have enabled FIPS mode, and configured SoftHSM 2 as my PKCS#11 provider. This is RHEL 8.9 and SoftHSM is 2.6.1 (from EPEL).

Whenever I run gradle, I always get (abbreviated stack trace):

Caused by: java.security.ProviderException: FIPS: error during the Token login required for the MessageDigest service.
        at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11$P11Service.newInstance(SunPKCS11.java:1489)
        at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
        at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
        at java.base/java.security.MessageDigest.getInstance(MessageDigest.java:185)
        at java.base/sun.security.provider.SecureRandom.init(SecureRandom.java:113)
        at java.base/sun.security.provider.SecureRandom.<init>(SecureRandom.java:86)
        at java.base/java.security.SecureRandom.getDefaultPRNG(SecureRandom.java:291)
        at java.base/java.security.SecureRandom.<init>(SecureRandom.java:225)
        at java.base/java.util.UUID$Holder.<clinit>(UUID.java:102)
        ... 32 more
Caused by: javax.security.auth.login.LoginException
        at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11.login(SunPKCS11.java:1779)
        at jdk.crypto.cryptoki/sun.security.pkcs11.Token.ensureLoggedIn(Token.java:244)
        at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11$P11Service.newInstance(SunPKCS11.java:1485)
        ... 40 more
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_ARGUMENTS_BAD
        at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11.C_Login(Native Method)
        at jdk.crypto.cryptoki/sun.security.pkcs11.SunPKCS11.login(SunPKCS11.java:1763)
        ... 42 more

My ~/.config/softhsm2/softhsm2.conf looks like this:

# SoftHSM v2 configuration file

directories.tokendir = /var/lib/softhsm/tokens/
objectstore.backend = file

# ERROR, WARNING, INFO, DEBUG
log.level = ERROR

# If CKF_REMOVABLE_DEVICE flag should be set
slots.removable = false

# Enable and disable PKCS#11 mechanisms using slots.mechanisms.
slots.mechanisms = ALL

# If the library should reset the state on fork
library.reset_on_fork = false

My Java 17 security file has these for the FIPS providers:

fips.provider.1=SUN
fips.provider.2=SunEC
fips.provider.3=SunJSSE
fips.provider.4=SunJCE
fips.provider.5=SunRsaSign
fips.provider.6=XMLDSig
fips.provider.7=SunPKCS11 ${user.home}/.pkcs11/pkcs11.cfg
fips.provider.8=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider

The PKCS11 config file (referenced in the fips provider directly above) looks like this:

name = SoftHSM
library = /usr/lib64/libsofthsm2.so
slotListIndex = 0
showInfo = true

In my ~/.gradle/gradle.properties, I have tried using system properties:

org.gradle.daemon=false
org.gradle.jvmargs=-Dfile.encoding=UTF-8 \
  -Djava.security.debug=all \
  -Djavax.net.ssl.keyStore=NONE \
  -Djavax.net.ssl.keyStoreType=PKCS11 \
  -Djavax.net.ssl.keyStorePassword=XXXXXXXXXXXXXXXX \
  -Djavax.net.ssl.keyStoreProvider=SunPKCS11-SoftHSM \
  -Djavax.net.ssl.trustStoreType=PKCS11 \
  -Djavax.net.ssl.trustStorePassword=XXXXXXXXXXXXXXXX \
  -Djavax.net.ssl.trustStoreProvider=SunPKCS11-SoftHSM

I have tried including these args in the JAVA_OPTS, GRADLE_OPTS, and DEFAULT_JVM_ARGS environment variables, and I have also even commented out the jvmargs in gradle.properties. In all cases, I keep getting this same error. You may have noticed showInfo = true above, and it always shows that it is accessing PKCS11:

Library info:
  cryptokiVersion: 2.40
  manufacturerID: SoftHSM                         
  flags: 0
  libraryDescription: Implementation of PKCS11        
  libraryVersion: 2.06
All slots: 2009819478, 1
Slots with tokens: 2009819478, 1
Slot info for slot 2009819478:
  slotDescription: SoftHSM slot ID 0x77cb6956                                      
  manufacturerID: SoftHSM project                 
  flags: CKF_TOKEN_PRESENT
  hardwareVersion: 2.06
  firmwareVersion: 2.06
Token info for token in slot 2009819478:
  label: mytoken1                        
  manufacturerID: SoftHSM project                 
  model: SoftHSM v2      
  serialNumber: 0c6dc663f7cb6956
  flags: CKF_RNG | CKF_LOGIN_REQUIRED | CKF_USER_PIN_INITIALIZED | CKF_RESTORE_KEY_NOT_NEEDED | CKF_TOKEN_INITIALIZED
  ulMaxSessionCount: CK_EFFECTIVELY_INFINITE
  ulSessionCount: CK_UNAVAILABLE_INFORMATION
  ulMaxRwSessionCount: CK_EFFECTIVELY_INFINITE
  ulRwSessionCount: CK_UNAVAILABLE_INFORMATION
  ulMaxPinLen: 255
  ulMinPinLen: 4
  ulTotalPublicMemory: CK_UNAVAILABLE_INFORMATION
  ulFreePublicMemory: CK_UNAVAILABLE_INFORMATION
  ulTotalPrivateMemory: CK_UNAVAILABLE_INFORMATION
  ulFreePrivateMemory: CK_UNAVAILABLE_INFORMATION
  hardwareVersion: 2.06
  firmwareVersion: 2.06
  utcTime: 2024012617425000
...  Followed by a bunch of mechanism details  ...

Does this seem like a bug? I have checked, double checked, triple-checked, and probably many more iterations of that, though I cannot find the problem. Can I provide any further information if this is worth looking into as a bug?

[Steve973]

I have some more information, in case it helps, and I pushed on a bit further. It turns out that it requires the pin to either be set in java.security, or as a system property on the command line. So, adding -Dfips.nssdb.pin=pin:XXXXXXXXXX resolves the bad arguments error message, but it ends up producing this:

Caused by: java.security.ProviderException: update() failed
        at jdk.crypto.cryptoki/sun.security.pkcs11.P11Digest.engineUpdate(P11Digest.java:242)
        at java.base/java.security.MessageDigest$Delegate.engineUpdate(MessageDigest.java:658)
        at java.base/java.security.MessageDigest.update(MessageDigest.java:349)
        at org.gradle.internal.hash.Hashing$MessageDigestHasher.update(Hashing.java:302)
        at org.gradle.internal.hash.Hashing$MessageDigestHasher.putInt(Hashing.java:318)
        at org.gradle.internal.hash.Hashing$DefaultHasher.putString(Hashing.java:413)
        at org.gradle.internal.hash.Hashing.signature(Hashing.java:78)
        at org.gradle.api.internal.changedetection.state.DefaultResourceSnapshotterCacheService.<clinit>(DefaultResourceSnapshotterCacheService.java:33)
        ... 107 more
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_GENERAL_ERROR
        at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11.C_DigestInit(Native Method)
        at jdk.crypto.cryptoki/sun.security.pkcs11.P11Digest.engineUpdate(P11Digest.java:224)
        ... 114 more

when I run gradle init in an empty directory. I have been searching a lot for information on this, but I cannot seem to find any. So, why is nssdb involved at all when using SoftHSM? I have to assume that plenty of people are using SoftHSM2 on RHEL8 with Java 17, and some of them must be using it in FIPS mode. Any ideas, or can I provide even more information?

[antoinelochet]

We run SoftHSM on Java 17 but not for Gradle.
What are you trying to achieve ?

[Steve973]

We run SoftHSM on Java 17 but not for Gradle. What are you trying to achieve ?

My team and I work on an enclave where we have future requirements of being FIPS compliant, and using something like SoftHSM for all of our X509 certificate accesses would help a lot.

[antoinelochet]

I don't think that SoftHSM has been FIPS certified. Contrary to OpenSSL which has a FIPS module.

Anyway, I will try reproducing your issue.
Does Gradle work with P11 but without the FIPS mode ?