Issues with older AMT PCs and TLS connections on Ubuntu 24.04

[marek26340]

Hello!

So, I've decided to try and set up MeshCentral in our environment of approx. 150 PCs. All of them are on Windows and some of them also support either Intel SM or AMT.
Unfortunately, I seem to be having quite a lot of issues with these older PCs, which only support older versions of TLS and thus I'm getting lots of errors.
When MC is trying to set a machine's AMT up, it sets up TLS as it should, waits the 10 seconds, then fails upon trying to connect (408) with TLS and falls back to an insecure connections, which succeeds (200). Then, when I'd check the config it applied using MeshCommander on my PC, I can see that MC did not set up CIRA and, for some reason, didn't even install it's trusted root cert (only the private key).

Another error that I'm getting in trace is: Error: 134820972243968:error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol:../deps/openssl/openssl/ssl/statem/statem_lib.c:1994:
After tons of googling, I tried to set up a separate OpenSSL config file (MinProtocol = TLSv1 and CipherString = DEFAULT:@SECLEVEL=1) and set it as an environment variable. That atleast let me past the Timeout error, but the error message itself still did not disappear. If MC is not using the system's OpenSSL and is using an internal one instead, that could explain why my changes haven't had much effect.
I also tried to add --tls-min-v1.0 to the launch parameters, with no changes.
I'm convinced this has to do with OpenSSL and TLS because when I ran OPENSSL_CONF=/etc/ssl/openssl-tls1.cnf openssl s_client -connect B106-01:16993, this was the output:

CONNECTED(00000003)
4037A930647C0000:error:0A000152:SSL routines:final_renegotiate:unsafe legacy renegotiation disabled:../ssl/statem/extensions.c:893:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 51 bytes and written 304 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.1
    Cipher    : 0000
    Session-ID: 63180000
    Session-ID-ctx:
    Master-Key:
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1732781739
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---

The "SSL handshake" part did not appear before I included that config file I made.
I've been struggling with this problem for a couple of days by now. Any advice will be greatly appreciated. Cheers!

Server Software:

• OS: Ubuntu Server 24.04.1
• Virtualization: bare-metal
• Network: LAN/WAN hybrid
• Version: 1.1.33 (also tried 1.1.34)
• Node: v16.20.2 (downgraded from v18 because I've read that v18 does not support the old ciphers at all?)

Remote Device:

• Device: HP EliteDesk 800 G1 SFF (as an example - it's happening on most, if not all of our machines)
• OS: Windows 10 22H2
• Network: Local to Meshcentral, planning for remote over WAN too
• Current Core Version: Dec 9 2022, 679808363

Your config.json file

{
  "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
  "__comment1__": "This is a simple configuration file, all values and sections that start with underscore (_) are i>
  "__comment2__": "See node_modules/meshcentral/sample-config-advanced.json for a more advanced example.",
  "settings": {
    "cert": "mcserver.example.cz",
    "_WANonly": true,
    "_LANonly": true,
    "_sessionKey": "MyReallySecretPassword1",
    "port": 443,
    "_aliasPort": 443,
    "redirPort": 80,
    "_redirAliasPort": 80,
    "agentPort": 4434,
    "mpsPort": 4433,
    "SelfUpdate": true,
    "exactPorts": true,
    "agentsInRam": true
  },
  "domains": {
    "": {
      "_title": "MyServer",
      "_title2": "Servername",
      "_minify": true,
      "_newAccounts": true,
      "_userNameIsEmail": true
    }
  },
  "_letsencrypt": {
    "__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before trying Let's Encrypt.",
    "email": "[email protected]",
    "names": "myserver.mydomain.com",
    "skipChallengeVerification": true,
    "production": false
  }
}

[si458]

you should really use a newer node version not roll back a version due to security risks!
you also need to verify the TLS/SSL that the OS system supports as NodeJS will use that
HOWEVER
the was 2 new options added months ago #5915
these let u customise what TLS/SSL ciphers you wanted to use which might help

"useNodeDefaultTLSCiphers": {
  "type": "boolean",
  "default": false,
  "description": "When true, get the default TLS ciphers from the node process, rather than using the recommended suites set up by meshcentral"
},
"tlsCiphers": {
  "type": [
    "string",
    "array"
  ],
  "default": null,
  "description": "Allows user to override the TLS ciphers used by meshcentral by default. If a string, should be a ':' separated list of ciphers to accept. If an array, should be an array of strings representing the ciphers to accept."
},

also we already include @SECLEVEL=0 in the TLS so it should allow TLS 1.0

MeshCentral/mpsserver.js

Line 51 in c16ff89

obj.server = tls.createServer({ key: certificates.mps.key, cert: certificates.mps.cert, minVersion: 'TLSv1', requestCert: true, rejectUnauthorized: false, ciphers: "HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!SRP:!CAMELLIA:@SECLEVEL=0", secureOptions: constants.SSL_OP_NO_SSLv2 | constants.SSL_OP_NO_SSLv3 | constants.SSL_OP_NO_COMPRESSION }, onConnection)

validated using https://testtls.com - use ur meshcentral dns name and port 4433 my.domain.com:4433

[marek26340]

Hi, thank you very much for your reply.

I rolled back the NodeJS version just as a test. I've rolled all of my changes back so that we can diagnose it as it should be used - no modifications to NodeJS (now using stock 18.19.1 which comes from Ubuntu Server 24.04 using apt) and OpenSSL (I deleted the modified config and regenerated it by reinstalling OpenSSL).

With my changes, I was able to connect to AMT with the only issue being the ssl_choose_client_version:unsupported protocol error message coming up every time I try to connect.
Now, with all changes reverted, this is what appears in the trace when I try to connect to the same HP EliteDesk as I mentioned above:
TCP relay error from 192.168.12.150:16993: Error: 00083B2689780000:error:0A000102:SSL routines:ssl_choose_client_version:unsupported protocol:../ssl/statem/statem_lib.c:1953:
TCP relay error from 192.168.12.150:16993: Error: 00083B2689780000:error:0A0000BF:SSL routines:tls_setup_handshake:no protocols available:../ssl/statem/statem_lib.c:104:
...and it never connects. It throws a timeout error to be exact.

Here's what I see in the tracing after I uninstalled the agent, unconfigured and re-setup ME (manually using MEBx) and reinstalled the agent.

10:34:49,"amt","A102-01,   Done."
10:34:48,"amt","A102-01,   Intel AMT connected."
10:34:48,"amt","A102-01,   Initial Contact Response, 200"
10:34:47,"amt","A102-01,   Direct-Connect, NoTLS, 192.168.12.150, admin"
10:34:47,"amt","A102-01,   Attempt Initial Local Contact, 3, 192.168.12.150"
10:34:47,"amt","A102-01,   Attempt Initial Contact, Local"
10:34:47,"amt","A102-01,   Initial Contact Response, 408"
10:34:45,"amt","A102-01,   Direct-Connect, TLS, 192.168.12.150, admin"
10:34:45,"amt","A102-01,   Attempt Initial Local Contact, 3, 192.168.12.150"
10:34:45,"amt","A102-01,   Attempt Initial Contact, Local"
10:34:35,"amt","A102-01,   Enabled TLS, holding 10 seconds..."
10:34:35,"amt","A102-01,   Performing Commit..."
10:34:34,"amt","A102-01,   Intel AMT connected."
10:34:34,"amt","A102-01,   Initial Contact Response, 200"
10:34:33,"amt","A102-01,   Direct-Connect, NoTLS, 192.168.12.150, admin"
10:34:33,"amt","A102-01,   Attempt Initial Local Contact, 3, 192.168.12.150"
10:34:33,"amt","A102-01,   Attempt Initial Contact, Local"
10:34:33,"amt","A102-01,   Initial Contact Response, 408"
10:34:31,"amt","A102-01,   Direct-Connect, TLS, 192.168.12.150, admin"
10:34:31,"amt","A102-01,   Attempt Initial Local Contact, 3, 192.168.12.150"
10:34:31,"amt","A102-01,   Attempt Initial Contact, Local"
10:34:31,"amt","A102-01,   Checking Intel AMT state..."
10:34:31,"amt","Start   Management,   node//dw2ljvrqkpsISG8yPPv7UoAlCgCKDeITyOAMh8MLWH3WF$3Mw5hx$P07azupyNLa,   3"
10:34:04,"agent","Verified   agent connection to   dw2ljvrqkpsISG8yPPv7UoAlCgCKDeITyOAMh8MLWH3WF$3Mw5hx$P07azupyNLa   (192.168.12.150:64386)."
10:34:01,"agent","New   agent at 192.168.12.150:64386"

Checking with MeshCommander, I can see that CIRA did not get set up and I can only see the private key in the security settings tab. TLS got enabled, but MC can't connect to it.

My only lead is pretty much just the fact that Ubuntu 24.04 might've tightened it's security by forcibly disabling older TLS versions..
And I'm sorry, but I'm not sure what I'm supposed to put in the "tlsCiphers" setting. The documentation isn't very clear on this setting. And I know there are many different ciphers to choose from - I have no idea which I do need, which I don't need and which I really shouldn't put in there.

Please advise. I really don't know how to proceed further here. Thank you.

[si458]

Where are you seeing the error ssl_choose_client_version:unsupported protocol ?
I didn't think u could debug amt on devices?

[marek26340]

It's right in the MeshCentral server tracing. I enabled main, webrelay, mps, mpscmd, agentupdate, agent, cert, db, amt and these messages showed up. These specifically are appearing from the "webrelay" tracing.

[si458]

never seen those errors before? thats new to me?
can you share logs at all from the tracing? (you can click the download button top right of the screen)
also doesnt help without AMT devices to hand i cant really debug things :(
i have an old amt 8 machine and its working fine on ubuntu 22.04 with node 20
BUT im using CIRA rather than AMT directly,
so might need to do some more testing possibly

[marek26340]

I could send you the whole debug log (--debug) if you want. There could be sensitive info in there, so what if I'll send you a link via Discord or email?

[si458]

@marek26340 email me if ur unsure (check my github page 👍 )

[marek26340]

Update: Just downgraded to Ubuntu Server 22.04. After importing the backup into MC and checking the debug logs while attempting to connect to different machines, it seems like the errors:
TCP relay error from 192.168.12.150:16993: Error: 00083B2689780000:error:0A000102:SSL routines:ssl_choose_client_version:unsupported protocol:../ssl/statem/statem_lib.c:1953:
TCP relay error from 192.168.12.150:16993: Error: 00083B2689780000:error:0A0000BF:SSL routines:tls_setup_handshake:no protocols available:../ssl/statem/statem_lib.c:104:
now occur like once or twice, then it proceeds with connecting to AMT just fine. Unfortunately, it still didn't help with the TLS -> 408 -> NoTLS -> 200 issue. And CIRA is still not getting set up automatically.

This pretty much confirms that Ubuntu 24.04 forcefully disabled TLS 1.0 somehow.
What "SECLEVEL" is MeshCentral setting? Level 1 or 0? I've read that as part of the deprecation process, they moved TLS 1.0 and 1.1 all the way down to SECLEVEL 0.

Going to get some sleep now.