Intermittent WebRelay Failures in Peer MeshCentral Setup

[rui-alves]

Describe the bug
WebRelay connections are intermittently failing. Error messages such as “Bad AESGCM cookie due to exception: Error: Unsupported state or unable to authenticate data” appear in the application logs. These issues started after implementing peering.

To Reproduce
Steps to reproduce the behavior:

1. Go to the MeshCentral Dashboard.
2. Attempt to establish a remote session using WebRelay.
3. Wait for a connection.
4. Observe error messages in the logs and the failure to establish a consistent session.

Expected behavior
The WebRelay feature should provide a stable connection to remote agents without intermittent failures or errors related to cookie authentication.

Screenshots

Server Software (please complete the following information):

• OS: Ubuntu 24.04 LTS
• Virtualization: Docker
• Network: Load balancer in GCP, agent and web ports in separate backends
• Version: 1.1.33
• Node: v20.15.1

Client Device (please complete the following information):

• Device: Laptop
• OS: Windows 11 Pro
• Network: Remote over WAN
• Browser: Google Chrome

Remote Device (please complete the following information):

• Device: Mini Pc
• OS: MX Linux 21
• Network: Remote over WAN
• Current Core Version : Nov 21 2022, 3188018466

Additional context
The infrastructure includes three MeshCentral servers connected by peer connections. Each server is deployed with Docker Compose on separate instances, with MongoDB configured in a replica set.

config.json file

{
  "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
  "settings": {
    "mongoDb": "mongodb://meshcentral01.mongodb.exemple.local:27017,meshcentral02.mongodb.exemple.local:27017,meshcentral03.mongodb.exemple.local:27017/?replicaSet=rs0",
    "mongoDbChangeStream": true,
    "mongoDbBulkOperations": true,
    "MongoDbCol": "meshcentral",
    "cert": "mc.admin.exemple.com",
    "WANonly": false,
    "LANonly": false,
    "sessionKey": "MYSECRETKEY",
    "port": 8080,
    "aliasPort": 443,
    "redirPort": 0,
    "mpsPort": 0,
    "agentPong": 300,
    "agentPort": 80,
    "agentAliasPort": 443,
    "agentAliasDNS": "mc.exemple.com",
    "agentPortTls": false,
    "TLSOffload": true,
    "SelfUpdate": false,
    "AllowFraming": true,
    "WebRTC": false,
    "logs": "debug",
    "MaxInvalidLogin": {
      "time": 5,
      "count": 5,
      "coolofftime": 30
    },
    "relayDNS": [
      "mc.relay01.exemple.com",
      "mc.relay02.exemple.com",
      "mc.relay03.exemple.com",
      "mc.relay04.exemple.com"
    ],
    "plugins": {
      "enabled": false
    },
    "allowLoginToken": true
  },
  "domains": {
    "": {
      "title": "Exemple SA",
      "title2": "MC01",
      "_maxDeviceView": 2000,
      "minify": true,
      "NewAccounts": false,
      "localSessionRecording": false,
      "userNameIsEmail": false,
      "loginKey": "5040302010",
      "agentInviteCodes": false,
      "agentCustomization": {
        "displayname": "Exemple",
        "description": "Exemple SA Remote Agent",
        "companyName": "Exemple",
        "serviceName": "Exemple Remote"
      },
      "agenttag": {
        "ServerName": 1,
        "ServerDesc": 1,
        "ServerTags": 1
      },
      "certUrl": "https://mc.admin.exemple.com:443"
    }
  },
  "peers": {
    "serverId": "meshcentral01",
    "servers": {
      "meshcentral01": {
        "url": "ws://meshcentral01.mongodb.exemple.local:8080/"
      },
      "meshcentral02": {
        "url": "ws://meshcentral02.mongodb.exemple.local:8080/"
      },
      "meshcentral03": {
        "url": "ws://meshcentral03.mongodb.exemple.local:8080/"
      }
    }
  }
}

docker-compose.yml file

networks:
  meshcentral-tier:
    driver: bridge

services:
  mongodb:
    restart: always
    container_name: mongodb
    image: mongo:8.0
    hostname: meshcentral01.mongodb.exemple.local
    privileged: true
    command: ["--replSet", "rs0", "--bind_ip_all", "--port", "27017", "--maxConns", "50000"]
    ports:
      - 27017:27017
    environment:
      - GLIBC_TUNABLES=glibc.pthread.rseq=0
    volumes:
      # mongodb data-directory - A must for data persistence
      - ./meshcentral-mongodb-data:/data/db
      - ./meshcentral-mongodb-config:/data/configdb
      - ./meshcentral-mongodb-dump:/data/dump
    networks:
      - meshcentral-tier

  meshcentral01:
    restart: always
    container_name: meshcentral01
    # use the official meshcentral container
    image: ghcr.io/ylianst/meshcentral:1.1.33
    depends_on:
      - mongodb
    ports:
      # MeshCentral will moan and try everything not to use port 80, but you can also use it if you so desire, just change the config.json according to your needs
      - 80:80
      - 8080:8080
    env_file:
      - .env
    environment:
      - CONFIG_FILE=../meshcentral-config/meshcentral.config.json
    volumes:
      - ./config:/opt/meshcentral/meshcentral-config
      # config.json and other important files live here. A must for data persistence
      - ./meshcentral/meshcentral-data:/opt/meshcentral/meshcentral-data
      # where file uploads for users live
      - ./meshcentral/meshcentral-files:/opt/meshcentral/meshcentral-files
      # location for the meshcentral-backups - this should be mounted to an external storage
      - ./meshcentral/meshcentral-backups:/opt/meshcentral/meshcentral-backups
      # location for site customization files
      - ./meshcentral/meshcentral-web:/opt/meshcentral/meshcentral-web
    networks:
      - meshcentral-tier

Some Line logs

meshcentral01  | WEB: webRelaySetup
meshcentral01  | COOKIE: Decoded AESGCM cookie: {"ruserid":"user//some.one","x":"Gk53U7bY","time":1731167490000,"dtime":85368}
meshcentral01  | WEBRELAY: CreateWebRelaySession, userid:user//some.one, addr:127.0.0.1, port:80
meshcentral01  | WEBRELAY: handleRequest, url:/
meshcentral01  | WEBRELAY: launchNewTunnel
meshcentral01  | RELAY: TCP: Request for web relay
meshcentral01  | COOKIE: Encoded AESGCM cookie: {"userid":"user//some.one","domainid":"","nodeid":"node//0l4@bZ@cmBgyT$xU3uwJW2ooo527o27lMhSbpa2PDGlh@NlJzDUCSVSgAD1TKozY","tcpport":80,"tcpaddr":"127.0.0.1","time":1731167575}
meshcentral01  | RELAY: TCP: Connection websocket to ws://localhost:8080/meshrelay.ashx?p=14&auth=UeMitJU7xP6rjwKMVV3Gt36olhXDj9ZMbNK8fiKw$Tfr1iSUX$ckmDo7ANLxKv2ZsX53MoppX4ecN0GarwSDYn2DoW8MYYsoiVhVR2T62t@BeEr@xvPTon7ULDCRYQGL78nE8DoXfoxJjgywUoztdgQa5Vu6iZgLzAnf8ekw3$0XKu5$jX04gDcmyT@gkTu1cphGmmyV0APgcrj@pwKDeJTP3kMX34zn9MPj9VcAfnjHCaS0I$vJkEzDXXMp$WhytpAnBAs$CI5mxXHDrQ==
meshcentral01  | COOKIE: Decoded AESGCM cookie: {"userid":"user//some.one","domainid":"","nodeid":"node//0l4@bZ@cmBgyT$xU3uwJW2ooo527o27lMhSbpa2PDGlh@NlJzDUCSVSgAD1TKozY","tcpport":80,"tcpaddr":"127.0.0.1","time":1731167575000,"dtime":535}
meshcentral01  | RELAY: TCP: Relay websocket open
meshcentral01  | COOKIE: Encoded AESGCM cookie: {"ruserid":"user//some.one","time":1731167575}
meshcentral01  | RELAY: Relay: Sending agent tunnel command: {"nodeid":"node//0l4@bZ@cmBgyT$xU3uwJW2ooo527o27lMhSbpa2PDGlh@NlJzDUCSVSgAD1TKozY","action":"msg","type":"tunnel","value":"*/meshrelay.ashx?id=5pk7f$HZD9KZ&rauth=6ed2NC$oaajm5p@KP8o9i4ANeb6m9eZEev1KE7XTdM4@SsFiKTKH08coK5Z$46HgqIPjlr2lJazdw9raEse2mMzuhmOzjF4$xv0V","tcpport":80,"tcpaddr":"127.0.0.1","soptions":{},"userid":"user//some.one"}
meshcentral01  | RELAY: Relay holding: 5pk7f$HZD9KZ (::1) Authenticated
meshcentral01  | COOKIE: Encoded AESGCM cookie: {"userid":"user//some.one","domainid":"","ps":1,"time":1731167575}
meshcentral01  | PEER: FTunnel meshcentral02: Start connect to ws://meshcentral02.mongodb.exemple.local:8080/meshrelay.ashx?p=14&auth=0ZC6hfLnn0H9MgswYMQXrfKyfm2yq4dDKx$McBluEK5Bsar4fttluZ8TmffCPk2BmWnOUaRQbtx1AxHkJ$rtE$7vsLO9bv$hx7mZIYXmv8NHOXgwsWIAAkGZ0E4cOvg=
meshcentral01  | PEER: FTunnel meshcentral02: Connected
meshcentral01  | PEER: FTunnel disconnect meshcentral02
meshcentral01  | PEER: FTunnel1: Soft disconnect
meshcentral01  | PEER: FTunnel2: Soft disconnect
meshcentral01  | RELAY: TCP: Relay websocket closed
meshcentral01  | WEBRELAY: tunnel-onclose
meshcentral01  | WEBRELAY: launchNewTunnel
meshcentral01  | RELAY: TCP: Request for web relay
meshcentral01  | COOKIE: Encoded AESGCM cookie: {"userid":"user//some.one","domainid":"","nodeid":"node//0l4@bZ@cmBgyT$xU3uwJW2ooo527o27lMhSbpa2PDGlh@NlJzDUCSVSgAD1TKozY","tcpport":80,"tcpaddr":"127.0.0.1","time":1731167605}
meshcentral01  | RELAY: TCP: Connection websocket to ws://localhost:8080/meshrelay.ashx?p=14&auth=WrR@W1KssEVecgbH0ynOkM5VXw2Bk43iyLnxs3XgTxQLLWjRHFTJzzVD1BFfpkgGjtj$gS0xwuIkG2bC2LqS55DlZW6@YdBjOekDWbrNVLBbVhtuci2GvTtPwxd9u8U@o8gKGwWmdhduu@hEv5IibqrUUoUr5A9FI7s1QLqiiZT2w3JrdCUs8Kj4HS4dMRKHnpGsAFi4iCqp@eC29E2nA3ruFamQv97nTzKiYMPh2IElwSoap16LmMFv@p0nCUHfY0$hcIAaq@5LawyoAg==
meshcentral01  | PEER: FTunnel disconnect meshcentral02
meshcentral01  | COOKIE: Decoded AESGCM cookie: {"userid":"user//some.one","domainid":"","nodeid":"node//0l4@bZ@cmBgyT$xU3uwJW2ooo527o27lMhSbpa2PDGlh@NlJzDUCSVSgAD1TKozY","tcpport":80,"tcpaddr":"127.0.0.1","time":1731167605000,"dtime":837}
meshcentral01  | RELAY: TCP: Relay websocket open
meshcentral01  | COOKIE: Encoded AESGCM cookie: {"ruserid":"user//some.one","time":1731167605}
meshcentral01  | RELAY: Relay: Sending agent tunnel command: {"nodeid":"node//0l4@bZ@cmBgyT$xU3uwJW2ooo527o27lMhSbpa2PDGlh@NlJzDUCSVSgAD1TKozY","action":"msg","type":"tunnel","value":"*/meshrelay.ashx?id=sxIprjOasNxe&rauth=prB8ZBsQQoyHkrUeIOe0Nldikp32HkMa3qYtKgI2LgJmOaIqqztLN3WPWn858Lc2tnm@1WaNV1v@oEd9ZTd6yA9HHIgXpfnbdZNM","tcpport":80,"tcpaddr":"127.0.0.1","soptions":{},"userid":"user//some.one"}
meshcentral01  | RELAY: Relay holding: sxIprjOasNxe (::1) Authenticated
meshcentral01  | COOKIE: Encoded AESGCM cookie: {"userid":"user//some.one","domainid":"","ps":1,"time":1731167606}
meshcentral01  | PEER: FTunnel meshcentral03: Start connect to ws://meshcentral03.mongodb.exemple.local:8080/meshrelay.ashx?p=14&auth=w4qPayW$f8syUcKcWx1TBe2lb$rFp85wmJEN0ydtrPmIPYqnDHnI8qziftu2BdBdkmix5ohBMOttiIT36alEBDBrI7xlE$56d0FfEMA5nhgHHzDNivwOZr2Hf3Zhk3c=
meshcentral01  | PEER: FTunnel meshcentral03: Connected
meshcentral01  | PEER: FTunnel disconnect meshcentral03
meshcentral01  | PEER: FTunnel1: Soft disconnect
meshcentral01  | PEER: FTunnel2: Soft disconnect
meshcentral01  | RELAY: TCP: Relay websocket closed
meshcentral01  | WEBRELAY: tunnel-onclose
meshcentral01  | WEBRELAY: launchNewTunnel
meshcentral01  | RELAY: TCP: Request for web relay
meshcentral01  | COOKIE: Encoded AESGCM cookie: {"userid":"user//some.one","domainid":"","nodeid":"node//0l4@bZ@cmBgyT$xU3uwJW2ooo527o27lMhSbpa2PDGlh@NlJzDUCSVSgAD1TKozY","tcpport":80,"tcpaddr":"127.0.0.1","time":1731167636}
meshcentral01  | RELAY: TCP: Connection websocket to ws://localhost:8080/meshrelay.ashx?p=14&auth=GVqL0TMkNY67XyLtS1DqUu5Sin0H@quuv3shNdldIXuFrMyj0gQD2riTB9t@NGy4f6wJVivW7tCr@$QTKZbjKgkCWWAPo1YuPdGuAwmyUMAVag0YHFtRZtjP4Pp@tVGriBzvRnCDjk511Ho4bjRX9w7Rzw2dcXwXJV$ALDHWC7YMjq7DJCzlAbSsIVt8Hp2cu281lg3jHVCRfzW$OrpVdj0$IdLT1MxoLpYzyMo6TdfT9xOuylpStMrm2@eEBGmnMuyh3mjjABs2hdntjw==
meshcentral01  | PEER: FTunnel disconnect meshcentral03
meshcentral01  | COOKIE: Decoded AESGCM cookie: {"userid":"user//some.one","domainid":"","nodeid":"node//0l4@bZ@cmBgyT$xU3uwJW2ooo527o27lMhSbpa2PDGlh@NlJzDUCSVSgAD1TKozY","tcpport":80,"tcpaddr":"127.0.0.1","time":1731167636000,"dtime":104}
meshcentral01  | RELAY: TCP: Relay websocket open
meshcentral01  | COOKIE: Encoded AESGCM cookie: {"ruserid":"user//some.one","time":1731167636}
meshcentral01  | RELAY: Relay: Sending agent tunnel command: {"nodeid":"node//0l4@bZ@cmBgyT$xU3uwJW2ooo527o27lMhSbpa2PDGlh@NlJzDUCSVSgAD1TKozY","action":"msg","type":"tunnel","value":"*/meshrelay.ashx?id=1NIL@WAsZlqa&rauth=BKxwo0t5nNkCaUoDZtbb66B0uZhF8gfxjkS1UjQrVhYlil3CXvXqZTYy1dqBN3PiiVyv6@z4$hRvcC4xB7$BIOBLA4T2MvTP1uTN","tcpport":80,"tcpaddr":"127.0.0.1","soptions":{},"userid":"user//some.one"}
meshcentral01  | RELAY: Relay holding: 1NIL@WAsZlqa (::1) Authenticated
meshcentral01  | COOKIE: Encoded AESGCM cookie: {"userid":"user//some.one","domainid":"","ps":1,"time":1731167636}
meshcentral01  | PEER: FTunnel meshcentral03: Start connect to ws://meshcentral03.mongodb.exemple.local:8080/meshrelay.ashx?p=14&auth=9MxOeTSKBDS2@Hg5GWLHGapG2xaJGfu8IA2J5$CFbX3Mf176FxzkG2GR0EghBL1lv3suE4suwJ9Lxu8Z7mrevDsggzgFcAenrvwUhYX6ZVcYgM5ICm9kJBiYgjA0Edo=
meshcentral01  | PEER: FTunnel meshcentral03: Connected
meshcentral02  | COOKIE: Decoded AESGCM cookie: {"ruserid":"user//some.one","time":1731167575000,"dtime":806}
meshcentral02  | RELAY: Relay holding: 5pk7f$HZD9KZ (187.180.178.121)
meshcentral02  | COOKIE: ERR: Bad AESGCM cookie due to exception: Error: Unsupported state or unable to authenticate data
meshcentral02  | COOKIE: ERR: Bad AESSHA cookie due to exception: Error: error:1C80006B:Provider routines::wrong final block length
meshcentral02  | COOKIE: Decoded AESGCM cookie: {"userid":"user//some.one","domainid":"","ps":1,"time":1731167575000,"dtime":824}
meshcentral02  | RELAY: Relay: Soft disconnect (192.168.100.85)
meshcentral02  | RELAY: Relay disconnect: 5pk7f$HZD9KZ (187.180.178.121)
meshcentral03  | COOKIE: Decoded AESGCM cookie: {"ruserid":"user//some.one","time":1731167605000,"dtime":1085}
meshcentral03  | RELAY: Relay holding: sxIprjOasNxe (187.180.178.121)
meshcentral03  | COOKIE: ERR: Bad AESGCM cookie due to exception: Error: Unsupported state or unable to authenticate data
meshcentral03  | COOKIE: ERR: Bad AESSHA cookie due to exception: Error: error:1C80006B:Provider routines::wrong final block length
meshcentral03  | COOKIE: Decoded AESGCM cookie: {"userid":"user//some.one","domainid":"","ps":1,"time":1731167606000,"dtime":97}
meshcentral03  | RELAY: Relay: Soft disconnect (192.168.100.85)
meshcentral03  | RELAY: Relay disconnect: sxIprjOasNxe (187.180.178.121)
meshcentral03  | COOKIE: Decoded AESGCM cookie: {"ruserid":"user//some.one","time":1731167636000,"dtime":355}
meshcentral03  | RELAY: Relay holding: 1NIL@WAsZlqa (187.180.178.121)
meshcentral03  | COOKIE: ERR: Bad AESGCM cookie due to exception: Error: Unsupported state or unable to authenticate data
meshcentral03  | COOKIE: ERR: Bad AESSHA cookie due to exception: Error: error:1C80006B:Provider routines::wrong final block length
meshcentral03  | COOKIE: Decoded AESGCM cookie: {"userid":"user//some.one","domainid":"","ps":1,"time":1731167636000,"dtime":366}
meshcentral03  | RELAY: Relay: Soft disconnect (192.168.100.85)