Startup not completing? (Z2M UNUSABLE!)

[corporategoth]

What happened?

I migrated from TrueNAS 24.04 -> 24.10, which meant my container went from being Kubernetes to Docker (which actually gives me more control).
I also moved from Mosquitto to EMQX for an MQTT broker.

Since doing so, Z2M is not completing it's startup.
Specifically, the Ember driver is coming up, and detects all my zigbee devices.
I also see Z2M logged into the EMQX server, and subscribed to zigbee2mqtt/#

HOWEVER:

1. I don't see any PUBLISH log lines to MQTT for the devices.
2. The Z2M front end does not start up (it doesn't even open a socket).
3. I am using HA, so I also don't see any auto-configuration going on (it does not subscribe to the HA topic either).

If I turn on debugging, I see communications with the Zigbee network, but nothing else.

What did you expect to happen?

I expected the Z2M UI to show up.
I expected it to publish the state of my devices.

How to reproduce it (minimal and precise)

Just start up the docker container. Here is my container config:

services:
    zigbee2mqtt:
        container_name: zigbee2mqtt
        image: koenkk/zigbee2mqtt:latest
        volumes:
          - /mnt/bigfast/app/zigbee2mqtt:/app/data
          - /run/udev:/run/udev:ro
        networks:
          bound_bond0:
          mqtt:
          web:
        ports:
          - 10103:8080
        environment:
          - TZ=America/New_York
        devices:
          - /dev/ttyACM0:/dev/ttyACM0
        restart: always

Any my configuration.yaml:

experimental:
  new_api: true
frontend:
  port: 8080
permit_join: false
mqtt:
  server: mqtt://emqx
  base_topic: zigbee2mqtt
  user: zigbee2mqtt
  password: [[REDACTED]]
  keepalive: 60
  reject_unauthorized: true
  version: 5
serial:
  port: /dev/ttyACM0
  adapter: ember
advanced:
  homeassistant_legacy_entity_attributes: false
  legacy_api: false
  legacy_availability_payload: false
  log_syslog:
    app_name: Zigbee2MQTT
    eol: /n
    host: localhost
    localhost: localhost
    path: /dev/log
    pid: process.pid
    port: 514
    protocol: udp4
    type: '5424'
  last_seen: ISO_8601_local
  cache_state_send_on_startup: true
  log_symlink_current: true
  log_rotation: false
  log_level: debug
device_options:
  legacy: false
devices:
  '0x6c5cb1fffe56cbd3':
    friendly_name: Basement Hallway Light
  '0xe0798dfffed92d59':
    friendly_name: Gym Lights Slave
  '0xe0798dfffecdd793':
    friendly_name: Gym Closet Light
  '0x6c5cb1fffe56d036':
    friendly_name: Basement Hallway Lights Slave
  '0xe0798dfffed930bd':
    friendly_name: Bar Lights
  '0x385b44fffee8fc2e':
    friendly_name: Rec Room Lights Slave
  '0x6c5cb1fffe56cbd6':
    friendly_name: Rec Room Lights
  '0xe0798dfffec7d0f3':
    friendly_name: Rec Room Fireplace Lights
  '0xe0798dfffed9355e':
    friendly_name: Pool Room Lights
  '0xe0798dfffed29180':
    friendly_name: Basement Door Light
  '0x385b44fffee8fc3f':
    friendly_name: Pool Table Lights
  '0xe0798dfffecdd7ad':
    friendly_name: Pool Room Lights Slave
  '0xe0798dfffec7cb95':
    friendly_name: Basement Stairs Lights
  '0x6c5cb1fffe5e7185':
    friendly_name: Laundry Room Lights Slave
  '0xe0798dfffed92d62':
    friendly_name: Basement Stairs Lights Slave
  '0xe0798dfffecdd77a':
    friendly_name: Breakfast Area Light
  '0x385b44fffeee0c05':
    friendly_name: Kitchen Lights Slave
  '0x385b44fffee8fbf1':
    friendly_name: Foyer Lights Slave
  '0xe0798dfffed92d4f':
    friendly_name: Main Stairs Lights Slave
  '0xe0798dfffed93305':
    friendly_name: Living Room Fireplace Lights
  '0xe0798dfffecdd5da':
    friendly_name: Living Room Lights
  '0x6c5cb1fffe56e8e7':
    friendly_name: Dining Room Lights
  '0xe0798dfffed2a9ab':
    friendly_name: Dining Room Lights Slave
  '0xe0798dfffec7d1aa':
    friendly_name: Pantry Light
  '0xe0798dfffed29308':
    friendly_name: Kitchen Lights
  '0xe0798dfffed28763':
    friendly_name: Kitchen Counter Lights (Right)
  '0x385b44fffee8fc2f':
    friendly_name: Garage Lights
  '0xe0798dfffec7d0e7':
    friendly_name: Kitchen Sink Light
  '0x6c5cb1fffe5e71b7':
    friendly_name: Kitchen Counter Lights (Left)
  '0xe0798dfffed9299f':
    friendly_name: Breakfast Area Light Slave
  '0x6c5cb1fffe5e8b21':
    friendly_name: Rear Door Lights
  '0x943469fffe089ca7':
    friendly_name: Deck Post Lights
  '0x6c5cb1fffe5e8b31':
    friendly_name: Rear Sensor Light
  '0x385b44fffee8fc72':
    friendly_name: Sitting Room Lamp
  '0x6c5cb1fffe56e570':
    friendly_name: Coat Closet Light
  '0x6c5cb1fffe56de47':
    friendly_name: Gym Lights
  '0x6c5cb1fffe56d035':
    friendly_name: Laundry Room Lights
  '0x6c5cb1fffe56d484':
    friendly_name: Foyer Lights
  '0x6c5cb1fffe56de3e':
    friendly_name: Dashiell's Closet Light
  '0x385b44fffeee0c34':
    friendly_name: Porch Lights
  '0x6c5cb1fffe5e7172':
    friendly_name: Driveway Lights Slave
  '0xe0798dfffecdd7a3':
    friendly_name: Garage Lights Slave
  '0xe0798dfffed29195':
    friendly_name: Driveway Lights
  '0x6c5cb1fffe5e7ca7':
    friendly_name: Main Stairs Lights
  '0x6c5cb1fffe5e7171':
    friendly_name: Dad's Closet Light
  '0x385b44fffee8fc34':
    friendly_name: Garden Lights
  '0xe0798dfffec7cb7b':
    friendly_name: Upstairs Hallway Lights Slave
  '0x6c5cb1fffe5e7187':
    friendly_name: Office Closet Light
  '0x6c5cb1fffe56d488':
    friendly_name: Upstairs Hallway Lights
  '0x385b44fffeee0c18':
    friendly_name: Master Bedroom Lamp (Left)
  '0xe0798dfffecdd798':
    friendly_name: Master Bedroom Lamp (Right)
  '0x385b44fffeee0c13':
    friendly_name: Master Bedroom Hallway Light
  '0x385b44fffeee0c0b':
    friendly_name: Master Bedroom Closet
  '0x6c5cb1fffe5e8560':
    friendly_name: Master Bathroom Bath Light
  '0x6c5cb1fffe56d030':
    friendly_name: Master Bathroom Shower Light
  '0x6c5cb1fffe56d894':
    friendly_name: Master Bathroom Lights
  '0x6c5cb1fffe56cbd4':
    friendly_name: Master Bedroom Vanity Lights
  '0x00124b002242693b':
    friendly_name: Media Lights
  '0xb0ce1814036d6943':
    friendly_name: Coat Closet Door
  '0xb0ce1814036d4e05':
    friendly_name: Pantry Door
  '0xb0ce1814036d56e8':
    friendly_name: Gym Left Closet Door
  '0xb0ce1814036d693a':
    friendly_name: Gym Right Closet Door
  '0x0022a3000019b9e0':
    friendly_name: Water Shutoff Valve
  '0x0c4314fffe49d4bf':
    friendly_name: Dad's Bedroom Left Blinds
  '0xb4e3f9fffe8fb903':
    friendly_name: Dad's Bedroom Right Blinds
  '0xb4e3f9fffed5e5f3':
    friendly_name: Master Bedroom Right Blinds
  '0x00124b0022426412':
    friendly_name: Behind TV Switch
  '0x00124b0022419e17':
    friendly_name: Basement Fireplace Switch
  '0x0c4314fffe49d8cf':
    friendly_name: Master Bedroom Left Blinds
  '0x0c4314fffe49dda7':
    friendly_name: Dashiell's Bedroom Blinds
  '0xb0ce1814036d5a78':
    friendly_name: Office Closet Door
  '0xb0ce18140016b637':
    friendly_name: Rear Door Left Light - Left
  '0xb0ce18140016c339':
    friendly_name: Rear Door Left Light - Right
  '0xb0ce18140016c64a':
    friendly_name: Rear Door Right Light - Left
  '0xb0ce18140016c48b':
    friendly_name: Rear Door Right Light - Right
  '0xb0ce18140016bdce':
    friendly_name: Deck Post Right Light - Tall
  '0xb0ce18140016c990':
    friendly_name: Deck Post Right Light - Medium
  '0xb0ce18140016b62d':
    friendly_name: Deck Post Right Light - Small
  '0xb0ce18140016b4e3':
    friendly_name: Deck Post Left Light - Tall
  '0xb0ce18140016bdf9':
    friendly_name: Deck Post Left Light - Medium
  '0xb0ce18140016c0e9':
    friendly_name: Deck Post Left Light - Small
  '0xb0ce18140016c33b':
    friendly_name: Porch Right Lights - Tall
  '0xb0ce18140016b50d':
    friendly_name: Porch Right Lights - Medium
  '0xb0ce18140016b8a4':
    friendly_name: Porch Right Lights - Small
  '0xb0ce18140016ca14':
    friendly_name: Porch Left Lights - Tall
  '0xb0ce18140016bf2a':
    friendly_name: Porch Left Lights - Medium
  '0xb0ce18140016c8a6':
    friendly_name: Porch Left Lights - Small
  '0xb0ce1814036d4a1f':
    friendly_name: Dad's Closet Door
  '0xb0ce1814036d594b':
    friendly_name: Dashiell's Closet Door
  '0x000d6f0018addefd':
    friendly_name: Back Door Lock
  '0x000d6f0018adddb9':
    friendly_name: Front Door Lock
  '0x000d6f0018addeaa':
    friendly_name: Basement Door Lock
  '0x0022a300002c9c54':
    friendly_name: Washing Machine Leak Sensor
  '0x0022a300002bfe14':
    friendly_name: Hot Water Heater Leak Sensor
  '0x0022a30000279b6d':
    friendly_name: Basement Bathroom Leak Sensor
  '0x0022a3000027b1fd':
    friendly_name: Garbage Disposal Leak Sensor
  '0x0022a3000019a383':
    friendly_name: Dishwasher Leak Sensor
  '0x0022a30000195d9a':
    friendly_name: Refrigerator Leak Sensor
  '0x0022a300002c9768':
    friendly_name: Master Bathroom Leak Sensor
  '0x0022a3000019744f':
    friendly_name: Attic Furnace Leak Sensor
  '0xb0ce1814036d5cee':
    friendly_name: Linen Closet Door
  '0x8cf681fffe34faba':
    friendly_name: Living Room Ikea Repeater
  '0xe0798dfffe7d650a':
    friendly_name: Post 3 Relay
  '0xe0798dfffe882ce5':
    friendly_name: Post 6 Relay
  '0x385b44fffeee0d85':
    friendly_name: Linen Closet Light
  '0xb0ce1814036d524d':
    friendly_name: Master Bedroom Closet Door
  '0xb0ce1814036d568f':
    friendly_name: Utility Room Door
  '0xb0ce1814036d572e':
    friendly_name: Shed Door
  '0xa4c138273042cc5a':
    friendly_name: Post 1 Switch
  '0xa4c138723d6fd056':
    friendly_name: Post 2 Switch
  '0xa4c138752b7ae3e6':
    friendly_name: Post 4 Switch
  '0xa4c13850d8b1f9bb':
    friendly_name: Post 5 Switch
  '0xb0ce1814036d5874':
    friendly_name: Free Little Library Door
  '0xb0ce1814001d7a6a':
    friendly_name: Post 1 Light - Tall
  '0xb0ce1814001d4ed3':
    friendly_name: Post 1 Light - Medium
  '0xb0ce1814001d4f3c':
    friendly_name: Post 1 Light - Small
  '0xb0ce18140016c3b0':
    friendly_name: Post 2 Light - Tall
  '0xb0ce1814001d776d':
    friendly_name: Post 2 Light - Medium
  '0xb0ce18140016bb98':
    friendly_name: Post 2 Light - Small
  '0xb0ce18140016c52d':
    friendly_name: Post 3 Light - Tall
  '0xb0ce18140016c952':
    friendly_name: Post 3 Light - Medium
  '0xb0ce18140016c8c5':
    friendly_name: Post 3 Light - Small
  '0xb0ce18140016b58d':
    friendly_name: Post 4 Light - Tall
  '0xb0ce18140016bf23':
    friendly_name: Post 4 Light - Medium
  '0xb0ce18140016bf4d':
    friendly_name: Post 4 Light - Small
  '0xb0ce18140016b80a':
    friendly_name: Post 5 Light - Medium
  '0xb0ce181400170ec1':
    friendly_name: Post 5 Light - Small
  '0xb0ce1814001d5517':
    friendly_name: Post 6 Light - Tall
  '0xb0ce1814001d46e2':
    friendly_name: Post 6 Light - Small
  '0xa4c1385fbe128b6e':
    friendly_name: Driveway Outlets
  '0x00124b0022426684':
    friendly_name: Bar Switch
  '0xa4c138b91fc3f3c8':
    friendly_name: Shed Outlets
  '0xa4c1386c5d6ca54f':
    friendly_name: Tree Light
  '0x00158d000316cca9':
    friendly_name: Garden Lights Left
  '0x00124b0025147c6f':
    friendly_name: Upstairs Hallway Motion Sensor
  '0xb4e3f9fffe8fb83b':
    friendly_name: Office Blinds
  '0x6c5cb1fffe5e0bc3':
    friendly_name: Shed Light
  '0x6c5cb1fffe5dde40':
    friendly_name: Utility Room Lights
  '0x04cd15fffe79b816':
    friendly_name: Pool Room Blinds Left
  '0x04cd15fffe79bbe5':
    friendly_name: Pool Room Blinds Right
  '0xb4e3f9fffe9f7b9c':
    friendly_name: Master Bedroom Back Blinds
  '0x6c5cb1fffe5e53c1':
    friendly_name: Sitting Room Lamp Slave
  '0x385b44fffee8fd30':
    friendly_name: Basement Door Light Slave
  '0x00158d00021658b5':
    friendly_name: Garden Lights Right
  '0x0022a300002ca0da':
    friendly_name: Upstairs Bathroom Leak Sensor
  '0x282c02bfffe8ba33':
    friendly_name: Plug 1
    debounce: 1
    homeassistant: {}
    legacy: false
    optimistic: true
  '0x282c02bfffed2118':
    friendly_name: Plug 2
    debounce: 1
    homeassistant: {}
    legacy: false
    optimistic: true
  '0x282c02bfffed2384':
    friendly_name: Plug 3
    debounce: 1
    homeassistant: {}
    legacy: false
    optimistic: true
  '0x282c02bfffed32bc':
    friendly_name: Plug 4
    debounce: 1
    homeassistant: {}
    legacy: false
    optimistic: true
  '0x70b3d52b60014ded':
    friendly_name: Smart Power Strip
  '0x048727fffe18b796':
    friendly_name: Living Room Fan
  '0x048727fffe1929d9':
    friendly_name: Main Bathroom Fan
  '0x6c5cb1fffe5e29b7':
    friendly_name: Main Bathroom Light
  '0x048727fffe192dbc':
    friendly_name: Basement Bathroom Fan
  '0x70ac08fffe70ce53':
    friendly_name: Basement Bathroom Light
  '0x048727fffe19559c':
    friendly_name: Rec Room Fan
  '0x70ac08fffe6c95d3':
    friendly_name: Office Light
  '0x048727fffe18bc94':
    friendly_name: Office Fan
  '0x6c5cb1fffe5dde37':
    friendly_name: Upstairs Bathroom Light
  '0x048727fffe1bdccd':
    friendly_name: Upstairs Bathroom Fan
  '0x70ac08fffe70c3f1':
    friendly_name: Dad's Bedroom Light
  '0x048727fffe195cf8':
    friendly_name: Dad's Bedroom Fan
  '0x6c5cb1fffe5dc427':
    friendly_name: Dashiell's Bedroom Light
  '0x048727fffe19f29b':
    friendly_name: Dashiell's Bedroom Fan
  '0x048727fffe19328d':
    friendly_name: Master Bedroom Fan
  '0x048727fffe1b9651':
    friendly_name: Master Bathroom Toilet Fan
  '0x0022a300002c8b59':
    friendly_name: Main Bathroom Leak Sensor
  '0x282c02bfffede33e':
    friendly_name: Chess Table Switch
    debounce: 1
    homeassistant: {}
    legacy: false
    optimistic: true
  '0x282c02bfffed98eb':
    friendly_name: Behind China Cabinet Switch
    debounce: 1
    homeassistant: {}
    legacy: false
    optimistic: true
  '0x282c02bfffed9f7f':
    friendly_name: Fish Tank Lights
    debounce: 1
    homeassistant: {}
    legacy: false
    optimistic: true
  '0x282c02bfffedf051':
    friendly_name: Dashiell's TV
    debounce: 1
    homeassistant: {}
    legacy: false
    optimistic: true
  '0x6c5cb1fffe5e67b7':
    friendly_name: Master Bathroom Toilet Light
  '0xb0ce18140016bd6a':
    friendly_name: Post 6 Light - Medium
  '0xb0ce18140016c627':
    friendly_name: Post 5 Light - Tall
groups:
  '1':
    friendly_name: deck_lights
    devices:
      - 0xb0ce18140016bdf9/1
      - 0xb0ce18140016c0e9/1
      - 0xb0ce18140016b4e3/1
      - 0xb0ce18140016c990/1
      - 0xb0ce18140016b62d/1
      - 0xb0ce18140016bdce/1
      - 0xb0ce18140016b637/1
      - 0xb0ce18140016c339/1
      - 0xb0ce18140016c64a/1
      - 0xb0ce18140016c48b/1
  '2':
    friendly_name: garden_lights
    devices:
      - 0x00158d000316cca9/1
      - 0x385b44fffee8fc34/2
      - 0x00158d00021658b5/1
  '3':
    friendly_name: rear_door_lights
    devices:
      - 0xb0ce18140016b637/1
      - 0xb0ce18140016c339/1
      - 0xb0ce18140016c64a/1
      - 0xb0ce18140016c48b/1
      - 0x6c5cb1fffe5e8b21/2
  '4':
    friendly_name: rear_door_left_lights
    devices:
      - 0xb0ce18140016b637/1
      - 0xb0ce18140016c339/1
  '5':
    friendly_name: rear_door_right_lights
    devices:
      - 0xb0ce18140016c64a/1
      - 0xb0ce18140016c48b/1
  '6':
    friendly_name: deck_post_lights
    devices:
      - 0xb0ce18140016bdf9/1
      - 0xb0ce18140016c0e9/1
      - 0xb0ce18140016b4e3/1
      - 0xb0ce18140016c990/1
      - 0xb0ce18140016b62d/1
      - 0xb0ce18140016bdce/1
      - 0x943469fffe089ca7/2
  '7':
    friendly_name: deck_post_left_lights
    devices:
      - 0xb0ce18140016bdf9/1
      - 0xb0ce18140016b4e3/1
      - 0xb0ce18140016c0e9/1
  '8':
    friendly_name: deck_post_right_lights
    devices:
      - 0xb0ce18140016b62d/1
      - 0xb0ce18140016c990/1
      - 0xb0ce18140016bdce/1
  '9':
    friendly_name: porch_lights
    devices:
      - 0xb0ce18140016c8a6/1
      - 0xb0ce18140016bf2a/1
      - 0xb0ce18140016ca14/1
      - 0xb0ce18140016b50d/1
      - 0xb0ce18140016b8a4/1
      - 0xb0ce18140016c33b/1
      - 0x385b44fffeee0c34/2
  '10':
    friendly_name: porch_left_lights
    devices:
      - 0xb0ce18140016c8a6/1
      - 0xb0ce18140016bf2a/1
      - 0xb0ce18140016ca14/1
  '11':
    friendly_name: porch_right_lights
    devices:
      - 0xb0ce18140016b8a4/1
      - 0xb0ce18140016b50d/1
      - 0xb0ce18140016c33b/1
  '12':
    friendly_name: driveway_lights
    devices:
      - 0xb0ce1814001d4ed3/1
      - 0xb0ce1814001d4f3c/1
      - 0xb0ce1814001d7a6a/1
      - 0xb0ce1814001d776d/1
      - 0xb0ce18140016bb98/1
      - 0xb0ce18140016c3b0/1
      - 0xb0ce18140016c952/1
      - 0xb0ce18140016c8c5/1
      - 0xb0ce18140016c52d/1
      - 0xb0ce18140016bf23/1
      - 0xb0ce18140016bf4d/1
      - 0xb0ce18140016b58d/1
      - 0xb0ce18140016b80a/1
      - 0xb0ce181400170ec1/1
      - 0xb0ce1814001d46e2/1
      - 0xb0ce1814001d5517/1
      - 0xe0798dfffed29195/2
      - 0xb0ce18140016c627/1
      - 0xb0ce18140016bd6a/1
  '13':
    friendly_name: post_1_lights
    devices:
      - 0xb0ce1814001d4f3c/1
      - 0xb0ce1814001d4ed3/1
      - 0xb0ce1814001d7a6a/1
  '14':
    friendly_name: post_2_lights
    devices:
      - 0xb0ce18140016bb98/1
      - 0xb0ce1814001d776d/1
      - 0xb0ce18140016c3b0/1
  '15':
    friendly_name: post_3_lights
    devices:
      - 0xb0ce18140016c8c5/1
      - 0xb0ce18140016c952/1
      - 0xb0ce18140016c52d/1
  '16':
    friendly_name: post_4_lights
    devices:
      - 0xb0ce18140016bf4d/1
      - 0xb0ce18140016bf23/1
      - 0xb0ce18140016b58d/1
  '17':
    friendly_name: post_5_lights
    devices:
      - 0xb0ce181400170ec1/1
      - 0xb0ce18140016b80a/1
      - 0xb0ce18140016c627/1
  '18':
    friendly_name: post_6_lights
    devices:
      - 0xb0ce1814001d46e2/1
      - 0xb0ce1814001d5517/1
      - 0xb0ce18140016bd6a/1
  '19':
    friendly_name: driveway_tall_lights
    devices:
      - 0xb0ce1814001d7a6a/1
      - 0xb0ce18140016c3b0/1
      - 0xb0ce18140016c52d/1
      - 0xb0ce18140016b58d/1
      - 0xb0ce1814001d5517/1
      - 0xb0ce18140016c627/1
  '20':
    friendly_name: driveway_medium_lights
    devices:
      - 0xb0ce1814001d4ed3/1
      - 0xb0ce1814001d776d/1
      - 0xb0ce18140016c952/1
      - 0xb0ce18140016bf23/1
      - 0xb0ce18140016b80a/1
      - 0xb0ce18140016bd6a/1
  '21':
    friendly_name: driveway_small_lights
    devices:
      - 0xb0ce1814001d4f3c/1
      - 0xb0ce18140016bb98/1
      - 0xb0ce18140016c8c5/1
      - 0xb0ce18140016bf4d/1
      - 0xb0ce181400170ec1/1
      - 0xb0ce1814001d46e2/1
  '28':
    friendly_name: driveway_1of6
    devices:
      - 0xb0ce1814001d7a6a/1
      - 0xb0ce18140016c952/1
      - 0xb0ce181400170ec1/1
  '29':
    friendly_name: driveway_2of6
    devices:
      - 0xb0ce18140016bf23/1
      - 0xb0ce1814001d46e2/1
      - 0xb0ce18140016c3b0/1
  '30':
    friendly_name: driveway_3of6
    devices:
      - 0xb0ce1814001d4ed3/1
      - 0xb0ce18140016c8c5/1
      - 0xb0ce18140016c627/1
  '31':
    friendly_name: driveway_4of6
    devices:
      - 0xb0ce1814001d776d/1
      - 0xb0ce18140016bf4d/1
      - 0xb0ce1814001d5517/1
  '32':
    friendly_name: driveway_5of6
    devices:
      - 0xb0ce1814001d4f3c/1
      - 0xb0ce18140016c52d/1
      - 0xb0ce18140016b80a/1
  '33':
    friendly_name: driveway_6of6
    devices:
      - 0xb0ce18140016bb98/1
      - 0xb0ce18140016b58d/1
      - 0xb0ce18140016bd6a/1
  '34':
    friendly_name: all_blinds
    devices:
      - 0xb4e3f9fffe8fb83b/1
      - 0x0c4314fffe49d8cf/1
      - 0xb4e3f9fffed5e5f3/1
      - 0xb4e3f9fffe9f7b9c/1
      - 0x0c4314fffe49d4bf/1
      - 0xb4e3f9fffe8fb903/1
      - 0x0c4314fffe49dda7/1
      - 0x04cd15fffe79b816/1
      - 0x04cd15fffe79bbe5/1
homeassistant:
  legacy_triggers: false
  status_topic: homeassistant/status

Zigbee2MQTT version

latest (1.41.0)

Adapter firmware version

8.0.2 [GA]

Adapter

Sonoff Dongle-E

Setup

Docker container in TrueNAS Scale

Debug log

log.log.gz

[corporategoth]

I thought this might be related to zigbee2mqtt/hassio-zigbee2mqtt#624 - however my Z2M is not 'functioning normally' (publishing to the MQTT server), and I don't have availability: true enabled.

[corporategoth]

@Koenkk @Nerivec any ideas on this?

[Nerivec]

I know several users had issues when moving to that broker and quickly reverted (i.e. never got any answers...).
And looks like MQTT is having troubles:

[2024-11-03 14:45:15] error: 	z2m: MQTT error: write ECONNRESET

[Koenkk]

I also moved from Mosquitto to EMQX for an MQTT broker.

Can you try moving back to mosquitto?

[corporategoth]

@Nerivec I noticed that it does get the MQTT error, disconnects, then connects the second time (I can verify from EMQX that zigbee2mqtt was connected). So there was definitely something weird about the connection then disconnection. HOWEVER, if it could not connect, I would expect it would shut down Z2M (which it does not). That is what happens if I have an incorrect passwrod or the MQTT server is not accessible at all.

@Koenkk Changing back to mosquitto fixed it, and I see stuff publishing to the MQTT server again, and I see a UI. So from a base functionality perspective, I guess it's working. But from a usability perspective I don't see why EMQX would not work (as it DOES make a connection, and maintains it, otherwise Z2M would die due to no MQTT connection). I would ideally like to move to EMQX as it has much better management and monitoring capabilities.

[corporategoth]

I figured out how to fix this.

When I traced the connection from the EMQX side, I saw:

2024-11-15T14:38:53.073870+00:00 [info] mfa: emqx_connection:parse_incoming/2(817), peername: 172.16.8.2:54638, clientid: zigbee2mqtt, reason: frame_too_large, at_state: <<"clean">>, input_bytes: <<49,216...>>, parsed_packets: []
2024-11-15T14:38:53.080239+00:00 [MQTT] [email protected]:54638 msg: mqtt_packet_received, packet: {frame_error,frame_too_large}
2024-11-15T14:38:53.080328+00:00 [MQTT] [email protected]:54638 msg: mqtt_packet_sent, packet: DISCONNECT(Q0, R0, D0, ReasonCode=149)
2024-11-15T14:38:53.080566+00:00 [SOCKET] [email protected]:54638 msg: socket_force_closed, reason: frame_too_large

The max packet size was set to the default on EMQX, which was 1MB.
I changed this to 5MB, and the connection was established and Z2M started up just fine.

So this yields two questions.

1. Why is Z2M sending a MQTT packet (or setting the max size above 1mb? I am not sure) >= 1MB on connection?
2. Why does this disconnect cause Z2M to get into a weird state where it reconnects to the MQTT server, but will not publish anything and won't start the UI even.

I guess now this is more a 'this is a bug, but no longer affecting me specifically' ticket.

[Nerivec]

It must be something specific to your network that creates such a huge packet. /bridge/definitions should be less than 150000 chars, and /bridge/devices, with 150 or so "regular" devices should be somewhere around 1-300000 chars (guesstimate, depends on clusters quite a bit).
Can you take a closer look at the logs, see what topic the packet was for? Since it's right after /bridge/state, it's one of the retained.

[corporategoth]

Here is, the 4100 bytes it prints in the log, presumably truncated:

1Ø¿␦zigbee2mqtt/bridge/devices[{"disabled":false,"endpoints":{"1":{"bindings":[{"cluster":"genOnOff","target":{"endpoint":1,"ieee_address":"0xe0798dfffea88b0a","type":"endpoint"}}],"clusters":{"input":["genBasic","genIdentify","genOnOff","genTime","genOta","26","lightingColorCtrl"],"output":["genBasic","genIdentify","genGroups","genScenes","genOnOff","genLevelCtrl","genPollCtrl","lightingColorCtrl","msIlluminanceMeasurement","msTemperatureMeasurement","msRelativeHumidity","msOccupancySensing","ssIasZone","haMeterIdentification","haApplianceStatistics","haElectricalMeasurement","seMetering","touchlink","manuSpecificLegrandDevices","manuSpecificSamsungAccelerometer"]},"configured_reportings":[],"scenes":[]},"242":{"bindings":[],"clusters":{"input":[],"output":["greenPower"]},"configured_reportings":[],"scenes":[]}},"friendly_name":"Coordinator","ieee_address":"0xe0798dfffea88b0a","interview_completed":true,"interviewing":false,"network_address":0,"supported":true,"type":"Coordinator"},{"date_code":"20240412","definition":{"description":"2-in-1 switch + dimmer","exposes":[{"features":[{"access":7,"description":"On/off state of this light","label":"State","name":"state","property":"state","type":"binary","value_off":"OFF","value_on":"ON","value_toggle":"TOGGLE"},{"access":7,"description":"Brightness of this light","label":"Brightness","name":"brightness","property":"brightness","type":"numeric","value_max":254,"value_min":0}],"type":"light"},{"access":1,"category":"diagnostic","description":"Instantaneous measured power","label":"Power","name":"power","property":"power","type":"numeric","unit":"W"},{"access":1,"description":"Sum of consumed energy","label":"Energy","name":"energy","property":"energy","type":"numeric","unit":"kWh"},{"access":3,"category":"config","features":[{"access":3,"description":"Animation Effect to use for the LEDs","label":"Effect","name":"effect","property":"effect","type":"enum","values":["off","solid","fast_blink","slow_blink","pulse","chase","open_close","small_to_big","aurora","slow_falling","medium_falling","fast_falling","slow_rising","medium_rising","fast_rising","medium_blink","slow_chase","fast_chase","fast_siren","slow_siren","clear_effect"]},{"access":3,"description":"Calculated by using a hue color circle(value/255*360) If color = 255 display white","label":"Color","name":"color","property":"color","type":"numeric","value_max":255,"value_min":0},{"access":3,"description":"Brightness of the LEDs","label":"Level","name":"level","property":"level","type":"numeric","value_max":100,"value_min":0},{"access":3,"description":"1-60 is in seconds calculated 61-120 is in minutes calculated by(value-60) Example a value of 65 would be 65-60 = 5 minutes - 120-254 Is in hours calculated by(value-120) Example a value of 132 would be 132-120 would be 12 hours. - 255 Indefinitely","label":"Duration","name":"duration","property":"duration","type":"numeric","value_max":255,"value_min":0}],"label":"Led effect","name":"led_effect","property":"led_effect","type":"composite"},{"access":3,"category":"config","features":[{"access":3,"description":"Individual LED to target.","label":"Led","name":"led","property":"led","type":"enum","values":["1","2","3","4","5","6","7"]},{"access":3,"description":"Animation Effect to use for the LED","label":"Effect","name":"effect","property":"effect","type":"enum","values":["off","solid","fast_blink","slow_blink","pulse","chase","falling","rising","aurora","clear_effect"]},{"access":3,"description":"Calculated by using a hue color circle(value/255*360) If color = 255 display white","label":"Color","name":"color","property":"color","type":"numeric","value_max":255,"value_min":0},{"access":3,"description":"Brightness of the LED","label":"Level","name":"level","property":"level","type":"numeric","value_max":100,"value_min":0},{"access":3,"description":"1-60 is in seconds calculated 61-120 is in minutes calculated by(value-60) Example a value of 65 would be 65-60 = 5 minutes - 120-254 Is in hours calculated by(value-120)  Example a value of 132 would be 132-120 would be 12 hours.

I have a lot (70? 80?) of Inovelli Blue switches, and I guess the description of them can be verbose once you factor in descriptions. And I have ~180 devices total.
I don't see this logged in my Z2M log files (though maybe it would @ debug? I don't know). Either way, as I said, increasing the max packet size worked.

[Nerivec]

Indeed, lots of features per device (multi-function), larger payload.
The combination of hitting the max, and having version: 5 triggers a DISCONNECT packet, which was not present in previous mqtt spec from what I gathered... 😅

though maybe it would @ debug?

Yes, from retained on startup.

[corporategoth]

Presumably the size of that could be calculated before you send the max packet size.
Then you could possibly send a max packet size of rounded up to next power of 2. Or * 1.2 rounded up to the next power of 2.

[corporategoth]

@Nerivec Unless I mistake the PR, even with the change, the max packet size would still need to be adjusted on the broker side to allow large packets - the Z2M setting is just the max packet size it will allow FROM the broker. At least, that's how the docs read.

If that's the case, there should probably be a section in the website docs on 'Broker setup' for what options may need to be changed (more than just documenting the field exists as a notation on the maximum_packet_size setting).

Or at least put it under the Zigbee2MQTT fails to start section.

[Nerivec]

Like this? Koenkk/zigbee2mqtt.io#3233 😉
PS: you can browse the plain markdown of the 2.0.0 updated docs here

And about the behavior, yes that's how I interpret the docs too, but from the behaviors I've seen, I'd say it's not respected/enforced in all brokers (yet). Guessing v5 still needs better support overall 😅

[corporategoth]

@Nerivec OK .. so you read my mind. Damn you ;)

Awesome :)