Empty devices list

[JohannCR]

Description of the issue

All devices disappeared from the addon's devices list. They are still present in the integration page, and still work.
Since 1.36.1-1 I've been uninstalling and reinstalling switching between the stable and edge versions to try to fix some ghost bindings that broke a couple of devices. Since then, I don't know why, the devices list is now empty. As if the database.db was corrupted in some way... When I open it, it looks good and contains all devices though.

Tried clearing cache, rebooting addon, reloading integration, rebooting HA, no change.

Is there a way to fix this ? Like removing the database, reboot and then putting it back or something like that ?

Addon version

edge. Can't find commit hash, but it's the same on stable anyway

Platform

Core
2024.4.1
Supervisor
2024.04.0
Operating System
12.1
Interface utilisateur
20240404.1

Logs of the issue (if applicable)

Nothing in the logs

[JohannCR]

For those wondering, I have not found any solution other than deleting everything and restarting from scratch repairing every device... That was a pain 🤣

[nrehn]

Hi. I have the same issue now.
Device list is empty but everything works.
It does not work to join devices either. When pressing the button in GUI nothing happens.

[martintm86]

Hello, for me it happens today after Update 2024.6.0. Everything works but i cant configure anything in the Zigbee2MQTT menu. All empty and without function...

Reboot and clearing cache does not fix that...

[Astromechdroide]

Hello, for me it happens today after Update 2024.6.0. Everything works but i cant configure anything in the Zigbee2MQTT menu. All empty and without function...

Reboot and clearing cache does not fix that...

I have the same problem. Is there a solution?

[JohannCR]

I have the same problem. Is there a solution?

Only one I tested successfully was to uninstall and delete all devices, reinstall and repair everything... Painful.
Plus I don't know why (may not be linked to this issue) but I have more devices disconnecting from the network since. Unfortunately I also changed my coordinator so I don't know what caused those disconnections. Again, may not be linked to z2m.
Try repairing all if your network isn't too big and don't hesitate to report back 😉

[martintm86]

After update to 2024.6.1 everything is fixed for me....

[Operrr]

My version is XA 2024.6.1, version Z2M
1.38.0-1 the list of devices is empty, but everything seems to work. Connecting new devices does not work, there are no logs, no information. Help me please!

[booroondook-eng]

The same problem, but I see empty devices list (and also empty Dashboard) in ingress mode only.
When I switch to normal mode (open http://xxx.xxx.xxx.xxx:8099 in the new browser window) I see completed list of my devices and working Dashboard).
In other words: The problem with empty devices list appears in the ingress mode only.

[Operrr]

I can't open the page directly

[Operrr]

Is it really possible that no one else has such a problem and there will be no help?

[martintm86]

For me the Problem comes with my Apache Reverse Proxy. In the Browser Console i have seen that it can't open a special WebSocket URL. After Adding it to reverse Proxy config anything is ok....
Before if i connect directly to HA by IP the Plugin was OK

ProxyPass /api/hassio_ingress ws://192.168.2.4:80/api/hassio_ingress
ProxyPassReverse /api/hassio_ingress ws://192.168.2.4:80/api/hassio_ingress

[pdm0]

Is it really possible that no one else has such a problem and there will be no help?

You are not alone in this. Unfortunately I am in the same situation and despite considerable efforts have not found any fix.

[Operrr]

I solved the problem. If you clear the "devices" section directly in "/homeassistant/zigbee2mqtt/configuration.yaml" then all device names and descriptions will disappear, but the device list will work correctly. After everything worked, I manually entered device descriptions using the old backup. Now everything is fine.

[booroondook-eng]

I found another way to resolve this problem.
My configuration.yaml contains the following string:
devices: devices.yaml
...and all devices' configuration stores in the devices.yaml file.
I noticed that this file has very large size: about 5 MB.
The reason for this size was that this file contains icons of all my devices in base64 format.
For example:

'0xa4c1380d5fXXXXXX':
  friendly_name: Zigbee Lamp 2
  icon: >-
    data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAJYAAACWCAYAAAA8.... etc

I removed all the lines with icons, the file was reduced to about 4 kilobytes, and the problem with the empty device list completely disappeared!

[Netpoet]

@booroondook-eng
Thanks a lot! I had the same issue, and your hint helped! I cleared all the image information lines from the config, restarted the coordinator and Home Assistant, and voilà, everything's back!
<3

[ahamro]

@booroondook-eng

I found another way to resolve this problem. My configuration.yaml contains the following string: devices: devices.yaml ...and all devices' configuration stores in the devices.yaml file. I noticed that this file has very large size: about 5 MB. The reason for this size was that this file contains icons of all my devices in base64 format. For example:

'0xa4c1380d5fXXXXXX':
  friendly_name: Zigbee Lamp 2
  icon: >-
    data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAJYAAACWCAYAAAA8.... etc

I removed all the lines with icons, the file was reduced to about 4 kilobytes, and the problem with the empty device list completely disappeared!

You're the best! In my case the devices are in zigbee2mqtt/configuration.yaml and after weeding out the icon lines the devices list and the dashboard were populated again. Including the icons. Great! Thanks for sharing your solution!

[xelemorf]

Great finding, I was getting nuts by bumping into this bug all the time!

I have identified that the "Localise device images" feature is doing this, flooding the Z2M configuration.yaml with embedded Base64 images to a point where Z2M just cannot handle.

In my case it also was not able to finish the localization of images as it was already too much with my 147 devices, failed around 100. I was also reconsidering my life choices while clearing the config manually by having a file lenght of 4 318 294 characters (4.11 MB), and the cleaned up file is 43 540 character (42.51k). Anyway, Notepad++ was a great help in this by collapsing the "icon: >-" segments and deleting each from the start of the next line.

@Koenkk, can you please remove this "Localise device images" feature completely from Z2M for now or add a warning/confirmation that things can go badly when having numerous devices?
I think embedding images and adding substantial amount of data into the main config file will always end up badly, maybe having them externally stored as jpg/png files in an dedicated z2m_img folder and referencing them by device id would be more fruitful. Thank you.

[Koenkk]

@xelemorf interesting finding, I guess z2m didn't startup anymore in your case?

[xelemorf]

@xelemorf interesting finding, I guess z2m didn't startup anymore in your case?

@Koenkk
Z2M could still start without showing any errmsg but none of my devices were present on the default UI, while they were still working as expected in all HA automations, scripts and on Lovelace.
Had to enable the front-end tcp-listen port 8099 to be able to see my devices in Z2M there (sidenote for the front-end 8099: it does not have any protection and could login without auth).

After the fix I've mentioned everything works as expected. Using currently available latest Z2M stable with latest HAOS.

Btw once had this issue, I also could not get the file from "request z2m backup" or "download state", and "network map" also never finished - works after the fix.

[Koenkk]

Interesting, do you still have the debug log of this? Could also be a frontend issue

See this on how to enable debug logging.

[xelemorf]

I'll try to find some time over the weekend to collect debug logs.

[silfax]

@xelemorf interesting finding, I guess z2m didn't startup anymore in your case?

@Koenkk Z2M could still start without showing any errmsg but none of my devices were present on the default UI, while they were still working as expected in all HA automations, scripts and on Lovelace. Had to enable the front-end tcp-listen port 8099 to be able to see my devices in Z2M there (sidenote for the front-end 8099: it does not have any protection and could login without auth).

After the fix I've mentioned everything works as expected. Using currently available latest Z2M stable with latest HAOS.

Btw once had this issue, I also could not get the file from "request z2m backup" or "download state", and "network map" also never finished - works after the fix.

I can confirm the same behavior with probably the same cause ('Localize device images'). Z2M appears to be working normally, device list only available by accessing it via port 8099. Didn't have the time to try cleaning devices.yaml (currently >5 MB). Will do it during weekend. Will also try to collect some logs before as well.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days

[xelemorf]

Interesting, do you still have the debug log of this? Could also be a frontend issue

See this on how to enable debug logging.

@Koenkk
Apologies for the delay, life just stepped in and did not have any spare time since, however I could make some time today and managed to peform a successful repro.

Z2M configuration.yaml debug setting (Z2M service restarted after changing):
log_debug_to_mqtt_frontend: true
log_level: debug

Size of configuration.yaml:
Orig: 42.6KB
Localised: 6.01MB

[Localtime: 10:25] The localization on the UI stalled marking items "done", however, all devices in config were processed.
[Localtime: 10:30] Devices still showing up in Z2M at this point.
[Localtime: 10:33] Stopped Z2M service.
[Localtime: 10:34] Started Z2M service.
[Localtime: 10:34] Devices are not showing up, all devices dissapeared. Repro successful.
[Localtime: 10:36] Stopped Z2M service:
[Localtime: 10:40] Config restored, Z2M service started, devices are shown

Showing the process for Localise device images, which appears to be stalled after "kitchen-mainlight":

Showing the empty device list after localise device images completed and a Z2M restart:

Showing a device list after the config restore:

Showing Z2M version and device count:

Could not upload 7z or logs directly, in-line would have been flooding so renamed it to txt extension. Please rename to Z2M_Localise.device.images_LOGS_u3.7z
Z2M_Localise.device.images_LOGS_u3.txt

PS: Would be great if the following sensitive data would not be exposed in the debug log by the way, each property have multiple types of exposure: extendedPanID, panID, channelList. I've marked all references as REDACTED.

[Nephiel]

Just had a Pi Zero 2 crash, tracked it down to Localise device images.
Apparently it's a bad idea to press that button on devices with little RAM.

For reference, the node process for Z2M on this Pi Zero 2 usually runs at 30% memory usage.

[xelemorf]

Interesting, do you still have the debug log of this? Could also be a frontend issue
See this on how to enable debug logging.

@Koenkk Apologies for the delay, life just stepped in and did not have any spare time since, however I could make some time today and managed to peform a successful repro.

Z2M configuration.yaml debug setting (Z2M service restarted after changing): log_debug_to_mqtt_frontend: true log_level: debug

Size of configuration.yaml: Orig: 42.6KB Localised: 6.01MB

[Localtime: 10:25] The localization on the UI stalled marking items "done", however, all devices in config were processed. [Localtime: 10:30] Devices still showing up in Z2M at this point. [Localtime: 10:33] Stopped Z2M service. [Localtime: 10:34] Started Z2M service. [Localtime: 10:34] Devices are not showing up, all devices dissapeared. Repro successful. [Localtime: 10:36] Stopped Z2M service: [Localtime: 10:40] Config restored, Z2M service started, devices are shown

Showing the process for Localise device images, which appears to be stalled after "kitchen-mainlight":

Showing the empty device list after localise device images completed and a Z2M restart:

Showing a device list after the config restore:

Showing Z2M version and device count:

Could not upload 7z or logs directly, in-line would have been flooding so renamed it to txt extension. Please rename to Z2M_Localise.device.images_LOGS_u3.7z Z2M_Localise.device.images_LOGS_u3.txt

PS: Would be great if the following sensitive data would not be exposed in the debug log by the way, each property have multiple types of exposure: extendedPanID, panID, channelList. I've marked all references as REDACTED.

@Koenkk: Looking for an update on this one.

[booroondook-eng]

mv devices.yaml devices.yaml.icons
cat devices.yaml.icons | grep -v "icon:" | grep -v "data:image" > devices.yaml

or

mv configuration.yaml configuration.yaml.icons
cat configuration.yaml.icons | grep -v "icon:" | grep -v "data:image" > configuration.yaml