[Support]: frigate.output pegging CPU cores at 100% - box is unresponsive - how to track down/diagnose?

[victorhooi]

Describe the problem you are having

I have a ARM64 host (256 cores), with a Coral USB TPU, running Frigate in a container.

I recently updated to the latest dev release, and also enable the new Semantic Search feature (not sure if this caused the issue or not)

At first, it all seemed to be working fine. However, the following day, I noticed that the machine running Frigate was very unresponsive:

• The Frigate web UI wasn't accessible via the browser.
• SSH-ing into the server itself took a long time - e.g. ~ 10 minutes to get to the password prompt, then maybe another 10 minutes after entering in the password to get to a prompt.
• top does show a few frigate.output processes that are pegging their respective cores at 100%.
  
• Trying to stop the Frigate container just seems to spin indefinitely:

victorhooi@ampere-01 ~/frigate-docker> docker compose stop
WARN[0000] /home/victorhooi/frigate-docker/compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion
[+] Stopping 0/1
 ⠴ Container frigate  Stopping                                                                                                                                                                                                           3153.6s

(About 52 minutes so far)

• Even trying to get docker logs from the container seems to hang indefinitely as well.

I'm assuming the box being unresponsive is related to the high CPU usage - would that be a reasonable hypothesis?

Does anybody know what exactly is frigate.output, or what it's responsible for? Is there an easy way to track down what specifically is causing the high CPU consumption? (I can try disabling the Semantic Search feature if that's the issue - but the reindexing only took around 70 seconds, so it didn't seem overly taxing on the box before, at least).

I haven't rebooted the box yet - but it seems I might have to, if I want to get logs - but is there any other diagnostic information I should collect first?

Version

ee816b2-standard-arm64

What browser(s) are you using?

No response

Frigate config file

mqtt:
  enabled: false

cameras:
  dummy_camera: # <--- this will be changed to your actual camera later
    enabled: false
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:554/rtsp
          roles:
            - detect

  bedroom:
    enabled: true
    ffmpeg:
      inputs:
        - path:
            rtsp://root:[email protected]:554/axis-media/media.amp?streamprofile=hdwitnessSecondary
          roles:
            - detect
        - path:
            rtsp://root:[email protected]:554/axis-media/media.amp?streamprofile=hdwitnessPrimary
          roles:
            - record
    # detect:
    #   width: 640 # <- optional, by default Frigate tries to automatically detect resolution
    #   height: 360 # <- optional, by default Frigate tries to automatically detect resolution

    zones: {}
    review: {}

  kids_bedroom:
    enabled: true
    ffmpeg:
      inputs:
        - path:
            rtsp://root:[email protected]:554/axis-media/media.amp?streamprofile=hdwitnessSecondary
          roles:
            - detect
        - path:
            rtsp://root:[email protected]:554/axis-media/media.amp?streamprofile=hdwitnessPrimary
          roles:
            - record
    # detect:
    #   width: 640 # <- optional, by default Frigate tries to automatically detect resolution
    #   height: 360 # <- optional, by default Frigate tries to automatically detect resolution

    zones: {}
    review: {}

  living_room:
    enabled: true
    ffmpeg:
      inputs:
        - path:
            rtsp://root:[email protected]:554/axis-media/media.amp?streamprofile=hdwitnessSecondary
          roles:
            - detect
        - path:
            rtsp://root:[email protected]:554/axis-media/media.amp?streamprofile=hdwitnessPrimary
          roles:
            - record
    # detect:
    #   width: 640 # <- optional, by default Frigate tries to automatically detect resolution
    #   height: 360 # <- optional, by default Frigate tries to automatically detect resolution

    zones: {}
    review: {}

  stairway:
    enabled: true
    ffmpeg:
      inputs:
        - path:
            rtsp://root:[email protected]:554/axis-media/media.amp?streamprofile=hdwitnessSecondary
          roles:
            - detect
        - path:
            rtsp://root:[email protected]:554/axis-media/media.amp?streamprofile=hdwitnessPrimary
          roles:
            - record
    # detect:
    #   width: 640 # <- optional, by default Frigate tries to automatically detect resolution
    #   height: 360 # <- optional, by default Frigate tries to automatically detect resolution

    zones: {}
    review: {}

version: 0.15-0

model:
  #path: plus://d31dec2d1734b0f0dfdd21f67599e721
  path: plus://7ec1cf3edb785d6d442aa030c6a10af5

objects:
  filters:
    dog:
      min_score: .7
      threshold: .9
    cat:
      min_score: .65
      threshold: .8
    face:
      min_score: .7
    package:
      min_score: .65
      threshold: .9
    license_plate:
      min_score: .6
    amazon:
      min_score: .75
    ups:
      min_score: .75
    fedex:
      min_score: .75
    person:
      min_score: .65
      threshold: .85
    car:
      min_score: .65
      threshold: .85
  track:
    - person
    - car
    - dog
    - cat
    - face
    - package
    - umbrella
    - waste_bin

detectors:
  coral:
    type: edgetpu
    device: usb

record:
  enabled: true
  retain:
    days: 7
    mode: motion
  alerts:
    retain:
      days: 30
  detections:
    retain:
      days: 30
snapshots:
  enabled: true
  retain:
    default: 30

ffmpeg:
  output_args:
    record: preset-record-generic-audio-aac
semantic_search:
  enabled: true
  reindex: false
  model_size: large

Relevant Frigate log output

Unable to get for now.

Relevant go2rtc log output

N/A

FFprobe output from your camera

N/A

Frigate stats

No response

Install method

Docker Compose

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    stop_grace_period: 30s # allow enough time to shut down the various services
    #    image: ghcr.io/blakeblackshear/frigate:stable
    #image: ghcr.io/blakeblackshear/frigate:b4acf4f-standard-arm64
    #image: ghcr.io/blakeblackshear/frigate:9c75123-standard-arm64 # crashes with some go2rtc index out of rnage issue
#    image: ghcr.io/blakeblackshear/frigate:039ab1c-standard-arm64
    #image: ghcr.io/blakeblackshear/frigate:47ad0ca-standard-arm64
#    image: ghcr.io/blakeblackshear/frigate:0b71cfa-standard-arm64
#    image: ghcr.io/blakeblackshear/frigate:ee8091b-standard-arm64
    #image: ghcr.io/blakeblackshear/frigate:0fc7999-standard-arm64
    #image: ghcr.io/blakeblackshear/frigate:eda52a3-standard-arm64
    #image: ghcr.io/blakeblackshear/frigate:5f77408-standard-arm64
    #image: ghcr.io/blakeblackshear/frigate:6294ce7-standard-arm64
    #image: ghcr.io/blakeblackshear/frigate:2137de3-standard-arm64
    #image: ghcr.io/blakeblackshear/frigate:d12c780-standard-arm64
    #image: ghcr.io/blakeblackshear/frigate:ed9c678-standard-arm64
    image: ghcr.io/blakeblackshear/frigate:ee816b2-standard-arm64

    shm_size: "64mb" # update for your cameras based on calculation above
    devices:
      - /dev/bus/usb:/dev/bus/usb  # Passes the USB Coral, needs to be modified for other versions
      #- /dev/apex_0:/dev/apex_0    # Passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      #- /dev/video11:/dev/video11  # For Raspberry Pi 4B
      #- /dev/dri/renderD128:/dev/dri/renderD128 # For intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home/victorhooi/frigate-docker/config:/config
      - /home/victorhooi/frigate-docker/storage:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "password"
      PLUS_API_KEY: ${PLUS_API_KEY}

Object Detector

Coral

Network connection

Wired

Camera make and model

Axis LV 3248

Screenshots of the Frigate UI's System metrics pages

No response

Any other information that may be helpful

No response

[NickM-27]

It's odd that there are multiple frigate.output processes as there should only be one. It would be helpful to see htop (it would need to be installed in the container) so we can see what specific threads are using CPU from this process.

[victorhooi]

@NickM-27 docker logs was hanging - but I was able to get the raw Docker log file from the filesystem 😀. Does that help, with the thread output?

Gist is here.

Excerpt is below - something about "Too many unprocessed recording segments in cache for <camera_name>"? Would that explain the hanging, and the 100% CPU?

{"log":"2024-12-02 00:12:52.980030029  [2024-12-02 00:12:52] frigate.record.maintainer      WARNING : Too many unprocessed recording segments in cache for bedroom. This likely indicates an issue with the detect stream, keeping the 6 most recent segments out of 7 and discarding the rest...\n","stream":"stdout","time":"2024-12-01T13:12:52.980206372Z"}
{"log":"2024-12-02 00:13:07.992016154  [2024-12-02 00:13:07] frigate.record.maintainer      WARNING : Too many unprocessed recording segments in cache for bedroom. This likely indicates an issue with the detect stream, keeping the 6 most recent segments out of 7 and discarding the rest...\n","stream":"stdout","time":"2024-12-01T13:13:07.992321378Z"}
{"log":"2024-12-02 00:13:18.000792477  [2024-12-02 00:13:18] frigate.record.maintainer      WARNING : Too many unprocessed recording segments in cache for bedroom. This likely indicates an issue with the detect stream, keeping the 6 most recent segments out of 7 and discarding the rest...\n","stream":"stdout","time":"2024-12-01T13:13:18.000952379Z"}
{"log":"2024-12-02 00:13:33.012195725  [2024-12-02 00:13:33] frigate.record.maintainer      WARNING : Too many unprocessed recording segments in cache for bedroom. This likely indicates an issue with the detect stream, keeping the 6 most recent segments out of 7 and discarding the rest...\n","stream":"stdout","time":"2024-12-01T13:13:33.012294586Z"}
{"log":"2024-12-02 00:13:43.020686907  [2024-12-02 00:13:43] frigate.record.maintainer      WARNING : Too many unprocessed recording segments in cache for bedroom. This likely indicates an issue with the detect stream, keeping the 6 most recent segments out of 7 and discarding the rest...\n","stream":"stdout","time":"2024-12-01T13:13:43.020821448Z"}

[NickM-27]

All I see is the usual jsmpeg streams that run all of the time, it is not clear what is actually using the CPU here

[victorhooi]

I ended up having to pull power to the server itself.

I can startup the Frigate containers again - but is there anything I should setup in advance to collect data etc, that would help with diagnostics?

(I can docker exec -it bash and install htop in the container - but I don't know if htop will respond if the container gets into that borked state? Is there something non-interactive that would enable you to see what happened afterwards?)

[NickM-27]

Not that I know of, at least not something that would be more likely to not hang

[victorhooi]

1. Would setting Frigate logs to debug help?

https://docs.frigate.video/configuration/advanced/#logging

If so - which modules should I set to debug? (The documentation page I linked to seems to suggest that list of Frigate modules isn't the full list).

2. I can also look at setting up Telegraf to collect some Docker system metrics/stats (Telegraf Docker input plugin) - at least until the point at which Frigate locks up the box, I guess.

3. After that, I guess I'd have to dip into tracing frameworks, or something like that (e.g. OpenTelemetry) - although I guess that would be more involved, and would require changes on the Frigate side to do properly.

I'm a bit loathe to start up Frigate again, aha, but I also want to get to the bottom of this issue.

Let me know if there's anything else I can gather that would be helpful for you =).

[leccelecce]

Are you sure frigate.output is the problem? Your top output suggests it's using 100%...of 4 cores. If you really have 256 cores that shouldn't remotely be an issue.

The load average of 145 suggests something else is eating CPU, possibly something in the kernel. What shows up on dmesg on the host?

[victorhooi]

Yes - I did think it was a bit weird that only 4 cores were pegged at 100% according to top.

Good tip about checking previous dmesg - I'd already had to pull power on the host - but thankfully Debian uses journalctl =), I was able to pull from previous boot with sudo journalctl --dmesg -b -1.

In there, I did see a bunch of messages about stuck CPUs - e.g.:

Dec 02 20:38:27 ampere-01 kernel: watchdog: BUG: soft lockup - CPU#23 stuck for 69204s! [frigate.output:2351605]
Dec 02 20:38:27 ampere-01 kernel: Modules linked in: tls bluetooth sha3_generic jitterentropy_rng drbg ansi_cprng ecdh_generic rfkill ecc mst_pciconf(OE) dm_mod dax uas usb_storage cdc_ether usbnet mii sha51>
Dec 02 20:38:27 ampere-01 kernel:  crc64_rocksoft xhci_pci crc64 libata xhci_hcd mlxfw crc_t10dif scsi_mod dimlib crct10dif_generic usbcore igb pci_hyperv_intf scsi_common crct10dif_ce usb_common i2c_algo_bi>
Dec 02 20:38:27 ampere-01 kernel: CPU: 23 PID: 2351605 Comm: frigate.output Tainted: P           OEL     6.10.11-arm64 #1  Debian 6.10.11-1
Dec 02 20:38:27 ampere-01 kernel: Hardware name: GIGABYTE R282-P91-00/MP92-FS0-00, BIOS F33j (SCP: 2.10.20230517) 08/26/2024
Dec 02 20:38:27 ampere-01 kernel: pstate: 20400009 (nzCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Dec 02 20:38:27 ampere-01 kernel: pc : contpte_ptep_get+0x48/0x88
Dec 02 20:38:27 ampere-01 kernel: lr : __handle_mm_fault+0x890/0xb58
Dec 02 20:38:27 ampere-01 kernel: sp : ffff8000de5237b0
Dec 02 20:38:27 ampere-01 kernel: x29: ffff8000de523850 x28: ffff40270051dc80 x27: ffff40270051dc80
Dec 02 20:38:27 ampere-01 kernel: x26: 0000000200100073 x25: ffffd21b5928d000 x24: ffff0816f81da800
Dec 02 20:38:27 ampere-01 kernel: x23: ffff08060a814000 x22: ffff082229d64000 x21: 0000000000000214
Dec 02 20:38:27 ampere-01 kernel: x20: ffff081186997ef0 x19: 00f040213187af83 x18: ffff8000de5236f8
Dec 02 20:38:27 ampere-01 kernel: x17: 000000000000000a x16: ffff082a330dc380 x15: ffff082a330dc3d0
Dec 02 20:38:27 ampere-01 kernel: x14: 0008000000000000 x13: ffff082a330dc400 x12: 0008000000000080
Dec 02 20:38:27 ampere-01 kernel: x11: 0010000000000001 x10: ff7c000000000bff x9 : ffffd21b58270e68
Dec 02 20:38:27 ampere-01 kernel: x8 : 0008000000000000 x7 : ffff082a330dc400 x6 : 0008000000000080
Dec 02 20:38:27 ampere-01 kernel: x5 : 0000000000000080 x4 : 00f040213187af83 x3 : 00f040213187af83
Dec 02 20:38:27 ampere-01 kernel: x2 : ffff082a330dc3d0 x1 : 00f040213187af83 x0 : 00f040213187af83
Dec 02 20:38:27 ampere-01 kernel: Call trace:
Dec 02 20:38:27 ampere-01 kernel:  contpte_ptep_get+0x48/0x88
Dec 02 20:38:27 ampere-01 kernel:  handle_mm_fault+0x16c/0x290
Dec 02 20:38:27 ampere-01 kernel:  do_page_fault+0x198/0x4c0
Dec 02 20:38:27 ampere-01 kernel:  do_mem_abort+0x4c/0xa8
Dec 02 20:38:27 ampere-01 kernel:  el1_abort+0x40/0x68
Dec 02 20:38:27 ampere-01 kernel:  el1h_64_sync_handler+0xa4/0xe8
Dec 02 20:38:27 ampere-01 kernel:  el1h_64_sync+0x64/0x68
Dec 02 20:38:27 ampere-01 kernel:  strnlen_user+0x84/0x198
Dec 02 20:38:27 ampere-01 kernel:  do_execveat_common.isra.0+0x14c/0x240
Dec 02 20:38:27 ampere-01 kernel:  __arm64_sys_execve+0x48/0x68
Dec 02 20:38:27 ampere-01 kernel:  invoke_syscall+0x78/0x108
Dec 02 20:38:27 ampere-01 kernel:  el0_svc_common.constprop.0+0x48/0xf0
Dec 02 20:38:27 ampere-01 kernel:  do_el0_svc+0x24/0x38
Dec 02 20:38:27 ampere-01 kernel:  el0_svc+0x3c/0x128
Dec 02 20:38:27 ampere-01 kernel:  el0t_64_sync_handler+0x120/0x130
Dec 02 20:38:27 ampere-01 kernel:  el0t_64_sync+0x190/0x198

or

Dec 02 20:38:47 ampere-01 kernel: watchdog: BUG: soft lockup - CPU#203 stuck for 69223s! [frigate.output:2351594]
Dec 02 20:38:47 ampere-01 kernel: Modules linked in: tls bluetooth sha3_generic jitterentropy_rng drbg ansi_cprng ecdh_generic rfkill ecc mst_pciconf(OE) dm_mod dax uas usb_storage cdc_ether usbnet mii sha51>
Dec 02 20:38:47 ampere-01 kernel:  crc64_rocksoft xhci_pci crc64 libata xhci_hcd mlxfw crc_t10dif scsi_mod dimlib crct10dif_generic usbcore igb pci_hyperv_intf scsi_common crct10dif_ce usb_common i2c_algo_bi>
Dec 02 20:38:47 ampere-01 kernel: CPU: 203 PID: 2351594 Comm: frigate.output Tainted: P           OEL     6.10.11-arm64 #1  Debian 6.10.11-1
Dec 02 20:38:47 ampere-01 kernel: Hardware name: GIGABYTE R282-P91-00/MP92-FS0-00, BIOS F33j (SCP: 2.10.20230517) 08/26/2024
Dec 02 20:38:47 ampere-01 kernel: pstate: 40400009 (nZcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Dec 02 20:38:47 ampere-01 kernel: pc : contpte_ptep_get_lockless+0xa4/0xf0
Dec 02 20:38:47 ampere-01 kernel: lr : __handle_mm_fault+0x570/0xb58
Dec 02 20:38:47 ampere-01 kernel: sp : ffff8000dfccb960
Dec 02 20:38:47 ampere-01 kernel: x29: ffff8000dfccba00 x28: ffff08047b3fa500 x27: ffff08047b3fa500
Dec 02 20:38:47 ampere-01 kernel: x26: 0000000200100073 x25: ffffd21b5928d000 x24: ffff081a65474100
Dec 02 20:38:47 ampere-01 kernel: x23: ffff07ffc9666000 x22: ffff080831f8b000 x21: 0000000000000214
Dec 02 20:38:47 ampere-01 kernel: x20: ffff400a9ef05a20 x19: 00000000000001d0 x18: ffff8000dfccb8a8
Dec 02 20:38:47 ampere-01 kernel: x17: 000000000000000a x16: ffff0800b9811380 x15: ffff0800b98113d0
Dec 02 20:38:47 ampere-01 kernel: x14: 0008000000000000 x13: ffff0800b9811400 x12: 0008000000000080
Dec 02 20:38:47 ampere-01 kernel: x11: 0010000000000001 x10: ff7c000000000bff x9 : 0000000000000080
Dec 02 20:38:47 ampere-01 kernel: x8 : 00f040213187af83 x7 : 0070000000000b83 x6 : 0000000402131876
Dec 02 20:38:47 ampere-01 kernel: x5 : 0000000402131876 x4 : 0070000000000b83 x3 : ffff0800b98113b8
Dec 02 20:38:47 ampere-01 kernel: x2 : 00f040213187af03 x1 : 00f0402131876b83 x0 : 00f040213187af83
Dec 02 20:38:47 ampere-01 kernel: Call trace:
Dec 02 20:38:47 ampere-01 kernel:  contpte_ptep_get_lockless+0xa4/0xf0
Dec 02 20:38:47 ampere-01 kernel:  handle_mm_fault+0x16c/0x290
Dec 02 20:38:47 ampere-01 kernel:  do_page_fault+0x198/0x4c0
Dec 02 20:38:47 ampere-01 kernel:  do_mem_abort+0x4c/0xa8
Dec 02 20:38:47 ampere-01 kernel:  el1_abort+0x40/0x68
Dec 02 20:38:47 ampere-01 kernel:  el1h_64_sync_handler+0xa4/0xe8
Dec 02 20:38:47 ampere-01 kernel:  el1h_64_sync+0x64/0x68
Dec 02 20:38:47 ampere-01 kernel:  strnlen_user+0x84/0x198
Dec 02 20:38:47 ampere-01 kernel:  do_execveat_common.isra.0+0x14c/0x240
Dec 02 20:38:47 ampere-01 kernel:  __arm64_sys_execve+0x48/0x68
Dec 02 20:38:47 ampere-01 kernel:  invoke_syscall+0x78/0x108
Dec 02 20:38:47 ampere-01 kernel:  el0_svc_common.constprop.0+0x48/0xf0
Dec 02 20:38:47 ampere-01 kernel:  do_el0_svc+0x24/0x38
Dec 02 20:38:47 ampere-01 kernel:  el0_svc+0x3c/0x128
Dec 02 20:38:47 ampere-01 kernel:  el0t_64_sync_handler+0x120/0x130
Dec 02 20:38:47 ampere-01 kernel:  el0t_64_sync+0x190/0x198

or some migration related ones:

Dec 02 01:20:35 ampere-01 kernel: watchdog: BUG: soft lockup - CPU#156 stuck for 4493s! [migration/156:806]
Dec 02 01:20:35 ampere-01 kernel: Modules linked in: tls bluetooth sha3_generic jitterentropy_rng drbg ansi_cprng ecdh_generic rfkill ecc mst_pciconf(OE) dm_mod dax uas usb_storage cdc_ether usbnet mii sha51>
Dec 02 01:20:35 ampere-01 kernel:  crc64_rocksoft xhci_pci crc64 libata xhci_hcd mlxfw crc_t10dif scsi_mod dimlib crct10dif_generic usbcore igb pci_hyperv_intf scsi_common crct10dif_ce usb_common i2c_algo_bi>
Dec 02 01:20:35 ampere-01 kernel: CPU: 156 PID: 806 Comm: migration/156 Tainted: P           OEL     6.10.11-arm64 #1  Debian 6.10.11-1
Dec 02 01:20:35 ampere-01 kernel: Hardware name: GIGABYTE R282-P91-00/MP92-FS0-00, BIOS F33j (SCP: 2.10.20230517) 08/26/2024
Dec 02 01:20:35 ampere-01 kernel: Stopper: multi_cpu_stop+0x0/0x190 <- migrate_swap+0xbc/0x180
Dec 02 01:20:35 ampere-01 kernel: pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Dec 02 01:20:35 ampere-01 kernel: pc : rcu_momentary_dyntick_idle+0x48/0xb8
Dec 02 01:20:35 ampere-01 kernel: lr : multi_cpu_stop+0x100/0x190
Dec 02 01:20:35 ampere-01 kernel: sp : ffff80009100bd60
Dec 02 01:20:35 ampere-01 kernel: x29: ffff80009100bd60 x28: 0000000000000001 x27: 0000000000000000
Dec 02 01:20:35 ampere-01 kernel: x26: 0000000000000000 x25: 00000000ffffffff x24: 0000000000000000
Dec 02 01:20:35 ampere-01 kernel: x23: 0000000000000001 x22: ffffd21b58d9cac8 x21: ffff8001d5bf3964
Dec 02 01:20:35 ampere-01 kernel: x20: ffff8001d5bf3940 x19: 0000000000000001 x18: 000000000000000e
Dec 02 01:20:35 ampere-01 kernel: x17: 0000000000000002 x16: 0000000000000001 x15: 0000000000000001
Dec 02 01:20:35 ampere-01 kernel: x14: 0000000000000006 x13: 0000000000000000 x12: ffff080c98b453b8
Dec 02 01:20:35 ampere-01 kernel: x11: ffffd21b59b2fa48 x10: 0000000000000bd0 x9 : ffffd21b580e0718
Dec 02 01:20:35 ampere-01 kernel: x8 : ffff3fff86e8b130 x7 : 0000000000000000 x6 : 00000000000000d0
Dec 02 01:20:35 ampere-01 kernel: x5 : 00000000410fd0c0 x4 : ffff6e22e6859000 x3 : ffff80009100bd60
Dec 02 01:20:35 ampere-01 kernel: x2 : 00000000fa42ee24 x1 : ffffd21b59a76980 x0 : 00000000fa42ee2c
Dec 02 01:20:35 ampere-01 kernel: Call trace:
Dec 02 01:20:35 ampere-01 kernel:  rcu_momentary_dyntick_idle+0x48/0xb8
Dec 02 01:20:35 ampere-01 kernel:  multi_cpu_stop+0x100/0x190
Dec 02 01:20:35 ampere-01 kernel:  cpu_stopper_thread+0xc8/0x190
Dec 02 01:20:35 ampere-01 kernel:  smpboot_thread_fn+0x140/0x190
Dec 02 01:20:35 ampere-01 kernel:  kthread+0xe8/0xf8
Dec 02 01:20:35 ampere-01 kernel:  ret_from_fork+0x10/0x20
Dec 02 01:20:39 ampere-01 kernel: watchdog: BUG: soft lockup - CPU#125 stuck for 4202s! [migration/125:649]
Dec 02 01:20:39 ampere-01 kernel: Modules linked in: tls bluetooth sha3_generic jitterentropy_rng drbg ansi_cprng ecdh_generic rfkill ecc mst_pciconf(OE) dm_mod dax uas usb_storage cdc_ether usbnet mii sha51>
Dec 02 01:20:39 ampere-01 kernel:  crc64_rocksoft xhci_pci crc64 libata xhci_hcd mlxfw crc_t10dif scsi_mod dimlib crct10dif_generic usbcore igb pci_hyperv_intf scsi_common crct10dif_ce usb_common i2c_algo_bi>
Dec 02 01:20:39 ampere-01 kernel: CPU: 125 PID: 649 Comm: migration/125 Tainted: P           OEL     6.10.11-arm64 #1  Debian 6.10.11-1
Dec 02 01:20:39 ampere-01 kernel: Hardware name: GIGABYTE R282-P91-00/MP92-FS0-00, BIOS F33j (SCP: 2.10.20230517) 08/26/2024
Dec 02 01:20:39 ampere-01 kernel: Stopper: multi_cpu_stop+0x0/0x190 <- migrate_swap+0xbc/0x180
Dec 02 01:20:39 ampere-01 kernel: pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Dec 02 01:20:39 ampere-01 kernel: pc : rcu_momentary_dyntick_idle+0x40/0xb8
Dec 02 01:20:39 ampere-01 kernel: lr : multi_cpu_stop+0x100/0x190
Dec 02 01:20:39 ampere-01 kernel: sp : ffff800090b23d60
Dec 02 01:20:39 ampere-01 kernel: x29: ffff800090b23d60 x28: 0000000000000001 x27: 0000000000000000
Dec 02 01:20:39 ampere-01 kernel: x26: 0000000000000000 x25: 00000000ffffffff x24: 0000000000000000
Dec 02 01:20:39 ampere-01 kernel: x23: 0000000000000001 x22: ffffd21b58d9c930 x21: ffff8001d5163964
Dec 02 01:20:39 ampere-01 kernel: x20: ffff8001d5163940 x19: 0000000000000001 x18: 000000000000000e
Dec 02 01:20:39 ampere-01 kernel: x17: 0000000000000002 x16: 0000000000000004 x15: 0000000000000004
Dec 02 01:20:39 ampere-01 kernel: x14: 0000000000000006 x13: 0000000000000000 x12: ffff07ff8df8d938
Dec 02 01:20:39 ampere-01 kernel: x11: ffffd21b59b2fa48 x10: 0000000000000bd0 x9 : ffffd21b580e0718
Dec 02 01:20:39 ampere-01 kernel: x8 : ffff07ff89e0c3b0 x7 : 0000000000000000 x6 : 00000000000000d0
Dec 02 01:20:39 ampere-01 kernel: x5 : 00000000410fd0c0 x4 : ffff36232653b000 x3 : ffff800090b23d60
Dec 02 01:20:39 ampere-01 kernel: x2 : 00000000d9db7214 x1 : ffffd21b59a76980 x0 : ffff083e7ffac0e8
Dec 02 01:20:39 ampere-01 kernel: Call trace:
Dec 02 01:20:39 ampere-01 kernel:  rcu_momentary_dyntick_idle+0x40/0xb8
Dec 02 01:20:39 ampere-01 kernel:  multi_cpu_stop+0x100/0x190

All of the above messages seem to be related to either the frigate.output process, or the "migration" process? (which I think is for distributing work among cores, right?)

Is it possible that this frigate.output, combined with whatever migration was doing, were able to lock up all the CPU cores on the box?

If so - what should I setup on the box, to monitor it, to help debug the issue?

And would setting CPU constraints prevent the box from being locked up here - or would the kernel's migration process still lock it up anyway?

[leccelecce]

Afraid I have no knowledge of ARM or super high core count systems, but this definitely shows the kernel knows something bad is going on. I can imagine all kinds of weird edge cases with 256 CPUs and a USB Coral. It shouldn't really be Frigate's fault as userspace Python app.

I would try limiting the Frigate container to 8 cores and see if that helps. I would do this using affinity i.e. cpuset 0-7 rather than just cpus: 8 to try and minimise jumping around cores.