Detect stream resolution and CPU usage

[lukasig]

Hi

I use frigate with three Reolink cameras.
Reolink cameras have 3 streaming resolutions:

main: 2560x1920
ext: 896x672
sub: 640x480

The issue is that the the two lower resolution streams are not enough for "detect".
My question is how to best set up the detect stream. Should I use the large (main) stream also for "detect" and reduce the detect with and height to something like 1920x1440 in frigate config? Will that help in any way the CPU usage? Or better use the native 2560x1920 since it would not require any resizing?

Also, since frigate 12 has go2rtc, is there a better way of setting this up through go2rtc?
Trying to have the lowest CPU usage and have a more capable object detection.

thanks

[NickM-27]

This is what I do as well when my camera does not provide a suitable resolution. You can have your GPU do the scaling so you will have reduced CPU usage (because motion detection does not run on as high of a resolution) but not use CPU for scaling. I do this in go2rtc so the substream is available for general streaming as well

as an example:

go2rtc:
  streams:
    backyard_cam: 
      - rtsp://{FRIGATE_RTSP_USER}:{FRIGATE_BACKYARD_PW}@192.168.50.155/Preview_01_main
    backyard_cam_sub:
      - "ffmpeg:backyard_cam#video=h264#width=1280#height=960#raw=-fpsmax 10#hardware=cuda"

[NickM-27]

the only reason you would do it in go2rtc is if you are going to use it outside of frigate and the sub stream from the camera won't work for that case

[NickM-27]

also worth mentioning I don't do this anymore since I don't have reolink cameras anymore, my dahua cameras have configurable sub streams built in

[ronbruins]

the only reason you would do it in go2rtc is if you are going to use it outside of frigate and the sub stream from the camera won't work for that case

So in both cases GPU is used if vaapi is set in either the ffmpeg line or in the frigate config. I have 4 Reolink cams. 3 and a doorbell, the sub stream is too small (640 at best) so I'm using the detect in frigate. Then I saw this. Thanks for the reply.

[NickM-27]

if you have reolink cameras, they offer a _ext stream which is typically somewhere around 900x600, so I just recommend using that

[ronbruins]

if you have reolink cameras, they offer a _ext stream which is typically somewhere around 900x600, so I just recommend using that

Unfortunately my E1 outdoor and track mix don't seem to have a _ext stream.

[Klorins]

This was very helpful for me as well, thank you.

Just want to add that if you want it to pass through the audio at the same time it would look like this:

go2rtc:
  streams:
    backyard_cam: 
      - rtsp://{FRIGATE_RTSP_USER}:{FRIGATE_BACKYARD_PW}@192.168.50.155/Preview_01_main
    backyard_cam_sub:
      - "ffmpeg:backyard_cam#video=h264#width=1280#height=720#audio=copy#raw=-fpsmax 10#hardware=vaapi"

I plan to use only the 720p stream for now at least. Then its nice to have the audio as well.

[MaBeniu]

Fine tuning the setup.
What benefits are gained when using only high resolution stream? My hardware is capable to handle 1 high resolution stream per camera for motion detection, object detection, recording and with go2rtc restreaming to HA and this resulting in approx 36% CPU load on average but with some occasional peaks getting to 80%. So far I am aware only for one benefit, as per documentation it states that small objects would be better detected with higher resolution due to first image is cropped and only afterwards is resized which I found to actually prove itself in practice. I finally am getting cat detected which was never a case with low res detection. This object I am interested to capture in 6 out of 8 cameras. So I am thinking maybe I shall use two different streams on the rest 2 cams if there is no other value and save me few CPU percent :)

What about motion detection is there any benefit for detecting motion when using high resolution stream? I already tried few recommendations how to improve motion detection but now from the stream resolution perspective was wondering what is better high res with 4fps or maybe low res with 10 fps to detect motion?
Any other potential benefits?

I am on Frigate 13 rc1 version

[NickM-27]

1. If you have a gpu then there is the middle ground option where you can use the high res stream but set detect to something like 1280x720. The GPU will be used to do the scaling and CPU won't be taxed so hard.

2. There are only downsides for motion detection, higher resolution can cause issues unless you also increase the frame height for motion detection but then that further increases cpu usage.

[MaBeniu]

thanks for feedback! could you elaborate a little more on the 1st option. here is one camera config example:

  kiemas:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/rtsp_kiem
          input_args: preset-rtsp-restream
          roles:
            - record
            - detect
    detect:
#      width: 640 # low res stream1 option
#      height: 480 # low res stream1 option
#      width: 2560 #high res stream0 option
#      height: 1440 #high res stream0 option
      fps: 4
    objects:
      track:
        - person
        - cat
    live:
      stream_name: rtsp_kiem

so if I specify resolution parameters as:

detect:
   width: 1280
   heigh: 720

that would detect motion based on original resolution 2560x1440 then motion area would be cropped to 1280x720 and only then rescaled to 300x300 (edgetpu default model res)???
my main point is that default model really lacks accuracy for objects like cats and the only option to get any value of that default model is to give it as higher res as possible for object detection hoping it would be able to identify. in my case cats can appear from 1m (if they climb on the car roof which gets close to camera) or 15meters if it is in the back of the area.

[NickM-27]

that would detect motion based on original resolution 2560x1440 then motion area would be cropped to 1280x720 and only then rescaled to 300x300 (edgetpu default model res)???

no, you should not be thinking about it in that way. When you set detect to 1280x720 that downsizes the stream before frigate sees it at all. also, frigate does not run object detection on the entire camera frame, the model resolution (ex: 300x300) comes from a region (portion of the image) that is cropped from the camera frame.

[cchewanik]

@NickM-27 thank you for this information I was confused about this also. In general is there a way for me to know that frigate / ffmpeg is downsizing a stream or performing additional operations to the stream because I haven't set the resolution correctly or have the wrong FPS...I notice certain cameras consume more of my CPU but I just assume it's because of resolution or higher FPS. Just curious if there is something I can check to make sure I haven't messed up my frigate config vs camera settings?

[NickM-27]

besides checking what you have in the config there is no automatic way, the biggest thing is to just make sure you use hwaccel with a gpu

[Thorsten1982]

Hello, I have two Reolink E1 Outdoor cameras and the POE Doorbell. If I change the substream on both cameras as described here, the Doorbell is shown as offline and the message "no frames have been received" appears. This does not happen with the E1 Outdoor camera. If you switch back to the "normal" substream configuration on the Doorbell, it is displayed normally again.

`go2rtc:
streams:
Haustuer:
- "ffmpeg:http://192.168.188.32/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=admin&password=xxx#video=copy#audio=copy#audio=opus"
Haustuer_camera_sub:
- ffmpeg:http://192.168.188.32/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=admin&password=xxx
#- "ffmpeg:Haustuer#video=h264#width=1280#height=960#raw=-fpsmax 10#hardware=cuda"

Einfahrt:
  - "ffmpeg:http://192.168.188.70/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=admin&password=xxx#video=copy#audio=copy#audio=opus"
Einfahrt_sub:
  #- ffmpeg:http://192.168.188.70/flv?port=1935&app=bcs&stream=channel10_ext.bcs&user=admin&password=xxx
  - "ffmpeg:Einfahrt#video=h264#width=1280#height=960#raw=-fpsmax 10#hardware=cuda"

cameras:
Haustuer: # <--- this will be changed to your actual camera later
enabled: true
ffmpeg:
inputs:
- path: rtsp://127.0.0.1:8554/Haustuer
input_args: preset-rtsp-restream
roles:
- record
- path: rtsp://127.0.0.1:8554/Haustuer_camera_sub
input_args: preset-rtsp-restream
roles:
- detect`

frigate-logs.txt