Why does traffic analysis output differ from the video in readme file?

[INF800]

Search before asking

• I have searched the Supervision issues and found no similar feature requests.

Question

I tried running the same code in readme to generate video outputs but accuracy is not upto mark.

Here are videos with different thresholds:

https://drive.google.com/drive/folders/1TFcEJcSvVSQXaMEYQTnhw2s-QNXP-LOz?usp=sharing

Additional

No response

[INF800]

FYI, I used roboflow API instead of running it locally.

cc: @onuralpszr

[INF800]

I see that default model id is vehicle-count-in-drone-video/6 was it supposed to be 8 instead of 6

cc: @onuralpszr

[INF800]

8 is giving resource error

[onuralpszr]

I see that default model id is vehicle-count-in-drone-video/6 was it supposed to be 8 instead of 6

cc: @onuralpszr

That is dataset not model so default 6 is correct

[onuralpszr]

8 is giving resource error

v8 as model not exist so it is correct

[onuralpszr]

FYI, I used roboflow API instead of running it locally.

cc: @onuralpszr

Out of curiosity what is your inference version ?

[INF800]

It's the same as in requirements.txt file

[INF800]

I got this warning message before running the script:

[10/14/24 07:10:56] WARNING  Your inference package version 0.9.17 is out of date! Please upgrade to version 0.23.0 of inference for    __init__.py:35
                             the latest features and bug fixes by running `pip install --upgrade inference`.

but ignored it anyway because that's what is being used in requirements.txt

cc: @onuralpszr

[INF800]

Just now I trained a new model using my custom data (just 2 4k images) and its results are wayyy better that current one

[INF800]

Just now I trained a new model using my custom data (just 2 4k images) and its results are wayyy better that current one

here are the results: https://drive.google.com/drive/folders/1bzHpSBSRi3nbFa7i-BdMZjTRbpqnp3nc?usp=sharing

ccL @onuralpszr

[onuralpszr]

@PawelPeczek-Roboflow can you help me out on this one please I was able to generate the same problematic video and I tried version in written req.txt and I tried latest version of inference to test and my thought was current code is not using conf and iou values then I used latest version with parameters I checked in documentation and still same results. Either model in roboflow universe problematic or inference code doing something weird

Latest version of inference changes I made in example code (code snippet not entire code)

from inference import get_model
...
self.model = get_model(model_id=model_id, api_key=roboflow_api_key)
...
results = self.model.infer(
    image=frame, confidence=self.conf_threshold, iou_threshold=self.iou_threshold
)[0]

cc @LinasKo
cc @SkalskiP (since you wrote initially you might be have better understanding of model side)

[onuralpszr]

I got this warning message before running the script:

[10/14/24 07:10:56] WARNING  Your inference package version 0.9.17 is out of date! Please upgrade to version 0.23.0 of inference for    __init__.py:35
                             the latest features and bug fixes by running `pip install --upgrade inference`.

but ignored it anyway because that's what is being used in requirements.txt

cc: @onuralpszr

I tested on both cases yes and I did the same thing too

[onuralpszr]

Just now I trained a new model using my custom data (just 2 4k images) and its results are wayyy better that current one

here are the results: https://drive.google.com/drive/folders/1bzHpSBSRi3nbFa7i-BdMZjTRbpqnp3nc?usp=sharing

ccL @onuralpszr

You used inference local model run right ?

[INF800]

Just now I trained a new model using my custom data (just 2 4k images) and its results are wayyy better that current one

here are the results: https://drive.google.com/drive/folders/1bzHpSBSRi3nbFa7i-BdMZjTRbpqnp3nc?usp=sharing
ccL @onuralpszr

You used inference local model run right ?

Nope, used Roboflow API with the same inference code as is with same version dependencies. The only change that I made is - replacing the model id with my newly trained model ID.

Based on this I think everything working just fine at code level.

[onuralpszr]

Just now I trained a new model using my custom data (just 2 4k images) and its results are wayyy better that current one

here are the results: https://drive.google.com/drive/folders/1bzHpSBSRi3nbFa7i-BdMZjTRbpqnp3nc?usp=sharing
ccL @onuralpszr

You used inference local model run right ?

Nope, used Roboflow API with the same inference code as is with same version dependencies. The only change that I made is - replacing the model id with my newly trained model ID.

Based on this I think everything working just fine at code level.

For code level yes It should be fine that I am sure I am converting this to Q&A discussion and continue from there.

Also you could you share your ID if it is public please

[INF800]

Yes the one that I used for this cusom-data/2 (with the typo).

cc: @onuralpszr

I'm using the model for running some inference jobs right now (15mins more remaining).

[PawelPeczek-Roboflow]

@onuralpszr do not really know the context, could you elaborate a little bit more?

[LinasKo]

Hi @INF800 👋

Based on the video, it's likely that InferenceSlicer was used to help detect small objects. Have you tried using it?
(guide, docs)

Next, it would help if you could share some code you used to run the model. Specifically, I'm curious what code you ran that produced poor results initially.

[onuralpszr]

@onuralpszr do not really know the context, could you elaborate a little bit more?

In basic context, inference example in here https://github.com/roboflow/supervision/tree/develop/examples/traffic_analysis doesn't show output of video in readme.

[PawelPeczek-Roboflow]

ok, taken a look - first of all, the model trained for ultralytics is a different class of model than the pointed one from Roboflow - not sure if the results were comparable in first place

[LinasKo]

Hey @PawelPeczek-Roboflow, what do you mean by "a different class of model"? What kind of difference are we looking at? Model family, size, or a different dataset?

[PawelPeczek-Roboflow]

not fully sure about details, but the weights size differ:

• one order of magnitude in favour of ultralytics model (it also runs 5x slower)
• additionally model from universe was trained on 100 images - we shall not expect it to be robust

[PawelPeczek-Roboflow]

also the Roboflow model was not that great in the metrics of original dataset

[LinasKo]

Curious.

@INF800, data points to either the video being used as an example, or additional tool use such as InferenceSlicer. For comparable results, I suggest using or fine-tuning one of yolov8 models from inference. If the objects are this small, I also suggest Inference Slicer (guide, docs).

I'll mark this as potentially misleading and investigate further later on. Thank you for reporting it.