Allow ByteTrack to track detections without IDs

[LinasKo]

Allow ByteTrack to track detections without IDs

Tip

Hacktoberfest is calling! Whether it's your first PR or your 50th, you’re helping shape the future of open source. Help us build the most reliable and user-friendly computer vision library out there! 🌱

For a long time, all of models we support would return a class_id which we set inside Detections (class_id is an array of N integers). However, we've seen more and more models where that isn't provided.

ByteTrack allows you to track objects in the scene. It uses the class and location of the detection to figure out where it is. It can track objects where all class IDs are the same. However, if you try passing something without an ID, it will fail. See this Colab.

Let's change ByteTrack so it can track detections, even if they didn't have a class_id. I see two ways of doing this:

1. Change the algorithm such that class_ids are not required
2. Set an internal class ID such as -1.

As a result, I expect the videos produced in the Colab to be visually indistinguishable.

It would also be useful to test on a video where some frames feature no detections (you can simulate with detections = sv.Detections.empty())

Also, check what happens when some detections have class_id and some do not.

---

Helpful links:

• Contribution guide
• ByteTrack: docs, code
• Detections: docs, code
• Supervision Cheatsheet
• Colab Starter Template
• Prior metrics test Colab

[LinasKo]

Contribution guidelines

If you would like to make a contribution, please check that no one else is assigned already. Then leave a comment such as "Hi, I would like to work on this issue". We're happy to answer any questions about the task even if you choose not to contribute.

Testing

Please share a Google Colab with minimal code to test the new feature. We know it's additional work, but it will speed up the review process. You may use the Starter Template. The reviewer must test each change. Setting up a local environment to do this is time-consuming. Please ensure that Google Colab can be accessed without any issues (make it public). Thank you! 🙏

[Kadermiyanyedi]

Hi, I would like to work on this issue. :)

[onuralpszr]

@Kadermiyanyedi task is yours, good luck ;)

[Kadermiyanyedi]

Hi,

I apologize for the delay. I couldn't work on the issue last week due to a busy schedule, but I've started it now. I'll submit a PR in a few days.

Thank you for your patience!

[LinasKo]

Hi @Kadermiyanyedi 👋

Thanks for letting me know! Feel free to take your time.

Your submissions are always of a very high quality so I'm happy either way, even if it takes longer :)

[AyuK03]

Hi, I would like to work on this issue

[onuralpszr]

Hi, I would like to work on this issue

This issue has taken already and if something changed we can but for now @Kadermiyanyedi is going the task, thank you for understanding

[LinasKo]

Hi @Kadermiyanyedi,

Quick note, there's been a few updates on the tracker today:

• I merged let ByteTrack to maintain track ID per instance #1528, which changes the behaviour when more than one ByteTrack is used simultaneously.
• I opened ByteTrack: Remove BaseTrack, refactor, add types, remove dead code, scope shared Kalman #1603 which involves some hefty refactoring, but the behaviour is the same.

I expect some merge conflicts; I'l leave it up to you if you want to handle them. If it proves beyond what you planned for this PR - feel free to complete the task with the current base, and I'll resolve any clashes.

[LinasKo]

Another update: It may be possible to just remove class_ids from STracks entirely. I didn't see it being used anywhere, nor does it appear in the original ByteTrack implementation.

[Kadermiyanyedi]

If it's not used anywhere, we can remove it.

I initialized a new NumPy array filled with -1 values in case the detections.class_id is None. The new array has the same length as detections.xyxy. The code runs correctly, but I'm not seeing the detected objects in the output, and I'm currently investigating this issue.

If we don't need the class id, I will remove and open PR quickly.

def detections2boxes(detections: Detections) -> np.ndarray:
    num_rows = detections.xyxy.shape[0]

    class_ids = np.full(num_rows, -1)
    if detections.class_id is not None:
        class_ids = detections.class_id
    return np.hstack(
        (
            detections.xyxy,
            detections.confidence[:, np.newaxis],
            class_ids[:, np.newaxis],
        )
    )

[LinasKo]

I recall seeing -1 as a special temporary value somewhere.
Are detections also gone when you use -2, or some large positive integer?

[Kadermiyanyedi]

I recall seeing -1 as a special temporary value somewhere. Are detections also gone when you use -2, or some large positive integer?

unfortunately it is same