How to increase object tracking speed (Yolov4 DeepSort)

Question

I recently started to get interested in computer vision technology and went through a couple of tutorials to solve a few problems in my business. There are 4 buildings in the area and the problem is to control the number of people entering and exiting them. And also it is necessary to take into account the movement of service personnel.

I tried using the following repository to solve these problems:

https://github.com/theAIGuysCode/yolov4-deepsort

And it seems to me that this will solve my problem. But now there is a question of processing speed of video recordings from CCTV cameras. I tried to run a ten second video fragment and the script completed in 211 seconds. Which in my opinion is very long.

What can I do to improve the processing speed? Tell me where to look for the answer.

Error when trying to install openvino

Building wheels for collected packages: tokenizers
  Building wheel for tokenizers (pyproject.toml) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3.6 /home/baurzhan/.local/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpr56p_xyt
       cwd: /tmp/pip-install-pfzy6bre/tokenizers_dce0e65cae1e4e7c9325570d12cd6d63
  Complete output (51 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/tokenizers
  copying py_src/tokenizers/__init__.py -> build/lib.linux-x86_64-3.6/tokenizers
  creating build/lib.linux-x86_64-3.6/tokenizers/models
  copying py_src/tokenizers/models/__init__.py -> build/lib.linux-x86_64-3.6/tokenizers/models
  creating build/lib.linux-x86_64-3.6/tokenizers/decoders
  copying py_src/tokenizers/decoders/__init__.py -> build/lib.linux-x86_64-3.6/tokenizers/decoders
  creating build/lib.linux-x86_64-3.6/tokenizers/normalizers
  copying py_src/tokenizers/normalizers/__init__.py -> build/lib.linux-x86_64-3.6/tokenizers/normalizers
  creating build/lib.linux-x86_64-3.6/tokenizers/pre_tokenizers
  copying py_src/tokenizers/pre_tokenizers/__init__.py -> build/lib.linux-x86_64-3.6/tokenizers/pre_tokenizers
  creating build/lib.linux-x86_64-3.6/tokenizers/processors
  copying py_src/tokenizers/processors/__init__.py -> build/lib.linux-x86_64-3.6/tokenizers/processors
  creating build/lib.linux-x86_64-3.6/tokenizers/trainers
  copying py_src/tokenizers/trainers/__init__.py -> build/lib.linux-x86_64-3.6/tokenizers/trainers
  creating build/lib.linux-x86_64-3.6/tokenizers/implementations
  copying py_src/tokenizers/implementations/sentencepiece_unigram.py -> build/lib.linux-x86_64-3.6/tokenizers/implementations
  copying py_src/tokenizers/implementations/base_tokenizer.py -> build/lib.linux-x86_64-3.6/tokenizers/implementations
  copying py_src/tokenizers/implementations/bert_wordpiece.py -> build/lib.linux-x86_64-3.6/tokenizers/implementations
  copying py_src/tokenizers/implementations/__init__.py -> build/lib.linux-x86_64-3.6/tokenizers/implementations
  copying py_src/tokenizers/implementations/sentencepiece_bpe.py -> build/lib.linux-x86_64-3.6/tokenizers/implementations
  copying py_src/tokenizers/implementations/byte_level_bpe.py -> build/lib.linux-x86_64-3.6/tokenizers/implementations
  copying py_src/tokenizers/implementations/char_level_bpe.py -> build/lib.linux-x86_64-3.6/tokenizers/implementations
  creating build/lib.linux-x86_64-3.6/tokenizers/tools
  copying py_src/tokenizers/tools/visualizer.py -> build/lib.linux-x86_64-3.6/tokenizers/tools
  copying py_src/tokenizers/tools/__init__.py -> build/lib.linux-x86_64-3.6/tokenizers/tools
  copying py_src/tokenizers/__init__.pyi -> build/lib.linux-x86_64-3.6/tokenizers
  copying py_src/tokenizers/models/__init__.pyi -> build/lib.linux-x86_64-3.6/tokenizers/models
  copying py_src/tokenizers/decoders/__init__.pyi -> build/lib.linux-x86_64-3.6/tokenizers/decoders
  copying py_src/tokenizers/normalizers/__init__.pyi -> build/lib.linux-x86_64-3.6/tokenizers/normalizers
  copying py_src/tokenizers/pre_tokenizers/__init__.pyi -> build/lib.linux-x86_64-3.6/tokenizers/pre_tokenizers
  copying py_src/tokenizers/processors/__init__.pyi -> build/lib.linux-x86_64-3.6/tokenizers/processors
  copying py_src/tokenizers/trainers/__init__.pyi -> build/lib.linux-x86_64-3.6/tokenizers/trainers
  copying py_src/tokenizers/tools/visualizer-styles.css -> build/lib.linux-x86_64-3.6/tokenizers/tools
  running build_ext
  running build_rust
  error: can't find Rust compiler
  
  If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.
  
  To update pip, run:
  
      pip install --upgrade pip
  
  and then retry package installation.
  
  If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

Are you running on CPU or GPU? – Mike B Sep 14 '22 at 06:22 — Mike B, Sep 14 '22 at 06:22
@MikelB, I am running on CPU – Baurzhan Sep 26 '22 at 09:52 — Baurzhan, Sep 26 '22 at 09:52

score 2 · Answer 1 · answered Oct 19 '22 at 05:46

Have you had a chance to check the spent time in sections of your code in order to find out where the bottleneck is? There are ways to improve grabbing&capturing frames from the camera, ways to improve decoding a compressed frame (e.g. from h.264 to raw-format) (and to prevent from copying the pixel-data from GPU-video-codec to the inference-enginve via GPU-Zero-Copy-mechanisms).

Then the inference itself - there are many different pre-trained object-/person-/pedestrian-detection models. Models can be optimized (e.g. via OpenVINO's tools) to run great on a specific underlying accelerator (like CPU, GPU, VPU, FPGA), like using INT8-optimized CPU-instruction-sets.

Models can be compressed, checked for sparsity.

Then the post-processing - like filtering the many detected objects (filtering confidence, filtering for overlapping via NMS) and then tracking.

Mike B · Answer 2 · 2022-09-26T15:34:28.093

1

DISCLAIMER, I am the creator and maintainer of https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet.

I see no export capabilities in that repository to any CPU friendly framework such as: onnx or openvino. Achieving fast inferences on CPU is conditioned by having models that run fast on CPU.

You can easily achieve 10FPS on CPU using my repo. Some random results run on CPU (Intel® Core™ i7-8850H CPU @ 2.60GHz × 12) with openvino models:

0: 640x640 2 persons, 1 chair, 1 refrigerator, Done. YOLO:(0.079s), StrongSORT:(0.025s)
0: 640x640 2 persons, 1 chair, 1 refrigerator, Done. YOLO:(0.080s), StrongSORT:(0.022s)
0: 640x640 2 persons, 1 chair, 1 refrigerator, Done. YOLO:(0.075s), StrongSORT:(0.022s)
0: 640x640 2 persons, 1 chair, 1 refrigerator, Done. YOLO:(0.078s), StrongSORT:(0.022s)
0: 640x640 2 persons, 1 chair, 1 refrigerator, Done. YOLO:(0.080s), StrongSORT:(0.022s)
0: 640x640 2 persons, 1 chair, 1 refrigerator, Done. YOLO:(0.112s), StrongSORT:(0.022s)
0: 640x640 2 persons, 1 chair, 1 refrigerator, Done. YOLO:(0.083s), StrongSORT:(0.022s)
0: 640x640 2 persons, 1 chair, 1 refrigerator, Done. YOLO:(0.078s), StrongSORT:(0.022s)
0: 640x640 2 persons, 1 chair, 1 refrigerator, Done. YOLO:(0.078s), StrongSORT:(0.024s)
0: 640x640 2 persons, 1 chair, 1 refrigerator, Done. YOLO:(0.085s), StrongSORT:(0.023s)

edited Sep 26 '22 at 15:34

answered Sep 26 '22 at 14:02

Mike B

2,136
2
12
31

I cloned your repository and started processing my video through your track.py script. The final results of the speed of work did not change much. I analyzed the main loop code, and in general, processing one frame does not take much time (1 second or a little more). At the moment, my conclusion is that the video consists of too many frames, which is the reason for the long processing. The 10 second video was split into 300 frames. The only way I managed to significantly speed up the video processing process is to process not every frame, but every tenth frame. – Baurzhan Nov 04 '22 at 08:37
But I'm not sure if this is the correct approach to solve this problem. – Baurzhan Nov 04 '22 at 08:37
Have you tried any of the CPU friendly trackers? For example: `python track.py --tracking-method ocsort`. OCSORT should achieve a minimum of 300FPS out of the box on CPU, but Yolo may be your bottleneck – Mike B Nov 07 '22 at 08:19
I have updated my question and attached a screenshot with my Yolo and Strongsort speed results. You can see that Yolo's work takes an average of half a second per frame, which is much longer compared to your results. What could be the reason? – Baurzhan Nov 16 '22 at 10:00
Are you running on an Intel CPU? In this case are you using OpenVINO models? – Mike B Nov 16 '22 at 10:09
Yes, on Intel Core i5 9th Gen. I tried to run through Openvino, but got an error: No module named openvino.runtime. And couldn't fix this error. I can attach a screenshot if needed. – Baurzhan Nov 16 '22 at 10:40
You have to install openvino-dev, check requirements.txt – Mike B Nov 16 '22 at 10:42
Yes, I tried to install openvino-dev but i can't solve the error. Attached a screenshot. I could not figure out what this error means and where the main text of the error is – Baurzhan Nov 16 '22 at 10:56
pip install openvino – Mike B Nov 16 '22 at 11:37
eventually installed openvino and openvino-dev, but still error "No module named 'openvino.runtime'". pip freeze --> openvino==2021.4.2, openvino-dev==2021.4.2 – Baurzhan Nov 18 '22 at 09:33
Update pip and install the newer versions. That may help – Mike B Nov 18 '22 at 10:39
Hi Mikel, Do I need to add any files to a project cloned from your repository in order to run track.py using the openvino model? – Baurzhan Nov 21 '22 at 11:16
You should only need to install openvino-dev – Mike B Nov 21 '22 at 12:32
But what about the model file. Shouldn't I instead specify the path to the openvino model file instead of the yolov5m.pt file in "--yolo-weights" argument? – Baurzhan Nov 21 '22 at 15:56
To be able to run an openvino model you first have to export to openvino – Mike B Nov 21 '22 at 21:43
do you mean that I need to get these files: yolov5s.bin, yolov5s.mapping, yolov5s.xml? I got these files using the instructions given in this repository: https://github.com/SamSamhuns/yolov5_export_cpu – Baurzhan Nov 22 '22 at 10:59
You have to do the same with the ReID model by its corresponding export script – Mike B Nov 22 '22 at 13:06
Upgrade pip to version 22.3.1, when try to install openvino==2022.2.0 got this: ERROR: Could not find a version that satisfies the requirement openvino==2022.2.0 (from versions: 2021.3.0, 2021.4.0, 2021.4.1, 2021.4.2) ERROR: No matching distribution found for openvino==2022.2.0 – Baurzhan Nov 29 '22 at 09:17
Hi Mikel, I failed to run tracking using openvino model on ubuntu16.04 So I started fresh on windows10, installed openvino2022.2.0 without problems I run it like this: python track.py --source test.mp4 --save-vid --yolo-weights yolov5s_openvino_model/ In yolov5s_openvino_model I put 3 files: .bin, .xml, .mapping I'm getting an error, I've attached a screenshot above – Baurzhan Dec 05 '22 at 11:28

How to increase object tracking speed (Yolov4 DeepSort)

2 Answers2