Skip to content

Commit

Permalink
Release v2.3.0, add GitHub Actions and SonarCloud
Browse files Browse the repository at this point in the history
* add normalize_images method as an abstract method in image_processors.py

* fix: change OpenCVVideo method name get_next_video_frames -> get_next_frames

* move out normalizing images from evalutor to extractors

* rename get_extractor() -> create_extractor() in ExtractorFactory

* add license to docstrings

* move ServiceShutdownSignal inside DockerManager

* make ServiceInitializer attributes protected

* add architecture section to README

* add created status to manager and tests

* add cpu-only flag

* add new quick_demo.bat files

* add new flag to readme

* change requirements in readme

* fix paths in docker-compose.yaml and change method 2 in readme

* add 'created at' badge

* change _get_image_evaluator() to returns ImageEvaluator insted of specific evaluator

* fix DIP in extractors

* add pylint to dependendcies

* move extractor service tests to root dir and fix imports

* move service_manager tests to root folder tests and fix paths

* fix extractors unit tests after fixing DIP in extractors

* move dependency injection to extractors manager

* add dependencies module

* change README files tests sections after moving tests to root folder

* use Depends() from FastAPI for better dependencies handling and refactor code and tests

* add tests for dependencies.py

* small changes in docstings, small refactor code changes

* change logging level to INFO

* change cpu_only to True in integration tests in service_manager

* add --cpu flag to service_manager e2e tests

* add .gitkeep to test_files

* fix test_video

* fix paths in test_add_prefix

* Add local server to run_tests.yml

* add skipif in service_manager e2e tests

* remove additional server in github actions

* change http to https

* add assert before isinstance()

* change 'is not None' to just 'assert extractor'

* fix logging check in test_list_input_directory_files

* Fix assertion for _dropout_rate to float with tolerance

* Add comment before pass in test_get_video_capture_failure for clarity

* remove magic values in test_get_next_video_frames

* remove overwriting client variable in image fixture

* fix removing test folders after tests

* add sleep 300 command as a constant for handling DRY

* fix test_container_status

* save log_lines as constants for handling DRY

* remove unused variable pop

* Remove the unused local variable 'container'

* add test video
  • Loading branch information
BKDDFS authored May 29, 2024
1 parent 54fe463 commit 6ad0b83
Show file tree
Hide file tree
Showing 70 changed files with 712 additions and 304 deletions.
24 changes: 19 additions & 5 deletions .github/README.pl.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
</div>
<div id="badges">
<p align="center">
<img alt="Github Created At" src="https://img.shields.io/github/created-at/BKDDFS/PerfectFrameAI">
<img alt="GitHub Downloads (all assets, all releases)" src="https://img.shields.io/github/downloads/BKDDFS/PerfectFrameAI/total?style=flat&color=blue">
<img alt="GitHub License" src="https://img.shields.io/github/license/BKDDFS/PerfectFrameAI">
<img alt="GitHub Release" src="https://img.shields.io/github/v/release/BKDDFS/PerfectFrameAI">
Expand Down Expand Up @@ -182,6 +183,24 @@
<td>bool</td>
<td>False</td>
</tr>
<tr>
<td>--all_frames</td>
<td></td>
<td>
Do pomijania oceniania klatek.
</td>
<td>bool</td>
<td>False</td>
</tr>
<tr>
<td>--cpu</td>
<td></td>
<td>
Wyłącza korzystanie z GPU. Musisz tego użyć jeśli nie masz GPU.
</td>
<td>bool</td>
<td>False</td>
</tr>
</tbody>
</table>
<p><strong>Przykład dla Best Frames Extraction:</strong></p>
Expand Down Expand Up @@ -434,11 +453,6 @@
Testy możesz uruchomić instalując zależności z <code>pyproject.toml</code>
i wpisując w terminal w lokalizacj projektu - <code>pytest</code>.
</p>
<blockquote>
Proszę zwrócić uwagę, że w projekcie są dwa foldery <code>tests/</code>.
<code>extractor_service</code> i <code>service_initializer</code> mają testy osobno.
W pliku common.py znajdują się pliki wpółdzielone przez testy i potrzebne do ich działania.
</blockquote>
<details id="unit">
<summary>jednostkowe</summary>
<p>
Expand Down
7 changes: 4 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ input_directory/*
output_directory/*
!input_directory/.gitkeep
!output_directory/.gitkeep
test_video.mp4
nima.h5
common/test_files/best_frames/*
common/test_files/top_images/*
tests/test_files/best_frames/*
tests/test_files/top_images/*
!tests/test_files/best_frames/.gitkeep
!tests/test_files/top_images/.gitkeep
51 changes: 38 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
</div>
<div id="badges">
<p align="center">
<img alt="Github Created At" src="https://img.shields.io/github/created-at/BKDDFS/PerfectFrameAI">
<img alt="GitHub last commit" src="https://img.shields.io/github/last-commit/BKDDFS/PerfectFrameAI">
<img alt="GitHub License" src="https://img.shields.io/github/license/BKDDFS/PerfectFrameAI">
<img alt="GitHub Tag" src="https://img.shields.io/github/v/tag/BKDDFS/PerfectFrameAI">
Expand Down Expand Up @@ -106,17 +107,19 @@
<h3>System Requirements:</h3>
<ul>
<li>Docker</li>
<li>Python 3.10 or higher (method 1 only)</li>
<li>Nvidia GPU (recommended)</li>
<li>10 GB free disk space</li>
</ul>
<li>Python 3.7+ (method 1 only)</li>
<li>8GB+ RAM</li>
<li>10GB+ free disk space</li>
</ul>
<p>Lowest tested specs - i5-4300U, 8GB RAM (ThinkPad T440) - 4k video, default 100img/batch.</p>
<p>Remember you can always decrease images batch size in schemas.py if you out of RAM.</p>
</blockquote>
<details>
<summary>Install Docker:</summary>
Docker Desktop: <a href="https://www.docker.com/products/docker-desktop/">https://www.docker.com/products/docker-desktop/</a>
</details>
<details>
<summary>Install Python v3.10+:</summary>
<summary>Install Python v3.7+:</summary>
MS Store: <a href="https://apps.microsoft.com/detail/9ncvdn91xzqp?hl=en-US&gl=US">https://apps.microsoft.com/detail/9ncvdn91xzqp?hl=en-US&gl=US</a><br>
Python.org: <a href="https://www.python.org/downloads/">https://www.python.org/downloads/</a>
</details>
Expand All @@ -140,9 +143,17 @@
<blockquote>
<p>
<strong>Hint for Windows users:</strong><br>
As a Windows user you can use <code>quick_demo.bat</code> file.
It will run <code>best_frames_extractor</code> with the default values. Just double click on it.
You can modify default values in config.py to adjust the application to your needs.
As a Windows user, you can use:<br>
<code>quick_demo_gpu.bat</code> or <code>quick_demo_cpu.bat</code>
if you don't have an Nvidia GPU.<br>
It will run <code>best_frames_extractor</code> with the default values.
Just double-click on it.
You can modify the default values in config.py to adjust the application to your needs.<br>
<strong>Warning!</strong><br>
Please note that when running the .bat file,
Windows Defender may flag it as dangerous.
This happens because obtaining a code-signing certificate
to prevent this warning requires a paid certificate...
</p>
</blockquote>
<p>Run <code>start.py</code> from the terminal.</p>
Expand Down Expand Up @@ -191,6 +202,24 @@
<td>bool</td>
<td>False</td>
</tr>
<tr>
<td>--all_frames</td>
<td></td>
<td>
For skipping frames evaluation part.
</td>
<td>bool</td>
<td>False</td>
</tr>
<tr>
<td>--cpu</td>
<td></td>
<td>
Uses only CPU for processing. If you, don't have GPU you must use it.
</td>
<td>bool</td>
<td>False</td>
</tr>
</tbody>
</table>
<p><strong>Example (Best Frames Extraction):</strong></p>
Expand All @@ -203,6 +232,7 @@
<blockquote><p><i>Does not require Python. Run using Docker Compose.</i></p></blockquote>
</summary>
<p>Docker Compose Docs: <a href="https://docs.docker.com/compose/">https://docs.docker.com/compose/</a></p>
<p>Remember to delete GPU part in docker-compose.yaml if you don't have GPU!</p>
<ol>
<li>Run the service: <br><code>docker-compose up --build -d</code></li>
<li>Send a request to the chosen endpoint.
Expand Down Expand Up @@ -415,11 +445,6 @@
You can run the tests by installing the dependencies from <code>pyproject.toml</code>
and typing in the terminal in the project location - <code>pytest</code>.
</p>
<blockquote>
Please note that there are two <code>tests/</code> folders in the project.
<code>extractor_service</code> and <code>service_initializer</code> have separate tests.
The common.py file contains shared files for the tests and necessary for their operation.
</blockquote>
<details id="unit">
<summary>unit</summary>
<p>
Expand Down
4 changes: 2 additions & 2 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ services:
ports:
- "8100:8100"
volumes:
- "B:/frames_evaluators/input_directory:/app/input_directory"
- "B:/frames_evaluators/output_directory:/app/output_directory"
- "./input_directory:/app/input_directory"
- "./output_directory:/app/output_directory"
environment:
- NVIDIA_VISIBLE_DEVICES=all
- NVIDIA_DRIVER_CAPABILITIES=compute,video,utility
Expand Down
1 change: 1 addition & 0 deletions extractor_service/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ RUN pip install --no-cache-dir -r requirements.txt
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,video,utility
ENV TF_CPP_MIN_LOG_LEVEL 3
ENV DOCKER_ENV=1

COPY . .

Expand Down
File renamed without changes.
95 changes: 95 additions & 0 deletions extractor_service/app/dependencies.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
"""
This module provides dependency management for extractors using FastAPI's dependency injection.
LICENSE
=======
Copyright (C) 2024 Bartłomiej Flis
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
"""
from dataclasses import dataclass
from typing import Type

from fastapi import Depends

from .image_evaluators import InceptionResNetNIMA
from .image_processors import OpenCVImage
from .video_processors import OpenCVVideo


@dataclass
class ExtractorDependencies:
"""
Data class to hold dependencies for the extractor.
Attributes:
image_processor (Type[OpenCVImage]): Processor for image processing.
video_processor (Type[OpenCVVideo]): Processor for video processing.
evaluator (Type[InceptionResNetNIMA]): Evaluator for image quality.
"""
image_processor: Type[OpenCVImage]
video_processor: Type[OpenCVVideo]
evaluator: Type[InceptionResNetNIMA]


def get_image_processor() -> Type[OpenCVImage]:
"""
Provides the image processor dependency.
Returns:
Type[OpenCVImage]: The image processor class.
"""
return OpenCVImage


def get_video_processor() -> Type[OpenCVVideo]:
"""
Provides the video processor dependency.
Returns:
Type[OpenCVVideo]: The video processor class.
"""
return OpenCVVideo


def get_evaluator() -> Type[InceptionResNetNIMA]:
"""
Provides the image evaluator dependency.
Returns:
Type[InceptionResNetNIMA]: The image evaluator class.
"""
return InceptionResNetNIMA


def get_extractor_dependencies(
image_processor=Depends(get_image_processor),
video_processor=Depends(get_video_processor),
evaluator=Depends(get_evaluator)
) -> ExtractorDependencies:
"""
Provides the dependencies required for the extractor.
Args:
image_processor (Type[OpenCVImage], optional): Dependency injection for image processor.
video_processor (Type[OpenCVVideo], optional): Dependency injection for video processor.
evaluator (Type[InceptionResNetNIMA], optional): Dependency injection for image evaluator.
Returns:
ExtractorDependencies: All necessary dependencies for the extractor.
"""
return ExtractorDependencies(
image_processor=image_processor,
video_processor=video_processor,
evaluator=evaluator
)
24 changes: 10 additions & 14 deletions extractor_service/app/extractor_manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@
along with this program. If not, see <https://www.gnu.org/licenses/>.
"""
import logging
from typing import Type

from fastapi import HTTPException, BackgroundTasks

from .dependencies import ExtractorDependencies
from .extractors import Extractor, ExtractorFactory
from .schemas import ExtractorConfig

Expand All @@ -35,7 +35,6 @@ class ExtractorManager:
maintaining system stability.
"""
_active_extractor = None
_config = None

@classmethod
def get_active_extractor(cls) -> str:
Expand All @@ -48,30 +47,28 @@ def get_active_extractor(cls) -> str:
return cls._active_extractor

@classmethod
def start_extractor(cls, background_tasks: BackgroundTasks, config: ExtractorConfig,
extractor_name: str) -> str:
def start_extractor(cls, extractor_name: str, background_tasks: BackgroundTasks,
config: ExtractorConfig, dependencies: ExtractorDependencies) -> str:
"""
Initializes the extractor class and runs the extraction process in the background.
Args:
config (ExtractorConfig): A Pydantic model with configuration
parameters for the extractor.
background_tasks: A FastAPI tool for running tasks in background,
which allows non-blocking operation of long-running tasks.
extractor_name (str): The name of the extractor that will be used.
background_tasks (BackgroundTasks): A FastAPI tool for running tasks in background.
config (ExtractorConfig): A Pydantic model with extractor configuration.
dependencies(ExtractorDependencies): Dependencies that will be used in extractor.
Returns:
str: Endpoint feedback message with started extractor name.
"""
cls._config = config
cls._check_is_already_extracting()
extractor_class = ExtractorFactory.create_extractor(extractor_name)
background_tasks.add_task(cls.__run_extractor, extractor_class, extractor_name)
extractor = ExtractorFactory.create_extractor(extractor_name, config, dependencies)
background_tasks.add_task(cls.__run_extractor, extractor, extractor_name)
message = f"'{extractor_name}' started."
return message

@classmethod
def __run_extractor(cls, extractor: Type[Extractor], extractor_name: str) -> None:
def __run_extractor(cls, extractor: Extractor, extractor_name: str) -> None:
"""
Run extraction process and clean after it's done.
Expand All @@ -81,10 +78,9 @@ def __run_extractor(cls, extractor: Type[Extractor], extractor_name: str) -> Non
"""
try:
cls._active_extractor = extractor_name
extractor(cls._config).process()
extractor.process()
finally:
cls._active_extractor = None
cls._config = None

@classmethod
def _check_is_already_extracting(cls) -> None:
Expand Down
Loading

0 comments on commit 6ad0b83

Please sign in to comment.