Introduces PTP tab and Job #2

dbrembilla · 2024-12-04T09:18:03Z

This MR introduces the first point to polygon functionalities, in particular:

Introduces a Tab in the volume view for the conversion
Creates a tab that can be used to create a PTP Job.

Once you have a Volume with point annotations, you can run a Point to Polygon job on a label with at least a point annotation. This will in turn run:

A Job computing the Expected areas for the label
A job uploading these expected areas
A job running the conversion, using the median of these expected areas as the expected area for the job
A job that uploads the new converted annotations

To test this MR:

Use the dev-modules-point-to-polygon branch for the biigle root
Add PTP to the composer, for now using the VCS (If you think I should add it to Packagist let me know)
Try to run the Point to Polygon conversion.

This commit introduces some of the basic logic in the PTP job, adding minimal functionality. Does not work still, but now it receives the image and annotations it needs to process and catches the path where the Image is.

Initialises PTP script, while still not working, this will be the starting point of the Python script that translates the points to polygons.

Initialises the new PTP controller that will launch the jobs and associates it to the API via a mock route (/test-ptp) to test the new features of PTP.

The PTP script now works and is able to run annotation conversions from point to polygons using SAM. Introduces also docstrings and type hints for the Python script

The Python Job can now be run via API using the controller, now loading the image path correctly.

Initialise the PTP sidebar. Before it used to simply replicate the MAIA tab. For now, it uses a placeholder index for executing, as the page will need to be developed similarly to the maia one.

This commit introduces the Point to Polygon View, along with its entry in the routes.php file.

This commit initialises the PTP tab that now correctly loads the point annotations, for now showing just the info about the annotation. The blade template correctly loads the vue templates/.

This commit introduces changes to the various templates. Now the template is able to show images, without the annotations, which will be introduced in a future commit, along with the fact that it will show one image per time and

This commit introduces now a system that loads images as well as their outlines on the label. Mockups of the buttons that will generate the PTP jobs are inserted. This commit also introduces the styling for the PTP module.

This commit makes it possible to spawn a PtpJob from the frontend via the `send-ptp-job` route, launching the Python scripts per label. The `ptp.py` is modified accordingly. Styling is also changed, to reflect this grouping and to allow the container to show the annotations well. A temporary directory system is introduced in the config.

This commit introduces the new job with the role of uploading the result of computing the expected area of the Point to Polygon conversion. This is done through a simple prediction on the image and taking the raw result/

This commit introduces several fixes and changes regarding the Point to Polygon pipeline. Essentially, there is now just one pipeline, triggered by the selected button, that first computes the expected area, then uploads it on the DB. The expected area is then use to apply the Point to Polygon conversion. In the next commits a Job that uploads the new annotations will be created, as well as a better system for the exchange of information between jobs.

This commit introduces the Job that uploads the new annotations found with the Python script to the DB, as well as fixing related scripts in order for it to work correctly.

This commit fixes some bugs that did not allow some of the strategies of the PTP conversion to work.

This commit introduces a fix on the Ptp process that updates the ImageAnnotationLabels table in order to have annotation information associated to them.

Removes useless components and API

mzur

Thanks! Mostly high-level comments first:

GPU or CPU?

Do you think these jobs should run on a GPU or is the runtime also decent on a CPU? We have to consider the following options:

On the GPU the whole volume is processed in a single job (because we have only one GPU for these long jobs). This may have the advantage of having to cache all files only once. Also, we may not need a temporary storage for the expected areas. The Python script can just get an input JSON with all annotation coordinates and label IDs as well as a directory of images and do everything else internally. The result is a JSON with the converted polygon coordinates.
On the CPU we can parallelize things and compute each image in a single job. This needs the temporary storage for the expected area and the different types of jobs (first process images for expected area, then do conversion).

Your current implementation of the jobs targets 2. Maybe it will be faster and easier to implement to do this on a GPU?

Volume Tab

I'd like this iteration of the feature to be implemented as an actual tab in the volume overview instead of a completely new view. You can take the biigle/laserpoints tab as an example.

Selection of a subset of labels should be optional. Either we don't offer this option at all for now (i.e. the user can only click the button "start" and all point annotations are converted) or we allow a selection of multiple labels. This can use the existing component to select multiple labels from label trees. Example from the clone volumes view:

Tests

You have to write tests for the controller and job classes.

Garbage files

There are still some files from biigle/maia in the repo (Requests, Jobs). Also the console publish command is not needed (it was used in the past by modules but not any more).

Branch

There is already the dev-modules-ptp branch of biigle/core with the module installed. However, the module is curently not loaded there. You should add an extra section to the composer.json to enable package discovery upon installation.

mzur · 2024-12-06T12:42:35Z

requirements.txt

+sahi==0.11.18
+shapely==2.0.6
+pycocotools==2.0.8


The last three are not really required, right?

mzur · 2024-12-06T12:53:22Z

src/Http/Controllers/Api/PtpController.php

+        $this->authorize('edit-in', $volume); Label::findOrFail($request->label_id);
+        $imageAnnotationArray = [];
+        $labelId = $request->label_id;
+        $pointShapeId = 1; //Find annotations with selected label in desired volume


No hard-coded IDs please 😉

Suggested change

$pointShapeId = 1; //Find annotations with selected label in desired volume

$pointShapeId = Shape::pointId(); //Find annotations with selected label in desired volume

mzur · 2024-12-06T13:21:11Z

src/Http/Controllers/Views/PtpController.php

+        $labels = Label::whereIn('label_tree_id', $labelTrees->pluck('id'))->get();
+
+        // ID of point shape
+        $pointAnnotationId = 1;


Suggested change

$pointAnnotationId = 1;

$pointAnnotationId = Shape::pointId();

mzur · 2024-12-06T13:22:37Z

src/Jobs/PtpJob.php

+use Illuminate\Queue\SerializesModels;
+use Storage;
+
+class PtpJob extends BaseJob implements ShouldQueue


This class has several attributes that were copy/pasted but are not relevant for this job.

mzur · 2024-12-06T13:25:22Z

src/Jobs/UploadConvertedAnnotationsJob.php

+    /**
+     * User responsible for the Point to Polygon conversion.
+     *
+     * @var User
+     */
+    public User $user;
+
+    /**
+     * The queue to push this job to.
+     *
+     * @var string
+     */
+    public string $inputDir;
+
+
+    public function __construct(string $inputDir, User $user)
+    {
+        $this->inputDir = $inputDir;
+        $this->user = $user;
+    }


With new PHP syntax you can do:

Suggested change

/**

* User responsible for the Point to Polygon conversion.

*

* @var User

*/

public User $user;

/**

* The queue to push this job to.

*

* @var string

*/

public string $inputDir;

public function __construct(string $inputDir, User $user)

{

$this->inputDir = $inputDir;

$this->user = $user;

}

/**

* @var $user User responsible for the Point to Polygon conversion.

* @var $inputDir ...

*/

public function __construct(public string $inputDir, public User $user)

{

//

}

mzur · 2024-12-06T13:35:48Z

src/resources/scripts/ptp.py

+    mask = mask * 255
+    mask = mask.astype(np.uint8)
+    contours, hierarchy = cv2.findContours(
+        mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE


We don't want to store coordinates for each pixel 😉

Suggested change

mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE

mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE

mzur · 2024-12-06T13:37:05Z

src/resources/scripts/ptp.py

I haven't read the whole script but maybe you can boil it down to only those features that are actually used by this BIIGLE module.

mzur · 2024-12-06T13:41:34Z

src/resources/views/volumesSidebar.blade.php

@@ -1,3 +1,3 @@
 @if (($user->can('edit-in', $volume) || $user->can('sudo')) && $volume->isImageVolume() && !$volume->hasTiledImages())
-    <sidebar-tab name="maia" icon="robot" title="Perform Machine Learning Assisted Image Annotation (MAIA)" href="{{route('volumes-maia', $volume->id)}}"></sidebar-tab>
+    <sidebar-tab name="ptp" icon="circle" title="Perform Point to Polygon conversion" href="{{route('volumes-ptp-conversion', $volume->id)}}"></sidebar-tab>


I thought that maybe we could use the same "magic hat" icon than MagicSAM, as the feature is very similar. It's basically MagicSAM on a volume level. @dlangenk what do you think?

mzur · 2024-12-06T13:43:03Z

src/Http/Controllers/Api/PtpController.php

+     */
+    public function generatePtpJob(Request $request) {
+        $this->validate($request, ['label_id' => 'integer', 'volume_id' => 'integer']);
+        $volume = Volume::findOrFail($request->volume_id);


You have to validate that the volume is an image volume and (maybe) that is does not have tiled images here. Video volumes are not supported.

@dlangenk Would the Python script work with a huge image mosaic?

mzur · 2024-12-06T13:46:20Z

src/Http/Controllers/Views/PtpController.php

+        //annotation with labels and that are points
+        $annotations = ImageAnnotation::join('image_annotation_labels','image_annotations.id', '=', 'image_annotation_labels.annotation_id')
+            ->join('images','image_annotations.image_id','=','images.id')
+            ->join('labels', 'image_annotation_labels.label_id','=','labels.id')
+            ->where('images.volume_id', $volume->id)
+            ->whereIn('image_annotation_labels.label_id', $labels->pluck('id'))
+            ->where('image_annotations.shape_id', $pointAnnotationId)
+            ->select('images.uuid', 'image_annotations.id', 'image_annotation_labels.label_id', 'labels.name AS label_name')->get();
+
+        //largo URL patches to show
+        $largoPatchesUrl = Storage::disk(config('largo.patch_storage_disk'))->url(':prefix/:id.'.config('largo.patch_format'));


These are not needed (any more).

dbrembilla added 30 commits November 8, 2024 15:47

Introduce logic image to annotation in Point to Polygon Job

b3fcbb9

This commit introduces some of the basic logic in the PTP job, adding minimal functionality. Does not work still, but now it receives the image and annotations it needs to process and catches the path where the Image is.

Initialise PTP script

eddabfc

Initialises PTP script, while still not working, this will be the starting point of the Python script that translates the points to polygons.

Add new mock PTP controller, API and Job

e384eba

Initialises the new PTP controller that will launch the jobs and associates it to the API via a mock route (/test-ptp) to test the new features of PTP.

Make the PTP python script functioning, add docstrings

d5f6b36

The PTP script now works and is able to run annotation conversions from point to polygons using SAM. Introduces also docstrings and type hints for the Python script

Add requirements.txt for Python

f327d51

Fix PTP job, now able to run the python script

38b4dea

The Python Job can now be run via API using the controller, now loading the image path correctly.

Apply black formatting to Python script

324bbc3

Sorts imports of Python script

f05d894

Initialse the PTP sidebar

7d4ede1

Initialise the PTP sidebar. Before it used to simply replicate the MAIA tab. For now, it uses a placeholder index for executing, as the page will need to be developed similarly to the maia one.

Introduce PTP view

a88ee9d

This commit introduces the Point to Polygon View, along with its entry in the routes.php file.

Initialise PTP tab, loading point annotations

540e09c

This commit initialises the PTP tab that now correctly loads the point annotations, for now showing just the info about the annotation. The blade template correctly loads the vue templates/.

Add ptp tab, now showing images

6ec484d

This commit introduces changes to the various templates. Now the template is able to show images, without the annotations, which will be introduced in a future commit, along with the fact that it will show one image per time and

Fix import and main.js for assets from largo

46269e5

Introduce images in the Ptp Tab, grouped per labels

4afcd80

This commit introduces now a system that loads images as well as their outlines on the label. Mockups of the buttons that will generate the PTP jobs are inserted. This commit also introduces the styling for the PTP module.

Allow styles to be published via Webpack

1b985d5

Fix logic, output dir in Python script

99d09f1

Add Ptp Expected Area Model

884bf3d

Remove double code from Python script

06c64bc

Introduce UploadConvertedAnnotationsJob

13d6e00

This commit introduces the Job that uploads the new annotations found with the Python script to the DB, as well as fixing related scripts in order for it to work correctly.

Fix ImageAnnotation import bug in Job

46db849

Introduce error message if no label selected in Ptp view

eb8d211

Fix bugs within PTP python script

35ab9cc

This commit fixes some bugs that did not allow some of the strategies of the PTP conversion to work.

Introduce update of ImageAnnotationLabels table

08bd68d

This commit introduces a fix on the Ptp process that updates the ImageAnnotationLabels table in order to have annotation information associated to them.

Clean up PTP container, improve style

18bba64

src/resources/assets/js/api/labelAnnotations.js

dd6486c

Removes useless components and API

Remove useless API file

a242820

Clean up PtpContainer component

dd6019b

dbrembilla added 4 commits December 3, 2024 16:11

Remove useless commit

a418c5d

Remove commits, add docstrings

4d61d77

Remove useless commit

aa0ae07

Add Publish command to Ptp

0012002

dbrembilla requested a review from mzur December 4, 2024 09:18

Try and fix test scripts

02f8ab9

mzur requested changes Dec 6, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduces PTP tab and Job #2

Introduces PTP tab and Job #2

dbrembilla commented Dec 4, 2024

mzur left a comment

mzur Dec 6, 2024

mzur Dec 6, 2024

mzur Dec 6, 2024

mzur Dec 6, 2024

mzur Dec 6, 2024

mzur Dec 6, 2024

mzur Dec 6, 2024

mzur Dec 6, 2024

mzur Dec 6, 2024

mzur Dec 6, 2024

	$pointShapeId = 1; //Find annotations with selected label in desired volume
	$pointShapeId = Shape::pointId(); //Find annotations with selected label in desired volume

	$pointAnnotationId = 1;
	$pointAnnotationId = Shape::pointId();

	mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE
	mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE

Introduces PTP tab and Job #2

Are you sure you want to change the base?

Introduces PTP tab and Job #2

Conversation

dbrembilla commented Dec 4, 2024

mzur left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment