Refactor scripts into a CLI #6

thompsonmj · 2024-09-18T21:03:44Z

This PR intends to provide functionality as a user-friendly CLI.

Preprocessing - resize images while preserving nested directory hierarchy (if any)
Segmentation
Prep for color calibration and landmarking

Installation can be done by cloning the repo or directly with pip install git+https://github.com/Imageomics/wing-segmentation.git@feature/start-cli.

thompsonmj · 2024-09-19T19:30:35Z

@ramirezmichelle, @egrace479 please let me know if this looks on track with how functionality should go. Refer to the info in the README for how to test and what expected behavior will be.

For example, using the default setting for the output directory and trying various other settings, I get the following organization with test data:

$ ls data/
Jiggins_data  Jiggins_data_resize  meier_images  meier_images_resize
$ ls data/Jiggins_data_resize/
128x128  256x256
$ ls data/Jiggins_data_resize/256x256/
area  cubic
$ ls -1 data/Jiggins_data_resize/256x256/area/
'Heliconius erato ssp. amalfreda'
...
'Heliconius melpomene ssp. vulcanus'
metadata.json
$ ls -1 data/Jiggins_data_resize/256x256/area/Heliconius\ erato\ ssp.\ amalfreda/
CAM021007_d.png
...
CAM021227_d.png
$ cat data/Jiggins_data_resize/256x256/area/metadata.json | jq
{
  "source_directory": "data/Jiggins_data/",
  "output_directory": "/local/scratch/thompson.4509/projects/wing-segmentation/data/Jiggins_data_resize/256x256/area",
  "resize_dimensions": [
    256,
    256
  ],
  "interpolation_method": "area",
  "num_workers": 16,
  "timestamp": "2024-09-19 14:38:56"
}

It automatically handles nested vs flat data under the source directory and provides a metadata.json file in the base of the output dir (in the parent of any nested subdirs) to keep track of what was done.

I will do segmentation next.

This large commit implements most desired functionality. Additional testing and fine-tuning is needed for user input validation.

thompsonmj · 2024-10-02T19:42:16Z

The functionality takes care of segmentation and cropped image prep for landmarking.

egrace479

First pass, these commands need to be updated to match changes in 53a147d.

README.md

egrace479

Also, looks a little better with "console" instead of "bash" formatting for this part.

README.md

thompsonmj · 2024-10-03T15:41:53Z

Thanks! Yes, the README will need rewritten once the user options are settled.

Still allows Segmenter to be imported by a user, but bypasses its heavy imports until called directly

For now, no num-workers option

egrace479

Some preliminary thoughts on presentation/argument structure.

egrace479 · 2024-12-04T16:58:24Z

README.md

+To use and continue building the CLI features, set up and activate a virtual environment, and build interactively with `pip install -e .[dev]`.
+
+> [!TIP] 
+> [`uv`](https://github.com/astral-sh/uv) is a fast, Rust-based package manager.
+> If using on an HPC system, you may install `uv` into your user path or a conda environment (latter illustrated here).
+
+```console
+conda create -n uv -c conda-forge --solver=libmamba python=3.10 uv -y
+```
+```console
+conda activate uv
+```
+```console
+uv venv
+```
+```console
+source .venv/bin/activate # or source .venv/Scripts/activate on Windows
+```
+```console
+uv pip install -e .[dev]
+```


Suggested change

To use and continue building the CLI features, set up and activate a virtual environment, and build interactively with `pip install -e .[dev]`.

> [!TIP]

> [`uv`](https://github.com/astral-sh/uv) is a fast, Rust-based package manager.

> If using on an HPC system, you may install `uv` into your user path or a conda environment (latter illustrated here).

```console

conda create -n uv -c conda-forge --solver=libmamba python=3.10 uv -y

```

```console

conda activate uv

```

```console

uv venv

```

```console

source .venv/bin/activate # or source .venv/Scripts/activate on Windows

```

```console

uv pip install -e .[dev]

```

To use and continue building the CLI features, set up and activate a virtual environment, and build interactively with

```

pip install -e .[dev]

```

I don't think this is an appropriate place to introduce uv. It would be a great addition to our Helpful Tools page in the Guide, and you could link to that with a suggestion that uv pip install -e .[dev] is a faster option.

Also, when I tried to run pip install -e .[dev] I got zsh: no matches found: .[dev], but pip install -e . worked as expected.

egrace479 · 2024-12-04T16:59:50Z

README.md

+Inspecting data in the `seg_viz/`, we can see that the 1024x1024 products have segmentation masks that differ from the 512x512 and 256x256 products.
+
+1024x1024 (Run 8e9ae0a2):
+
+![1024 resize segmentation result visualization](readme_images/STRI_WOM_0011_V_viz_1024.png)
+
+512x512 (Run 3354acb9):
+
+![512 resize segmentation result visualization](readme_images/STRI_WOM_0011_V_viz_512.png)
+
+256x256 (Run 0f27d745):
+
+![256 resize segmentation result visualization](readme_images/STRI_WOM_0011_V_viz_256.png)
+
+A potential fix for this could be to add padding to the bounding boxes (with the `--bbox-padding` option) wherever results are inconsistent with expectations.


The underlying image should be cited. I'm still waiting on Christopher for the appropriate licensing and citation for the STRI images (see PR #5).

egrace479 · 2024-12-04T17:34:44Z

src/wing_segmenter/cli.py

+    output_group.add_argument('--outputs-base-dir', default=None, help='Base path to store outputs.')
+    output_group.add_argument('--custom-output-dir', default=None, help='Fully custom directory to store all output files.')


What exactly is the difference between these?

egrace479 · 2024-12-04T17:39:33Z

src/wing_segmenter/cli.py

+    resize_group.add_argument('--resize-mode', choices=['distort', 'pad'], default=None,
+                              help='Resizing mode. "distort" resizes without preserving aspect ratio, "pad" preserves aspect ratio and adds padding if necessary.')


Suggested change

resize_group.add_argument('--resize-mode', choices=['distort', 'pad'], default=None,

help='Resizing mode. "distort" resizes without preserving aspect ratio, "pad" preserves aspect ratio and adds padding if necessary.')

resize_group.add_argument('--resize-mode', choices=['distort', 'pad'], default=None,

help='Resizing mode. "distort" resizes without preserving aspect ratio, "pad" preserves aspect ratio and adds padding if necessary. Required with --size.')

egrace479 · 2024-12-04T17:40:16Z

README.md

+  --size SIZE [SIZE ...]
+                        Target size. Provide one value for square dimensions or two for width and height. (default: None)
+  --resize-mode {distort,pad}
+                        Resizing mode. "distort" resizes without preserving aspect ratio, "pad" preserves aspect ratio and adds padding if necessary.


Suggested change

Resizing mode. "distort" resizes without preserving aspect ratio, "pad" preserves aspect ratio and adds padding if necessary.

Resizing mode. "distort" resizes without preserving aspect ratio, "pad" preserves aspect ratio and adds padding if necessary. Required with --size.

Matching suggestion made at arg definition.

egrace479 · 2024-12-04T17:48:09Z

src/wing_segmenter/cli.py

+        if args.custom_output_dir and args.outputs_base_dir:
+            parser.error('Cannot specify both --outputs-base-dir and --custom-output-dir. Choose one.')


Is this not handled by the mutually exclusive group?

thompsonmj added 4 commits September 18, 2024 16:52

Begin CLI packaging

5a161d1

Begin documenting CLI development section

a0f6d25

Implement resizing

a2d570a

Document resizing

9bfd5c5

ramirezmichelle marked this pull request as ready for review September 24, 2024 18:27

ramirezmichelle approved these changes Sep 24, 2024

View reviewed changes

Functionality checkpoint

53a147d

This large commit implements most desired functionality. Additional testing and fine-tuning is needed for user input validation.

egrace479 requested changes Oct 2, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

egrace479 reviewed Oct 2, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

thompsonmj added 2 commits October 3, 2024 12:06

Defer import of Segmenter to speed up help message printing

0f4ef69

Still allows Segmenter to be imported by a user, but bypasses its heavy imports until called directly

Add completely custom output directory option

fad9b67

thompsonmj force-pushed the feature/start-cli branch from e4a5e02 to fad9b67 Compare October 4, 2024 14:35

thompsonmj added 6 commits October 4, 2024 10:38

Delete stray comment

dc6f9a2

Move resize interpolations to constants file

edc1acb

Housekeeping and argument validation fixes

afec2f8

Provide output logging and clean up run scanner

676477a

Only use serial processing

344c270

For now, no num-workers option

Rename background removal options to be more intuitive

d5d6773

thompsonmj force-pushed the feature/start-cli branch from 8af0232 to d5d6773 Compare November 25, 2024 21:33

thompsonmj added 5 commits November 25, 2024 17:23

Add usage and examples

7869894

Code blocks with console

739b4ff

Add bbox padding suggestiong

c0731fa

Edit development installation instructions

0715a32

Add CLI development disclaimer

9841170

thompsonmj requested a review from egrace479 December 2, 2024 20:16

egrace479 reviewed Dec 4, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor scripts into a CLI #6

Refactor scripts into a CLI #6

thompsonmj commented Sep 18, 2024 •

edited

Loading

thompsonmj commented Sep 19, 2024

thompsonmj commented Oct 2, 2024

egrace479 left a comment

egrace479 left a comment

thompsonmj commented Oct 3, 2024

egrace479 left a comment

egrace479 Dec 4, 2024

egrace479 Dec 4, 2024

egrace479 Dec 4, 2024

egrace479 Dec 4, 2024

egrace479 Dec 4, 2024

egrace479 Dec 4, 2024

		output_group.add_argument('--outputs-base-dir', default=None, help='Base path to store outputs.')
		output_group.add_argument('--custom-output-dir', default=None, help='Fully custom directory to store all output files.')

		resize_group.add_argument('--resize-mode', choices=['distort', 'pad'], default=None,
		help='Resizing mode. "distort" resizes without preserving aspect ratio, "pad" preserves aspect ratio and adds padding if necessary.')

	Resizing mode. "distort" resizes without preserving aspect ratio, "pad" preserves aspect ratio and adds padding if necessary.
	Resizing mode. "distort" resizes without preserving aspect ratio, "pad" preserves aspect ratio and adds padding if necessary. Required with --size.

		if args.custom_output_dir and args.outputs_base_dir:
		parser.error('Cannot specify both --outputs-base-dir and --custom-output-dir. Choose one.')

Refactor scripts into a CLI #6

Are you sure you want to change the base?

Refactor scripts into a CLI #6

Conversation

thompsonmj commented Sep 18, 2024 • edited Loading

thompsonmj commented Sep 19, 2024

thompsonmj commented Oct 2, 2024

egrace479 left a comment

Choose a reason for hiding this comment

egrace479 left a comment

Choose a reason for hiding this comment

thompsonmj commented Oct 3, 2024

egrace479 left a comment

Choose a reason for hiding this comment

egrace479 Dec 4, 2024

Choose a reason for hiding this comment

egrace479 Dec 4, 2024

Choose a reason for hiding this comment

egrace479 Dec 4, 2024

Choose a reason for hiding this comment

egrace479 Dec 4, 2024

Choose a reason for hiding this comment

egrace479 Dec 4, 2024

Choose a reason for hiding this comment

egrace479 Dec 4, 2024

Choose a reason for hiding this comment

thompsonmj commented Sep 18, 2024 •

edited

Loading