CliRunner and click.prompt doesn't work for testing

[yves-chevallier]

Summary

When testing a Click command with CliRunner, it appears that click.prompt behaves unexpectedly in functional tests, especially when the terminal is not a true TTY. Specifically, when using CliRunner to simulate input in tests, click.prompt blocks indefinitely if more prompts are expected than input provided, making it impossible to test these commands correctly. This behavior is different from running the same code in a real shell environment, where input ends with EOF and the program gracefully handles the prompt.

Minimal Reproducible Example (MRE)

Here is a minimal reproducible example that demonstrates the issue:

import click
from click.testing import CliRunner
from pytest import fixture


@click.command()
@click.option("-n", "--count", default=2)
def main(count):
    sum = 0
    for i in range(count + 1):  # +1 is a bug in the code
        sum += click.prompt(f"Enter number {i + 1}", type=int)
    click.echo(f"Sum: {sum}")


@fixture
def runner():
    return CliRunner()


def test_program(runner):
    result = runner.invoke(main, args=["-n2"], input="1\n2")
    assert "3" in result.output
    assert result.exit_code == 0


if __name__ == "__main__":
    main()

Expected Behavior

In this example, there's a bug in the code (the for loop runs for count + 1 iterations). When testing using the CliRunner, we expect it to behave similarly to how the program runs in a real shell environment. Specifically, when EOF is reached (input ends), the test should either handle the prompt and output accordingly, or raise an appropriate error that can be captured in a test.

For example, when running the test program in a shell script:

#!/bin/bash

output=$(echo -e "1\n2\n" | python main.py)

if [[ "$output" == *"Sum: 3"* ]]; then
  echo "Test passed!"
  exit 0
else
  echo "Test failed!"
  echo "Expected output: 'Sum: 3'"
  echo "Actual output: $output"
  exit 1
fi

The test does not block and can process input, even though there's one extra prompt due to the bug.

Actual Behavior

When running the test with pytest using CliRunner, it blocks indefinitely because click.prompt is waiting for additional input, even though the input stream has ended:

$ python -mpytest main.py
===== test session starts =====
platform linux -- Python 3.13.0, pytest-8.3.3, pluggy-1.5.0
rootdir: /click-test
collected 1 item

main.py (blocking...)

Root Cause

It seems that CliRunner and Click's input handling behave differently in non-TTY environments, especially around handling click.prompt. When the terminal is not a true TTY, Click disables certain features (like colors) and handles input differently. In this case, click.prompt appears to ignore EOF signals, resulting in the test blocking indefinitely.

In contrast, when piping input in a real shell, the EOF signal is respected, and the program behaves as expected.

Possible Solutions

• Improve CliRunner behavior: CliRunner could simulate EOF and handle click.prompt more gracefully, aligning with behavior in actual terminal sessions.
• Provide a clearer way to handle prompts in non-TTY environments: Currently, there is no obvious way to handle situations like this during functional testing without a TTY. There could be an enhancement to handle prompts better in non-interactive tests.
• Document this behavior: If this is the intended behavior, it should be clearly documented that CliRunner behaves differently in non-TTY environments, especially in relation to prompts and input streams.

Environment Details

• Python 3.13.0
• Click 8.x
• pytest 8.3.3