Feature: Ignore files not under version control

[mdeweerd]

Context:
I created a script to add lines to the file ignore list that I can run if there are only false codespell positives.

I want to exclude files that are not under git version control as these do not need to be added as exceptions.

It seems appropriate and useful that 'codespell' has an option to do so.

For example, the following shell command filters to only files that are under version control:
git ls-files $(ls -p | grep -v -E '/$').

I could apply something like that to the script by using the codespell output two times or running it a second time on the filtered files. Done.

However, excluding files that are not under version controls seems like a good filter for codespell.

EDIT: updating the ignored lines file could even be a feature in codespell.

[DimitriPapadopoulos]

Currently codespell is a simple piece of software focused on processing the arguments it is given, without filters — except for hidden files. I am not certain adding filters is worth the extra complexity. You could instead modify the arguments passed to codespell. Any way, I'll let maintainers decide.

[mdeweerd]

The use case is that when running it from the CLI with local files used for development, these files are codespell'ed as well and when running the script to ignore line that are misspelled, lines from files not under version control also got added.
I updated my script to avoid that, but a CLI codespell run still shows these "false positives".

[yarikoptic]

this might also be a very nice alternative to needing to explicit list to ignore all of those other folders (.git/ , .mypy_cache/) etc. And it would also help to avoid side-effects of fixing some files not under git, while committing only the ones under git. So overall I would have liked to have such mode and making it configurable via config.

edits:

• it would also help to avoid dealing with Behavior of --skip with folder names #1915

[DimitriPapadopoulos]

OK, you convinced me. Which version control though? Not everyone uses git.

[mdeweerd]

I vote for git - I moved almost all my subversion projects to git, rcs/cvs has been long gone, and it's what I encounter the most.

[mdeweerd]

Note that git can do this:

git ls-files -- ':*php'
git ls-files -- . ':!*sql' ':!*bin'

So the file glob patterns could all be provided to git ls-files and then codespell just needs to iterate over the result.

[yarikoptic]

well, could be a setting ignore_vcs = git and then puke NotImplementedError for anything but git ATM so keeping it open for others to add andling for what they care. Minor note: for the test, make sure you have some files with obscure filenames [*] and use git ls-files -z and separate based on \000 not newline , and also I think that would avoid you need to deal with quotes:

use without -z

❯ mkdir /tmp/test; cd /tmp/test; git init; echo 1 > "new
line"; touch empty; touch '"quoted"'; git add *; git commit -m "added files"; git ls-files
Initialized empty Git repository in /tmp/test/.git/
[master (root-commit) 8bd959f] added files
 3 files changed, 1 insertion(+)
 create mode 100644 "\"quoted\""
 create mode 100644 empty
 create mode 100644 "new\nline"
"\"quoted\""
empty
"new\nline"

with -z but then translating \000 into a | for visualization

❯ rm -rf /tmp/test; mkdir /tmp/test; cd /tmp/test; git init; echo 1 > "new
line"; touch empty; touch '"quoted"'; git add *; git commit -m "added files"; git ls-files -z | tr '\000' '|'
Initialized empty Git repository in /tmp/test/.git/
[master (root-commit) 28da778] added files
 3 files changed, 1 insertion(+)
 create mode 100644 "\"quoted\""
 create mode 100644 empty
 create mode 100644 "new\nline"
"quoted"|empty|new
line|%

[*] In DataLad we programmatically compose "most obscure" (although still without new line IIRC) filename for a filesystem at hands during testing: https://github.com/datalad/datalad/blob/maint/datalad/tests/utils_pytest.py#L1453 . So on my laptop it is :

❯ python -c 'from datalad.tests.utils_pytest import OBSCURE_FILENAME; print(repr(OBSCURE_FILENAME))'
' |;&%b5{}\'"<>ΔЙקم๗あ .datc '

NB if you use debian package -- there would be no unicode part.

so note spaces leading/trailing it to ensure we do not strip them anywhere...