[JOSS Review] elk: A Python package to elicit latent knowledge from LLMs

[isdanni]

Thanks for submitting to JOSS! Note: this is a review thread, more checklist items will be added.

tracking review progress: openjournals/joss-reviews#6511

Documentation

• In the https://github.com/EleutherAI/ccs/tree/main?tab=readme-ov-file#quick-start section, it is usually recommended to install all the required packages in a specific virtual environment instead of system-wide.
• Please merge https://github.com/EleutherAI/ccs/tree/main?tab=readme-ov-file#development and https://github.com/EleutherAI/ccs/tree/main?tab=readme-ov-file#contributing-to-this-repository to Contribution Guidelines. It should at least include answers to 3 questions in the JOSS review principles: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support. Here are some useful templates(There's no need to include everything but this section should at least include the basic answers. For example, how to submit an issue, how to create a pull request and how to set up the development environment, and is there any difference from the production or main branch): https://github.com/auth0/open-source-template/blob/master/GENERAL-CONTRIBUTING.md, https://gist.github.com/briandk/3d2e8b3ec8daf5a27a62, https://github.com/pytorch/pytorch/blob/main/CONTRIBUTING.md, https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/setting-guidelines-for-repository-contributors.
• Are there any examples for using the codebase? With real-world analysis questions preferably.

Minor issues

• LICENSE.md.
• The bold markdown syntax did not work when an URL is wrapped around the text.
  

[isdanni]

Tests

Please ignore the previous comments. Using pip install -e . solved the package issue.

• When running pytest on a single-core machine, there are 5 failed tests related to not having more than 1 GPU. If the elk package can only run in a multi-core environment, please specify it in the documentation.

[isdanni]

Paper

• There's a typo in the header column: "machine leaarning"
• Please elaborate on the "enhanced version of the CSS method" and how it compares to existing implementations/repositories(if any).
• More details of the VINC method from the original paper. For example, why it's important, who would use it and how it's different from the other methods.
• Is Contrastive Representation Clustering (CRC) listed in References?
• "Statement of the Need" could use a more in-depth introduction

[lauritowal]

@isdanni thanks for the review! I'll update the things and let you know once it that's done

[lauritowal]

@isdanni I've done the changes and merged the latest main into the joss-paper branch. The pull-request will merged soon: #9

[isdanni]

@isdanni I've done the changes and merged the latest main into the joss-paper branch. The pull-request will merged soon: #9

Thanks for the update! I checked off most of the task items. Just one last thing(very minor issue):

In the Introduction section of README.md, the bold syntax does not work with URL; Unless this is intended I'd suggest removing it for better readability.

[lauritowal]

@isdanni thanks: https://github.com/EleutherAI/ccs/pull/10/files :)