This is a one-off project to test that redirects are in place, before a site is migrated to a new location.
There are a few steps.
Ensure you have Python 3.9 installed, as well as pipenv. Then run the following:
pipenv install
This scrapes the existing site to get a list of page links that should have redirects from them. Run
pipenv run scrapy runspider scraper.py -o ./output/output.csv
Add a line to your hosts file (using something like sudo nano /etc/hosts) for the site that you're trying to test. This is needed because the DNS will point to the old website and will not enable you to test anything.
Run
pipenv run python test_redirects.py
The output will go to ./output/redirect_test_output.xlsx