Fix Hugo Build & Deploy to correctly generate robots.txt

[stumbo]

Describe the bug
robots.txt is supposed to be automatically generated when production is set to true. This would then overwrite the default robots.txt file -- which disallows indexing. This is not working.

Running locally the robots.txt file is correctly generated. This isn't happening in the github action. Figure out where the workflow is messed up.

[stumbo]

Draft PR 239 resolves this issue. Need some final clean up and documentation before converting from draft.

[masinter]

A fix for this was approved and merged. Is there a way of checking whether it was effective? Do the web crawlers now crawl all our pages correctly?

[stumbo]

There are several online tools that claim to scan websites and identify issues. 

From seomator.com

URL	Info	Crawlable	Indexable
/	Google crawl rule: Allow: Bing crawl rule: Allow: Google index rule: none Bing index rule: none	Google Bing	Google Bing

Google Search Console - access limited to Interlisp.org site maintainers - notes the new robots.txt file was
successfully fetched on 10/3/2024.

https://interlisp.org/robots.txt	10/3/24, 2:07 AM	check_circle_outline Fetched	49 bytes

I requested a recrawl, which will show any issues with the robots.txt file.  But, given tools such as seomator seem
to think the robots.txt file is good and allows both Google and Bing to crawl I don't expect any issues.