New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Summarizing crawl logs #2

Open

ato opened this issue Apr 3, 2019 · 1 comment

Labels

crawl-logs-stream

Member

ato commented Apr 3, 2019 •

edited

Loading

(e.g. for each crawled domain or per page) and report on encountered errors

Server errors
Robots.txt blocked
Timeouts
Truncated files
Slow download speed
Domains seen
Mime types (most common)
Size histogram

ato added crawl-logs crawl-logs-stream and removed crawl-logs labels

Member Author

ato commented Apr 3, 2019

@kris-sigur's crawl reviewer UI (as an example):

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment