rebuild NWIS, WQP, Ecosheds data #52
Conversation
…d task table to include pull date so pull date retriggers pull of each year chunk. Closes USGS-R#50
…h fewer sites and obs than the inventory suggested there were (674k sites and 6.2 mil records)
…etain all temperature data and report any location info we can from the column name.
…ure column names correctly
…erature of grab samples. See USGS-R#56
…ites and obs as expected.
limnoliver
changed the title
|
dcd9a64 - This now includes an Ecosheds rebuild after receiving a new snapshot of Ecosheds from Jeff Walker. This produced increases in both records and sites as expected. Also note, NorWest is a static database, so no updates to those data. |
|
Looks good to me, Sam. Only one question on retaining some commented-out code. The use of Side note, I converted this PR from draft to review. Feel free to merge. |
| select(-agency_cd, -count_nu) | ||
|
|
||
| if (!all(names(fixed_dups) %in% | ||
| # fixed_dups <- dat_long %>% |
There was a problem hiding this comment.
Is there a need to hold on to this commented-out code. Do you foresee going back on this change in the future?
There was a problem hiding this comment.
Thanks for spotting this. I think I just did this during development, but will delete now.
There was a problem hiding this comment.
Actually, keeping in commented code so targets stay "clean".
|
Looks good. Without running it, it looks like it does what you're saying. |
This PR repulls NWIS and WQP temperature data for the nation.
For data updates:
38094d6 - For WQP data, note there was a gain in sites (17k) but a loss in the number of records (792k). I compared the old data file to the new data file to try to explain some of this. Note, the pull includes sub-daily data, and there was a gain in the number of unique site-days (311k). On the surface this is worrisome, but after parsing the differences, I'm not worried about
Of the missing sites, only one of those was in in the inventory/partition files (e.g., was returned from whatWQPsites). I spot checked a few of the sites missing with the most data by looking at the WQP homepage. Some appeared to have MonitoringLocationIdentifier changes (e.g., BTMUA-INTAKE became BTMUA-Intake) while others seem to no longer exist (KENAI_WQX-10000117). It doesn't seem like we systematically missed them in our pull (e.g., a bad partition pull or whatWQPsites changes).
b0017c9 - expected increases in sites and data from NWIS.
For moderate code changes:
42086ab - retaining all temperature columns that are returned from NWIS instead of a priori selecting one.
909988d - pulls HUC22 which includes US territories, ef1784d modifies some spatial filters to ensure those site are retained.
682466f - use .qs files instead of .rds files for temporary files. Forgot to do this with the WQP pull -- will do this next time!
9bd0aa8 - modified the task tables for WQP pulls to include the pull date so new pulls would be triggered.