NEWS.md

NEWS-ContDataQC

#> Last Update: 2017-11-30 12:44:37

Planned Updates

• Spell out "AW"" and other abbreviations (e.g., AirWater). 20170308. On hold.

• Gaps in data not always evident in the plots. 20170308.

• Use futile.logger to better log output for user. Issue #29. 20170606.

• Debug Aggregate operation. 20170919.

• Create CDFs. Similar to code already used in previous analyses by Lei. 20170919.

• PeriodStats(), add number and/or percent of observations above given threshold. 20170919.

• Update Vignette after any updates to help or documentation. 20170919.

• Run devtools::check(). 20170926.

• Fix mixed case issue with filenames in "file" versions of QCRaw, Aggregate, and Stats. 20170929.

  • More check data stuff.
  • Update vignette
  • Threshold number or pct on plot
  • CSV issue
  • Excel file update (see 9/28 email)

Version History

v2.0.1.9056

2017-11-30

• PeriodStats

  • Added data input requirements. 20171130.

  • Updated example for default report from DOCX to HTML. 20171130.

  • Updated Vignette. 20171130.

v2.0.1.9055

2017-11-30

• Export.IHA

  • Updated example to use system username. 20171130.

  • Repeated SiteID line in example. 20171130.

  • Updated Vignette. 20171130.

v2.0.1.9054

2017-11-30

• Export.StreamThermal

  • Update Excel file with category definitions from journal article. 20171130.

  • Add metric definition Excel file to extdata so can use when create data export. 20171130.

  • Update examples in vignette. 20171130.

v2.0.1.9053

2017-11-29

• Readme, fix typo in github install function. 20171129.

• Update example in Export.StreamThermal so is self contained. Vignette did not need to be updated. Included example code for export to Excel (same as in Export.IHA example). 20171129.

v2.0.1.9052

2017-11-29

• Update "=" to "<-" in Export.IHA example in function and vignette. 20171129.

• SummaryStats (File) example for non-ContDataQC file. Add minimum necessary fields to a USGS download. 20171129.

• Update vignette:

  • Revised config.R thresholds. 20171129.

  • SummaryStats (File) for non-ContDataQC file. 20171129.

v2.0.1.9051

2017-11-29

• Reports, typo in "Generated by:". 20171129.

v2.0.1.9050

2017-11-27

• Updated some QC thresholds in config.R. 20171127.

v2.0.1.9049

2017-11-16

• Fix error in myReport.format and then examples. 20171115.

• Renamed report RMD files and altered reference in PeriodStats(). 20171115.

• Added myReport.Format to config.R and set the default there rather than in code. 20171115.

• Add "username" to reports. 20171116.

• Update vignette. 20171116.

v2.0.1.9048

2017-11-15

• Add variable for report format (docx or html). Default will be "docx". Needed for Shiny app since won't have Pandoc. Pandoc now ships with RStudio but still don't have when run package via Shiny.

• Test version. Not complete. Vignette needs to be rebuilt.

v2.0.1.9047

2017-09-29

• Aggregate.File. Example file mispelled (AW vs. Aw). 20170928.

• File versions of QCRaw, Aggregate, and Stats; change input and output directory name. 20170929.

• File QC. Turn off report. Mixed case issue in file names. 20170929.

v2.0.1.9046

2017-09-28

• Add packages used in vignette to "Suggests" in DESCRIPTION.

• Fixed function example lines wider than 100 characters.

• Moved necessary packages from "Suggests" to "Imports" in DESCRIPTION.

• DESCRIPTION, License, out of sync with GitHub. Fixed.

v2.0.1.9045

2017-09-28

• Add knitr:: to all kable commands in RMD files. Had error after moving knitr from Depends to Suggests. 20170928.

• Renamed RMD files with ' - ' to '_'. Recommended in check(). 20170928.

v2.0.1.9044

2017-09-28

• CompSiteCDF funtion complete. 20170928.

• Vignette updated. 20170928.

v2.0.1.9043

2017-09-27

• DESCRIPTION. As suggested by check(), move packages from Depends to Suggests. 20170927.

• Update ReadMe. 20170927.

v2.0.1.9042

2017-09-21

• CompSiteCDF() to compare multiple sites on a site CDF. 20170921.

v2.0.1.9041

2017-09-20

• Export.StreamThermal.

• Rename from Export.ST() to Export.StreamThermal() .20170920.

• Complete function and documentation. 20170920.

• Update Vignette. 20170920.

• Export.IHA. Tweak opening text. 20170920.

v2.0.1.9040

2017-09-20

• Vignette.
  • Tweaked language. 20170919.
  • Added section about requiring dataRetrieval v2.7.3 that isn't on CRAN yet. 20170919.

v2.0.1.9039

2017-09-19

• Vignette added. 20170919.

• Added daily mean visualization Excel file to . 20170919.

v2.0.1.9038

2017-09-12

• Rename fun.Export.IHA() to Export.IHA(). 20170912.

v2.0.1.9037

2017-09-12

• Added function, rarify(), from RIVPACS code by John van Sickles (USEPA). Modified to be a stand alone function and with a seed parameter for reproducible results. Included example dataset with 2 samples. 20170912.

• Added "Bug Reports" to DESCRIPTION file. 20170912.

• Modified fun.Export.IHA() argument fun.myCol.Discharge to fun.myCol.Parameter. 20170912.

v2.0.1.9036

2017-09-11

• Added function, fun.Export.IHA(), to export data in format ready to use in IHA package. 20170911.

v2.0.1.9035

2017-09-11

• Updated fun.GageData.R. USGS updated their webservices to use https instead of http. dataRetrieval package updated to v2.7.3 on GitHub. Some functions modified within dataRetrieval as had to modify some code in ContDataQC package. Version number of dataRetrieval not added to DESCRIPTION (not available on CRAN yet). 20170911.

v2.0.1.9034

2017-09-06

• Added PeriodStats() to calculate daily stats (mean, min, max, etc) for a period (N) before a given date. Outputs a CSV of data and DOCX report. 20170905.

v2.0.1.9033

2017-08-24

• Changed reports back to DOCX. Will add a user parameter to change in the future. 20170824.

v2.0.1.9032

2017-08-23

• Added rmarkdown to dependencies in Description. It should install with Pandoc but some users had to install separately. 20170823

• Added install.packages("installr") to example in ContDataQC() for installing Pandoc. 20170823.

v2.0.1.9031

2017-08-23

• Reload on GitHub. Shiny can't download. 20170823.

v2.0.1.9030

2017-08-23

• Another DOCX reference fix. 20170823.

v2.0.1.9029

2017-08-23

• Change reference to DOCX variable to .ext to easier to change in the future. 20170823.

v2.0.1.9028

2017-08-23

• Convert Reports from DOCX to HTML. Needed for use with Shiny. 20170823.

v2.0.1.9027

2017-08-23

• Rebuild before update to GitHub. 20170823.

v2.0.1.9026

2017-08-23

• Ensure use file.path when read and save files. 20170823.

v2.0.1.9025

2017-07-31

• Thresholds. Tweaked config.R. Issue #36. 20170731.

v2.0.1.9024

2017-07-27

• Thresholds. Updated config.R with more informed thresholds after working with more data. Issue #36. 20170728.

• DESCRIPTION. Added Jen Stamp as contributer and added my middle initial. 20170728.

v2.0.1.9023

2017-06-22

• QC report (and aggregate report), fixed water temp vs. sensor depth legend. Colors were swapped. Issue # 35. 20170622.

v2.0.1.9022

2017-06-21

• QC report header for water temp and sensor depth says water level. Fixed. Issue #34. 20170621.

v2.0.1.9021

2017-06-21

• Dec 31 not being assigned a season in summar stats. Fixed. Issue #33. 20170621.

v2.0.1.9020

2017-06-09

• QC of gage data not using all parameters present. Fixed. Issue #32. 20170609.

• Added air and water pressure in psi to gage download. fun.GageData.R. 20170609.

• ContData.env$myNames.Order missing new fields. Was dropping fields not included (e.g., gage height). config.R. 20170609.

• fun.QC.R missing QC routine for gage height. 20170609.

v2.0.1.9019

2017-06-07

• Updated data flow for "aggregate" step so the QC report is generated "after" the compiled file is created. Issue #31. 20170607.

v2.0.1.9018

2017-06-07

• Updated main function to allow for a single file or a vector of files. New "file" versions of most scripts. Some updates to the Reports (rmd files). Issue #24. 20170607.

v2.0.1.9017

2017-06-06

• Report_QC.rmd. Correct typo in "Flags" descriptions. Issue #28. 20170606.

v2.0.1.9016

2017-05-31

• File delimiter for strsplit(). Also a regex issue with "." (dot). Created special check. 20170531.

v2.0.1.9015

2017-05-31

• File delimiter. Configuration file allows for user defined values. However, the QC check for StationIDs containing this value uses regular expressions. Added a special condition for the dot, ".", delimiter as regex treats this character as a match for (almost) any character. 20170531.

v2.0.1.9014

2017-05-26

• Time interval. Fixed calculation of time interval. Certain date time formats would stop processing or be reported as zero. fun.QC.R, Report_Aggregate.rmd, Report_QC.rmd, and Report_Stats.rmd. 20170526.

v2.0..1.9013

2017-05-24

• fun.Stats.R. Had previously commented out stats and plots due to errors. Single typo fixed in year season stats and reworked daily summaryBy stats section. 20170524.

v2.0..1.9012

2017-05-24

• fun.Gage.R. Changed variable "myTZ" to "fun.myTZ" for consistency. 20170524.

v2.0.1.9011

2017-05-23

• Remove unused arguement "fun.myFile.Prefix" from help file for ContDataQC(). 20170523.

v2.0.1.9010

2017-05-22

• Modify input and export directories so can be any folder. The default directory is now the working directory. Modified input parameters for ContDataQC() by removing "fun.myDir.BASE". Modified examples and all other necessary scripts.

v2.0.1.9009

2017-05-19

• Report_Aggregate.rmd. Fix sampling interval (seconds to minutes,line 23). Left plots as dots instead of lines. 20170519.

• ContDataQC(). Update examples to with date range. After adding in "test4" the dates don't match for "test2". 20170519.

• Update "test4" dataset. Change "Water Level ft"" to "Sensor Depth ft". 20170519.

• Summary Stats quit with error. Fixed in summaryBy statements. 20170519.

v2.0.1.9008

2017-05-18

• Add example for offset data for QC in main function (ContDataQC()). 20170518.

• Fix plots (legend) in Report_QC.R (remove box as it is only partially displaying). 20170518.

• Config file. Upate Sensor Depth and Water Pressure display names. 20170518.

• Label code chunks in RMD files. 20170518.

• Fix plots (blank) in Report_QC.R for "offset" data. With point type of "line" the NAs in the data caused nothing to be plotted. Fixed with na.omit(as.numeric(x)) when subsetted data for plotting. Then had to fix date range on x-axis. 20170518.

v2.0.1.9007

2017-05-17

• Add additional config file. Modified config.R to include OlsonNames() in timezone section. Added /extdata/config.TZ.central.R with only time zone change. Modified examples in zfun.ContaDataQC.R to include a USGS gage in the central time zone. Modified ContDataQC() with optional parameter to use helper functions to include different config files. Will only need paramters that are different. User provides path to the new R file for new variables to include in the environment ContData.env.

v2.0.1.9006

2017-05-17

• zfun.ContDataQC.R. Add in "test4" data (offset times) to examples. 20170515.

• config.R. Fix typo in ContData.env$myName.GageHeight; missing "." between name and unites. 20170517.

• fun.QC.R. Line 1034 bad reference for difftime; renamed myTimeDiff.all to myT.diff.all.

• Comnfirmed that knitr is silent so the user does not get code scrolling across screen. 20170517.

• Confirmed that steps/tasks in the process are independant of each other. That is, someone can use some or all routines as each is a separate function. 20170517.

v2.0.1.9005

2017-05-15

• Add data.R to desribe data inlcuded in library. 20170508.

• fun.QC.R. Fix typo in myT.diff.all operation (line 965). 20170508.

• Create data process script in "data-raw" for three example "test1" files. 20170509.

• Revised fun.OffsetCollectionCheck() in fun.Helper.R. Return value is not boolean. 20170509.

• Revised fun.CalcQCStats() in fun.QC.R. Added ability to handle offset timing for Air/Water data files. 20170509.

• Fixed typo in finding time interval in markdown files; Report_Aggregate.RMD, Report_Stats.RMD, and Report_QC.RMD. 20170510.

• Offset time collections (e.g., Air and Water data in same file but not starting at the same time). Fixed fun.QC.R(fun.QC() and fun.CalcQCStats()) to account for this disparity. Previous iteration flagged every other row. Rows without data are no longer flagged (unless the data at the regular time interval was missing). 20170512.

• Applied fix from fun.QC.R for difftime to report markdown files; Report_Aggregate.RMD, Report_Stats.RMD, and Report_QC.RMD.. 20170512.

• Modified example files in /data-raw/ to use "Sensor Depth" instead of "Water Level". Reran ProcessData scripts to recreate data files. 20170512.

• fun.QC(); moved "Working" message inside of IF statement to only reports to user if that data type is present in the data. Avoids printing to the console all 11 parameters being checked. 20170512.

• config.R; ContData.envmyNam**e.Turbiditymodifiedvalu**e, ContDat**a.envmyName.Chlorophylla modified typo in name (and changed value), ContData.envmyLab.Chlorophyllamodifiedvaluetypo, ContDat**a.envmyThresh.RoC.SD.number.Chlorophylla name fix, ContData.envmyNam**e.Fla**g.Chlorophyllanamefix, ContDat**a.envmyUnits.Chlorophylla name fix. 20170512.

• Offset data files. Fix overall data flags for each parameters. Currently giving "NA" if any "NA". And have "NA" due to offset data. fun.QC.R. 20170512.

v2.0.1.9004

2017-05-05

• Reformat file comments for proper outlining in RStudio. 20170505.

• Fix development version from 2.0.1.0004 to 2.0.1.9004. 20170505.

• Update Description with version number (last edit was 2.0.1.0000). 20170505.

v2.0.1.0003

2017-05-05

• Change WaterLevel to SensorDepth; config.R, config_COLD.R, Report_Aggregate.RMD, Report_QC.RMD, fun.QC.R, fun.Stats.R, zfunAggregateData.R. 20170505.

• Changed WaterLevel to GageHeight in fun.GageData.R and added new parameter to config.R. Added to config.R with thresholds. 20170505.

• Corrected flag names for Turbidity and Chlorophyll a; config.R. 20170505.

v2.0.1.0002

2017-05-04

• Corrected error in Julian Day (0:364 instead of 1:365). Fixed in fun.Stats.R (line 131) by adding "+1". 20170320.

• Added 5 parameters (Conductivity, Dissolved Oxygen, pH, Turbidity, and Chrlorphyl a) to those that can be evaluated. Changes made to env.UserDefinedValues.R and fun.QC.R. 20170323.

• Renamed env.UserDefinedValues.R to config.R. 20170421.

• Added "config" functions to load user configuration (e.g., thresholds specific to coldwater streams) [fun.CustomConfig.R] 20170421.

• Tweak Reports (Report_Aggregate.RMD, Report_QC.RMD, and Report_Stats.RMD) and fun.QC.R for determining time frequency for sampling. Was using interval for 10th and 11th samples (or 4th and 5th in fun.QC.R). Now using median of all time differences. Will always work as long as have at least 1 sample. Previously could fail with less than 5 samples (or 11).

v2.0.1.0001

2017-03-08

• Bug fix for missed references to new environment for variables introduced in v2.0.1. Multiple files affected. Gage data, QC, and Aggregate working but not Stats. One table in the QC and Aggregate reports has been commented out. (To be fixed later).

v2.0.1

2017-03-06

• Bug fix for variable names not showing up.

• Created environment ContData.env and assigned all variables in fun.UserDefinedValues.R file to it.

• Renamed fun.UserDefinedValues.R to env.UserDefinedValues.R.

• Converted all RMD and R files in package to use ContData.env$.

• Added NEWS.md

• Edited earlier fix to fun.QC.R for adding day, month, year to data. Did not use variable names in previous edit.

• Create RMD files for ReadMe and NEWS. Added RMD to ignore list.

v2.0.0

2017-02-28

• Released on GitHub as a fully documented package.

• Update for TN, Modified QC report (RMD) to summarize number of samples differently.

• Included tables in Aggregate report (RMD).

• Reworked scripts to be run as a library.

• Uploaded to GitHub.

v1.2.1

2017-01-16

• Update for NJ, fixed date/time issue when resaving files in Excel.

• Added date/time QC process to be run again at the aggregate step.

v1.2.0

2016-05-04

• NWQMC 2016, Tampa, FL

v1.1.0

2016-03-31

• AMAAB 2016, Cacapon, WV

• Minor

v1.0.0

2015-10-26

• SWPBA 2015, Myrtle Beach, SC

• Initial public release.