NEWS.rmd

---
title: "NEWS-ContDataQC"
output: github_document
  #md_document:
   # variant: markdown_github
---

<!-- NEWS.md is generated from NEWS.Rmd. Please edit that file -->

```{r, echo = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "NEWS-"
)
```

```{r, echo = FALSE}
cat(paste0("Last Update: ",Sys.time()))
```

# Planned Updates
  
* ~~Spell out "AW"" and other abbreviations (e.g., AirWater).  20170308.  On hold.~~

* ~~Gaps in data not always evident in the plots.  20170308.~~

* ~~Use futile.logger to better log output for user.  Issue #29. 20170606.~~

* Debug Aggregate operation.  20170919.

* Create CDFs.  Similar to code already used in previous analyses by Lei.  20170919.

* PeriodStats(), add number and/or percent of observations above given threshold.  20170919.

* Update Vignette after any updates to help or documentation.  20170919.

* Run devtools::check().  20170926.

* Fix mixed case issue with filenames in "file" versions of QCRaw, Aggregate, and Stats.  20170929.

    + More check data stuff.
    + Update vignette
    + Threshold number or pct on plot
    + CSV issue
    + Excel file update (see 9/28 email)

# Version History

## v2.0.1.9056
2017-11-30

* PeriodStats

    + Added data input requirements.  20171130.
    
    + Updated example for default report from DOCX to HTML.  20171130.
    
    + Updated Vignette.  20171130.
    
## v2.0.1.9055
2017-11-30

* Export.IHA

    + Updated example to use system username.  20171130.
    
    + Repeated SiteID line in example.  20171130.
    
    + Updated Vignette.  20171130.

## v2.0.1.9054
2017-11-30

* Export.StreamThermal

    + Update Excel file with category definitions from journal article.  20171130.
    
    + Add metric definition Excel file to extdata so can use when create data export.  20171130.
    
    + Update examples in vignette.  20171130.

## v2.0.1.9053
2017-11-29

* Readme, fix typo in github install function.  20171129.

* Update example in Export.StreamThermal so is self contained. Vignette did not need to be updated. Included example code for export to Excel (same as in Export.IHA example).  20171129.

## v2.0.1.9052
2017-11-29

* Update "=" to "<-" in Export.IHA example in function and vignette.  20171129.

* SummaryStats (File) example for non-ContDataQC file.  Add minimum necessary fields to a USGS download.  20171129.

* Update vignette:

    + Revised config.R thresholds.  20171129.

    + SummaryStats (File) for non-ContDataQC file.  20171129.

## v2.0.1.9051
2017-11-29

* Reports, typo in "Generated by:".  20171129.

## v2.0.1.9050
2017-11-27

* Updated some QC thresholds in config.R.  20171127.

## v2.0.1.9049
2017-11-16

* Fix error in myReport.format and then examples.  20171115.

* Renamed report RMD files and altered reference in PeriodStats().  20171115.

* Added myReport.Format to config.R and set the default there rather than in code.  20171115.

* Add "username" to reports.  20171116.

* Update vignette.  20171116.

## v2.0.1.9048
2017-11-15

* Add variable for report format (docx or html).  Default will be "docx".  Needed for Shiny app since won't have Pandoc.  Pandoc now ships with RStudio but still don't have when run package via Shiny.

* Test version.  Not complete.  Vignette needs to be rebuilt.

## v2.0.1.9047
2017-09-29

* Aggregate.File.  Example file mispelled (AW vs. Aw).  20170928.

* File versions of QCRaw, Aggregate, and Stats; change input and output directory name.  20170929.

* File QC.  Turn off report.  Mixed case issue in file names.  20170929.

## v2.0.1.9046
2017-09-28

* Add packages used in vignette to "Suggests" in DESCRIPTION.

* Fixed function example lines wider than 100 characters.

* Moved necessary packages from "Suggests" to "Imports" in DESCRIPTION.

* DESCRIPTION, License, out of sync with GitHub.  Fixed.

## v2.0.1.9045
2017-09-28

* Add knitr:: to all kable commands in RMD files.  Had error after moving knitr from Depends to Suggests. 20170928.

* Renamed RMD files with ' - ' to '_'.  Recommended in check().  20170928.

## v2.0.1.9044
2017-09-28

* CompSiteCDF funtion complete.  20170928.

* Vignette updated.  20170928.

## v2.0.1.9043
2017-09-27

* DESCRIPTION.  As suggested by check(), move packages from Depends to Suggests.  20170927.

* Update ReadMe.  20170927.

## v2.0.1.9042
2017-09-21

* CompSiteCDF() to compare multiple sites on a site CDF.  20170921.

## v2.0.1.9041
2017-09-20

* Export.StreamThermal.  
+ Rename from Export.ST() to Export.StreamThermal() .20170920.
+ Complete function and documentation. 20170920.
+ Update Vignette.  20170920.

* Export.IHA.  Tweak opening text.  20170920.

## v2.0.1.9040
2017-09-20

* Vignette. 
    + Tweaked language.  20170919.
    + Added section about requiring dataRetrieval v2.7.3 that isn't on CRAN yet.  20170919.

## v2.0.1.9039
2017-09-19

* Vignette added.  20170919.

* Added daily mean visualization Excel file to \extdata.  20170919.

## v2.0.1.9038
2017-09-12

* Rename fun.Export.IHA() to Export.IHA().  20170912.

## v2.0.1.9037
2017-09-12

* Added function, rarify(), from RIVPACS code by John van Sickles (USEPA).  Modified to be a stand alone function and with a seed parameter for reproducible results.  Included example dataset with 2 samples.  20170912.

* Added "Bug Reports" to DESCRIPTION file.  20170912.

* Modified fun.Export.IHA() argument fun.myCol.Discharge to fun.myCol.Parameter.  20170912.

## v2.0.1.9036
2017-09-11

* Added function, fun.Export.IHA(), to export data in format ready to use in IHA package.  20170911.

## v2.0.1.9035
2017-09-11

* Updated fun.GageData.R.  USGS updated their webservices to use https instead of http.  dataRetrieval package updated to v2.7.3 on GitHub.  Some functions modified within dataRetrieval as had to modify some code in ContDataQC package.  Version number of dataRetrieval *not* added to DESCRIPTION (not available on CRAN yet).  20170911.

## v2.0.1.9034
2017-09-06

* Added PeriodStats() to calculate daily stats (mean, min, max, etc) for a period (N) before a given date.  Outputs a CSV of data and DOCX report.  20170905.

## v2.0.1.9033
2017-08-24

* Changed reports back to DOCX.  Will add a user parameter to change in the future.  20170824.

## v2.0.1.9032
2017-08-23

* Added rmarkdown to dependencies in Description.  It should install with Pandoc but some users had to install separately.  20170823

* Added install.packages("installr") to example in ContDataQC() for installing Pandoc.  20170823.

## v2.0.1.9031
2017-08-23

* Reload on GitHub.  Shiny can't download.  20170823.

## v2.0.1.9030
2017-08-23

* Another DOCX reference fix. 20170823.

## v2.0.1.9029
2017-08-23

* Change reference to DOCX variable to .ext to easier to change in the future.  20170823.

## v2.0.1.9028
2017-08-23

* Convert Reports from DOCX to HTML.  Needed for use with Shiny.  20170823.

## v2.0.1.9027
2017-08-23

* Rebuild before update to GitHub.  20170823.

## v2.0.1.9026
2017-08-23

* Ensure use file.path when read and save files.  20170823.

## v2.0.1.9025
2017-07-31

* Thresholds.  Tweaked config.R.  Issue #36.  20170731.

## v2.0.1.9024
2017-07-27

* Thresholds.  Updated config.R with more informed thresholds after working with more data.  Issue #36.  20170728.

* DESCRIPTION.  Added Jen Stamp as contributer and added my middle initial.  20170728.

## v2.0.1.9023
2017-06-22

* QC report (and aggregate report), fixed water temp vs. sensor depth legend.  Colors were swapped.  Issue # 35.  20170622.

## v2.0.1.9022
2017-06-21

* QC report header for water temp and sensor depth says water level.  Fixed.  Issue #34.  20170621.

## v2.0.1.9021
2017-06-21

* Dec 31 not being assigned a season in summar stats.  Fixed.  Issue #33. 20170621.

## v2.0.1.9020
2017-06-09

* QC of gage data not using all parameters present.  Fixed. Issue #32. 20170609.

* Added air and water pressure in psi to gage download.  fun.GageData.R. 20170609.

* ContData.env$myNames.Order missing new fields.  Was dropping fields not included (e.g., gage height).  config.R.  20170609.

* fun.QC.R missing QC routine for gage height. 20170609.

## v2.0.1.9019
2017-06-07

* Updated data flow for "aggregate" step so the QC report is generated "after" the compiled file is created.  Issue #31. 20170607.

## v2.0.1.9018
2017-06-07

* Updated main function to allow for a single file or a vector of files.  New "file" versions of most scripts.  Some updates to the Reports (rmd files). Issue #24. 20170607.

## v2.0.1.9017
2017-06-06

* Report_QC.rmd.  Correct typo in "Flags" descriptions.  Issue #28.  20170606.

## v2.0.1.9016
2017-05-31

* File delimiter for strsplit().  Also a regex issue with "." (dot).  Created special check.  20170531.

## v2.0.1.9015
2017-05-31

* File delimiter.  Configuration file allows for user defined values.  However, the QC check for StationIDs containing this value uses regular expressions.  Added a special condition for the dot, ".", delimiter as regex treats this character as a match for (almost) any character.  20170531.

## v2.0.1.9014
2017-05-26

* Time interval.  Fixed calculation of time interval.  Certain date time formats would stop processing or be reported as zero.  fun.QC.R, Report_Aggregate.rmd, Report_QC.rmd, and Report_Stats.rmd.  20170526.

## v2.0..1.9013
2017-05-24

* fun.Stats.R. Had previously commented out stats and plots due to errors.  Single typo fixed in year season stats and reworked daily summaryBy stats section.  20170524.

## v2.0..1.9012
2017-05-24

* fun.Gage.R.  Changed variable "myTZ" to "fun.myTZ" for consistency.  20170524.

## v2.0.1.9011
2017-05-23

* Remove unused arguement "fun.myFile.Prefix" from help file for ContDataQC().  20170523.

## v2.0.1.9010
2017-05-22

* Modify input and export directories so can be any folder.  The default directory is now the working directory.  Modified input parameters for ContDataQC() by removing "fun.myDir.BASE".  Modified examples and all other necessary scripts.

## v2.0.1.9009
2017-05-19

* Report_Aggregate.rmd.  Fix sampling interval (seconds to minutes,line 23).  Left plots as dots instead of lines.  20170519.

* ContDataQC().  Update examples to with date range.  After adding in "test4" the dates don't match for "test2".  20170519.

* Update "test4" dataset.  Change "Water Level ft"" to "Sensor Depth ft".  20170519.

* Summary Stats quit with error.  Fixed in summaryBy statements.  20170519.

## v2.0.1.9008
2017-05-18

* Add example for offset data for QC in main function (ContDataQC()).  20170518.

* Fix plots (legend) in Report_QC.R (remove box as it is only partially displaying).  20170518.

* Config file.  Upate Sensor Depth and Water Pressure display names.  20170518.

* Label code chunks in RMD files.  20170518.

* Fix plots (blank) in Report_QC.R for "offset" data.  With point type of "line" the NAs in the data caused nothing to be plotted.  Fixed with na.omit(as.numeric(x)) when subsetted data for plotting.  Then had to fix date range on x-axis.  20170518.

## v2.0.1.9007
2017-05-17

* Add additional config file.  Modified config.R to include OlsonNames() in timezone section.  Added /extdata/config.TZ.central.R with only time zone change.  Modified examples in zfun.ContaDataQC.R to include a USGS gage in the central time zone.  Modified ContDataQC() with optional parameter to use helper functions to include different config files.  Will only need paramters that are different.  User provides path to the new R file for new variables to include in the environment ContData.env.

## v2.0.1.9006
2017-05-17

* zfun.ContDataQC.R.  Add in "test4" data (offset times) to examples.  20170515.

* config.R.  Fix typo in ContData.env$myName.GageHeight; missing "." between name and unites.  20170517.

* fun.QC.R.  Line 1034 bad reference for difftime; renamed myTimeDiff.all to myT.diff.all.

* Comnfirmed that knitr is silent so the user does not get code scrolling across screen.  20170517.

* Confirmed that steps/tasks in the process are independant of each other.  That is, someone can use some or all routines as each is a separate function.  20170517.


## v2.0.1.9005
2017-05-15

* Add data.R to desribe data inlcuded in library.  20170508.

* fun.QC.R.  Fix typo in myT.diff.all operation (line 965).  20170508.

* Create data process script in "data-raw" for three example "test1" files.  20170509.

* Revised fun.OffsetCollectionCheck() in fun.Helper.R.  Return value is not boolean.  20170509.

* Revised fun.CalcQCStats() in fun.QC.R.  Added ability to handle offset timing for Air/Water data files.  20170509.

* Fixed typo in finding time interval in markdown files; Report_Aggregate.RMD, Report_Stats.RMD, and Report_QC.RMD.  20170510.

* Offset time collections (e.g., Air and Water data in same file but not starting at the same time).  Fixed fun.QC.R(fun.QC() and fun.CalcQCStats()) to account for this disparity.  Previous iteration flagged every other row.  Rows without data are no longer flagged (unless the data at the regular time interval was missing).  20170512.

* Applied fix from fun.QC.R for difftime to report markdown files; Report_Aggregate.RMD, Report_Stats.RMD, and Report_QC.RMD..  20170512.

* Modified example files in /data-raw/ to use "Sensor Depth" instead of "Water Level". Reran ProcessData scripts to recreate data files.  20170512.

* fun.QC(); moved "Working" message inside of IF statement to only reports to user if that data type is present in the data.  Avoids printing to the console all 11 parameters being checked.  20170512.

* config.R;  ContData.env$myName.Turbidity modified value,  ContData.env$myName.Chlorophylla modified typo in name (and changed value), ContData.env$myLab.Chlorophylla modified value typo, ContData.env$myThresh.RoC.SD.number.Chlorophylla  name fix, ContData.env$myName.Flag.Chlorophylla name fix, ContData.env$myUnits.Chlorophylla name fix.  20170512.

* Offset data files. Fix overall data flags for each parameters.  Currently giving "NA" if any "NA".  And have "NA" due to offset data.  fun.QC.R.  20170512.

## v2.0.1.9004
2017-05-05

* Reformat file comments for proper outlining in RStudio.  20170505.

* Fix development version from 2.0.1.0004 to 2.0.1.9004.  20170505.

* Update Description with version number (last edit was 2.0.1.0000).  20170505.

## v2.0.1.0003
2017-05-05

* Change WaterLevel to SensorDepth; config.R, config_COLD.R, Report_Aggregate.RMD, Report_QC.RMD, fun.QC.R, fun.Stats.R, zfunAggregateData.R.  20170505.

* Changed WaterLevel to GageHeight in fun.GageData.R and added new parameter to config.R. Added to config.R with thresholds.  20170505. 

* Corrected flag names for Turbidity and Chlorophyll a; config.R.  20170505.

## v2.0.1.0002
2017-05-04

* Corrected error in Julian Day (0:364 instead of 1:365).   Fixed in fun.Stats.R (line 131) by adding "+1".  20170320.

* Added 5 parameters (Conductivity, Dissolved Oxygen, pH, Turbidity, and Chrlorphyl a) to those that can be evaluated.  Changes made to env.UserDefinedValues.R and fun.QC.R.  20170323.

* Renamed env.UserDefinedValues.R to config.R. 20170421.

* Added "config" functions to load user configuration (e.g., thresholds specific to coldwater streams) [fun.CustomConfig.R] 20170421.

* Tweak Reports (Report_Aggregate.RMD, Report_QC.RMD, and Report_Stats.RMD) and fun.QC.R for determining time frequency for sampling.  Was using interval for 10th and 11th samples (or 4th and 5th in fun.QC.R).  Now using median of all time differences.  Will always work as long as have at least 1 sample.  Previously could fail with less than 5 samples (or 11).

## v2.0.1.0001
2017-03-08

* Bug fix for missed references to new environment for variables introduced in v2.0.1.  Multiple files affected.  Gage data, QC, and Aggregate working but not Stats.  One table in the QC and Aggregate reports has been commented out.  (To be fixed later).

## v2.0.1
2017-03-06

* Bug fix for variable names not showing up.

* Created environment ContData.env and assigned all variables in fun.UserDefinedValues.R file to it.

* Renamed fun.UserDefinedValues.R to env.UserDefinedValues.R.

* Converted all RMD and R files in package to use ContData.env$.

* Added NEWS.md

* Edited earlier fix to fun.QC.R for adding day, month, year to data.  Did not use variable names in previous edit.

* Create RMD files for ReadMe and NEWS.  Added RMD to ignore list.

## v2.0.0
2017-02-28

* Released on GitHub as a fully documented package.

* Update for TN, Modified QC report (RMD) to summarize number of samples differently.  

* Included tables in Aggregate report (RMD).  

* Reworked scripts to be run as a library.  

* Uploaded to GitHub.

## v1.2.1
2017-01-16

* Update for NJ, fixed date/time issue when resaving files in Excel.  

* Added date/time QC process to be run again at the aggregate step.

## v1.2.0
2016-05-04

* NWQMC 2016, Tampa, FL

## v1.1.0
2016-03-31

* AMAAB 2016, Cacapon, WV

* Minor 

## v1.0.0
2015-10-26

* SWPBA 2015, Myrtle Beach, SC

* Initial public release.