Quicklook

[rstaten]

This PR:

• Simplifies the QuickLook configuration file (an example yaml file desispec/data/example-config-r0-00000002.yaml is provided).
• Provides the user an option to produce either a simplified or fully expanded version of a QuickLook configuration file via desi_quicklook command line arguments.
• Fixes an index issue in the SkyPeaks QuickLook QA.

[sbailey]

@afausti can you take a look at this to confirm that it is consistent with the configuration from QLF? Thanks.

[rstaten]

Below is an example of how to run QuickLook based on the most recent commit using the example configuration file provided in desispec/data/quicklook:

desi_quicklook --config_file example-config-r0-00000002.yaml --night 20160728 --camera r0 --expid 2

Unless otherwise specified using the keyword --rawdata_dir, desi and fibermap files must be located in $QL_SPEC_DATA/night.

[rkehoe]

@sbailey. I recommend holding on merging this PR until the next set of updates which will fix several of the outstanding items raised by @afausti, as well as addressing other feature requests, etc. These should arrive early In the week.

[gdhungana]

@sbailey I am not fully convinced to rebasing this branch to my last commit, as now the master has been merged and there are too many commits. While @rstaten 's latest push is safe locally and on progress, and will be updated next time, we decided to open a new branch, to avoid any potential headache. I'll send a separate PR from there soon. @rkehoe Please confirm if we need to keep this PR/branch?

[rkehoe]

@rstaten we should continue with xwsigma devel. in this branch.

[rkehoe]

@sbailey This branch can now be considered for merge with one caveat noted below. It incorporates changes that accomplish:

• Update to XWSigma QA which implements a 2D Gaussian psf in the testing.
• incorporates a pipeline configuration for arcs processing to 'PSF' files, including with resolution information.
• implements run-time params and metrics that follow discussions from Utah (May) and the June Collab. Meeting. These are mostly placeholders for the moment for the purpose of manifesting the data to QLF for development and testing, and updates based on the Hack Day will come in the next update.
• remove code causing IndexError seen in test_ql_qa on Trivial change to support loadable modules on OS X, where the extensi… #413 by @tskisner

The remaining issue with this PR concerns the error:

ImportError: numpy.core.multiarray failed to import

Which appears to be with the offline pipeline, not QL. Can @sbailey suggest a solution here? Once that is addressed, this PR should be ready to merge.

[tskisner]

The numpy.core.multiarray failure seems to be some feature of the travis environment. It does not show up locally for me, and it also shows up in a branch that has only trivial changes from master. Seems like something changed in the last ~month in the test environment. I will try to figure that out today some time unless someone beats me to it.

[geordie666]

Thanks, @tkisner. I'm seeing the same errors on my latest branches of desitarget:

https://travis-ci.org/desihub/desitarget/jobs/249323894

So this appears to be general to the test environment.

[rkehoe]

@sbailey It appears @tskisner's update may have fixed the numpy problem with this branch. However, tests are still failing in a travis job_stages error (line 54) where one of the QL unit tests
Is being Killed for an unspecified reason. No changes to QL code, including this test, were implemented. @rstaten has noted that this error seems to happen on other github areas when there is a memory leak. Are there other anomalies showing up in Travis that could lead to a fix. I would rather not turn off this test to permit a merge.

[sbailey]

The travis updates did change from a very old Ubuntu version (12.04) to a merely old version (14.04). Travis docs say this has 4 GB of memory, but we have hit memory limits before from memory hungry specsim (see desihub/specsim#67 and desihub/specsim#68). Options:

• Isolate which test(s) are causing the problem and temporarily disable them on Travis, and then chase this issue in a separate PR. i.e. put this in front of any memory hungry tests: @unittest.skipIf('TRAVIS' in os.environ)
• refactor tests to use coarser or shorter wavelength grid to use less memory
• refactor tests to not use multiprocessing so that they remaining one core instead of 2 (might help, might not depending upon what is going on when we run out of memory).

Note that due to stdout caching, the last line in the log may not be the thing that was running when the job was killed. Previously I have debugged this with running the tests in one window on my laptop and watching "top" in another window, and noting what was running when the memory spikes.

[sbailey]

Having cited memory hungry specsim, do the QL tests use specsim to generate input data? I don't think so... in which case the memory usage problems are elsewhere (resolution matrix is always a suspicious one to look at). If you can identify which test is using lots of memory that will give us further clues of how to proceed.

[rstaten]

The memory issue was not with any particular test, but with the input files being created by the test setup. In particular, a pix file for the xwsigma test was being created more times than necessary, leading to the increase in memory. I moved this to the test itself, so that the file is only created once. This seems to have solved the issue as the travis tests now pass.

[rkehoe]

@sbailey I will merge this shortly.

[sbailey]

thanks for cleaning up the memory usage in the tests. merge when ready.

[tkisner]

I don’t think these emails are for me. From: rstaten [mailto:[email protected]] Sent: Wednesday, July 12, 2017 12:49 PM To: desihub/desispec <[email protected]> Cc: Travis Kisner <[email protected]>; Mention <[email protected]> Subject: Re: [desihub/desispec] Quicklook (#389) The memory issue was not with any particular test, but with the input files being created by the test setup. In particular, a pix file for the xwsigma test was being created more times than necessary, leading to the increase in memory. I moved this to the test itself, so that the file is only created once. This seems to have solved the issue as the travis tests now pass. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#389 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKC6J67gAwwfZDWf16fBO5OAXrRdAfeks5sNQcWgaJpZM4Npti3> .

[tskisner]

@geordie666 , my user name is @tskisner, not "tkisner", which is some other user.