Check for flat field errors on 20231214, 20240108

[schlafly]

There were reports of arc/flat amplitudes in the cals on 20231214 and 20240108. On 20240108, we learned that the cause in that case was that the telescope was not pointed directly at the white spot, but was instead ~10 deg away. We then took a third set of cals that resolved this issue.

For 20231214 we just ran with the nominal set of cals. THRUFRAC on that night look a bit odd. We should look at the flat field and decide if we should rerun with flats (and maybe arcs?) from a different night.

There is another question as to whether we vignetted the beam during the science exposures. One thing to look for there would be low transparencies despite good conditions.

[sybenzvi]

@schlafly, following up from the survey-ops telecon, I ran some requested checks on the individual flat programs (CALIB LED-00, 01, 02, 03) for the second half of 2023.

The good news is that it looks like the outlier nights of 20231214 and 20240108 have valid calibrations on the nights before and after.

The bad news is that we have two extended runs of iffy calibrations from 20231004-20231015 and again from 20231217-20240101 and they may not be so easy to recover. I am attaching a couple of diagnostic plots showing the ratio of the temperature-corrected integral fluxes from the flat exposures in all cameras and spectrographs, expressed relative to the median T-corrected flux from the second half of 2023.

The first example shows a good night (20230924):

The color scale is chosen such that the petals are white when the flux ratio is 1, blue when the flux ratio is < 1, and red when the flux ratio is > 1. If the T-corrected flux drifts by more than 5% from the reference, the petal outline will appear orange. If the difference is > 10% the outline turns red.

Here is what a bad night (20231010) looks like:

On nights like this, the drift from the reference shows a position dependence in the focal plane that is common to all 4 LED sequences. I posted an animation of all LED exposures taken since September if you want to do a quick visual check of many nights.

I did not yet get a chance to check the combined fiberflatnight files.

[schlafly]

Thanks Segev. That animation is great. Two things I don't understand:

1. by eye, the gradients are sort of piecewise gradients that are constant in a petal.
2. the angle is not in our expected left-right direction but instead some more vertical direction

The second one is interesting but maybe we don't care. Maybe it makes me wonder if we are also displaced in dec.

The first one is kind of important. We fit out a petal-constant throughput calibration in every exposure based on comparison with calibration stars. If this petal constant removes all of the signal, we sort of don't care about the flat field anymore. So if it's somehow the case that these gradients are petal-constants, that would explain their apparently modest effect on the data. But physically I have no understanding for how a dome mispositioning would lead to petal-constant offsets, so I'm confused.

[sybenzvi]

Hi Eddie, just to clarify, the piecewise constant images are an artifact of how I extracted the data. I used existing integrated flux QA metrics calculated by Nightwatch, which compresses everything into integrated fluxes reported at the level of each petal rather than each fiber. By construction, any gradients within a petal are erased.

I did confirm that these plots qualitatively match the fiber fluxes by comparing to focal plane displays in Nightwatch... and I figured these compressed data would be good enough to check for the east-west vs. north-south gradients across the focal plane, so I took a shortcut and used the existing Nightwatch database.

Sorry for not making this clear at the start. If you think it's worth comparing fiber fluxes against a reference night, we can make those plots too. (We'll have to do that anyway to look at the fiberflatnight data.)

[schlafly]

Ah, got it. That makes sense. Too bad. If you could check at least one case (maybe in each of the two problematic time windows?) and verify that it looks like the smooth gradient we expect, that would be great.

It does also still make sense to see if somehow the fiber flats take this out, but since I see similar gradients in the THRUFRAC in QA I don't think that's likely. I guess a last question would be whether fitting out a single gradient would be adequate to repair these flats; we could imagine fitting those out.

[sybenzvi]

@schlafly, I got around to checking the flatfibernight files this morning and they seem to tell a similar story as the petal-level integrated fluxes. For example, here are the ratios of the fiberflatnights in Oct. 2023 relative to 2023-09-29 (see also here). The smooth gradient is there so zeroing it out with a simple fit could be possible.

The December data look a bit noisier (see here) but also have a smooth gradient.

[schlafly]

Perfect, okay. I'm guessing now the reason this doesn't cause huge problems is that we do fit a per-petal sky gradient. That's intended to soak up physical variation in the sky but can also soak up unintended instrumental systematics in the flat field. So maybe that's how we appear to have dodged this bullet.

@julienguy , @sbailey , we can try harder to assess whether these nights were actually bad in some way, but it is going to be subtle; I certainly hadn't noticed anything in QA. The flat fields look complicated enough that it's not obvious to me that we can just fit out a gradient and expect to be absorbing only problematic signal. But we could do it anyway given that the sky subtraction gradient will help to remove any ill effects a gradient introduces.

Thoughts about how we want to proceed here? I guess we could fit a gradient out from the calibration stars...