Implementation of ATLAS_WJ_8TEV in the new format

[achiefa]

Re-implementation of the ATLAS_WJ_8TEV in the new commondata format. I summarise the relevant information below, also as a reminder for myself.

$(x, Q^2)$ map and data-theory comparisons

Legacy reports: [legacy], [legacy_ATLAS], [legacy_NP]
New reports: [default HepData v1], [default HepData v2]
default = symmetrized,

Description

This dataset provides a correlation matrix for statistical uncertainties. Systematic uncertainties are given for diagonal entries only. Quoting from the ATLAS paper:

All uncertainties of a statistical nature, such as the statistical uncertainty of the data, the statistical uncertainty of simulated samples used in the background estimate, or the uncertainty from limited sample size of the signal simulation used in the unfolding are treated as uncorrelated between bins and between $W^+$ and $W^−$ production. All other systematic uncertainties are treated as fully correlated between bins and between the production of $W^+$ and $W^−$ bosons.

The statistical uncertainties provided in the HepData table are:

Source	Type	Corr. b/w bins and $W^{\pm}$	Correalted with other experiments
JetScaleEff1	Systematic	Yes	No
JetScaleEff2	Systematic	Yes	No
JetScaleEff3	Systematic	Yes	No
JetScaleEff4	Systematic	Yes	No
JetScaleEff5	Systematic	Yes	No
JetScaleEff6	Systematic	Yes	No
JetScaleEta1	Systematic	Yes	No
JetScaleEta2	Systematic	Yes	No
JetScaleHighPt	Systematic	Yes	No
JetScaleMC	Systematic	Yes	No
JetScalePileup1	Systematic	Yes	Yes
JetScalePileup2	Systematic	Yes	Yes
JetScalePileup3	Systematic	Yes	No
JetScalePileup4	Systematic	Yes	No
JetScaleFlav1Known	Systematic	Yes	Yes
JetScaleFlav2	Systematic	Yes	Yes
JetScaleBjet	Systematic	Yes	No
JetScalepunchT	Systematic	Yes	Yes
JetResolution10	Systematic	Yes	No
JetSFBeff	Systematic	Yes	No
JetSFCeff	Systematic	Yes	No
JetSFLmistag	Systematic	Yes	No
JetSFHighPt	Systematic	Yes	No
JetJVFcut	Systematic	Yes	Yes
ElScaleR12	Systematic	Yes	No
ElScaleZee	Systematic	Yes	No
ElScalePS	Systematic	Yes	No
ElResolution	Systematic	Yes	No
ElSFReco	Systematic	Yes	No
ElSFId	Systematic	Yes	No
ElSFTrigger	Systematic	Yes	No
ElSFIso	Systematic	Yes	No
ElSFChargeMisID	Systematic	Yes	No
METScale	Systematic	Yes	No
METResLong	Systematic	Yes	No
METResTrans	Systematic	Yes	No
PileupWeight	Systematic	Yes	No
QCDlowRange	Theory	Yes	No
QCDhighRange	Theory	Yes	No
QCDvarIso	Theory	Yes	No
QCDvarElID	Theory	Yes	No
QCDfitUncert	Theory	Yes	No
QCDotherGen	Theory	Yes	No
QCDfitRebin	Theory	Yes	No
XsecZ	Theory	Yes	No
XsecTop	Theory	Yes	No
XsecDibos	Theory	Yes	No
BkgTtbarNorm	Theory	Yes	No
BkgMCstat	Systematic	Yes	No
WHFmodel	Theory	Yes	No
LumiUncert	Systematic	Yes	Yes
UnfoldMCstat	Stat	No	No
UnfoldOtherGen	Stat	No	No
UnfoldReweight	Stat	No	No

Please, @enocera @scarlehoff, feel free to check if this breakdown has flaws, as matching with the paper was not trivial.

There are a few things to mention:

1. The experimentalists deliver the correlation matrix for the statistical uncertainties. Hence, there will be 16 additive and correlated artificial uncertainties.
2. Some systematic uncertainties correlate between bins and between $W^+$ and $W^-$ productions, as can be read from the second column. There are 50 + 1 (lumi) correlated systematic uncertainties in total and possibly 3 uncorrelated statistical uncertainties (if I'm not wrong). Since the systematic uncertainties between $W^{\pm}$ productions correlate, I will use the same type of label in the common data (ATLASWX).
3. The majority of the uncertainties are asymmetric. In the legacy implementation, these were treated using the CMS prescription (eq. 6 of 1703.01630). Roughly speaking, in this prescription the upper and lower bounds of the asymmetric uncertainties are treated as different artificial sources. The same argument applies to symmetric uncertainties, the only exception being the uncertainty associated with the luminosity. Hence, the legacy implementation has $(50+3) * 2 (sys)+ 16 (stat.) + 1 (lumi.) = 123$ artificial sources. @enocera suggested using the symmetric prescription as default. I will implement this CMS prescription as a variant.
4. There is a third column in the table. This column expresses the inter-experiment correlations, explained in this paper. I post the relevant table below. Accounting for these inter-experiment correlations necessitates common uncertainty type flags for 8 TeV $W$ + jets, 7 TeV $W$ + jets, $t\bar{t}$ lepton + jets data at 8 TeV and 13 TeV, and inclusive jets at 8 TeV datasets. If you agree, I will start by neglecting these correlations and leave them for a second iteration across the commondata.

For the record, there is a third variant in the legacy implementation, which accounts for non-perturbative corrections (NP) and that I will ignore for the moment since they were not used in NNPDF4.0.

[achiefa]

I have implemented the two variants I discussed in the description - default (symmetrized) and CMS_prescription (see above). However, the $\chi^2$ is notably worse than the legacy implementation. However, this could not be a problem since the HepData tables have been updated and modified since the first legacy implementation (see 'Version 2 modifications' in HepData). Below you can find data-theory comparisons with the new implementation for the two variants.
New reports: [sym], [cms]

[achiefa]

I attach the comparison between the two covmats that should be the same (but they are not).

[scarlehoff]

What do you mean they should be equal? If they are not, does that means there's a bug in the old, in the new...?

[achiefa]

I mean that the legacy implementation adopted the cms prescription to treat the asymmetric uncertainties as default. Hence, the variant CMS_prescription should be the same as the legacy one. Now, I have the following observations:

1. The HepData tables have been updated since the legacy implementation. In particular, they changed the systematic luminosity uncertainty.
2. I agree that there could be a bug in the old implementation, but that is hard to say from my point of view.
3. There could be a bug in my implementation, but I have not found any.
   However, ERN and I agreed to use the symmetrised variant as default for NNPDF4.1. The CMS prescription will no longer be used, and that variant was just a plus that I implemented.

[scarlehoff]

The HepData tables have been updated since the legacy implementation. In particular, they changed the systematic luminosity uncertainty.

Ok, then they should not be equal, right?

[achiefa]

Yes, they should not be equal

[scarlehoff]

So, the implementation itself I don't see anything wrong however, I'm slightly worried about the chi2 changes. The CMS prescription should've kept the results at least similar, however the change in the chi2 is huge (from ~1 to ~1.5), in the "default" it goes all the way to chi2 ~ 3... and you left a few TODO in the filter_utils and filter files. Are those to-do no longer applicable or some of those checks would change the results?

@enocera is this expected? I am happy with merging this as it is and revising possible problems in the future if you are ok with it.

@achiefa please, remove the old kinematics file

[achiefa]

and you left a few TODO in the filter_utils and filter files. Are those to-do no longer applicable or some of those checks would change the results?

You're right. I'll check whether these TODO's affect the results. Thank you for pointing this out.

[enocera]

@achiefa @scarlehoff This is unexpected. @achiefa If I understand correctly, there are two versions on Hepdata. what happens if you use the other version?

[achiefa]

@enocera It doesn't improve with the older version either. BTW, I was looking carefully at the single sources of systematic uncertainties and I observed that, in some cases, the plus-minus signs are inverted (see the snapshot below). Do these uncertainties require a particular treatment?

[enocera]

@achiefa The case that you have highlighted has symmetric uncertainties upside down. In this case you should also use the symmetrisation formula by D'Agostini, wich will return a "symmetrised" uncertainty of -4 (and not +4). This means that the uncertainty is anticorrelated. Are you sure that the sign is correctly taken into account?

[achiefa]

Ok, I think I was mistakenly taking the absolute value of some numbers. Now the symmetrized version (which is going to become the default variant in the new implementation) gives much better results [see here].

[scarlehoff]

Great! This is much better. Not only that, it is close to the numbers one gets with the NP variant which I think itis expected, right?

[enocera]

Great! This is much better. Not only that, it is close to the numbers one gets with the NP variant which I think itis expected, right?

Indeed!

[achiefa]

Wait, the link to legacy_NP points to the wrong report. Let me update it.

[achiefa]

Ok, updated. Honestly, I don't see much difference between legacy and legacy_NP.

[achiefa]

Ok, now I have uploaded the data-theory comparisons for version 1 and version 2 of the HepData tables.

[scarlehoff]

Nice. Is this ready then?

[achiefa]

Yes, it is. The inter-experiment correlations are still missing, but my understanding is that we will address them in a second iteration. Maybe it is worth making a note pointing to this PR for reference. In the description I put all the necessary information to include these correlations.

[scarlehoff]

Perfect! Thanks

[scarlehoff]

Sorry, just one question, then the CMS version of the prescription we don't want it? Is it because it is exactly the same as before?

[achiefa]

The CMS prescription was the one adopted in the legacy implementation and I didn't manage to reproduce it. However, after a chat with @enocera, we agreed to neglect this prescription so that its implementation is entirely unnecessary.