Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: bad datetimes in DE2 RPA data result in non-monotonic series #221

Open
jklenzing opened this issue Dec 19, 2023 · 0 comments
Open

BUG: bad datetimes in DE2 RPA data result in non-monotonic series #221

jklenzing opened this issue Dec 19, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@jklenzing
Copy link
Member

Description

Some of the data files for DE2 RPA improperly store the time epoch. This results in a non-monotonic index

To Reproduce this bug:

Steps to reproduce the behavior:

import pysat
rpa = pysat.Instrument('de2', 'rpa', strict_time_flag=False)
rpa.load(1983, 29)
rpa.data

yields

                         dataQuality      x           y            z  ionTemperature  ...       glat        glon       ilat   mlt         alt
1983-01-29 23:59:55.969            0  411.0  105.610001   310.600006          1463.0  ...  68.669998   30.740000  65.360001  3.23  350.589996
1983-01-29 23:59:59.969           60  126.0   83.610001   343.600006          1385.0  ...  68.940002   30.730000  65.610001  3.29  350.850006
1983-01-29 00:00:03.967           60  226.0   92.610001   337.600006          1357.0  ...  69.199997   30.709999  65.860001  3.32  351.109985
1983-01-29 00:00:07.967           60  279.0  103.610001   402.600006          1343.0  ...  69.459999   30.700001  66.110001  3.34  351.369995
1983-01-29 00:00:11.967            0  276.0   91.610001   440.600006          1505.0  ...  69.720001   30.680000  66.360001  3.36  351.619995
...                              ...    ...         ...          ...             ...  ...        ...         ...        ...   ...         ...
1983-01-29 19:10:17.039            0    1.0   12.610000   -95.599998          1131.0  ... -33.669998  102.120003  46.459999  1.55  252.960007
1983-01-29 19:10:20.039            0  -43.0   34.610001  -103.599998          1041.0  ... -33.470001  102.110001  46.230000  1.55  253.009995
1983-01-29 19:10:21.039            0   11.0   45.610001   -93.599998          1136.0  ... -33.400002  102.099998  46.160000  1.55  253.020004
1983-01-29 19:10:24.039            0    0.0   23.610001  4544.600098           993.0  ... -33.200001  102.089996  45.930000  1.55  253.070007
1983-01-29 19:10:25.039            0    0.0   23.610001  4587.600098          1112.0  ... -33.130001  102.089996  45.849998  1.55  253.089996

Manual inspection of the data here and on the preceeding day confirm that the first two data points should be for 1983-01-28. Loading the file with cdflib independently of pysat and plotting the raw epoch value confirms this is an issue with the file, not the datetime conversion in either package.

Test configuration

  • OS: Ventura 13.6.1
  • Python 3.10.9

Additional context

Since this is outside of our control, an optional routine to scrub non-monotonic files. Over the 1.5 months of 1983 data inspected, this affects four data points (including the two shown here), so dropping these points is a potential fix.

@jklenzing jklenzing added the bug Something isn't working label Dec 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant