You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Raw sensor value range is up to 32767, so sampleENU is up to 4915050. calibration.scale.axis is around 1024, so the CALIBRATED_SAMPLE calculation can involve numbers greater than 5 billion, and overflow will occur just above 2 billion.
To see it happening, add trace as follows, and move a magnet close to the sensor.
The CALIBRATED_SAMPLE calculation can overflow 32bit signed integer arithmetic.
Raw sensor value range is up to 32767, so sampleENU is up to 4915050. calibration.scale.axis is around 1024, so the CALIBRATED_SAMPLE calculation can involve numbers greater than 5 billion, and overflow will occur just above 2 billion.
To see it happening, add trace as follows, and move a magnet close to the sensor.
https://github.com/lancaster-university/codal-core/blob/master/source/drivers/LSM303Magnetometer.cpp#L181
DMESGN( "raw %d enu %d ", (int) (*z), (int) sampleENU.z);
https://github.com/lancaster-university/codal-core/blob/master/source/driver-models/Compass.cpp#L320
DMESG( "calibrated %d ", (int) sampleENU.z);
These figures are without calibration, but calibrated calibration.centre.xyz are small compared to these sampleENU.xyz values.
Related: lancaster-university/microbit-dal#472
The text was updated successfully, but these errors were encountered: