Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider filtering LU7AA #10

Open
TheSkorm opened this issue Jun 7, 2023 · 20 comments
Open

Consider filtering LU7AA #10

TheSkorm opened this issue Jun 7, 2023 · 20 comments

Comments

@TheSkorm
Copy link
Member

TheSkorm commented Jun 7, 2023

Why?

  • Often timestamp issues that ignored because "it works on APRS.fi"
    • Causes the map to mess up
  • Packets are often transmitted multiple times with new timestamps which breaks calculating speeds with current setup
  • Uploads to APRS are performed by web clients
    • Old versions of the site can still be running uploading bad data
    • Web client issues can cause bad data to be uploaded
    • Users can choose to upload really old packets - making a balloon look like it recently transmitted when really it hasn't
    • Endpoint used for APRS uploads is completely unauthenticated
    • Data is only uploaded when a web client is viewing the page and box ticked - so a lot of data is missed
    • Data is not rich - does not contain list of receivers, SNRs, gridsquares, raw data, telemetry
  • WSPR balloons might want to upload directly to SondeHub and will require filter / preferring direct uploads rather than APRS

Why not?

  • Many WSPR balloons are on LU7AA
  • Many WSPR balloons otherwise not interested in getting data into SondeHub (is this a problem? probably not, but it seems like a waste not having their data)
@SteveRan
Copy link

SteveRan commented Jun 7, 2023

Some thoughts:

"Many WSPR balloons otherwise not interested in getting data into SondeHub" - until the process of getting a WSPR balloon up on Sonehub is near automatic that will continue.

A few months ago (following the Chinese balloon incident and subsequent picoballoon shootdown & scrambles) great play was made of Sondehub being the definitive, single source way for the authorities to find where picoballoons are. While that's that's just an aspiration without a proper feed of WSPR picoballoons a huge chunk is being missed out.
Where Sondehub wins is bringing all the various balloon types together APRS, LoRa & WSPR (plus all the latex stuff).

I don't think its in LU7AA's interests to get this working properly.

WSPR->LU7AAserver->LU7AAclient ->APRS->SondeHub is so tenuous, we need a direct WSPR->Sondehub gateway.

@TheSkorm
Copy link
Member Author

TheSkorm commented Jun 7, 2023

SondeHub certainly has funding and infrastructure that would enable us to run a WSPR to SondeHub gateway like we do for APRS.

For that we would need:

  • api access / approval to use an API to gather WSPR data
  • documentation on how wspr balloons encode their messages (this appears to be the tricky part - I've seen some scattered documents on some, but it seems like everyone does it a new and fun exciting way)
  • ideally someone to build the decoders, but I'm sure with enough details from the point above we could work something out slowly

Maybe a consideration with disabling the LU7AA APRS ingestion is that people will be more forthcoming with above to get their respective balloons on SondeHub?

@darksidelemm
Copy link
Member

For this would also then need to maintain a registry of what timeslots/channels are in use by what callsign, and deal with the U4B frequency approach (which I still don't completely understand...).

@SteveRan
Copy link

SteveRan commented Jun 7, 2023

The APIs to WSPR and Sondehub are pretty well defined. Extracting the data from the WSPR encodings is the challenging bit. For whatever reason the encoding is not published (again I suspect its not the the manufacturers interests to make it open source). Obviously LU7AA worked it out for the main 3 types many moons ago and has evolved it as things changed - they have had the advantage of evolving with it. The traquito guy (https://traquito.github.io/) seems to have worked a lot of it out. There is detailed discussion about it in [picoballoon] posts a few months back - but I haven't been taking much notice (WSPR is of no interest to me and in my mind picoballoons are misusing it), That said its very popular and can't be ignored by Sondehub.

Your right darkside there is going to be some per flight fixed data that needs to be captured - callsigns / postfix / band / timeslots / channels /encoding scheme/ station information...... This can be used to both decode from WSPR and exclude data coming via APRS.fi Probably going to need a couple of admin guys to mange occasionally.

@SteveRan
Copy link

So what if anything are we going to do about this? Shall I invest some time into understanding how the WSPR encodings work - with a view to producing a gateway. I'd rather not duplicate any other effort - are alternatives being looked at ? - I'd rather not waste my time if its not viable / won't happen / there is a viable alternative.

@darksidelemm
Copy link
Member

Nothing is going to happen quickly with this.

I think that at least documenting the WSPR encodings is still a valuable effort however, as it will make any future efforts that much easier.

@SteveRan
Copy link

So I've started digging into it. there is a surprising amount of info embedded in the emails of the last few months and links to sites - slowly wading through them. Not been able to find much on Zachtek yet.

@TheSkorm
Copy link
Member Author

Traquito : https://traquito.github.io/js/WSPREncoded.js
Zachtek : Can probably work out from firmware? https://github.com/HarrydeBug/WSPR-transmitters/blob/master/Standard%20Firmware/Beta/WSPR-TX0.95.ino

what are the other common transmitters?

@SteveRan
Copy link

Think I have got Zachtek, SkyTracker and U4B nailed! Ask me any questions you have about it to see if I have the answers. I'm writing it up, but what I may do now to prove, is to create a gateway for a limited number of Payloads and put them up on Sonehub as test - make sure they produce identical results as the LU7AA-APRS-Sondehub feed.

It looks likely that a "payload detector" script could be written to identify new payloads as they pop up. By looking for telemetry messages and then working back to the previous message (via rx_stations, frequency and time) Think some of this is already being done by Doug Malnati in his channel map.

@darksidelemm
Copy link
Member

So I'm still hoping that we see an API from QRPLabs, which would cover the U4B trackers. I would be happy taking data from an API of theirs, as at least then the accuracy of the data becomes their problem. Hans has indicated he is interested in contributing data to SondeHub on the picoballoon list, so we should at give him the chance to do this.

The Zachtek and WB8ELK trackers would be a good place to start on a gateway then I think.
I guess we still don't completely get away from having to know what channel and model of tracker each callsign is using?

@darksidelemm
Copy link
Member

The other thing we need to figure out is how to identify these on the tracker, especially in the case where there are multiple trackers with the same callsign (e.g. the VE3OCL case, though this is a U4B where i hope this will be handled by Hans).

One option could be to just use CALLSIGN-channel_number, and we can also include the channel number and other useful metadata in what is uploaded.

@TheSkorm
Copy link
Member Author

If you write something that's opensource that we can package into a docker container (we can help with that) we can run any sort of gateway in our infra to save any sort of infra worries, costs and ensure long term reliability

@SteveRan
Copy link

Darkside Said: "I guess we still don't completely get away from having to know what channel and model of tracker each callsign is using?"

I'm pretty sure you can automatically detect what tracker is being used. Zachtek is easy as its the only one to use a type 3 WSPR message (with 6 character locator) as the 2nd message. Skytracker repeats the 4 character locator - so its very unlikely (1 in 32,400) that will happen for a U4B message. There are also other clues - some Skytracker values would be invalid in U4B encoding. I'm pretty sure if you applied those rules you could auto detect with a very high degree of confidence (particularly over time).

Zachtek and Sky tracker don't really use "channels" like UB4 - but there seems to be some informal agreement not to use Frequency and Timeslots that clash. Skytracker does use telemetry callsigns that start with "Q" and hence might conflict with U4B - I'm pretty sure (but need to confirm) that Skytracker is configuring a fixed Callsign character 3 in the telemetry - as another way of fitting in with U4B channels. Need to speak to Bill B to confirm.

@SteveRan
Copy link

SteveRan commented Jun 17, 2023 via email

@KevWal
Copy link

KevWal commented Feb 8, 2024

Team, did this make any progress please? Id love to see a WSPR to Sondehub flow that keeps as much of the original data intact as possible, rather than the convulted route it goes today ;)

@darksidelemm
Copy link
Member

This hasn't gone anywhere. Nothing has really changed in the state of WSPR balloon telemetry unfortunately.
At one point the QRPLabs dev had promised to develop an API of some sort allowing telemetry data to be obtained for the 'fleet' of U4B trackers, but that hasn't happened either.

@KevWal
Copy link

KevWal commented Feb 9, 2024

So my view of the architecture of this would be a program pulling the wspr data from wsprnet.org every 2 mins, finding the balloons, adding the second packet telemetry and feeding this data into the sondehub api.

Personally, I think doing this standalone, in code that perhaps could run on sondehub infrastructure would be the best result - not relying on u4b or lu7aa or any other providers.

There is already code that does that to an extent, but it needs work to A) work with the different encodings and chanel schemes we now have, and B) needs a way of finding what balloons to search for.

Re finding the balloons to search for, we can scrape the lu7aa site, could scrape the u4b site to get initial lists, but I would advocate sondehub having its own dB of balloons to search for, with a front end to that that allows people to register their balloons, much the same way as lu7aa does today.

I don't think I could succeed in getting all of that working in code myself, but I'm certainly willing to go and do some leg work on it collecting all the work others have done so we have the best starting point and then progress it from there.

Are people supportive of that architecture?

Thanks, Kevin

@xssfox
Copy link

xssfox commented Feb 10, 2024

if someone can build it sondehub can host it. We can create some api endpoints for updating a DB with this data (we somewhat have this already for changing prediction parameters). If you can give me an idea of what the schema looks like I can build that. If you want to start off with an offline db (even if it's just a static file) I can modify the code to read from our DB.

@KevWal
Copy link

KevWal commented Feb 20, 2024

Just to document where my research got to:

Existing Info / code:

Steve Randall - G8KHW / AJ4XE

  • Lots of collected info
  • Some example SQL code for collecting from wspr.live

Bill Brown – WB8ELK

  • Working code for decoding Skytracker and U4B (inc Traquito)
  • including doing the matching frequency between the callsign and the telemetry sequence
  • Only uploading to APRS.FI, (and Sondehub picks it up from there.)
  • Bill is working on also uploading to sondehub using their API but hasn't had time to make that mod.

Mikael Dagman SA6BSS / Stefan DG4NOB / David Lundberg SM0ULC

Mike Pate - K5MAP - https://github.com/k5map/BalloonTelemetry

  • Uploads to Sondehub,
  • Deals with all tracker types – Zachtek, AB5SS pico, LightAPRS-W, QRP-Labs U4B/Tranquito
  • Only deals with a single balloon per instance
  • Doesn’t seem to deal with frequency for u4b / traquito? – Raised Issue

@KevWal
Copy link

KevWal commented Feb 20, 2024

And then if I was able to impliment this, my thinking would be:

Find Balloons

  1. Start with a fixed text file list
  2. Scrape list from http://lu7aa.org/wsprset.asp - Whilst scraping isn't nice, why not use that as a starting point?
  3. Also scrape list from https://www.qrp-labs.com/track/tracking.html - Whilst scraping isn't nice, why not use that as a starting point?
  4. Move to storing in a database
  5. Web Interface to add / edit list?
  6. Autogenerate by searching wspr data?

Grab WSPR Data

  • A) Scrape from wsprnet.org using BeautifulSoup? (SM0ULC and Bill example) - scraping isnt nice, but seems to be the only way to get the data from the source?
  • Or B) SQL query from wspr.live? (Steve examples) - SQL is much nicer, but not the master source of the data?

Decode Balloon Data

  1. Start with Find first packet positions
  2. Find second packet positions and telemetry
    U4B / Traquitio / Etc

Upload to SondeHub

  1. Start with Upload first packet positions
  2. Then replace with Upload full positions
  3. Add Uploading telemetry
  4. Add Uploading receivers
    https://github.com/projecthorus/pysondehub/wiki/SondeHub-Amateur-Uploader-Class-Usage

What about positions not uploaded to wspr until later than we first check?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants