ValueError: Failed to read CAR header. Invalid uvarint

[adonoho]

Traceback (most recent call last):
  File "/Users/awd/miniforge3/envs/bsky/lib/python3.10/site-packages/atproto_firehose/client.py", line 221, in _process_message_frame
    await self._on_message_callback(frame)
  File "/Users/awd/Projects/PhenomML/BSky2GBQ/main.py", line 91, in async_on_message
    blocks = atproto.CAR.from_bytes(message.blocks).blocks
  File "/Users/awd/miniforge3/envs/bsky/lib/python3.10/site-packages/atproto_core/car/car.py", line 51, in from_bytes
    header, blocks = libipld.decode_car(data)
ValueError: Failed to read CAR header. Invalid uvarint

Happy to help and dig deeper.

Anon,
Andrew

[adonoho]

# <== is the marker to the line that generated this ValueError.

async def async_on_message(message, test_function, handler):
    message = hose.parse_subscribe_repos_message(message)
    if isinstance(message,
                  atproto.models.ComAtprotoSyncSubscribeRepos.Commit):
        blocks = atproto.CAR.from_bytes(message.blocks).blocks  # <== This is the line that creates the above trace.
        for op in message.ops:
            uri = atproto.AtUri.from_str("at://" + message.repo \
                                         + "/" + op.path)
            raw = blocks.get(op.cid)
            if raw:
                record = get_or_create(raw, strict=False)
                if record.py_type is not None:
                    rdict = record.model_dump()
                    item = {
                        "repo": message.repo,
                        "revision": message.rev,
                        "sequence": message.seq,
                        "timestamp": message.time,
                        "action": op.action,
                        "cid": str(op.cid),
                        "path": op.path,
                        "collection": uri.collection,
                        "record": rdict
                    }
                    if test_function(item):
                        await IOLoop.current().run_in_executor(None, handler, item)

[MarshalX]

Hi! Today is the day when everyone tries devil payloads over the network. These are special malformed messages that raise issues like that. It would help if you would catch the value of .blocks and write to the file. And pls share. Otherwise, just catch the ValueError exception and ignore this commit

[adonoho]

OK, I'll try to get to this after dinner.

[MarshalX]

File "/Users/awd/Projects/PhenomML/BSky2GBQ/main.py", line 91, in async_on_message
blocks = atproto.CAR.from_bytes(message.blocks).blocks

I think it is worth to add check that message.blocks is not empty :)