add opt-in strict string format

[zzstoatzz]

adds pydantic validation per #406 (comment). see the usage example

format	example	pattern/constraints
did	did:plc:z72i7hdynmk6r22z27h6tvur	did:[a-z]+:[a-zA-Z0-9._%-]+ (max 2KB, no /?#[]@)
handle	alice.bsky.social	Domain name: 2+ segments, ASCII alphanumeric/hyphens, 1-63 chars per segment, max 253 total. Last segment no leading digit.
at-uri	at://alice.bsky.social/app.bsky.feed.post/3jxtb5w2hkt2m	at:// + handle/DID + optional /collection/rkey. Max 8KB. No query/fragment in Lexicon.
datetime	2024-11-24T06:02:00Z	ISO 8601/RFC 3339 with required 'T', seconds, timezone (Z/±HH:MM). No -00:00.
nsid	app.bsky.feed.post	3+ segments: reversed domain (lowercase alphanum+hyphen) + name (letters only). Max 317 chars.
tid	3jxtb5w2hkt2m	13 chars of [2-7a-z]. First byte's high bit (0x40) must be 0.
record-key	3jxtb5w2hkt2m	1-512 chars of [A-Za-z0-9._:~-]. "." and ".." forbidden.
uri	https://example.com/path	RFC-3986 URI with letter scheme + netloc/path/query/fragment. Max 8KB, no spaces.
did:plc	did:plc:z72i7hdynmk6r22z27h6tvur	Method identifier must be 24 chars of base32 ([a-z2-7]). Cannot include underscore.

does not address open questions regarding the language types or re2. happy to chat on those if they'd be blocking here

may close #406

notes

I ran this to generate models

uvx poetry run atproto gen models

and this to update docs

uvx poetry run make -s -C docs gen

and this to run the new tests

uvx poetry run pytest tests/test_atproto_client/models/tests/test_string_formats.py

if desirable, I can follow up this PR to add contribution instructions to the current README section (for uv at least, and perhaps a TODO for proper poetry usage, since I don't really use that) - let me know!

---

in/valid examples for tests are sourced from https://github.com/bluesky-social/atproto/tree/main/interop-test-files/syntax

[MarshalX]

awesome hard work! thank you! speaking of your gotten knowledges about how to contribute we need to write CONTRIBUTING.md someday. but this will be separated PR for sure

[zzstoatzz]

thanks for the review @MarshalX! comments should be addressed in 6ad722a

[MarshalX]

could we parametrize it as well? to use all the test cases from .txt files instead of the first one

[zzstoatzz]

aha good call. yeah this definitely opened a new can of worms 🙂

there are a couple straggler cases I couldn't quite figure out right away that I've explicitly skipped for now, just a few of the many. let me know if anything jumps at you, ill come back to dig more later

[MarshalX]

sorry for the long response! i am busy at work :(

as I can see this regex is different to official one:

https://github.com/bluesky-social/atproto/blob/53fcc2fbcbda9b325eeffb5705126831baf2a427/packages/syntax/src/aturi.ts#L5)

We use official in our AtUri class:

atproto/packages/atproto_core/uri/uri.py

Line 9 in e011bb4

_ATP_URI_REGEX = r'^(at:\/\/)?((?:did:[a-z0-9:%-]+)|(?:[a-z0-9][a-z0-9.:-]*))(\/[^?#\s]*)?(\?[^#\s]+)?(#[^\s]+)?$'

(btw probably you could just use AtUri class to validate string instead of duplicating regex)

Probably, your regex come from re-impl mentioned in the issues. But it would be awesome to align with official one! And as I can see you can get everything from the syntax package! https://github.com/bluesky-social/atproto/tree/53fcc2fbcbda9b325eeffb5705126831baf2a427/packages/syntax/src. With it all the tests should pass and no need for excluding test cases

[zzstoatzz]

sorry for the long response! i am busy at work :(

no worries! so am I 🙂

ack! I missed that - thank you!

I will update the regexs to follow those as soon as I can 👍

[zzstoatzz]

hi @MarshalX - ok all of the cases are passing with no skipping!

I found that the spec and test cases in the interop valid/invalid .txt examples were not entirely covered by the types in atproto_core, so for now I have tended towards duplicating regex in string_formats.py and documenting with comments why I included the logic I did.

For example, looking at the AT-URI validation, I noticed what appear to be few differences between the regex I added and atproto_core's AtUri that seem important for passing the interop tests:

1. The spec requires "at://" prefix (see tests like "a://did:plc:asdf123" and "at:/did:plc:asdf123" being invalid)
2. The path must be a valid NSID format (invalid like "at://did:plc:asdf123/12345" and "at://did:plc:asdf123/short/stuff" show this)
3. The handle authority section has some idiosyncrasies - tests fail cases like "at://name", "at://name.0", "at://bsky" that a general URI parser might accept

let me know if you think i'm missing something here or if you think I should move some of these requirements into the types in atproto_core instead

also, just in case it comes up: i had a tricky problem with loading the test cases where I was calling .strip() on each line of the .txt files, but at least one of the test cases was checking for whitespace 🙃