You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The changes to allow for these new characters need to be made in the tokenizer's string method.
Currently there is a check for "is_new_line_not_cr" that will need to be updated to only check for '\n' as the other two characters in that method are now allowed with the JSON superset proposal.
A good test for this would include the following strings:
The test should correctly evaluate these to the appropriate RawToken and after parsing each the buffer should be empty. A good reference for this is the tokenizer_strings test in the same module.
Allow line separator and paragraph separator in string literals.
proposal
The text was updated successfully, but these errors were encountered: