Repository navigation
lexer: ignore carriage returns - #189
Conversation
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.
837ddc8 to
f8139dd
Compare
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.
Alonely0
left a comment
There was a problem hiding this comment.
Thanks for your contribution!
After some experimentation, it seems that, more generally, we should eat carriage returns. It does seems gawk ignores them:
$ gawk $'BEGIN {\r\n print "foo"\r" bar"}\r\n'
foo barBut, it also accepts escaped newlines with a preceding carriage return.
$ gawk $'BEGIN {\r\n print "foo"\\r\n" bar"}\r\n'
foo barso, you should modify the code to make the lexer eat all bare \r as well, while keeping the optional \r before an escaped \n in the ignoring patterns. Then, it should be possible to drop the change to the Newline token regex, since the carriage would be ignored.
A program with Windows line endings stopped on the first carriage return. Like gawk, treat a bare \r as white space, and accept one before the newline of a backslash line continuation.
f8139dd to
e5a3dcb
Compare
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.
|
Thanks for the review! The lexer now ignores all bare carriage returns (an optional one before the newline of an escaped newline is still accepted), the |
Alonely0
left a comment
There was a problem hiding this comment.
Looks good, thanks you so much for your contribution!
Fixes #177
A program with Windows line endings (
BEGIN {\r\n print "ok"\r\n}\r\n) was rejected at the first\r. As suggested in the review, the lexer now treats a bare\ras white space (like gawk), and still accepts one before the newline of a backslash line continuation (\+\r\n). TheNewlinetoken is unchanged, since the\rbefore a\nis simply skipped. A\rinside a string or regex literal is not affected.Tests
crlf_program(from the issue),bare_carriage_return_is_white_space(print "foo"\r" bar"printsfoo bar) andescaped_newline_with_carriage_returnintests/by-util/test_awk.rs, plus a lexer unit test comparing CR/CRLF sources with their LF/space equivalents.cargo test --workspacepasses,cargo clippy --workspace --all-targets -- -D warningsandcargo fmt --all --checkare clean.