Skip to content

lexer: ignore carriage returns - #189

Merged
Alonely0 merged 1 commit into
uutils:mainfrom
emreocell:fix-crlf-program
Nov 20, 2026
Merged

Alonely0 merged 1 commit into
uutils:mainfrom
emreocell:fix-crlf-program

Conversation

@emreocell

@emreocell emreocell commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #177

A program with Windows line endings (BEGIN {\r\n print "ok"\r\n}\r\n) was rejected at the first \r. As suggested in the review, the lexer now treats a bare \r as white space (like gawk), and still accepts one before the newline of a backslash line continuation (\ + \r\n). The Newline token is unchanged, since the \r before a \n is simply skipped. A \r inside a string or regex literal is not affected.

Tests

  • crlf_program (from the issue), bare_carriage_return_is_white_space (print "foo"\r" bar" prints foo bar) and escaped_newline_with_carriage_return in tests/by-util/test_awk.rs, plus a lexer unit test comparing CR/CRLF sources with their LF/space equivalents.
  • cargo test --workspace passes, cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --all --check are clean.
  • I could not compare against gawk (not installed here); expected output is taken from the issue and the review.

Copilot AI balanced review requested due to automatic review settings November 19, 2026 03:18

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

Copilot AI balanced review requested due to automatic review settings November 20, 2026 01:33

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

@Alonely0 Alonely0 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution!

After some experimentation, it seems that, more generally, we should eat carriage returns. It does seems gawk ignores them:

$ gawk $'BEGIN {\r\n  print "foo"\r" bar"}\r\n'
foo bar

But, it also accepts escaped newlines with a preceding carriage return.

$ gawk $'BEGIN {\r\n  print "foo"\\r\n" bar"}\r\n'
foo bar

so, you should modify the code to make the lexer eat all bare \r as well, while keeping the optional \r before an escaped \n in the ignoring patterns. Then, it should be possible to drop the change to the Newline token regex, since the carriage would be ignored.

A program with Windows line endings stopped on the first carriage return.
Like gawk, treat a bare \r as white space, and accept one before the
newline of a backslash line continuation.
Copilot AI balanced review requested due to automatic review settings November 20, 2026 02:05
@emreocell emreocell changed the title lexer: treat CRLF as a line ending lexer: ignore carriage returns Nov 20, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

@emreocell

emreocell commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for the review! The lexer now ignores all bare carriage returns (an optional one before the newline of an escaped newline is still accepted), the Newline regex change is gone, and I added tests for CRLF, a bare CR mid-line and the escaped newline with a CR.

@Alonely0 Alonely0 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks you so much for your contribution!

@Alonely0
Alonely0 merged commit de3a482 into uutils:main Nov 20, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

awk: a program file with CRLF line endings is rejected

3 participants