Repository navigation
Conversation
Merging this PR will degrade performance by 14.96%
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
| if let Some(pos) = chunk.iter().rposition(|&b| matches!(b, b'.' | b'!' | b'?' | b':')) { | ||
| return Some(pos + 1); | ||
| } | ||
| if let Some(pos) = chunk.iter().rposition(|&b| matches!(b, b',' | b';')) { | ||
| return Some(pos + 1); | ||
| } | ||
| if let Some(pos) = chunk | ||
| .iter() | ||
| .rposition(|b| b.is_ascii_whitespace() && *b != CR) | ||
| { | ||
| return Some(pos + 1); | ||
| } | ||
| if let Some(pos) = chunk | ||
| .iter() | ||
| .rposition(|b| !b.is_ascii_alphanumeric() && *b != CR) | ||
| { | ||
| return Some(pos + 1); | ||
| } |
There was a problem hiding this comment.
It feels inefficient to iterate the chunk four times, when once could suffice
| if ctx.break_mode != BreakMode::Hard | ||
| && line[idx].is_ascii_whitespace() | ||
| && line[idx] != CR | ||
| { | ||
| *ctx.last_space = Some(ctx.output.len() - 1); | ||
| } else if !ctx.spaces { | ||
| } else if ctx.break_mode == BreakMode::Hard { | ||
| *ctx.last_space = None; | ||
| } |
There was a problem hiding this comment.
If conditions can be simplified
| if ctx.break_mode != BreakMode::Hard | |
| && line[idx].is_ascii_whitespace() | |
| && line[idx] != CR | |
| { | |
| *ctx.last_space = Some(ctx.output.len() - 1); | |
| } else if !ctx.spaces { | |
| } else if ctx.break_mode == BreakMode::Hard { | |
| *ctx.last_space = None; | |
| } | |
| if ctx.break_mode == BreakMode::Hard { | |
| *ctx.last_space = None; | |
| } else if line[idx].is_ascii_whitespace() && line[idx] != CR { | |
| *ctx.last_space = Some(ctx.output.len() - 1); | |
| } |
| if ctx.spaces { | ||
| *ctx.last_space = Some(ctx.output.len()); | ||
| } else { | ||
| if ctx.break_mode == BreakMode::Hard { |
There was a problem hiding this comment.
I've seen the ctx.break_mode == BreakMode::Hard really often, maybe consider implementing fn is_hard(self) -> bool to use as a shorthand ?
| let cur_pos = ctx.output.len(); | ||
| match ctx.break_mode { | ||
| BreakMode::Punctuation => match byte { | ||
| b'.' | b'!' | b'?' | b':' => { | ||
| *ctx.last_sentence_end = Some(cur_pos); | ||
| } | ||
| b',' | b';' => { | ||
| *ctx.last_clause_end = Some(cur_pos); | ||
| } | ||
| b if b.is_ascii_whitespace() && b != CR => { | ||
| *ctx.last_space = Some(cur_pos); | ||
| } | ||
| b if !b.is_ascii_alphanumeric() => { | ||
| *ctx.last_non_alnum = Some(cur_pos); | ||
| } | ||
| _ => {} | ||
| }, | ||
| BreakMode::Spaces => { | ||
| if byte.is_ascii_whitespace() && byte != CR { | ||
| *ctx.last_space = Some(cur_pos); | ||
| } | ||
| } | ||
| BreakMode::Hard => {} | ||
| } |
There was a problem hiding this comment.
This is duplicated at least 3 times, please move it to a specific function
yeah sure, thank you for your review, I will handle these issues. |
|
@oech3 I dropped the short flag; it's
|
Introduces --punctuation (no short), mutually exclusive with -s. When a line hits the width limit, break at the latest tracked position in this priority: sentence-ending punctuation (. ! ? :), clause punctuation (, ;), whitespace, then any non-alphanumeric byte. Falls back to a hard cut at width when no candidate exists. Byte sets are ASCII; non-ASCII scalars fall into the non-alphanumeric tier via char::is_alphanumeric. Refactors in response to review on uutils#13327: - BreakMode::is_hard() helper replaces repeated enum comparisons. - find_punctuation_break (bytewise path) does a single reverse pass tracking every tier at once, in place of four separate rposition scans. - track_punctuation() centralises the per-byte tier dispatch that was duplicated across the ASCII, UTF-8 and non-UTF-8 processors. - clear_break_trackers() replaces inconsistent per-processor newline clears, fixing a bug where a mid-line sentence-end could steal the position of an immediately following input newline. - last_space branches use is_hard() to flatten nested if/else-if. Benches on a local machine show no regression against upstream: fold_custom_width 9.01 -> 8.31 ms median, fold_many_lines 23.03 -> 21.31 ms.
| let mut non_alnum = None; | ||
| for (i, &b) in chunk.iter().enumerate().rev() { | ||
| match b { | ||
| b'.' | b'!' | b'?' | b':' => return Some(i + 1), |
There was a problem hiding this comment.
I don't prefer restricting supported letters since it is very specialized. Please support any set of char or u8 array instead and remove that list.
|
I really don't want to hope have different feature with GNU. Should this be really part of coreutils instead of 3rd pirty? |
|
GNU testsuite comparison: |
| if ch.is_ascii() { | ||
| track_punctuation(ctx, ch as u8, pos); | ||
| } else if !ch.is_alphanumeric() { | ||
| *ctx.last_non_alnum = Some(pos); |
There was a problem hiding this comment.
pos + 1 will cut a multibyte char like — in the middle, producing invalid UTF-8
please add a test with non-ASCII punctuation
| // could clash with one of them. | ||
| Arg::new(options::PUNCTUATION) | ||
| .long(options::PUNCTUATION) | ||
| .help("break at punctuation boundaries (tiered: sentence-end, clause, space, non-alphanumeric)") |
There was a problem hiding this comment.
please use translate!() and add it to en-US.ftl
| } | ||
|
|
||
| /// Record `byte` at position `pos` as the newest candidate break point for | ||
| /// whichever tier it belongs to. This replaces the per-byte match that was |
There was a problem hiding this comment.
please remove the history part of the comment, it won't make sense once merged
| .stdout_is_bytes(expected); | ||
| } | ||
|
|
||
| // --punctuation (-p was dropped to avoid clashing with future GNU short options). |
| let pos = ctx.output.len(); | ||
| match ctx.break_mode { | ||
| BreakMode::Punctuation => track_punctuation(ctx, byte, pos), | ||
| BreakMode::Spaces if byte.is_ascii_whitespace() && byte != CR => { |
There was a problem hiding this comment.
CR is already handled above, && byte != CR is dead
Introduces --punctuation (no short), mutually exclusive with -s. When a line hits the width limit, break at the latest tracked position in this priority: sentence-ending punctuation (. ! ? :), clause punctuation (, ;), whitespace, then any non-alphanumeric byte. Falls back to a hard cut at width when no candidate exists. Byte sets are ASCII; non-ASCII scalars fall into the non-alphanumeric tier via char::is_alphanumeric. Refactors in response to review on uutils#13327: - BreakMode::is_hard() helper replaces repeated enum comparisons. - find_punctuation_break (bytewise path) does a single reverse pass tracking every tier at once, in place of four separate rposition scans. - track_punctuation() centralises the per-byte tier dispatch that was duplicated across the ASCII, UTF-8 and non-UTF-8 processors. - clear_break_trackers() replaces inconsistent per-processor newline clears, fixing a bug where a mid-line sentence-end could steal the position of an immediately following input newline. - last_space branches use is_hard() to flatten nested if/else-if. Benches on a local machine show no regression against upstream: fold_custom_width 9.01 -> 8.31 ms median, fold_many_lines 23.03 -> 21.31 ms.
Introduces --punctuation (no short), mutually exclusive with -s. When a line hits the width limit, break at the latest tracked position in this priority: sentence-ending punctuation (. ! ? :), clause punctuation (, ;), whitespace, then any non-alphanumeric byte. Falls back to a hard cut at width when no candidate exists. Byte sets are ASCII; non-ASCII scalars fall into the non-alphanumeric tier via char::is_alphanumeric. Refactors in response to review on uutils#13327: - BreakMode::is_hard() helper replaces repeated enum comparisons. - find_punctuation_break (bytewise path) does a single reverse pass tracking every tier at once, in place of four separate rposition scans. - track_punctuation() centralises the per-byte tier dispatch that was duplicated across the ASCII, UTF-8 and non-UTF-8 processors. - clear_break_trackers() replaces inconsistent per-processor newline clears, fixing a bug where a mid-line sentence-end could steal the position of an immediately following input newline. - last_space branches use is_hard() to flatten nested if/else-if. Benches on a local machine show no regression against upstream: fold_custom_width 9.01 -> 8.31 ms median, fold_many_lines 23.03 -> 21.31 ms.
The collapsed closure in rebase_tracker exceeded the line width, so rustfmt reflowed it back to block form; Style/lint flagged it on the previous commit. Also drops "alnum" from the file-local spell-checker ignore list: it is already present in .vscode/cspell.dictionaries/jargon.wordlist.txt, so the entry was redundant.
- translate the --punctuation help (en-US, fr-FR) - anchor non-ASCII break chars on their last byte so the cut never splits a multibyte char (e.g. an em dash); add tests with non-ASCII punctuation - drop a dead CR check and comments that only described the PR history
- byte-level paths (-b, invalid UTF-8) only take ASCII bytes as non-alphanumeric break points, so --punctuation no longer splits a UTF-8 sequence; add a test that -b --punctuation cuts where plain -b does when no ASCII break exists - drop two dead CR checks (CR never reaches those branches) - reword the BreakMode doc; drop a comment that described review history
As I said in an earlier comment in the issue 13316, I've implemented -p / --punctuation as a standalone flag (no argument). When the line hits the width limit, it searches backwards through
the buffered content for the best break point using this priority: sentence-ending punctuation (. ! ? :) -> clause punctuation (, ;) -> whitespace -> any non-alphanumeric character -> hard cut at width.
The rightmost match of the highest-priority tier wins, which maximizes line usage while still preferring semantically meaningful breaks. This is mutually exclusive with -s (enforced by clap). The tiered fallback means a single -p invocation replaces what would otherwise require piping through fold four times with different break characters.
The character sets are hardcoded to ASCII for now but the structure makes it straightforward to extend with locale-aware or CJK punctuation (e.g., 。 、 ?) in a follow-up if needed.
This has changed many things in the way that fold used to work. Since the only attribute that requires 'back tracking' of spaces is space itself, we only had a true or false switch for that, which is perfectly fine, but for an intelligent fold like the one suggested by Dmole, we need more than that, hence the big changes to the fold context.