Skip to content

fold: add -p/--punctuation flag for tiered break-point folding - #13327

Open
abdorah wants to merge 4 commits into
uutils:mainfrom
abdorah:main
Open

abdorah wants to merge 4 commits into
uutils:mainfrom
abdorah:main

Conversation

@abdorah

@abdorah abdorah commented Jul 9, 2026

Copy link
Copy Markdown

As I said in an earlier comment in the issue 13316, I've implemented -p / --punctuation as a standalone flag (no argument). When the line hits the width limit, it searches backwards through
the buffered content for the best break point using this priority: sentence-ending punctuation (. ! ? :) -> clause punctuation (, ;) -> whitespace -> any non-alphanumeric character -> hard cut at width.

The rightmost match of the highest-priority tier wins, which maximizes line usage while still preferring semantically meaningful breaks. This is mutually exclusive with -s (enforced by clap). The tiered fallback means a single -p invocation replaces what would otherwise require piping through fold four times with different break characters.

The character sets are hardcoded to ASCII for now but the structure makes it straightforward to extend with locale-aware or CJK punctuation (e.g., 。 、 ?) in a follow-up if needed.

This has changed many things in the way that fold used to work. Since the only attribute that requires 'back tracking' of spaces is space itself, we only had a true or false switch for that, which is perfectly fine, but for an intelligent fold like the one suggested by Dmole, we need more than that, hence the big changes to the fold context.

@codspeed

codspeed Bot commented Jul 9, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will degrade performance by 14.96%

❌ 3 regressed benchmarks
✅ 1 untouched benchmark
⏩ 480 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation fold_many_lines[100000] 84.3 ms 107.5 ms -21.55%
❌ CPU Simulation fold_custom_width[50000] 36.1 ms 44.5 ms -18.84%
❌ Memory fold_many_lines[100000] 26.6 KB 27.5 KB -3.39%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing abdorah:main (33f1033) with main (2775292)

Open in CodSpeed

Footnotes

  1. 480 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Comment thread src/uu/fold/src/fold.rs Resolved

@RenjiSann RenjiSann left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, this needs to be rebased. @abdorah can you take care of it ?

Comment thread src/uu/fold/src/fold.rs Outdated
Comment on lines +226 to +243
if let Some(pos) = chunk.iter().rposition(|&b| matches!(b, b'.' | b'!' | b'?' | b':')) {
return Some(pos + 1);
}
if let Some(pos) = chunk.iter().rposition(|&b| matches!(b, b',' | b';')) {
return Some(pos + 1);
}
if let Some(pos) = chunk
.iter()
.rposition(|b| b.is_ascii_whitespace() && *b != CR)
{
return Some(pos + 1);
}
if let Some(pos) = chunk
.iter()
.rposition(|b| !b.is_ascii_alphanumeric() && *b != CR)
{
return Some(pos + 1);
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It feels inefficient to iterate the chunk four times, when once could suffice

Comment thread src/uu/fold/src/fold.rs Outdated
Comment on lines 492 to 499
if ctx.break_mode != BreakMode::Hard
&& line[idx].is_ascii_whitespace()
&& line[idx] != CR
{
*ctx.last_space = Some(ctx.output.len() - 1);
} else if !ctx.spaces {
} else if ctx.break_mode == BreakMode::Hard {
*ctx.last_space = None;
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If conditions can be simplified

Suggested change
if ctx.break_mode != BreakMode::Hard
&& line[idx].is_ascii_whitespace()
&& line[idx] != CR
{
*ctx.last_space = Some(ctx.output.len() - 1);
} else if !ctx.spaces {
} else if ctx.break_mode == BreakMode::Hard {
*ctx.last_space = None;
}
if ctx.break_mode == BreakMode::Hard {
*ctx.last_space = None;
} else if line[idx].is_ascii_whitespace() && line[idx] != CR {
*ctx.last_space = Some(ctx.output.len() - 1);
}

Comment thread src/uu/fold/src/fold.rs Outdated
if ctx.spaces {
*ctx.last_space = Some(ctx.output.len());
} else {
if ctx.break_mode == BreakMode::Hard {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've seen the ctx.break_mode == BreakMode::Hard really often, maybe consider implementing fn is_hard(self) -> bool to use as a shorthand ?

Comment thread src/uu/fold/src/fold.rs Outdated
Comment on lines +741 to +764
let cur_pos = ctx.output.len();
match ctx.break_mode {
BreakMode::Punctuation => match byte {
b'.' | b'!' | b'?' | b':' => {
*ctx.last_sentence_end = Some(cur_pos);
}
b',' | b';' => {
*ctx.last_clause_end = Some(cur_pos);
}
b if b.is_ascii_whitespace() && b != CR => {
*ctx.last_space = Some(cur_pos);
}
b if !b.is_ascii_alphanumeric() => {
*ctx.last_non_alnum = Some(cur_pos);
}
_ => {}
},
BreakMode::Spaces => {
if byte.is_ascii_whitespace() && byte != CR {
*ctx.last_space = Some(cur_pos);
}
}
BreakMode::Hard => {}
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is duplicated at least 3 times, please move it to a specific function

@abdorah

abdorah commented Nov 11, 2026

Copy link
Copy Markdown
Author

Sorry, this needs to be rebased. @abdorah can you take care of it ?

yeah sure, thank you for your review, I will handle these issues.

@abdorah

abdorah commented Nov 13, 2026

Copy link
Copy Markdown
Author

@oech3 I dropped the short flag; it's --punctuation only now, with a comment
at the Arg noting that no short form is reserved so a future GNU short option
can't clash.

@RenjiSann

  • Four rposition scans: find_punctuation_break now does a single reverse
    pass, recording a candidate for each tier as it goes. A sentence-end
    short-circuits immediately, since no lower tier can outrank it.
  • Nested last_space conditions: flattened using the new helper below.
  • is_hard() helper: I added BreakMode::is_hard(self) -> bool; it replaces
    the repeated == BreakMode::Hard comparisons.
  • Duplicated match: I extracted it into track_punctuation(ctx, byte, pos),
    shared by the ASCII, UTF-8 and non-UTF-8 processors.

sylvestre pushed a commit to abdorah/coreutils that referenced this pull request Nov 16, 2026
Introduces --punctuation (no short), mutually exclusive with -s. When a
line hits the width limit, break at the latest tracked position in this
priority: sentence-ending punctuation (. ! ? :), clause punctuation
(, ;), whitespace, then any non-alphanumeric byte. Falls back to a hard
cut at width when no candidate exists. Byte sets are ASCII; non-ASCII
scalars fall into the non-alphanumeric tier via char::is_alphanumeric.

Refactors in response to review on uutils#13327:

- BreakMode::is_hard() helper replaces repeated enum comparisons.
- find_punctuation_break (bytewise path) does a single reverse pass
  tracking every tier at once, in place of four separate rposition
  scans.
- track_punctuation() centralises the per-byte tier dispatch that was
  duplicated across the ASCII, UTF-8 and non-UTF-8 processors.
- clear_break_trackers() replaces inconsistent per-processor newline
  clears, fixing a bug where a mid-line sentence-end could steal the
  position of an immediately following input newline.
- last_space branches use is_hard() to flatten nested if/else-if.

Benches on a local machine show no regression against upstream:
fold_custom_width 9.01 -> 8.31 ms median, fold_many_lines
23.03 -> 21.31 ms.
Comment thread src/uu/fold/src/fold.rs
let mut non_alnum = None;
for (i, &b) in chunk.iter().enumerate().rev() {
match b {
b'.' | b'!' | b'?' | b':' => return Some(i + 1),

@oech3 oech3 Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't prefer restricting supported letters since it is very specialized. Please support any set of char or u8 array instead and remove that list.

@oech3

oech3 commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

I really don't want to hope have different feature with GNU. Should this be really part of coreutils instead of 3rd pirty?

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

GNU testsuite comparison:

Congrats! The gnu test tests/tail/follow-name is no longer failing!
Congrats! The gnu test tests/tail/pid-pipe is no longer failing!

Comment thread src/uu/fold/src/fold.rs Outdated
if ch.is_ascii() {
track_punctuation(ctx, ch as u8, pos);
} else if !ch.is_alphanumeric() {
*ctx.last_non_alnum = Some(pos);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pos + 1 will cut a multibyte char like — in the middle, producing invalid UTF-8
please add a test with non-ASCII punctuation

Comment thread src/uu/fold/src/fold.rs Outdated
// could clash with one of them.
Arg::new(options::PUNCTUATION)
.long(options::PUNCTUATION)
.help("break at punctuation boundaries (tiered: sentence-end, clause, space, non-alphanumeric)")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please use translate!() and add it to en-US.ftl

Comment thread src/uu/fold/src/fold.rs Outdated
}

/// Record `byte` at position `pos` as the newest candidate break point for
/// whichever tier it belongs to. This replaces the per-byte match that was

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please remove the history part of the comment, it won't make sense once merged

Comment thread tests/by-util/test_fold.rs Outdated
.stdout_is_bytes(expected);
}

// --punctuation (-p was dropped to avoid clashing with future GNU short options).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this comment can go

Comment thread src/uu/fold/src/fold.rs Outdated
let pos = ctx.output.len();
match ctx.break_mode {
BreakMode::Punctuation => track_punctuation(ctx, byte, pos),
BreakMode::Spaces if byte.is_ascii_whitespace() && byte != CR => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CR is already handled above, && byte != CR is dead

sylvestre pushed a commit to abdorah/coreutils that referenced this pull request Nov 16, 2026
Introduces --punctuation (no short), mutually exclusive with -s. When a
line hits the width limit, break at the latest tracked position in this
priority: sentence-ending punctuation (. ! ? :), clause punctuation
(, ;), whitespace, then any non-alphanumeric byte. Falls back to a hard
cut at width when no candidate exists. Byte sets are ASCII; non-ASCII
scalars fall into the non-alphanumeric tier via char::is_alphanumeric.

Refactors in response to review on uutils#13327:

- BreakMode::is_hard() helper replaces repeated enum comparisons.
- find_punctuation_break (bytewise path) does a single reverse pass
  tracking every tier at once, in place of four separate rposition
  scans.
- track_punctuation() centralises the per-byte tier dispatch that was
  duplicated across the ASCII, UTF-8 and non-UTF-8 processors.
- clear_break_trackers() replaces inconsistent per-processor newline
  clears, fixing a bug where a mid-line sentence-end could steal the
  position of an immediately following input newline.
- last_space branches use is_hard() to flatten nested if/else-if.

Benches on a local machine show no regression against upstream:
fold_custom_width 9.01 -> 8.31 ms median, fold_many_lines
23.03 -> 21.31 ms.
Introduces --punctuation (no short), mutually exclusive with -s. When a
line hits the width limit, break at the latest tracked position in this
priority: sentence-ending punctuation (. ! ? :), clause punctuation
(, ;), whitespace, then any non-alphanumeric byte. Falls back to a hard
cut at width when no candidate exists. Byte sets are ASCII; non-ASCII
scalars fall into the non-alphanumeric tier via char::is_alphanumeric.

Refactors in response to review on uutils#13327:

- BreakMode::is_hard() helper replaces repeated enum comparisons.
- find_punctuation_break (bytewise path) does a single reverse pass
  tracking every tier at once, in place of four separate rposition
  scans.
- track_punctuation() centralises the per-byte tier dispatch that was
  duplicated across the ASCII, UTF-8 and non-UTF-8 processors.
- clear_break_trackers() replaces inconsistent per-processor newline
  clears, fixing a bug where a mid-line sentence-end could steal the
  position of an immediately following input newline.
- last_space branches use is_hard() to flatten nested if/else-if.

Benches on a local machine show no regression against upstream:
fold_custom_width 9.01 -> 8.31 ms median, fold_many_lines
23.03 -> 21.31 ms.
The collapsed closure in rebase_tracker exceeded the line width, so
rustfmt reflowed it back to block form; Style/lint flagged it on the
previous commit.

Also drops "alnum" from the file-local spell-checker ignore list: it is
already present in .vscode/cspell.dictionaries/jargon.wordlist.txt, so
the entry was redundant.
- translate the --punctuation help (en-US, fr-FR)
- anchor non-ASCII break chars on their last byte so the cut never splits
  a multibyte char (e.g. an em dash); add tests with non-ASCII punctuation
- drop a dead CR check and comments that only described the PR history
- byte-level paths (-b, invalid UTF-8) only take ASCII bytes as non-alphanumeric
  break points, so --punctuation no longer splits a UTF-8 sequence; add a test
  that -b --punctuation cuts where plain -b does when no ASCII break exists
- drop two dead CR checks (CR never reaches those branches)
- reword the BreakMode doc; drop a comment that described review history

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants