Skip to content

0.7.0: string storage per column, faster cell writing, re-measured benchmarks - #43

Merged
glenntosco merged 42 commits into
mainfrom
string-storage
Nov 18, 2026
Merged

glenntosco merged 42 commits into
mainfrom
string-storage

Conversation

@glenntosco

Copy link
Copy Markdown
Collaborator

What this ships (0.7.0)

  • PackageOptions.StringStorage (Auto by default, Shared, Inline): text goes in the cell or in the shared string table, whichever suits each column, so unique text is no longer hashed and remembered.
  • Faster cell writing: rows are composed as UTF-8 straight in the zip entry buffer, numbers are formatted into a reused buffer, text is encoded in one pass. Nothing is allocated per cell.
  • Smaller sheet files: a cell that follows the previous one in its row is written without its r attribute (valid under ECMA-376). PackageOptions.WriteCellReferences = true restores it. NPOI 2.7.6's SAX reader, Excel OleDb ACE and MiniExcel 1.46.0 misread such files; the README and changelog say so.
  • Fixes: a carriage return in text now reads back as written, text over 32,767 characters is refused when written, Skip and cell writes refuse a column past XFD, a second sheet is judged on its own text under Auto, a formula with a NaN or infinite result stores no result.
  • Benchmarks re-measured on a clean Azure VM with three launches per row and SpreadCheetah at the same zip level next to every row; raw results are in benchmarks/published-results. The comparison is mixed and written as measured: ahead of SpreadCheetah on long, wide, very wide, million-row and styled sheets, level on mixed text, about 2% behind on all-unique text at Optimal; SpreadCheetah allocates less.

Checked before this PR

  • dotnet build and dotnet test on the final tree: 13 + 48 + 682 (net10.0) and 682 (net8.0) tests pass.
  • Deep reader fuzz with seeds 1, 42 and 2026 (3,000 iterations), writer-side fuzz over about 9,500 seeds, Excel corpus tests (34/34 on both frameworks), the 64 MB memory probe, .NET Framework 4.8 tests, tests/excel-check.ps1 on the samples and a 100,000-row workbook.
  • Benchmark correctness guard passes for every library.
  • DocFX build with warnings as errors: 0 warnings.
  • Both packages pack at 0.7.0.

After the build check is green and this is merged, tagging v0.7.0 publishes both packages to nuget.org through Trusted Publishing.

SpreadCheetah 1.29.0 joins the write scenarios at its default compression (Fastest) and at Optimal, so each row has a SheetStream row at the same level. The suite now targets net10.0. Results and the README follow once measured on a quiet machine.
A cell in the column right after the previous cell of its row no longer
carries its address (r is optional in ECMA-376; a cell without it belongs in
the next column, the first of a row in column A). A cell after a gap from
Skip() or an unstyled empty cell still writes r, so positions stay exact.

TemplateStyle now finds cells that have no r. Tests that read cell addresses
through the Open XML SDK work them out from position, and new tests cover
Skip, AutoFit, Cells[,], Edit and template mode.
… per cell

Numbers, style ids, shared string indexes and row numbers are written through
a per-sheet char buffer (WriteChars) rather than XmlWriter.WriteValue, which
builds a new string every call. The text is the same as XmlConvert produces.
netstandard2.0 cannot format a double into a buffer and keeps the old call for
doubles and decimals. The row number text is no longer built per row.

Tests pin the formatted text against XmlConvert and assert that writing cells
allocates nothing per cell.
The negative-style-index test found cells by their r attribute, which the
writer now leaves off consecutive cells, so it would have mutated almost
nothing. It matches the cell tag itself now and checks that every cell was
changed.
…cell

The default still leaves the r attribute off cells that follow the previous
cell, which keeps files smaller and writing faster. Setting the option to true
writes r on every cell, in every way of writing cells (Row, Cells[,], AutoFit,
Worksheets.Edit and rows filled into a template), for tools that need it: the
NPOI event (SAX) reader, the Excel ADO / OleDb provider, Microsoft Spreadsheet
Compare and Open XML SDK code that reads Cell.CellReference.

The setting is read once per row, so each cell still pays one bool test.
…docs

Replaces the changelog sentence that said NPOI's event reader fails with the
option that fixes it.
… per piece

Numbers, dates, durations, booleans, shared and inline strings and styled empty
cells are now composed as finished XML in a fixed-size, reused buffer and passed to
the XML writer with a single WriteRaw call per batch. This avoids the writer's
per-element and per-attribute checks and state tracking for every cell.

The file is unchanged. Text is escaped by hand exactly as the XML writer escaped it
(&, < and >; everything else, including line breaks, is left alone), and the
buffer is flushed before the row ends and before any cell that still goes through
the XML writer (formulas, rich text, error cells, cells copied from a template),
so the order of the output is preserved.

New tests write the same scripts of cells through the library and through a
reference built on a real XmlWriter and compare the sheet XML character for
character: every numeric type and extreme, dates, booleans, shared and inline
text with every special character, every UTF-16 code unit, text around the buffer
size, gaps, both address settings, AutoFit, and seeded random scripts. The
allocation test also covers escaped text, gaps and the other cell kinds.
Rows are composed as bytes in the write buffer of the part (a new PartStream
replaces BufferedStream, one buffer shared by all parts) instead of being
handed to XmlWriter.WriteRaw, which re-encoded and copied them again. The row
element, formulas, error cells and cached results are written the same way;
rich text still renders its runs with an XmlWriter. Template and edited
sheets keep the XmlWriter in charge and receive the same bytes as text.

Short decimals are formatted by a small routine (Utf8Numbers) that gives the
text of the runtime's shortest round-trip formatter and leaves everything it
cannot decide to the runtime; on .NET Framework the runtime formats all
doubles as before.

Output is character-identical (differential tests against a real XmlWriter,
now covering formulas, rich text, row options, small buffers and template
sheets). Writing a sheet allocates about 150 KB instead of about 690 KB.
The reader benchmark file is written with cell references so every reader
under test can read it.
Rows used to go through the part's XmlWriter, which is dead after any failure,
so a caller that caught an error and carried on could never save a damaged
sheet. Two ways around that existed on the new path and are closed:

- PartStream remembers a failed write to the zip entry and fails every later
  write, drain and close.
- A rich-text cell is rendered before any of it is put in the row, so a run
  that cannot be rendered (a font name XML cannot carry) leaves nothing behind
  instead of half a cell that made the whole sheet unreadable.

Also: the README and docs no longer say rows go through an XmlWriter.
Every library in the suite now writes three more scenarios from the same
deterministic data: SpreadCheetah's own published shape (100,000 x 10 distinct
strings), a realistic string mix with repeated and unique columns and text that
needs XML escaping, and a styled sheet of dates, date-times, money and
percentages. The --verify guard compares every cell value (and, for the styled
sheet, every number format) with the generator, and prints each string
scenario's data profile. --peak runs the new scenarios too.
…a clean Windows machine

Prints the hardware, operating system, SDK and package versions, builds,
stops if the correctness check fails, then runs the chosen scenarios and the
peak-memory measurement and copies the reports and a hardware.txt into a
time-stamped results folder. -DryRun only prints the plan. A double-click
launcher is included for machines that block .ps1 files by default.
…nd fairness rules

Documents SpreadCheetahOwn, StringHeavy and StyledDates (including the data
profile of the string scenarios), how SpreadCheetah and MiniExcel are fed, the
stricter correctness guard, and the run-on-fresh-vm script. Also fixes the
documented filters: matching ignores letter case, so '*Read*' had been
selecting every SpreadCheetah benchmark too.
…at a buffer boundary

The text buffer used to turn staged bytes back into characters was one char per byte, but a flush that ends inside a multi-byte character carries bytes into the next flush, which can then decode to one char more than the buffer holds. The buffer now has the decoder's worst-case size. A failure while handing bytes to the XML writer now makes the package refuse to save instead of leaving a half-written cell behind, and the rich-text render stream is emptied even when copying fails.
…or an infinity

The cached-value overload wrote text that Excel cannot use, and a different text on .NET than on .NET Framework. The formula is now written with no stored result on every build, so Excel calculates it on open.
…d write

When a write to the output failed, the row writer reset its buffer before the error left it, so the next row closed the row by stepping back over a byte that was no longer there and every later row threw an index error instead of the I/O error, as did Save. Starting or closing a row now reports the stored failure first, and the row is marked as having cells before anything can be flushed. The fail-stop tests now require an I/O error from every row after the first failure and from Save, including for an output that fails for good after a given size.
…ames the original failure, cover the template boundary on .NET Framework
…, Shared and Inline

Text cells used to go through the shared string table whatever they were, which cost a hash lookup per cell, kept every
distinct text in memory until the save and made text-heavy sheets slow. StringStorage.Auto looks at the first 128 to 512
cells of each column and then either keeps sharing it (values repeat and are long or varied) or writes it inline with no
lookup. Shared is the old behaviour, Inline never shares; UseSharedStrings = false still means inline.

Inline text is also written in one pass, eight characters at a time on net8.0, instead of three scans and a copy. The
differential tests against a real XmlWriter cover every length and every kind of special character at every position.
…add the Auto differential test, document the setting in the changelog
…at repeats among thousands of values is shared

A column in which none of the first 128 cells repeated used to be written inline for good. A column that repeats among tens of thousands of distinct values (customer or product names) shows no repeat in its first cells, so it stayed inline and its file came out 3 to 7 times larger than with Shared, and the write was slower.

Such a column now gets two more looks of 512 cells, at its 512th and 8,192nd cell. A look counts how many cells repeat a text the column already put in the table, estimates the number of different values from that, and shares the column when the estimated vocabulary (values times average length) is more than the 24 KB the zip compressor can remember. A vocabulary that fits stays inline. Measured, 100,000 rows of 20,000 different values: 25 characters 6.4 MB to 1.9 MB, 80 characters 17.1 MB to 2.6 MB, same size as Shared, same speed as Shared.
- An unstyled inline cell: its 24 opening bytes and 13 closing bytes are written with three and two 8-byte stores (.NET 8 and later) instead of a block copy and a byte loop, the closing tags are written inside the text pass so the cell needs one room check, a plain row is tested with one flag, and the text check reads sixteen characters per step.
- The shared string table is looked up and added to with one hash (.NET 8 and later).
- sharedStrings.xml is composed as text in chunks and handed to the XML writer raw, instead of one XML writer call per element.
The files are the same byte for byte.
…a cell outside columns 1 to 16,384 before anything of it is staged

Skip added its count to NextColumn unchecked, so int.MaxValue wrapped the cursor negative. The column check in BeginCellRaw let a negative column through and the cell's opening bytes were already in the buffer when the column letters threw, and in Auto mode the column state array was indexed out of range. Skip now compares the count against the room left, and every text, formula, hyperlink and raw cell entry point checks the column before it stages a byte, adds to the shared table or registers a link, so a caller that catches the exception and keeps writing still gets a valid sheet.
…eet only from its second use there

The sample of a column on a second sheet found the first sheet's entries in the shared table and counted them as repeats, so a unique column written on two sheets was shared and the table grew about eleven times. Each sheet now takes a scope (the table size when its first text arrives and a stamp): entries below that size are reported as new on first use and as repeats afterwards, so the policy sees what it would see with an empty table per sheet. A sheet of a new package has watermark 0 and is byte for byte unchanged. Also adds the .NET Framework tests for the Skip fix and this one, and the changelog entries for both.
Excel rejects the whole workbook when one cell holds more, and the library saved it without a word. Every text entry point (plain and inline text, rich text runs together, the cached text of a formula, hyperlink text, the Cells indexer, and through them template, edit and object mapping) now throws an ArgumentException in plain words before any byte is staged, a link registered or a table entry added. 32,767 itself still works.
A raw CR in element text is read back as LF by every XML parser, so CRLF came back as LF. The character reference reads back as CR in Excel (through COM), ExcelReader and openpyxl; Excel's own _x000D_ is shown literally by openpyxl, so it was not chosen. Done in the inline encoder (vector and scalar loops), the general text appender, the shared string table (short and long entries), rich-text runs and the template copy. The differential tests now compare with an XmlWriter set to Entitize, which writes the same reference in text.
…ed cell addresses as we measured them, fuzz a seed with addresses

StringStorage and the docs said every reader reads inline and shared text alike; openpyxl shows the escapes of control characters in both and of a literal _xHHHH_ in inline text, so say that Excel and ExcelReader read them alike. WriteCellReferences: list what was checked on 2026-11-16 (EPPlus, ClosedXML, ExcelDataReader, NPOI XSSF read gaps correctly; MiniExcel 1.46.0, NPOI SAX and the ACE OleDb provider do not) and correct the earlier claim for MiniExcel. Utf8Numbers: the .NET Core 1 and 2 claim was untrue (they describe themselves as .NET Core 4.6.x). CHANGELOG: a sheet under 128 rows has its text stored as Shared does. Benchmarks README: dates of the new scenarios, no numbers. ReaderFuzzTests: a seed written with WriteCellReferences so cell references still get mutated.
…filler cell after the rich text

A cell holds at most 32,767 characters, so with the text alone the block after the first flush was about 16,030 bytes and the decoder could never overflow, which let the old buffer sizing pass. A 2,000 character cell after the rich text makes that block a whole buffer; with the old sizing restored the tests fail ('output char buffer is too small') in both test projects. Also corrects the comments on the long text in the allocation test and gives it a 16 KB write buffer so it is longer than the buffer again.
…late copy

Edit re-escaped the text of untouched cells with only & < >, so an inline cell, a cached formula text or a rich-text run came back with a raw CR (read as LF), including in our own files, where Auto writes unique columns inline. Template copies wrote text made only of white space (which XmlReader reports as a whitespace node) through WriteWhitespace, which also writes a raw CR. Both now write the character reference. The test covers Shared, Inline and Auto files, white-space-only texts, formula results and rich-text runs through both Edit and OpenTemplate.
… and empty the rendering buffer on every exit

WriteRichText rendered the runs into a buffer the next rich-text cell reuses before it checked the column, and only the copy into the row emptied it, so a refused cell's runs came out in front of the next cell's text ('STALEfresh'), on new sheets and template sheets. The column is checked first and the buffer is emptied in a finally that covers the whole write. The other places that stage text in a reused buffer were checked: no other code keeps one between cells.
BeginScope resized the stamp array to the exact table size at every sheet, so a package of many small sheets reallocated and copied it once per sheet (115 MB of allocation for 300 sheets of 200 rows by five unique columns). It now doubles, capped at what the budget (plus the template's entries) can hold. The comment now says what the memory costs for template entries, which the budget does not charge.
Counting them as new on first use, like the entries of an earlier sheet, made a template-filled column that cycles through the template's 300 phrases inline, and the file 38% larger (130 KB to 180 KB; 14% for 3,000 names). The template's entries are in the file anyway, so using them costs the table nothing; they count as repeats again, which gives the d2a3682 output for a single template-filled sheet (same cell counts and file sizes in 9 scenarios). Entries of earlier sheets keep the scope rule. Pinned by a test; the CHANGELOG now states what is and is not unchanged.
A StringBuilder or a type with its own ToString passed the setter and was refused when the sheet was sealed, after which every Save failed and the package was lost. Such objects are now converted by the rule the writer uses (invariant culture for IFormattable, ToString otherwise) and measured when assigned, so the assignment throws and the cell is left as it was. The common types (numbers, booleans, dates, Guid, enums, strings) cost one type test and allocate nothing.
WriteEmpty() with no style and a null text counted on past 16,385, which contradicted what Skip promises and made Skip(0) fail afterwards with a message about a column that does not exist.
One launch let a row land in a slightly slower process state (about 5 ms on a 55 ms write, seen in the same binary across launches), so a single-launch row could flip between two runs. Three launches put that spread into the StdDev column.
…the figures in the README and docs up to date

The tables in the README, benchmarks/README.md and two docs pages came from a developer
workstation and an earlier version. They now come from one Azure Standard_D8s_v6 machine
(2026-11-18), three launches per row for the five standard write scenarios, with SpreadCheetah
at the same zip level next to every SheetStream row.

The comparison is stated as it measured: ahead on long, wide, very wide, million-row and styled
sheets, level on mixed text, about 2% behind on all-unique text at the Optimal zip level, and
SpreadCheetah allocates less. MiniExcel is faster than the default SheetStream settings on
StyledDates only. Two Fastest-zip rows show a start-up effect in their mean; the page explains
it and compares medians there.

benchmarks/published-results holds the BenchmarkDotNet reports, peak.log, verify.log and the
hardware record of this run and of two earlier full runs, and the three-run ratio table.
The changelog heading is dated in the previous commit.
@glenntosco
glenntosco merged commit 9da3863 into main Nov 18, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant