Repository navigation
0.7.0: string storage per column, faster cell writing, re-measured benchmarks - #43
Merged
Merged
Conversation
SpreadCheetah 1.29.0 joins the write scenarios at its default compression (Fastest) and at Optimal, so each row has a SheetStream row at the same level. The suite now targets net10.0. Results and the README follow once measured on a quiet machine.
A cell in the column right after the previous cell of its row no longer carries its address (r is optional in ECMA-376; a cell without it belongs in the next column, the first of a row in column A). A cell after a gap from Skip() or an unstyled empty cell still writes r, so positions stay exact. TemplateStyle now finds cells that have no r. Tests that read cell addresses through the Open XML SDK work them out from position, and new tests cover Skip, AutoFit, Cells[,], Edit and template mode.
… per cell Numbers, style ids, shared string indexes and row numbers are written through a per-sheet char buffer (WriteChars) rather than XmlWriter.WriteValue, which builds a new string every call. The text is the same as XmlConvert produces. netstandard2.0 cannot format a double into a buffer and keeps the old call for doubles and decimals. The row number text is no longer built per row. Tests pin the formatted text against XmlConvert and assert that writing cells allocates nothing per cell.
The negative-style-index test found cells by their r attribute, which the writer now leaves off consecutive cells, so it would have mutated almost nothing. It matches the cell tag itself now and checks that every cell was changed.
…ard2.0 still allocates
…cell The default still leaves the r attribute off cells that follow the previous cell, which keeps files smaller and writing faster. Setting the option to true writes r on every cell, in every way of writing cells (Row, Cells[,], AutoFit, Worksheets.Edit and rows filled into a template), for tools that need it: the NPOI event (SAX) reader, the Excel ADO / OleDb provider, Microsoft Spreadsheet Compare and Open XML SDK code that reads Cell.CellReference. The setting is read once per row, so each cell still pays one bool test.
…docs Replaces the changelog sentence that said NPOI's event reader fails with the option that fixes it.
… per piece Numbers, dates, durations, booleans, shared and inline strings and styled empty cells are now composed as finished XML in a fixed-size, reused buffer and passed to the XML writer with a single WriteRaw call per batch. This avoids the writer's per-element and per-attribute checks and state tracking for every cell. The file is unchanged. Text is escaped by hand exactly as the XML writer escaped it (&, < and >; everything else, including line breaks, is left alone), and the buffer is flushed before the row ends and before any cell that still goes through the XML writer (formulas, rich text, error cells, cells copied from a template), so the order of the output is preserved. New tests write the same scripts of cells through the library and through a reference built on a real XmlWriter and compare the sheet XML character for character: every numeric type and extreme, dates, booleans, shared and inline text with every special character, every UTF-16 code unit, text around the buffer size, gaps, both address settings, AutoFit, and seeded random scripts. The allocation test also covers escaped text, gaps and the other cell kinds.
…cks, for measurement only
Rows are composed as bytes in the write buffer of the part (a new PartStream replaces BufferedStream, one buffer shared by all parts) instead of being handed to XmlWriter.WriteRaw, which re-encoded and copied them again. The row element, formulas, error cells and cached results are written the same way; rich text still renders its runs with an XmlWriter. Template and edited sheets keep the XmlWriter in charge and receive the same bytes as text. Short decimals are formatted by a small routine (Utf8Numbers) that gives the text of the runtime's shortest round-trip formatter and leaves everything it cannot decide to the runtime; on .NET Framework the runtime formats all doubles as before. Output is character-identical (differential tests against a real XmlWriter, now covering formulas, rich text, row options, small buffers and template sheets). Writing a sheet allocates about 150 KB instead of about 690 KB. The reader benchmark file is written with cell references so every reader under test can read it.
Rows used to go through the part's XmlWriter, which is dead after any failure, so a caller that caught an error and carried on could never save a damaged sheet. Two ways around that existed on the new path and are closed: - PartStream remembers a failed write to the zip entry and fails every later write, drain and close. - A rich-text cell is rendered before any of it is put in the row, so a run that cannot be rendered (a font name XML cannot carry) leaves nothing behind instead of half a cell that made the whole sheet unreadable. Also: the README and docs no longer say rows go through an XmlWriter.
Every library in the suite now writes three more scenarios from the same deterministic data: SpreadCheetah's own published shape (100,000 x 10 distinct strings), a realistic string mix with repeated and unique columns and text that needs XML escaping, and a styled sheet of dates, date-times, money and percentages. The --verify guard compares every cell value (and, for the styled sheet, every number format) with the generator, and prints each string scenario's data profile. --peak runs the new scenarios too.
…a clean Windows machine Prints the hardware, operating system, SDK and package versions, builds, stops if the correctness check fails, then runs the chosen scenarios and the peak-memory measurement and copies the reports and a hardware.txt into a time-stamped results folder. -DryRun only prints the plan. A double-click launcher is included for machines that block .ps1 files by default.
…nd fairness rules Documents SpreadCheetahOwn, StringHeavy and StyledDates (including the data profile of the string scenarios), how SpreadCheetah and MiniExcel are fed, the stricter correctness guard, and the run-on-fresh-vm script. Also fixes the documented filters: matching ignores letter case, so '*Read*' had been selecting every SpreadCheetah benchmark too.
…at a buffer boundary The text buffer used to turn staged bytes back into characters was one char per byte, but a flush that ends inside a multi-byte character carries bytes into the next flush, which can then decode to one char more than the buffer holds. The buffer now has the decoder's worst-case size. A failure while handing bytes to the XML writer now makes the package refuse to save instead of leaving a half-written cell behind, and the rich-text render stream is emptied even when copying fails.
…or an infinity The cached-value overload wrote text that Excel cannot use, and a different text on .NET than on .NET Framework. The formula is now written with no stored result on every build, so Excel calculates it on open.
…d write When a write to the output failed, the row writer reset its buffer before the error left it, so the next row closed the row by stepping back over a byte that was no longer there and every later row threw an index error instead of the I/O error, as did Save. Starting or closing a row now reports the stored failure first, and the row is marked as having cells before anything can be flushed. The fail-stop tests now require an I/O error from every row after the first failure and from Save, including for an output that fails for good after a given size.
…ames the original failure, cover the template boundary on .NET Framework
…, Shared and Inline Text cells used to go through the shared string table whatever they were, which cost a hash lookup per cell, kept every distinct text in memory until the save and made text-heavy sheets slow. StringStorage.Auto looks at the first 128 to 512 cells of each column and then either keeps sharing it (values repeat and are long or varied) or writes it inline with no lookup. Shared is the old behaviour, Inline never shares; UseSharedStrings = false still means inline. Inline text is also written in one pass, eight characters at a time on net8.0, instead of three scans and a copy. The differential tests against a real XmlWriter cover every length and every kind of special character at every position.
…fter the string storage change
…add the Auto differential test, document the setting in the changelog
…at repeats among thousands of values is shared A column in which none of the first 128 cells repeated used to be written inline for good. A column that repeats among tens of thousands of distinct values (customer or product names) shows no repeat in its first cells, so it stayed inline and its file came out 3 to 7 times larger than with Shared, and the write was slower. Such a column now gets two more looks of 512 cells, at its 512th and 8,192nd cell. A look counts how many cells repeat a text the column already put in the table, estimates the number of different values from that, and shares the column when the estimated vocabulary (values times average length) is more than the 24 KB the zip compressor can remember. A vocabulary that fits stays inline. Measured, 100,000 rows of 20,000 different values: 25 characters 6.4 MB to 1.9 MB, 80 characters 17.1 MB to 2.6 MB, same size as Shared, same speed as Shared.
- An unstyled inline cell: its 24 opening bytes and 13 closing bytes are written with three and two 8-byte stores (.NET 8 and later) instead of a block copy and a byte loop, the closing tags are written inside the text pass so the cell needs one room check, a plain row is tested with one flag, and the text check reads sixteen characters per step. - The shared string table is looked up and added to with one hash (.NET 8 and later). - sharedStrings.xml is composed as text in chunks and handed to the XML writer raw, instead of one XML writer call per element. The files are the same byte for byte.
…ehind the second look
…a cell outside columns 1 to 16,384 before anything of it is staged Skip added its count to NextColumn unchecked, so int.MaxValue wrapped the cursor negative. The column check in BeginCellRaw let a negative column through and the cell's opening bytes were already in the buffer when the column letters threw, and in Auto mode the column state array was indexed out of range. Skip now compares the count against the room left, and every text, formula, hyperlink and raw cell entry point checks the column before it stages a byte, adds to the shared table or registers a link, so a caller that catches the exception and keeps writing still gets a valid sheet.
…eet only from its second use there The sample of a column on a second sheet found the first sheet's entries in the shared table and counted them as repeats, so a unique column written on two sheets was shared and the table grew about eleven times. Each sheet now takes a scope (the table size when its first text arrives and a stamp): entries below that size are reported as new on first use and as repeats afterwards, so the policy sees what it would see with an empty table per sheet. A sheet of a new package has watermark 0 and is byte for byte unchanged. Also adds the .NET Framework tests for the Skip fix and this one, and the changelog entries for both.
Excel rejects the whole workbook when one cell holds more, and the library saved it without a word. Every text entry point (plain and inline text, rich text runs together, the cached text of a formula, hyperlink text, the Cells indexer, and through them template, edit and object mapping) now throws an ArgumentException in plain words before any byte is staged, a link registered or a table entry added. 32,767 itself still works.
A raw CR in element text is read back as LF by every XML parser, so CRLF came back as LF. The character reference reads back as CR in Excel (through COM), ExcelReader and openpyxl; Excel's own _x000D_ is shown literally by openpyxl, so it was not chosen. Done in the inline encoder (vector and scalar loops), the general text appender, the shared string table (short and long entries), rich-text runs and the template copy. The differential tests now compare with an XmlWriter set to Entitize, which writes the same reference in text.
…ed cell addresses as we measured them, fuzz a seed with addresses StringStorage and the docs said every reader reads inline and shared text alike; openpyxl shows the escapes of control characters in both and of a literal _xHHHH_ in inline text, so say that Excel and ExcelReader read them alike. WriteCellReferences: list what was checked on 2026-11-16 (EPPlus, ClosedXML, ExcelDataReader, NPOI XSSF read gaps correctly; MiniExcel 1.46.0, NPOI SAX and the ACE OleDb provider do not) and correct the earlier claim for MiniExcel. Utf8Numbers: the .NET Core 1 and 2 claim was untrue (they describe themselves as .NET Core 4.6.x). CHANGELOG: a sheet under 128 rows has its text stored as Shared does. Benchmarks README: dates of the new scenarios, no numbers. ReaderFuzzTests: a seed written with WriteCellReferences so cell references still get mutated.
…filler cell after the rich text
A cell holds at most 32,767 characters, so with the text alone the block after the first flush was about 16,030 bytes and the decoder could never overflow, which let the old buffer sizing pass. A 2,000 character cell after the rich text makes that block a whole buffer; with the old sizing restored the tests fail ('output char buffer is too small') in both test projects. Also corrects the comments on the long text in the allocation test and gives it a 16 KB write buffer so it is longer than the buffer again.
…late copy Edit re-escaped the text of untouched cells with only & < >, so an inline cell, a cached formula text or a rich-text run came back with a raw CR (read as LF), including in our own files, where Auto writes unique columns inline. Template copies wrote text made only of white space (which XmlReader reports as a whitespace node) through WriteWhitespace, which also writes a raw CR. Both now write the character reference. The test covers Shared, Inline and Auto files, white-space-only texts, formula results and rich-text runs through both Edit and OpenTemplate.
… and empty the rendering buffer on every exit
WriteRichText rendered the runs into a buffer the next rich-text cell reuses before it checked the column, and only the copy into the row emptied it, so a refused cell's runs came out in front of the next cell's text ('STALEfresh'), on new sheets and template sheets. The column is checked first and the buffer is emptied in a finally that covers the whole write. The other places that stage text in a reused buffer were checked: no other code keeps one between cells.
BeginScope resized the stamp array to the exact table size at every sheet, so a package of many small sheets reallocated and copied it once per sheet (115 MB of allocation for 300 sheets of 200 rows by five unique columns). It now doubles, capped at what the budget (plus the template's entries) can hold. The comment now says what the memory costs for template entries, which the budget does not charge.
Counting them as new on first use, like the entries of an earlier sheet, made a template-filled column that cycles through the template's 300 phrases inline, and the file 38% larger (130 KB to 180 KB; 14% for 3,000 names). The template's entries are in the file anyway, so using them costs the table nothing; they count as repeats again, which gives the d2a3682 output for a single template-filled sheet (same cell counts and file sizes in 9 scenarios). Entries of earlier sheets keep the scope rule. Pinned by a test; the CHANGELOG now states what is and is not unchanged.
A StringBuilder or a type with its own ToString passed the setter and was refused when the sheet was sealed, after which every Save failed and the package was lost. Such objects are now converted by the rule the writer uses (invariant culture for IFormattable, ToString otherwise) and measured when assigned, so the assignment throws and the cell is left as it was. The common types (numbers, booleans, dates, Guid, enums, strings) cost one type test and allocate nothing.
WriteEmpty() with no style and a null text counted on past 16,385, which contradicted what Skip promises and made Skip(0) fail afterwards with a message about a column that does not exist.
One launch let a row land in a slightly slower process state (about 5 ms on a 55 ms write, seen in the same binary across launches), so a single-launch row could flip between two runs. Three launches put that spread into the StdDev column.
…the figures in the README and docs up to date The tables in the README, benchmarks/README.md and two docs pages came from a developer workstation and an earlier version. They now come from one Azure Standard_D8s_v6 machine (2026-11-18), three launches per row for the five standard write scenarios, with SpreadCheetah at the same zip level next to every SheetStream row. The comparison is stated as it measured: ahead on long, wide, very wide, million-row and styled sheets, level on mixed text, about 2% behind on all-unique text at the Optimal zip level, and SpreadCheetah allocates less. MiniExcel is faster than the default SheetStream settings on StyledDates only. Two Fastest-zip rows show a start-up effect in their mean; the page explains it and compares medians there. benchmarks/published-results holds the BenchmarkDotNet reports, peak.log, verify.log and the hardware record of this run and of two earlier full runs, and the three-run ratio table.
The changelog heading is dated in the previous commit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this ships (0.7.0)
PackageOptions.StringStorage(Autoby default,Shared,Inline): text goes in the cell or in the shared string table, whichever suits each column, so unique text is no longer hashed and remembered.rattribute (valid under ECMA-376).PackageOptions.WriteCellReferences = truerestores it. NPOI 2.7.6's SAX reader, Excel OleDb ACE and MiniExcel 1.46.0 misread such files; the README and changelog say so.Skipand cell writes refuse a column past XFD, a second sheet is judged on its own text underAuto, a formula with a NaN or infinite result stores no result.benchmarks/published-results. The comparison is mixed and written as measured: ahead of SpreadCheetah on long, wide, very wide, million-row and styled sheets, level on mixed text, about 2% behind on all-unique text atOptimal; SpreadCheetah allocates less.Checked before this PR
dotnet buildanddotnet teston the final tree: 13 + 48 + 682 (net10.0) and 682 (net8.0) tests pass.tests/excel-check.ps1on the samples and a 100,000-row workbook.After the
buildcheck is green and this is merged, taggingv0.7.0publishes both packages to nuget.org through Trusted Publishing.