Repository navigation
Use _construct_direct in UxDataArray.to_xarray() and UxDataset.to_xarray() - #1860
Conversation
for efficiency
ASV BenchmarkingBenchmark Comparison ResultsBenchmarks that have improved:
Benchmarks that have stayed the same:
Benchmarks that have got worse:
|
also adds test_nested_mutables_match_old, which Claude recommended after being prompted for anything else that might cause an issue.
dylannelson
left a comment
There was a problem hiding this comment.
Looks good and not seeing any blockers. Only one small thing came up when the PR was tested using different versions. Not blocking though since it could be addressed later if it actually shows up. Thanks!
cmdupuis3
left a comment
There was a problem hiding this comment.
For the ds._close suggestion, you can also consider adding this unit test:
@pytest.mark.parametrize("grid_format", ["UGRID", "HEALPix"])
def test_close_not_shared(self, grid_format):
"""Closing the result never closes the original; the old path gave ``_close=None``."""
uxds = self.get_uxds()
closed = []
uxds.set_close(lambda: closed.append(True)) # UxDataset.__init__ drops _close, so set one
result = uxds.to_xarray(grid_format)
assert result._close is None
result.close()
assert not closedCo-authored-by: Christopher Dupuis <[email protected]>
|
@cmdupuis3 Thank you for the review! I incorporated some of your suggestions directly (github automatically marked the corresponding threads as resolved), and the rest of them I incorporated with some slight refactoring. I believe everything has been fixed; if you see any other issues please let me know! |
Closes #1794
Overview
Uses
xr.Dataset._construct_direct()andxr.DataArray._construct_direct()to speed upto_xarray()calls, instead of passing inputs intoxr.Dataset()andxr.DataArray()which go through all of xarray's initialization validation checks. Every UxDataset / UxDataArray is already a valid xr.Dataset / xr.DataArray, so there's no need to repeat these validation checks.This probably isn't a particularly hot path because these methods don't tend to take that long compared to other operations. But, the validation checks scale with the number of coordinates and variables. For example, ASV benchmarks are currently showing ~300x speedup (from ~100ms to ~0.3ms) for a dataset with 100 data_vars and 10 coords, and ~3000x speedup (from 7.6s to 2.7ms) for a dataset with 1000 data_vars and 100 coords.
This PR adds a lot of regression tests to ensure the new behavior matches the "naive" implementation of using xr.Dataset() or xr.DataArray() directly. Iterated with Claude to help generate tests I might not have thought of, and debug various little issues I found along the way, but I also tried to limit the tests to a minimum set that remains human-readable and (hopefully) easy enough to read and review.
PR Checklist
General
Testing & Benchmarking
Documentation and Examples
docs/api.rst; internal (private) function names start with an underscore (_)AI Disclosure
AI Usage: GitHub Copilot's inline code suggestions, plus lots of iteration with Claude on the implementation and test details.