Repository navigation
Conversation
cmdupuis3
left a comment
There was a problem hiding this comment.
-
Grid.__eq__can't tell when only one of the coordinates is wrong. Ingrid.pyline 814 it should beandinstead ofor. This isn't new in this PR but the changes here are the first to actually rely on it. -
The test doesn't actually test the part that would trigger nondeterminism, because
chunks="auto"leaves the data as a single dask block.
Wow, great point… I opened #1881 to improve visibility for this bug. My preference would be to solve it via a separate PR. Since the PR here is just adding tests I think it's fine to block this one until that issue gets fixed.
I'm pretty confident that the |
|
Agree with @cmdupuis3 that #1881 should land first to fix the As you both noted, one small follow-up here: could we update |
Sure, I've added a check for I'll re-request reviews here once #1881 gets fixed! |
|
I believe this is ready for re-reviews now that #1889 has merged! |
cmdupuis3
left a comment
There was a problem hiding this comment.
I'm still not seeing reliable non-determinism in the tests, although we're getting closer. I'm seeing more consistent non-determinism by setting dask.config.set({"dataframe.shuffle.method": "disk"})
Reverts the `test_scrip_dask_lazy_dedup_matches_eager` and `test_scrip_dask_lazy_dedup_matches_eager_radians` tests to their versions from `main`, now that they started to fail on this branch (reported as #1905). Adds `test_scrip_dask_and_eager_dedup_assign_identical_node_ids` to ensure that the SCRIP reader's dedup algorithms themselves aren't causing issues. Adds (with xfail) `test_scrip_dask_and_eager_grids_match_with_default_settings` as a regression test for #1905; remove the xfail marker once #1905 is solved.
The non-determinism in the original issue report was in the non-dask case, which is why I didn't focus on trying to find a way to make non-deterministic dask results. But, I do agree that with that dask config setting, you start to see non-deterministic results (when you checkout to the commit just before PR #1775 merged). So, it makes sense to have the regression test guard against that config as well. I've updated I also had to make other updates due to #1905 which was recently introduced into I believe this is ready again for re-review; please let me know if you find any other issues! |
cmdupuis3
left a comment
There was a problem hiding this comment.
I am seeing reliable nondeterminism where I should be. Approving, with a typo fix.
Co-authored-by: Christopher Dupuis <[email protected]>
|
@rajeeja just a quick ping, I believe this is now ready for you to re-review if you get the chance! |
Closes #1738
Overview
Issue #1738 was apparently already solved by PR #1775, though this wasn't realized at the time. E.g., running the MCVE from that issue report currently succeeds on main. It succeeds when checking out to the commit which merged 1775 to main, and it fails when checking out to the commit just before 1775 was merged. The PR here adds
test_scrip_load_is_deterministicas a regression test to confirm the bug initially reported in 1738 is actually solved.Discussion in thread of 1775 (see e.g. #1775 (comment)) also scoped 1738 to additionally check whether the dask and eager paths produce the same ordering as each other, and possibly make changes if the orderings disagreed. A close investigation across a few examples shows that these do indeed already produce the same ordering; this just wasn't checked directly in 1775.
So, the PR here also updatestest_scrip_dask_lazy_dedup_matches_eagerandtest_scrip_dask_lazy_dedup_matches_eager_radiansto assert exact equality for grid_lazy == grid_eager, instead of comparing after "normalizing" the order of each based on the winding order from face_node_connectivity. The orders are already exactly the same in both paths, before normalizing.EDIT: I reverted the changes which are now crossed-out in the previous sentences; they revealed that issue #1905 was recently introduced into
main. In their place, I've addedtest_scrip_dask_and_eager_dedup_assign_identical_node_idswhich assertsgrid_lazy == grid_eagerwhen usingmerge_coincident_nodes=False(to avoid issue 1905), confirming that at least the SCRIP reader's deduplication algorithms agree between eager/dask, even though an open_grid call with default settings does not agree currently due to 1905. (I also addedtest_scrip_dask_and_eager_grids_match_with_default_settings, currently marked withxfail, as a additional tracker for solving issue 1905.)PR Checklist
General
Testing & Benchmarking
Documentation and Examples
docs/api.rst; internal (private) function names start with an underscore (_)AI Disclosure
AI Usage: GitHub Copilot / Claude Sonnet 5.5, to help with understanding: the problem, which parts of the commits from #1775 solved it, whether the existing tests demonstrated it was solved (they did not), and which tests to consider updating/adding. Claude Opus 5.5 for more debugging, updates, and planning how to proceed, upon discovering the bug reported in #1905.