Skip to content

Fix Quantity merging across concatenation semantics - #1979

Open
emanueldellipaoli01-a11y wants to merge 1 commit into
NeuralEnsemble:masterfrom
emanueldellipaoli01-a11y:fix-1978-quantity-merge-compat
Open

emanueldellipaoli01-a11y wants to merge 1 commit into
NeuralEnsemble:masterfrom
emanueldellipaoli01-a11y:fix-1978-quantity-merge-compat

Conversation

@emanueldellipaoli01-a11y

@emanueldellipaoli01-a11y emanueldellipaoli01-a11y commented Nov 19, 2026 •

Copy link
Copy Markdown

Fixes #1978.

This PR addresses a compatibility issue between Neo and the upcoming changes to quantities in [python-quantities PR #293](python-quantities/python-quantities#293).

1. Concatenation of Quantity arrays

Currently, np.concatenate, np.hstack, np.vstack, np.stack and np.append can discard the units of Quantity arrays.

Neo has several operations that rely on this behaviour by explicitly rescaling quantities and reattaching their units after concatenation.

For example:

othertimes = other.times.rescale(self.times.units)
times = np.hstack([self.times, othertimes]) * self.times.units

The upcoming change in quantities makes these NumPy functions preserve units and automatically rescale compatible quantities.

Consequently, the existing Neo implementation would attach the units twice, producing incorrect dimensionality (e.g. s**2 instead of s).

This incompatibility was confirmed when running Neo's core tests against the proposed changes in python-quantities PR #293. The following tests failed:

  • test_Epoch_merge
  • test_Event_merge
  • TestSpikeTrainList::test_multiplexed

The solution is to explicitly concatenate the magnitudes and reattach the reference units, ensuring compatibility with both behaviours of quantities.

2. Incorrect unit conversion in SpikeTrain annotations

There is also an existing issue in SpikeTrain.merge().

The result of other_ann.rescale(self_ann.units) is currently discarded. Consequently, annotations expressed in different compatible units can be concatenated without proper conversion.

For example, merging an annotation containing 1 s with another containing 500 ms may produce [1, 500] s instead of the correct [1, 0.5] s.

This PR fixes that behaviour and handles UncertainQuantity annotations while preserving their uncertainty information.

3. Regression tests

New regression tests cover:

  • Event and Epoch merging.
  • SpikeTrain merging and array annotations.
  • Quantity and UncertainQuantity unit conversion.
  • SpikeTrainList operations.
  • IrregularlySampledSignal concatenation.
  • Relevant IO serialization behaviour.

The goal is to ensure that Neo produces correct results independently of the concatenation semantics provided by different versions of quantities.

@zm711 zm711 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll leave @apdavison to comment on this. But for a PR like this could you point at an actual problem that is happening between quantities and Neo? I don't see any issues linked and I see no MREs in your description. This is quite a number of code changes in core bits of Neo so I think we will need a bit more motivation for the benefit of this before we review this.

@emanueldellipaoli01-a11y

Copy link
Copy Markdown
Author

I'll leave @apdavison to comment on this. But for a PR like this could you point at an actual problem that is happening between quantities and Neo? I don't see any issues linked and I see no MREs in your description. This is quite a number of code changes in core bits of Neo so I think we will need a bit more motivation for the benefit of this before we review this.

Thanks for pointing this out. There's actually already an issue for this: #1978.

I've updated the PR description to link to it and added some more context about the compatibility issues with the upcoming changes in python-quantities#293.

The quantities maintainers confirmed that testing the proposed concatenation changes against Neo's core tests leads to failures in Event.merge(), Epoch.merge() and SpikeTrainList.multiplexed, because units end up being attached twice.

I also mentioned the existing bug in SpikeTrain.merge(), where the result of rescale() is ignored, causing incorrect values in merged array annotations.

I've added regression tests for these cases, as well as the related IO behaviour.

Thanks again for taking the time to review this. Hopefully the extra context makes it clearer why these changes are needed.

@emanueldellipaoli01-a11y

Copy link
Copy Markdown
Author

Hi @apdavison,

I saw Zach's comment and updated the PR description with a link to #1978 and a more detailed explanation of the quantities compatibility issue.

The core tests I ran pass locally (401 passed, 9 skipped). The two IO test files report 2 passed and 17 skipped.

The NeoCoreTest and NeoIoTest-automatic-trigger workflows are showing action_required. Do these need approval from a maintainer before they can run?

The Read the Docs build also failed, but the log says it was terminated due to a timeout after 30 minutes and 50 seconds. I'm not sure what caused it.

I'd appreciate your thoughts on the changes whenever you have time.

Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make merging of Event, Epoch, SpikeTrainList and array annotations independent of how quantities handles np.concatenate

2 participants