diff --git a/docs/migration/ravendb-to-sql-migration-instructions.md b/docs/migration/ravendb-to-sql-migration-instructions.md index dbcf16c25e..27e456c9c0 100644 --- a/docs/migration/ravendb-to-sql-migration-instructions.md +++ b/docs/migration/ravendb-to-sql-migration-instructions.md @@ -190,7 +190,7 @@ Keep RavenDB running until every category is Done or Abandoned. If RavenDB goes ## Step 8: Finish the migration 1. Stop ServiceControl. -2. Run `--migration-verify`. It counts the rows in RavenDB again, and compares them with the rows that the copy read. It also shows the skipped and merged rows. It exits with 0 only when every category is Done or Abandoned. +2. Run `--migration-verify`. It counts the rows in RavenDB again, and compares them with the rows that the copy read. It also shows the skipped rows. Merged and changed rows are not shown here. Each one is logged as a warning by the copy run that merged or changed it, and that run's summary counts them. It exits with 0 only when every category is Done or Abandoned. 3. Start ServiceControl. Finished checkpoints remain in SQL, but no background copy runs and RavenDB is not opened. Verify reads all the data in RavenDB again, but not the message bodies. On a large instance this takes time, and ServiceControl is stopped during that time. diff --git a/docs/migration/ravendb-to-sql-migration-overview.md b/docs/migration/ravendb-to-sql-migration-overview.md index 43c8f20f73..36c4d99b4d 100644 --- a/docs/migration/ravendb-to-sql-migration-overview.md +++ b/docs/migration/ravendb-to-sql-migration-overview.md @@ -155,8 +155,8 @@ There are eighteen *categories*: sixteen required and two optional. | Licensing report masks | Required | | | Uploaded licensed endpoint details file | Required | Nothing recomputes it. Losing it means downloading it again from the licence portal and uploading it again | | Subscriptions | Required | | -| Integration events still waiting to be sent | Required | Each is an event a subscriber has not received yet. There are only any if the old instance fell behind or could not reach the broker. They are sent once ServiceControl opens, later than they would have been. RavenDB sends them in no particular order anyway, so no ordering is lost | -| Custom checks, with their last status | Required | Not every check reports again. An endpoint that is down never does, and a check with no repeat interval reports only when its endpoint starts. Leaving one out could hide a known failure until that endpoint restarts | +| Integration events still waiting to be sent | Required | Each is an event a subscriber has not received yet. There are only any if the old instance fell behind or could not reach the broker. They are sent once ServiceControl opens, later than they would have been, except a `MessageFailed` event for a message archived or resolved before the move (see [Rows removed after the copy](#rows-removed-after-the-copy)). RavenDB sends them in no particular order anyway, so no ordering is lost | +| Custom checks, with their last status | Required | Not every check reports again. An endpoint that is down never does, and a check with no repeat interval reports only when its endpoint starts. Leaving one out could hide a known failure until that endpoint restarts. Results of the six checks the RavenDB persister runs about its own database, matched on id and category, are left out, as harmless skips | | Failed error imports, with bodies | Required | Messages taken off the error queue but not ingested. They exist nowhere else and never expire. After the move the "Error Message Ingestion" custom check keeps flagging them and `--import-failed-errors` imports them into SQL. Importing them on RavenDB first is better still | | Group comments | Required | Copied after unresolved failed messages, so a note like "do not retry" is there when ServiceControl opens. Blank comments are not copied. Some comments are removed after ServiceControl opens, see [Open decisions](#rows-removed-after-the-copy) | | Event log | Optional | The history behind the ServicePulse activity feed. On a busy instance it holds a row for every failure, retry and submission | @@ -174,7 +174,7 @@ There are eighteen *categories*: sixteen required and two optional. **Written new by the copy rather than copied:** -- Retry claims in `FailedMessageRetries`. A claim ties a message to the retry that sent it. The copy writes one for each retry-issued message it inserts, see [Open decisions](#values-that-change-or-merge-in-sql). +- Retry claims in `FailedMessageRetries`. A claim ties a message to the retry that sent it. The copy writes one for each retry-issued message it inserts that RavenDB still held a claim for, so a retry by id sends or holds back a message just as it did before the move. ### Optional data windows @@ -186,8 +186,13 @@ There are eighteen *categories*: sixteen required and two optional. ### Data that changes shape -- **Event log items, historic retry operations and pending integration events get new numbers.** Nothing refers to the old ones, so this is safe. -- **Other values change or merge on the way into SQL.** Each is listed under [Open decisions](#values-that-change-or-merge-in-sql). +- **Event log items, historic retry operations and pending integration events get new numbers.** Nothing refers to the old ones, so this is safe. For historic retry operations and pending integration events, the log says so once per category, in an information line. +- **Other values change or merge on the way into SQL.** Each is listed under [Open decisions](#values-that-change-or-merge-in-sql), with the line the log shows for it. + - A row copied with a changed value logs a warning starting `Changed in category on the way into SQL:`, followed by what changed and what was kept. + - A row merged into another logs a warning starting `Merged in category into a row with the same key:`, naming both keys, the source row kept, or the row SQL already held, and the source row dropped. + - The category's summary at the end of the copy counts both, apart from the rows already present, for example `Subscriptions: 14 copied (0 changed), 0 already present, 1 merged, nothing skipped`. The summary is a warning when anything was changed or merged. The counts cover only the current run: a category resumed after a crash counts only the changes and merges of the run that finished it, and counts earlier merges as already present. + - The summary's already present count leaves out the merged rows. If the copy reports more merged rows than SQL found already present in total, the summary adds a warning naming both counts, because the copy's prediction of which keys SQL treats as equal then disagrees with SQL. + - The per-row lines are written after each batch commits. If the process stops between a batch's commit and its log lines, those lines are lost. A resumed copy does not repeat them, because it finds those rows already present. ### Skipped rows @@ -200,7 +205,7 @@ A skip is one of three kinds: - **Fault, a retry can fix it.** Something outside the data went wrong. Fix the cause and retry. - **Fault, no retry can fix it.** The RavenDB document cannot become a SQL row, so a retry skips it again. Only [abandoning](#retry-and-abandon) gets past it. -- **Harmless.** A row SQL would have removed anyway. +- **Harmless.** A row SQL would have removed anyway, or a result of a check the source's own storage runs about itself (the RavenDB source flags its own checks, and the target skips flagged rows). A value SQL can hold only after changing it is not a skip. Those rows are copied, and are listed under [Open decisions](#values-that-change-or-merge-in-sql). @@ -228,6 +233,7 @@ A value SQL can hold only after changing it is not a skip. Those rows are copied | --- | | A blank group comment, which SQL never stores | | An archived or resolved message, or event log item, already past its retention period | +| A result of one of the six checks the RavenDB persister runs about its own database, such as index lag or free disk space, matched on the check's id and category. Only a RavenDB primary instance reports these checks, so nothing updates them after the move. A check that reuses one of these ids under another category is copied, such as a RavenDB audit instance's dirty memory check | - **Any fault skip makes the category [Failed](#failed-categories)**, even one. Harmless skips leave it Done. - **A malformed document does not stop the copy.** It is skipped and counted, and the rest of the category carries on. If skips pass the [early-stop threshold](#failed-categories), the category stops where it is, and abandoning it then gives up every unread row too. Raise the threshold or fix the cause, and retry. @@ -283,11 +289,11 @@ flowchart LR ### Writing to SQL - **A failed message is written as one row**, with its status unchanged. Its older attempts are dropped, see [Open decisions](#values-that-change-or-merge-in-sql). -- **A failed message keeps its failure groups and its last-modified time**, so filtered lists show the same messages as before. +- **A failed message keeps its failure groups and its last-modified time**, so filtered lists show the same messages as before. A failure group with no id is the exception, see [Open decisions](#values-that-change-or-merge-in-sql). - **`UniqueMessageId` keeps its value** but becomes a `uniqueidentifier`. It is the primary key, the ServicePulse URL, the retry key and the body key. - **`StatusChangedAt` is rebuilt.** For archived and resolved messages it is `@expires` minus the current `ServiceControl/ErrorRetentionPeriod` (wrong if that changed, see [Open decisions](#values-that-change-or-merge-in-sql)). For unresolved and retry-issued messages it is the newest attempt's time, because the column is `NOT NULL`. - **Bodies go through `IBodyStoragePersistence`**, which handles compression and the body store. The copier keeps a body inline when it is within `ServiceControl/MaxBodySizeToStore` (102,400 bytes by default), as ingestion does. -- **Throughput rows are written directly, and each day's count is set, not added**, so a resumed copy cannot double-count. The copied rows also stop the collectors gathering the same days again. +- **Throughput rows are written directly, and each day's count is added to the stored count**, as ServiceControl's own throughput write adds it. Endpoints whose names the SQL key treats as one have their counts for a day summed into one row: SQL Server compares names under the key column's own collation, and PostgreSQL compares them as its deterministic collations do, character for character. A resumed copy cannot double-count, because each batch and its checkpoint commit together. The copied rows also stop the collectors gathering the same days again. ### Batch size and throttling @@ -366,7 +372,7 @@ Stop ServiceControl before either command. In a container, run it as a one-off ` **What a retry does:** - **It re-reads the whole category from the start**, leaving rows already in SQL alone. Every body is read again, so a retry takes about as long as the first copy. -- **Historic retry operations and pending integration events** get new keys on every insert, so a retry first deletes what the category copied, in the same save as the reset. This is safe because both are required, so the target started empty. +- **Historic retry operations, pending integration events and throughput history** are deleted first. The first two get new keys on every insert, and throughput history adds each day's count to the stored one, so a re-read would copy them a second time. The retry deletes what the category copied, in the same save as the reset. This is safe because all three are required, so the target started empty and ServiceControl has not opened on it. Once ServiceControl has opened on the database, a retry of any of them is refused and deletes nothing. - **The event log cannot be retried.** It also gets new keys, but it copies after ServiceControl opens, when copied rows cannot be told apart from live ones. It can only be abandoned. **To abandon:** @@ -464,29 +470,35 @@ Each case below changes, merges or drops data, or lets a loss pass without faili ### Values that change or merge in SQL -- **Attempt history collapses to the newest attempt.** SQL keeps the newest attempt and the attempt count, so only the newest attempt keeps its headers and exception. -- **Message redirects, subscriptions and pending retry operations merge when their keys match in SQL.** The copy keeps one row, counts the other as already present, and logs a warning naming both. +- **Attempt history collapses to the newest attempt.** SQL keeps the newest attempt and the attempt count, so only the newest attempt keeps its headers and exception. This is not logged and not counted as a change, because SQL keeps only the newest attempt for every message that failed more than once, by design. +- **A failure group with no id is dropped from its message, and a group with no title or type is stored with an empty one.** SQL keys a group by its id, so the message is missing from that group in ServicePulse, and a group retry or archive leaves it out. ServiceControl's classifiers always set an id, so this needs hand-made data. The log names each message, in a warning whose detail starts `Dropped the failure groups with no id` or `Stored an empty title or type for the failure groups that had none`. +- **Two failure groups of one message whose ids SQL treats as one key become one**, keeping the first group's title and type. ServiceControl never gives a message the same group twice. The log names each message, in a warning whose detail starts `Kept the failure group '' titled '' and dropped`. +- **Message redirects, subscriptions, pending retry operations, group comments and failed error imports merge when their keys match in SQL.** The copy keeps one row, counts the other as already present, and logs a warning starting `Merged <source id> in category <category> into a row with the same key:` naming both. On SQL Server the copy predicts these merges from the key column's collation settings for case, accents, kana type and width, plus SQL Server's trailing-space rule, and reports the merges those settings make. It compares text with .NET's culture rules rather than SQL Server's own tables, so an unusual character the two treat differently can merge without a line. - Redirects merge on source addresses the key column treats as equal. On SQL Server, by default, that includes addresses that differ only in case. - - Subscriptions also merge when they differ only in message-type version, because the SQL key leaves the version out. - - Pending retry operations with the same retry type and request id merge, keeping the latest. -- **On SQL Server, endpoint settings whose names differ only in case merge into one row**, because the name column compares without case by default. PostgreSQL keeps both. + - Subscriptions also merge when they differ only in message-type version, because the SQL key leaves the version out. The warning names the version and endpoint of the row kept and of the row dropped. + - Pending retry operations with the same retry type and request id merge, keeping the latest. The warning names the source row kept and the one dropped, and each source id ends with that operation's completion time. + - On SQL Server, group comments merge when their group ids differ only in trailing spaces. ServiceControl makes every group id from a GUID, so only a comment written on a hand-made group id can merge. PostgreSQL keeps both. + - Failed error imports of the same message merge. Version 6.18 and earlier stored each failed import attempt as its own document, and SQL keeps one row per message. The copy keeps the first it reads, and for those documents RavenDB order is effectively random, so it can keep an older attempt's exception. The warning names both documents, or the stored row's failure time and this one's. +- **On SQL Server, endpoint settings whose names differ only in case, kana type or width merge into one row**, because the name column's default collation ignores all three. A column whose collation also ignores accents merges those too. PostgreSQL keeps both. - The copy keeps the setting whose name exactly matches a known endpoint, because ServiceControl deletes the other. Otherwise it keeps the first in RavenDB order, which is effectively arbitrary. - The choice is made within one batch of up to a thousand. Across batches, the first one met wins. - - The copy logs a warning, and the dry run lists each pair. -- **A message type over 450 characters is cut to its last 450 characters**, as ingestion does. The row is copied, not skipped. + - The copy logs a warning starting `Merged EndpointSettings/<name>` that names both names and the one kept, and the dry run lists each pair. +- **A message type over 450 characters is cut to its last 450 characters**, as ingestion does. The row is copied, not skipped. The log names each message, in a warning whose detail starts `Cut the message type from <length> characters to its last 450`. - **On SQL Server, licensing endpoints whose names differ only in trailing spaces become one row**, and a day both counted holds the sum. PostgreSQL keeps both. - - This is not reported as a merge: the copy logs no warning, and the dry run does not list it. - - A key column whose collation also ignores, for example, accents still makes one row, but keeps only one endpoint's counts. The other's counts are lost. -- **`StatusChangedAt` is wrong for an archived or resolved message if `ServiceControl/ErrorRetentionPeriod` changed since RavenDB stamped it.** It is off by the difference, but the message still ages out of SQL when RavenDB would have deleted it. -- **Every retry-issued message gets a retry claim, even one whose claim RavenDB had already released.** While the claim exists, retrying the message from the failed messages list sends nothing. -- **Failed message edit locks are not copied.** On RavenDB, an edited message that fails again can never be edited again. After the move it can be edited once more. + - The copy keeps the endpoint it meets first, counts the other as already present and logs a warning starting `Merged <source id> in category LicensingEndpoints` naming both. Within one batch the endpoint met first is the one whose source id sorts first, and an endpoint SQL already holds is met before any in the batch. The dry run does not list it. + - A key column whose collation also ignores, for example, accents, kana type or width makes one row in the same way, a day holds the sum of every endpoint the column treats as one, and the copy logs the same warning for it. +- **`StatusChangedAt` is wrong for an archived or resolved message if `ServiceControl/ErrorRetentionPeriod` changed since RavenDB stamped it.** It is off by the difference, but the message still ages out of SQL when RavenDB would have deleted it. Archived and resolved messages are not copied yet, so no log line flags these. +- **Failed message edit locks are not copied.** On RavenDB, an edited message that fails again can never be edited again. After the move it can be edited once more. No log line lists these messages. +- **A failed message's sending and receiving endpoints are rebuilt from its headers** when RavenDB hands back the stored endpoint without its type, following ingestion's own rules on the same headers. A message whose headers name no endpoint gets none. No log line names these messages. +- **A failed error import's failure time is when RavenDB last modified its document**, because RavenDB stores no failure time for an import. ServiceControl never changes an import after storing it, so this is when the import was stored. No log line names imports for this. A missing exception or body is stored empty, and the log names each such import, in a warning whose detail starts `Stored an empty exception`, `Stored an empty body` or `Stored an empty exception and an empty body`. The dry run counts the subscription and endpoint settings merges before anything moves. ### Rows removed after the copy -- **A group comment on a group whose messages are all archived or resolved is removed soon after ServiceControl opens**, because that group has no failed messages in SQL until the background copy brings them. -- **Unresolved messages from version 6.18 or earlier can expire during the copy.** On those versions, a message that failed again after being archived or resolved kept its expiry. One deleted before the copy reads it is not copied and not counted. +- **A group comment on a group whose messages are all archived or resolved is removed soon after ServiceControl opens**, because that group has no failed messages in SQL until the background copy brings them. The same happens when the group's messages are gone from RavenDB or were skipped by the copy. The copy names each such comment, in a warning starting `Changed GroupComment/<group id> in category GroupComments on the way into SQL: The group '<group id>' has no copied failed message`, which also quotes the whole comment. +- **A `MessageFailed` event still waiting to be sent for a message archived or resolved before the move is never sent.** ServiceControl looks the message up when it sends the event. Archived and resolved messages arrive only with the background copy, so the lookup finds nothing, and the event is deleted with no log line. RavenDB would have sent it. +- **Unresolved messages from version 6.18 or earlier can expire during the copy.** On those versions, a message that failed again after being archived or resolved kept its expiry. One deleted before the copy reads it is not copied and not counted. No log line names these messages yet. ### What the checks do not catch @@ -518,7 +530,7 @@ The dry run counts the subscription and endpoint settings merges before anything - **Abandoned:** you gave up on what a category had not copied, or on one it depends on. Final. - **Already present:** a row the copier found in SQL already, so it left it alone. - **Fault skip:** a row lost because something went wrong. -- **Harmless skip:** a row left behind because SQL would have removed it anyway. +- **Harmless skip:** a row left behind because SQL would have removed it anyway, or because it is a result of a check the source's own storage runs about itself (the RavenDB source flags its own checks, and the target skips flagged rows). - **Free abort:** going back to RavenDB at no cost, possible only while ServiceControl is still closed. - **Dry run:** a read-only rehearsal that reports what would move, what would be skipped and how long the outage would be. diff --git a/docs/migration/ravendb-to-sql-migration-system-design.md b/docs/migration/ravendb-to-sql-migration-system-design.md index f35991fdc7..d066366a1b 100644 --- a/docs/migration/ravendb-to-sql-migration-system-design.md +++ b/docs/migration/ravendb-to-sql-migration-system-design.md @@ -58,7 +58,7 @@ Dependencies point inward: each persister knows only its own store, the neutral Supporting assemblies: -- `SqlServerMigrationSqlDialect` and `PostgreSqlMigrationSqlDialect` implement `IMigrationSqlDialect` and are the only provider-specific migration code. SQL Server reads key-column collations in `Open` and inserts with `MERGE ... WITH (HOLDLOCK)`, de-duplicating the batch with `ROW_NUMBER() OVER (PARTITION BY <key> COLLATE <column collation>)`. PostgreSQL uses `INSERT ... ON CONFLICT DO NOTHING RETURNING`. Both implement `SetLicensingEndpointThroughput`. Each provider assembly carries its own `AddMigrationCheckpoints` EF migration and model snapshot, so a checkpoint schema change needs a migration in both. +- `SqlServerMigrationSqlDialect` and `PostgreSqlMigrationSqlDialect` implement `IMigrationSqlDialect` and are the only provider-specific migration code. SQL Server reads key-column collations in `Open`, builds each key column's `KeyComparer` from the collation's comparison style (case, accent, kana type and width, plus SQL Server's trailing-space padding), and inserts with `MERGE ... WITH (HOLDLOCK)`, de-duplicating the batch with `ROW_NUMBER() OVER (PARTITION BY <key> COLLATE <column collation>)`. PostgreSQL uses `INSERT ... ON CONFLICT DO NOTHING RETURNING`. Both implement `AddLicensingEndpointThroughput`, which groups each statement's rows by key and sums their counts before adding them to the stored day, so the database decides which names are one day. SQL Server groups under the key column's collation; PostgreSQL's deterministic collations make a plain `GROUP BY` the same equality. Each provider assembly carries its own `AddMigrationCheckpoints` EF migration and model snapshot, so a checkpoint schema change needs a migration in both. - `ServiceControl.DomainEvents` holds `MigrationStarted`, `MigrationCategoryHalted`, `MigrationFinished` and `MigrationSourceUnreachable`. The RavenDB persister registers no `IMigrationTarget`, `IMigrationTargetReadiness` or `IMigrationCheckpointStore`, so the guards below resolve `null` on RavenDB and return. `MigrationPairIsSupportedCheck` is what rejects RavenDB as a target. @@ -144,8 +144,8 @@ Each position in the sequence is deliberate: 9. **Checkpoint initialization.** The command seeds `NotStarted` rows for required and selected optional categories before copying. `MigrationStartup.RunRequiredCategories` also ensures required rows exist, then runs each required category under its own linked `CancellationTokenSource`, awaited with `WaitAsync` on that token. 10. **`ClosedWindowProgress`** polls the checkpoint store every 30 seconds and logs progress. When the running category has committed nothing for 30 minutes, measured from the later of `LastProgressAt` and its run start, it cancels that category's token. `WaitAsync` returns even if the call underneath ignores cancellation, the category settles `Halted` with the stall in `LastError`, and the next category runs. The abandoned call keeps running until the host refuses and the process exits. A host shutdown leaves the row `InProgress`. 11. **`CopyOrExplainWhyItStopped`** maps `MigrationCheckpointConflictException` to a message about a second instance writing the same checkpoints. -12. **`ReportWhatTheCopyLeftBehind`** logs skips per category. It offers `--migration-retry` only for an `IsFailed` row, and reports harmless skips on a `Complete` row as harmless. -13. **`RefuseIfAnyCategoryDidNotComplete`** keeps the host closed. One message lists every Failed required category with stored state, counts, skips by reason (permanent reasons flagged as not retryable through `IsPermanent`), `LastError` and both commands. A Copying required category gets a line without a command. Rollback advice comes last. +12. **`ReportWhatTheCopyLeftBehind`** logs skips per category. It offers `--migration-retry` only for an `IsFailed` row, and reports harmless skips on a `Complete` row with the explanation `HarmlessSkipExplanations` holds for each harmless reason. A test fails unless that map covers exactly the `IsBenign` reasons. +13. **`RefuseIfAnyCategoryDidNotComplete`** keeps the host closed. One message lists every Failed required category with stored state, counts, skips by reason (permanent reasons flagged as not retryable through `IsPermanent`, and harmless reasons flagged `, harmless` through `IsBenign`), `LastError` and both commands. A Copying required category gets a line without a command. Rollback advice comes last. 14. **`RecordHostOpenedOnTarget`** is registered on normal startup, never on the offline command. In `StartedAsync` it writes `Migration/HostOpenedOnTarget` the first time a host opens on a database that holds checkpoint rows, so the marker means "a host opened since the copy began", not "a host ran here once". The marker ends the free abort. ## The background copy @@ -177,18 +177,19 @@ Windowing: `MigrationEngineOptions.WindowFor` supplies the window and the reader | Read after cursor | `IMigrationCategoryReader`, usually via `RavenDocumentStream.ByPrefix`; `FailedMessagesReader` streams a query on `FailedMessageViewIndex` ordered by `id()`, after waiting for the index to be up to date, and falls back to `ByPrefix` when the index is errored, disabled or holds fewer failed messages with a copied status than the collection does; the two licensing endpoint readers use `ThroughputDocuments.Endpoints`; single-document readers load by id | No-tracking session. A cursor whose document no longer exists is refused, because RavenDB would otherwise resume after the missing id and skip rows. The cursor advances on every document seen. Both unresolved paths resume on the same document id cursor | | Fetch bodies | `MigrationBodyLoader` over `IMigrationSource.ReadBody` | Rows with a null body only, 8 concurrent, `MigrationEngine.MaxBodyReadAttempts` (3) with `MigrationEngineOptions.BodyRetryBackoff` (200 ms). `IsDefect` exceptions such as `NotSupportedException` are not retried and reach the batch catch. A missing attachment is skipped as `BodyMissing` after one attempt, a permanent fault no retry fixes, since ingestion writes an attachment for every failed message. Body skips stay off the checkpoint until the write commits | | Map documents to rows | `IMigrationCategoryWriter.Prepare`, returning `PreparedBatch` | Only the writer knows `NOT NULL` columns, key caps and what the product deletes anyway. Rows are inserted later inside the target transaction. Body-carrying writers write external bodies here, before the transaction, only for messages the target does not hold. A crash leaves orphan bodies, see [open decisions](ravendb-to-sql-migration-overview.md#what-the-checks-do-not-catch) | -| Insert and checkpoint | `EFCoreMigrationTarget.Write`: `AccountForEveryRow`, then in one transaction under the execution strategy `PreparedBatch.Insert`, `AlreadyPresentIn`, `MigrationCheckpoint.Extend`, `MigrationCheckpointExtensions.UpsertCheckpoint` | Most writers use the dialect's `InsertMissing` built on `MigrationInsert`. Store-keyed rows use `StoreKeyedInsert`, because `MigrationInsert` rejects store-generated keys. Throughput days use `SetLicensingEndpointThroughput`. Each attempt clears the change tracker. Skips and merges are logged after commit, and `Extend` adds the batch's `PreparedBatch.Merges` to `MergedCount`. See [progress and checkpoints](ravendb-to-sql-migration-overview.md#progress-and-checkpoints) | +| Insert and checkpoint | `EFCoreMigrationTarget.Write`: `AccountForEveryRow`, then in one transaction under the execution strategy `PreparedBatch.Insert`, `AlreadyPresentIn`, `MigrationCheckpoint.Extend`, `MigrationCheckpointExtensions.UpsertCheckpoint` | Most writers use the dialect's `InsertMissing` built on `MigrationInsert`. Store-keyed rows use `StoreKeyedInsert`, because `MigrationInsert` rejects store-generated keys. Throughput days use `AddLicensingEndpointThroughput`. Each attempt clears the change tracker. Skips, merges and changes are logged after commit. `MigrationWriteResult.Merged` and `Changed` carry the batch's counts of each, which the engine adds up per category for the current run in `MigrationEngine.MergedAndChangedThisRun`; the checkpoint stores neither. See [progress and checkpoints](ravendb-to-sql-migration-overview.md#progress-and-checkpoints) | | Post-commit reconciliation | Engine compares `Write`'s saved checkpoint with the counts it restated | Mismatch settles `Halted`, either kind | | Early stop | `HaltThreshold.Exceeded` on per-run counters | Fault skips (reasons not `IsBenign`, plus body skips) must exceed both `Migration/HaltThresholdPercent` and `Migration/HaltThresholdMinimum`. Counters are not persisted, so the threshold is per run | | Settle | `MigrationEngine.Settle`, after `IMigrationTarget.Count` at the end of the source | Saves the category's SQL row count on the row it settles, as `TargetCountAtSettle`. A halt saves without it, because the target may be what failed, and `--migration-abandon` reads it later. Logs before saving, because the checkpoint store shares the target database and a failed save would hide the cause | -Body placement lives in `FailedMessagesWriter`, shared by `ArchivedAndResolvedFailedMessagesWriter`. `MessageBodyClassifier.Classify` decides placement and `FailedMessageDocumentTranslator.SetBody` fills the columns: text up to `MaxBodySizeToStore` (100 KB default) inline; longer text inline prefix plus external copy; binary, invalid UTF-8 or NUL-containing text external only; empty bodies not stored. External writes go through `IBodyStoragePersistence.WriteBody`, which owns compression and the filesystem, Azure Blob or S3 backend, and complete before the transaction opens. +Failed message body placement lives in `FailedMessagesWriter`, shared by `ArchivedAndResolvedFailedMessagesWriter`. `MessageBodyClassifier.Classify` decides placement and `FailedMessageDocumentTranslator.SetBody` fills the columns: text up to `MaxBodySizeToStore` (100 KB default) inline; longer text inline prefix plus external copy; binary, invalid UTF-8 or NUL-containing text external only; empty bodies not stored. External writes go through `IBodyStoragePersistence.WriteBody`, which owns compression and the filesystem, Azure Blob or S3 backend, and complete before the transaction opens. `FailedErrorImportsWriter` places a failed error import's body by the SQL import store's own rule, inline up to `MaxBodySizeToStore` and otherwise external only, under `FailedErrorImportEntity.ExternalBodyId`, also before the transaction opens. Invariants enforced across class boundaries: - `MigrationCheckpoint.Extend` throws unless the per-`MigrationSkipReason` counts sum exactly to the skipped count. `EFCoreMigrationTarget.AccountForEveryRow` throws unless prepared rows plus skips equal the batch size, catching rows dropped during preparation. `AlreadyPresentIn` throws when a writer over-reports. Every row is copied, skipped or already present. -- `MigrationSkipReasonExtensions.IsBenign` is fixed: `PastRetention`, `BlankGroupComment`. Everything else is a fault, `Unknown` included. -- `PreparedBatch.Merges` reports rows whose key collides with another under the target's collation. `EndpointSettingsWriter` detects them with `IMigrationSqlDialect.KeyComparer` against the batch and the existing table, and `EFCoreMigrationTarget` logs one warning per merge naming both keys and the one kept. The dropped row counts as already present, and also in `MergedCount`, the part of already present that merges explain. Every writer whose key can fold two source rows into one reports them here, or `MergedCount` understates. +- `MigrationSkipReasonExtensions.IsBenign` is fixed: `PastRetention`, `BlankGroupComment`, `SourceStorageCheck`. Everything else is a fault, `Unknown` included. +- `PreparedBatch.Merges` reports rows whose key collides with another under the target's collation. Writers with a text key detect them with `KeyMerges.In` or `KeyMerges.Split`, which compare with `IMigrationSqlDialect.KeyComparer` against the batch and the existing table. `FailedErrorImportsWriter`, whose key is a GUID, reports a second import of the same message in the batch, and one the table holds with another failure time. `EFCoreMigrationTarget` logs one warning per merge naming both keys, the source row kept, or the stored row, and the source row dropped. The dropped row counts as already present, and also in `MigrationWriteResult.Merged`, so the category summary `MigrationStartup.ReportWhatTheCopyLeftBehind` logs shows merges apart from the rest of already present. Every writer whose key can fold two source rows into one reports them here, or the summary understates merges. +- `PreparedBatch.Changes` reports rows the insert adds whose stored value differs from the source's on purpose: a failed message's cut message type, dropped or blank failure groups, and failure groups one key folds together; a failed error import's missing exception or body stored empty; and a group comment on a group with no copied failed message, which the retention sweep deletes. A failed message keeping only its newest processing attempt is not a change, because every row does that by design. Writers report only rows the target does not already hold, so a re-read reports nothing twice. `EFCoreMigrationTarget` logs one warning per change after the commit, and `MigrationWriteResult.Changed` counts them. A process that stops between the commit and the log loses that batch's lines for good. `MigrationStartup.ReportWhatTheCopyLeftBehind` warns, naming both counts, when a category's merged count exceeds its already-present count, because the key comparer then disagrees with the database. `PreparedBatch.RenumberedRows` names rows the database gives new ids, and the target logs it once per copy of a category, on the batch whose checkpoint has copied nothing yet. - `TargetCountAtSettle` is read with `IMigrationTarget.Count` just before the save that settles a category at the end of the source, or that abandons it, and is never updated afterwards. For a required category it is exactly what the copy left in SQL: the host is closed, `TargetHoldsNoServiceControlDataCheck` started the target empty, and a second copier would move the row's version so the settle save is refused. That is why the count needs no shared transaction with the save. For an optional category it also includes what ServiceControl wrote while the copy ran. ## Category states @@ -224,7 +225,7 @@ Settlement at end of source, in order: - `CanBeRetried`: an `IsFailed` row, or an optional `NotStarted` or `InProgress` row with `LastError` (which covers `RecordSourceOutage` rows). Never `Blocked` or a missing row. `MigrationOperatorCommands.Retry` rejects `EventLog` before consulting it. - `CanBeAbandoned`: a required row that `IsFailed` or is `InProgress` with copied plus skipped plus already-present above 0, treating an id unknown to the registry as required; an optional category in any state except `Complete` or `Abandoned`, including no row. -Checkpoint schema: one row per category in `MigrationCheckpoints` (`migration_checkpoints` on PostgreSQL), primary key `CategoryId`. Columns: `State`, `Cursor`, `CopiedCount`, `SkippedCount`, `AlreadyPresentCount`, `MergedCount`, `SkipReasons` (JSON keyed by enum name; unknown names deserialize to `Unknown`, a fault), `StartedAt`, `LastProgressAt`, `SettledAt`, `LastError`, `StartedWindowSeconds`, `TargetCountAtSettle` (null until the category first settles) and `Version`, the optimistic concurrency token. No attempt counter and no decision record. `AddMigrationCheckpoints` creates the table in both providers and has not been released, so `TargetCountAtSettle` replaces the never-written `SourceTotal` column inside it, and `MergedCount` is added there too, rather than in a second migration. Run-level facts are `Settings` rows under `Migration/`, such as `Migration/HostOpenedOnTarget`. +Checkpoint schema: one row per category in `MigrationCheckpoints` (`migration_checkpoints` on PostgreSQL), primary key `CategoryId`. Columns: `State`, `Cursor`, `CopiedCount`, `SkippedCount`, `AlreadyPresentCount`, `SkipReasons` (JSON keyed by enum name; unknown names deserialize to `Unknown`, a fault), `StartedAt`, `LastProgressAt`, `SettledAt`, `LastError`, `StartedWindowSeconds`, `TargetCountAtSettle` (null until the category first settles) and `Version`, the optimistic concurrency token. No attempt counter and no decision record. `AddMigrationCheckpoints` creates the table in both providers and has not been released, so `TargetCountAtSettle` replaces the never-written `SourceTotal` column inside it rather than in a second migration. This build does not store merged or changed counts: `MigrationEngine.MergedAndChangedThisRun` holds them for the current run only, so the summary of a resumed category counts only that run's. Phase 13 adds them to the checkpoint so that verify can show them. Run-level facts are `Settings` rows under `Migration/`, such as `Migration/HostOpenedOnTarget`. ## Commands @@ -240,12 +241,13 @@ Every migration command except the source report builds a host with `AddPersiste | `--migration-retry <category>` | `MigrationRetryCommand`, `MigrationOperatorCommands` | No | `Retry` checks `CanBeRetried` and builds the reset row; `IMigrationTarget.ResetForRetry` persists it | | `--migration-abandon <category>` | `MigrationAbandonCommand`, `MigrationOperatorCommands` | No | Checks `CanBeAbandoned`, then writes `Abandoned` rows through the checkpoint store only | -`MigrationOperatorCommands.Retry` builds the reset row without I/O: state `NotStarted`; `Cursor`, the four counts, `SkipReasons`, `SettledAt`, `TargetCountAtSettle` and `LastError` cleared; `StartedAt` and `StartedWindowSeconds` kept. Change retry semantics there. `ResetForRetry` only persists the row it receives, in one transaction under the execution strategy. Store-keyed categories cannot be re-read idempotently, so their writers implement `IStoreKeyedCategoryWriter.DeleteCopiedRows` and the reset deletes their rows in the same transaction: +`MigrationOperatorCommands.Retry` builds the reset row without I/O: state `NotStarted`; `Cursor`, the four counts, `SkipReasons`, `SettledAt`, `TargetCountAtSettle` and `LastError` cleared; `StartedAt` and `StartedWindowSeconds` kept. Change retry semantics there. `ResetForRetry` only persists the row it receives, in one transaction under the execution strategy. A category whose rows a re-read cannot recognise as already copied cannot be re-read idempotently: store-keyed rows come back under new keys, and throughput counts are added again. Its writer implements `IStoreKeyedCategoryWriter.DeleteCopiedRows` and the reset deletes its rows in the same transaction: - `RetryOperationsWriter`: `HistoricRetryOperations`, then `UnacknowledgedRetryOperations`. - `PendingIntegrationEventsWriter`: `ExternalIntegrationDispatchRequests`. +- `LicensingThroughputWriter`: `LicensingEndpointThroughput`. -Both are required categories, so the host has not opened and the target started empty. `EventLog` is also store-keyed but copies after the host opens, when its rows are indistinguishable from live ones, so it is abandon-only. +All three are required categories, and `CanBeRetried` takes a required category only when it is Failed, which keeps the host closed, so the host has not opened and the target started empty. `ResetForRetry` also checks this itself: inside the reset transaction it reads the `Migration/HostOpenedOnTarget` setting, and when it is set it refuses a reset that would delete, naming the category, and deletes and saves nothing. A reset that deletes nothing, such as an optional category's, is not refused. `EventLog` is also store-keyed but copies after the host opens, when its rows are indistinguishable from live ones, so it is abandon-only. `--migration-abandon` first re-saves the named row unchanged, when it exists, to bump its version and fence off a concurrent copy. It then abandons dependants along `MigrationCategory.MustFollowIsDataLink` (`EndpointSettings` after `KnownEndpoints`, `LicensingThroughput` after `LicensingEndpoints`), deepest first, skipping `Complete` and `Abandoned` dependants and inserting `Abandoned` rows for dependants without one, and saves the named row last. `GroupComments` follows the unresolved failed messages for ordering only and is not cascaded. An optional category with no row gets a new `Abandoned` row. Every row it saves `Abandoned` carries `TargetCountAtSettle`, read just before the save, as a settle does. No domain event is raised, because ServiceControl is stopped. @@ -257,7 +259,7 @@ Both operator commands rely on `MigrationCheckpointConflictException` to detect - **RavenDB lower than the sum** is rows RavenDB removed after the copy read them, normal for the event log, the archive and a few unresolved messages with a stale expiry. Printed, not judged. - **Expiry can hide the same number of unread rows** in the categories it touches, so a flagged number is a floor. - **The count streams the same reader the copy used** (`RavenMigrationSource.Count`), so verify finds rows one run missed, such as a resume that skipped a range or a stream that ended early. It cannot find a reader that always misses the same rows, or a source pointed at the wrong database. -- **No live SQL count is compared with anything.** Once the host opens, normal use rewrites most categories' tables within minutes. `MergedCount` and `TargetCountAtSettle` are printed beside the sum instead. +- **No live SQL count is compared with anything.** Once the host opens, normal use rewrites most categories' tables within minutes. `TargetCountAtSettle` is printed beside the sum instead. Merged rows are printed there only once phase 13 stores their count on the checkpoint; until then they appear only in the log of the run that made them. - **An abandoned category's sum covers its last pass only.** Rows an earlier pass copied before a retry are in `TargetCountAtSettle` but not in the sum, so RavenDB minus the sum overstates what was never copied. For rows never read on a Done category, verify prints `ROWS NOT READ`, names the category under the table and tells the operator to keep RavenDB, because those rows exist only there. The exit code ignores them, and neither operator command takes a Done row. Whether to fix them is an [open decision](ravendb-to-sql-migration-overview.md#what-the-checks-do-not-catch). @@ -277,15 +279,15 @@ Every reader derives from `MigrationCategoryReader<TDocument>` and every writer | 5 | `NotificationSettings` | Required | `NotificationSettingsReader`, `NotificationSettingsWriter` | One `Settings` row via `SettingRowMapper` | | 6 | `TrialEndDate` | Required | `TrialEndDateReader`, `TrialEndDateWriter` | One `Settings` row via `SettingRowMapper` | | 7 | `RetryOperations` | Required | `RetryOperationsReader`, `RetryOperationsWriter` | Two tables: historic rows store-keyed via `StoreKeyedInsert`, unacknowledged rows via `InsertMissing`, which sizes the batch. Deletes its rows on retry | -| 8 | `LicensingEndpoints` | Required | `LicensingEndpointsReader`, `LicensingEndpointsWriter` | Throughput database. Reads only the `EndpointDocuments` collection, through `ThroughputDocuments.Endpoint`, because the throughput database gives its documents no id prefix and also holds the settings documents. With no prefix RavenDB streams in reverse etag order and ignores `startAfter`, so `ThroughputDocuments.Endpoints` reads every endpoint, sorts the ids in ordinal order and skips past the cursor itself | -| 9 | `LicensingThroughput` | Required | `LicensingThroughputReader`, `LicensingThroughputWriter` | Throughput database. Data link to `LicensingEndpoints`. The reader gives one batch per group from `ThroughputDocuments.EndpointGroups` and ignores the batch size, because one endpoint can hold a year or more of days. A group is the endpoints of one source whose names match ignoring case and trailing spaces, and its cursor is its last id in ordinal order, so a resume never splits it. The writer adds the counts of a group's endpoints whose names the key column treats as the same. `SetLicensingEndpointThroughput` splits the batch into statements and sets each day's count rather than incrementing, so re-reads are idempotent | +| 8 | `LicensingEndpoints` | Required | `LicensingEndpointsReader`, `LicensingEndpointsWriter` | Throughput database. Reads only the `EndpointDocuments` collection, through `ThroughputDocuments.Endpoint`, because the throughput database gives its documents no id prefix and also holds the settings documents. With no prefix RavenDB streams in reverse etag order and ignores `startAfter`, so `ThroughputDocuments.Endpoints` reads every endpoint, sorts the ids in ordinal order and skips past the cursor itself. Reports through `KeyMerges.In` each endpoint folded into another's key. `LicensingThroughputWriter` reports no merge for the days, because the fold is reported here | +| 9 | `LicensingThroughput` | Required | `LicensingThroughputReader`, `LicensingThroughputWriter` | Throughput database. Data link to `LicensingEndpoints`. The reader gives one batch per endpoint from `ThroughputDocuments.Endpoints` with a batch size of one, and ignores the requested batch size, because one endpoint can hold a year or more of days. Each batch's cursor is its endpoint's id. The writer keeps the last row of a day repeated for one source endpoint, compared ordinally. `AddLicensingEndpointThroughput` splits the batch into statements and adds each day's count to the stored one, summing endpoints whose names the key column treats as the same, in one batch or across batches. A re-read is not idempotent, so a resume relies on each batch committing with its checkpoint, and a retry deletes the copied rows first | | 10 | `LicensingReportMasks` | Required | `LicensingReportMasksReader`, `LicensingReportMasksWriter` | Throughput database. One `Settings` row via `SettingRowMapper` | | 11 | `LicensedEndpointDetails` | Required | `LicensedEndpointDetailsReader`, `LicensedEndpointDetailsWriter` | Throughput database. One `Settings` row via `SettingRowMapper` | -| 12 | `UnresolvedAndRetryIssuedFailedMessages` | Required | `FailedMessagesReader`, `FailedMessagesWriter` | Carries bodies. `FailedMessageStream` projection, `FailedMessageDocumentTranslator` | +| 12 | `UnresolvedAndRetryIssuedFailedMessages` | Required | `FailedMessagesReader`, `FailedMessagesWriter` | Carries bodies. `FailedMessageStream` projection, `FailedMessageDocumentTranslator`. `FailedMessageStream.ResolveRetryClaims` looks up each batch's RavenDB retry claims in one request and gives every retry-issued row the `RetryClaimHeld` metadata key. The writer writes a claim only for a retry-issued message it inserts whose flag is true, so the claims mirror RavenDB's, and fails the batch when the flag is missing or not a bool | | 13 | `PendingIntegrationEvents` | Required | `PendingIntegrationEventsReader`, `PendingIntegrationEventsWriter` | Store-keyed via `StoreKeyedInsert`. Deletes its rows on retry | -| 14 | `CustomChecks` | Required | `CustomChecksReader`, `CustomChecksWriter` | | +| 14 | `CustomChecks` | Required | `CustomChecksReader`, `CustomChecksWriter` | The RavenDB reader flags a check whose id and category match one of the RavenDB persister's own six checks with the `SourceStorageCheck` metadata key, and the writer skips a flagged row as `SourceStorageCheck` without knowing any id. A RavenDB audit instance's dirty memory result reports under another category, so it is copied | | 15 | `FailedErrorImports` | Required | `FailedErrorImportsReader`, `FailedErrorImportsWriter` | Carries bodies inline in `Read`, never calls `ReadBody` | -| 16 | `GroupComments` | Required | `GroupCommentsReader`, `GroupCommentsWriter` | Ordered after `UnresolvedAndRetryIssuedFailedMessages`, not a data link. Skips blanks as `BlankGroupComment` | +| 16 | `GroupComments` | Required | `GroupCommentsReader`, `GroupCommentsWriter` | Ordered after `UnresolvedAndRetryIssuedFailedMessages`, not a data link. Skips blanks as `BlankGroupComment`. Reports group ids that differ only in trailing spaces as merges on SQL Server. Reports each copied comment on a group with no copied failed message as a change, because the retention sweep deletes it | | 1 | `EventLog` | Optional | `EventLogReader`, `EventLogWriter` | Windowed, store-keyed, abandon-only. Skips as `PastRetention` | | 2 | `ArchivedAndResolvedFailedMessages` | Optional | `ArchivedAndResolvedFailedMessagesReader`, `ArchivedAndResolvedFailedMessagesWriter` | Windowed, carries bodies, shares `FailedMessagesWriter` preparation. Skips as `PastRetention` | diff --git a/src/ServiceControl.Migration.AcceptanceTests/CustomCheckMigrationTests.cs b/src/ServiceControl.Migration.AcceptanceTests/CustomCheckMigrationTests.cs new file mode 100644 index 0000000000..920c6e2cce --- /dev/null +++ b/src/ServiceControl.Migration.AcceptanceTests/CustomCheckMigrationTests.cs @@ -0,0 +1,97 @@ +namespace ServiceControl.Migration.AcceptanceTests; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.Contracts.CustomChecks; +using ServiceControl.Operations; +using ServiceControl.Persistence; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.Infrastructure; + +[TestFixture] +[NonParallelizable] +class CustomCheckMigrationTests : MigrationAcceptanceTest +{ + static readonly EndpointDetails Endpoint = new() { Name = "Sales", Host = "MACHINE", HostId = Guid.NewGuid() }; + + static Guid IdOf(EndpointDetails endpoint, string customCheckId) => + new CustomCheckDetail { OriginatingEndpoint = endpoint, CustomCheckId = customCheckId, Category = "Ping" }.GetDeterministicId(); + + async Task<IList<CustomCheckView>> CopiedChecks() => (await CustomChecks.GetStats(new PagingInfo())).Results!; + + // One of each status, because Status.Fail is the enum's zero and a status that never deserializes would read as Fail. + [Test] + public async Task Copies_a_failing_and_a_passing_custom_check_with_their_deterministic_ids_and_reported_status() + { + await SeedSourceCustomCheck(Endpoint, "Sales.HealthCheck", "Ping", failed: true, reason: "timed out"); + await SeedSourceCustomCheck(Endpoint, "Sales.DiskCheck", "Disk", failed: false); + + var result = await CopyCategory("CustomChecks"); + + var copied = await CopiedChecks(); + var failing = copied.Single(check => check.CustomCheckId == "Sales.HealthCheck"); + var passing = copied.Single(check => check.CustomCheckId == "Sales.DiskCheck"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(2)); + Assert.That(await Target.Count(MigrationCategoryRegistry.Find(MigrationCategoryIds.CustomChecks)!), Is.EqualTo(2), "Count is what --migration-verify reads"); + Assert.That(Guid.Parse(failing.Id!), Is.EqualTo(IdOf(Endpoint, "Sales.HealthCheck")), "a live report of the same check upserts on this id, so it replaces the copied row rather than adding a second"); + Assert.That(failing.Status, Is.EqualTo(Status.Fail)); + Assert.That(failing.Category, Is.EqualTo("Ping")); + Assert.That(failing.FailureReason, Is.EqualTo("timed out")); + Assert.That(failing.ReportedAt, Is.EqualTo(SeededReportedAt)); + Assert.That((failing.OriginatingEndpoint!.Name, failing.OriginatingEndpoint.Host, failing.OriginatingEndpoint.HostId), Is.EqualTo((Endpoint.Name, Endpoint.Host, Endpoint.HostId))); + Assert.That(Guid.Parse(passing.Id!), Is.EqualTo(IdOf(Endpoint, "Sales.DiskCheck"))); + Assert.That(passing.Status, Is.EqualTo(Status.Pass), "a status read back as the enum's zero would show every copied check red"); + Assert.That(passing.Category, Is.EqualTo("Disk")); + Assert.That(passing.FailureReason, Is.Null); + } + } + + [Test] + public async Task A_check_with_no_originating_endpoint_is_skipped_as_a_required_value_missing() + { + await SeedSource(($"CustomChecks/{Guid.NewGuid()}", new { CustomCheckId = "Sales.HealthCheck", Category = "Ping", Status = (int)Status.Pass, ReportedAt = DateTime.UtcNow })); + + var result = await CopyCategory("CustomChecks"); + var checkpoint = await ReadCheckpoint("CustomChecks"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Saved.State, Is.EqualTo(MigrationCategoryState.CompleteWithErrors), "one fault skip ends a required category Failed"); + Assert.That(checkpoint.CopiedCount, Is.Zero); + Assert.That(checkpoint.SkipReasons?.GetValueOrDefault(MigrationSkipReason.RequiredValueMissing), Is.EqualTo(1), "CustomChecks.OriginatingEndpointName is NOT NULL, and a throw would stop the batch rather than count the row"); + } + } + + [Test] + public async Task Only_a_check_whose_id_and_category_match_a_RavenDB_persister_check_is_skipped_as_harmless_and_the_category_ends_done() + { + var serviceControl = new EndpointDetails { Name = "Particular.ServiceControl", Host = "MACHINE", HostId = Guid.NewGuid() }; + var audit = new EndpointDetails { Name = "Particular.ServiceControl.Audit", Host = "MACHINE", HostId = Guid.NewGuid() }; + await SeedSourceCustomCheck(serviceControl, "Error Database Index Lag", "ServiceControl Health", failed: true, reason: "index lag"); + await SeedSourceCustomCheck(audit, "Audit Database Index Lag", "ServiceControl.Audit Health", failed: true, reason: "index lag"); + await SeedSourceCustomCheck(audit, "RavenDB dirty memory", "ServiceControl.Audit Health", failed: true, reason: "dirty memory"); + await SeedSourceCustomCheck(Endpoint, "Sales.HealthCheck", "Ping", failed: true, reason: "timed out"); + await SeedSourceCustomCheck(Endpoint, "Message Ingestion Process", "Sales", failed: true, reason: "stuck"); + + var result = await CopyCategory("CustomChecks"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Saved.State, Is.EqualTo(MigrationCategoryState.Complete), "a harmless skip leaves a required category Done, so ServiceControl can open"); + Assert.That(result.Saved.SkipReasons, Is.EqualTo(new Dictionary<MigrationSkipReason, long> { [MigrationSkipReason.SourceStorageCheck] = 1 }), "only the primary's index lag check is the RavenDB persister's own"); + Assert.That((await CopiedChecks()).Select(check => (check.CustomCheckId, check.Category)), Is.EquivalentTo(new[] + { + ("Audit Database Index Lag", "ServiceControl.Audit Health"), + ("RavenDB dirty memory", "ServiceControl.Audit Health"), + ("Sales.HealthCheck", "Ping"), + ("Message Ingestion Process", "Sales") + }), "the skip matches the id and the category, so the audit instance's checks and a customer check with a colliding id keep their failing results"); + } + } +} diff --git a/src/ServiceControl.Migration.AcceptanceTests/FailedErrorImportMigrationTests.cs b/src/ServiceControl.Migration.AcceptanceTests/FailedErrorImportMigrationTests.cs new file mode 100644 index 0000000000..cef7c85604 --- /dev/null +++ b/src/ServiceControl.Migration.AcceptanceTests/FailedErrorImportMigrationTests.cs @@ -0,0 +1,162 @@ +namespace ServiceControl.Migration.AcceptanceTests; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.EntityFrameworkCore; +using NUnit.Framework; +using ServiceControl.Operations; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.Entities; + +[TestFixture] +[NonParallelizable] +class FailedErrorImportMigrationTests : MigrationAcceptanceTest +{ + [Test] + public async Task Copies_a_small_body_inline_with_the_time_RavenDB_stored_the_import_as_its_failure_time() + { + await SeedSourceFailedErrorImport("native-1", [1, 2, 3], "Deserialization failed"); + + var result = await CopyCategory(MigrationCategoryIds.FailedErrorImports); + var import = await GetImport("native-1"); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Saved.State, Is.EqualTo(MigrationCategoryState.Complete)); + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(await Target.Count(Category), Is.EqualTo(1), "Count is what --migration-verify reads"); + Assert.That(import.Body, Is.EqualTo(new byte[] { 1, 2, 3 })); + Assert.That(import.BodyStoredExternally, Is.False); + Assert.That(import.ExceptionInfo, Is.EqualTo("Deserialization failed")); + Assert.That(import.FailedAt, Is.EqualTo(await SourceLastModified("native-1")).Within(TimeSpan.FromMilliseconds(1)), "replay runs oldest first by this time, so the time of the copy would reorder a customer's stuck imports"); + Assert.That(RecordedBodies.Written, Is.Empty); + } + } + + [Test] + public async Task Migrated_bodies_replay_through_the_products_own_import_path() + { + byte[] small = [1, 2, 3]; + var large = LargeBody(); + await SeedSourceFailedErrorImport("native-3", small, "Deserialization failed"); + await SeedSourceFailedErrorImport("native-4", large, "Deserialization failed"); + + await CopyCategory(MigrationCategoryIds.FailedErrorImports); + + var external = await GetImport("native-4"); + Assert.That(external.BodyStoredExternally, Is.True, "otherwise this proves nothing about the external placement"); + Assert.That(external.Body, Is.Empty, "the product keeps no inline copy of a body it stores in body storage"); + + var replayed = new Dictionary<string, FailedTransportMessage>(); + await FailedImportStore.ProcessFailedErrorImports((message, _) => + { + replayed[message.Id] = message; + return Task.CompletedTask; + }, TestContext.CurrentContext.CancellationToken); + + using (Assert.EnterMultipleScope()) + { + Assert.That(replayed.GetValueOrDefault("native-3")?.Body, Is.EqualTo(small), "an inline body is read from the row"); + Assert.That(replayed.GetValueOrDefault("native-4")?.Body, Is.EqualTo(large), "an external body is read from body storage under the body id the SQL import store uses for that key"); + Assert.That(replayed.GetValueOrDefault("native-3")?.Headers, Is.EqualTo(HeadersFor("native-3")), "the replay ingests the message under the headers the copy stored"); + Assert.That(replayed.GetValueOrDefault("native-4")?.Headers, Is.EqualTo(HeadersFor("native-4"))); + } + } + + [Test] + public async Task A_retry_finds_every_import_already_present_and_writes_no_body_a_second_time() + { + await SeedSourceFailedErrorImport("native-2", [9], "Deserialization failed"); + await SeedSourceFailedErrorImport("native-7", LargeBody(), "Deserialization failed"); + await CopyCategory(MigrationCategoryIds.FailedErrorImports); + var bodiesWritten = RecordedBodies.Written.Count; + + await Target.ResetForRetry(RetryResetMigrationTests.ResetForRetry(await ReadCheckpoint(MigrationCategoryIds.FailedErrorImports))); + var second = await CopyCategory(MigrationCategoryIds.FailedErrorImports); + + using (Assert.EnterMultipleScope()) + { + Assert.That(bodiesWritten, Is.EqualTo(1), "the large body is the only one the first copy writes to body storage"); + Assert.That(second.AlreadyPresent, Is.EqualTo(2), "--migration-retry re-reads from the start, and the rows the first copy wrote are left alone"); + Assert.That(await Target.Count(Category), Is.EqualTo(2)); + Assert.That(RecordedBodies.Written, Has.Count.EqualTo(bodiesWritten), "a body is written only for an import the target does not hold yet"); + } + } + + [Test] + public async Task Only_the_first_import_of_a_key_in_a_batch_has_its_body_written() + { + var first = LargeBody(); + var second = LargeBody(); + await SeedSource((DocumentId("native-8-a"), ImportOf("native-8", first)), (DocumentId("native-8-b"), ImportOf("native-8", second))); + + var result = await CopyCategory(MigrationCategoryIds.FailedErrorImports); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(result.AlreadyPresent, Is.EqualTo(1), "both documents derive one key, and the insert keeps the first row of a key"); + Assert.That(RecordedBodies.Written.Select(written => written.Body), Is.EqualTo(new[] { first }), "a second body under the same id would replace the body of the row the insert kept"); + } + } + + [Test] + public async Task An_import_with_no_message_is_skipped_as_a_required_value_missing() + { + await SeedSource((DocumentId("native-5"), new FailedErrorImport { Id = "native-5", Message = null, ExceptionInfo = "Deserialization failed" })); + await SeedSourceFailedErrorImport("native-6", [6], "Deserialization failed"); + + var result = await CopyCategory(MigrationCategoryIds.FailedErrorImports); + Assert.That(result.Saved.State, Is.EqualTo(MigrationCategoryState.CompleteWithErrors), "one fault skip ends a required category Failed"); + + var checkpoint = await ReadCheckpoint(MigrationCategoryIds.FailedErrorImports); + + using (Assert.EnterMultipleScope()) + { + Assert.That(checkpoint.CopiedCount, Is.EqualTo(1), "a skip copies the rest of the batch, where a throw would have stopped it"); + Assert.That(checkpoint.SkippedCount, Is.EqualTo(1)); + Assert.That(checkpoint.SkipReasons, Is.EqualTo(new Dictionary<MigrationSkipReason, long> { [MigrationSkipReason.RequiredValueMissing] = 1 }), + "FailedErrorImports.MessageId is NOT NULL; the source attaches an empty body, so nothing is skipped as BodyMissing or BodyUnreadable"); + } + } + + static readonly MigrationCategory Category = MigrationCategoryRegistry.Find(MigrationCategoryIds.FailedErrorImports)!; + + static string DocumentId(string nativeMessageId) => $"FailedErrorImports/{nativeMessageId}"; + + static Dictionary<string, string> HeadersFor(string nativeMessageId) => new() { ["NServiceBus.MessageId"] = nativeMessageId }; + + // One byte over the limit, because the store moves a body out only when it is larger than the limit. + byte[] LargeBody() + { + var body = new byte[EFSettings.BodyStorage.MaxBodySizeToStore + 1]; + Random.Shared.NextBytes(body); + return body; + } + + Task SeedSourceFailedErrorImport(string nativeMessageId, byte[] body, string exceptionInfo) => + SeedSource((DocumentId(nativeMessageId), ImportOf(nativeMessageId, body, exceptionInfo))); + + static FailedErrorImport ImportOf(string nativeMessageId, byte[] body, string exceptionInfo = "Deserialization failed") => new() + { + Id = nativeMessageId, + Message = new FailedTransportMessage { Id = nativeMessageId, Headers = HeadersFor(nativeMessageId), Body = body }, + ExceptionInfo = exceptionInfo + }; + + async Task<DateTime> SourceLastModified(string nativeMessageId) + { + using var session = SourceStore.OpenAsyncSession(Source.PrimaryDatabase); + + return session.Advanced.GetLastModifiedFor(await session.LoadAsync<FailedErrorImport>(DocumentId(nativeMessageId)))!.Value.ToUniversalTime(); + } + + Task<FailedErrorImportEntity> GetImport(string nativeMessageId) + { + var key = FailedErrorImport.DeriveKey(HeadersFor(nativeMessageId), nativeMessageId); + + return QueryTarget(dbContext => dbContext.FailedErrorImports.AsNoTracking().SingleAsync(row => row.UniqueMessageId == key)); + } +} diff --git a/src/ServiceControl.Migration.AcceptanceTests/GroupCommentsMigrationTests.cs b/src/ServiceControl.Migration.AcceptanceTests/GroupCommentsMigrationTests.cs new file mode 100644 index 0000000000..0d5d7751b2 --- /dev/null +++ b/src/ServiceControl.Migration.AcceptanceTests/GroupCommentsMigrationTests.cs @@ -0,0 +1,51 @@ +namespace ServiceControl.Migration.AcceptanceTests; + +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.MessageFailures; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.Infrastructure; +using ServiceControl.Persistence.Tests; + +[TestFixture] +[NonParallelizable] +class GroupCommentsMigrationTests : MigrationAcceptanceTest +{ + static readonly MigrationCategory GroupCommentsCategory = MigrationCategoryRegistry.Find(MigrationCategoryIds.GroupComments)!; + + [Test] + public async Task Copies_each_comment_onto_its_group_and_an_orphaned_one_too_and_skips_a_null_or_blank_one_as_harmless() + { + var kept = new IngestedFailure(); + var cleared = new IngestedFailure(); + var neverWritten = new IngestedFailure(); + var spaces = new IngestedFailure(); + await SeedSourceWithBodies([.. new[] { kept, cleared, neverWritten, spaces }.Select(failure => (failure.ToFailedMessage(), failure.Body, failure.ContentType))]); + await SeedSourceGroupComments( + (kept.Groups.Single().Id, "Waiting on the payments team"), + (cleared.Groups.Single().Id, ""), + (neverWritten.Groups.Single().Id, null), + (spaces.Groups.Single().Id, " "), + ("group-with-no-copied-message", "Raised with the shipping team")); + await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); + + var result = await CopyCategory(MigrationCategoryIds.GroupComments); + var comments = (await GroupsStore.GetUnresolvedGroupsByClassifier(kept.Groups.Single().Type, null, new PagingInfo())).ToDictionary(view => view.Id, view => view.Comment); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Saved.State, Is.EqualTo(MigrationCategoryState.Complete), "SQL never stores a blank comment, so leaving one behind loses nothing and must not keep ServiceControl closed"); + Assert.That(result.Copied, Is.EqualTo(2), "a comment whose group has no copied message is copied too, and left to the retention sweeper"); + Assert.That(await Target.Count(GroupCommentsCategory), Is.EqualTo(2), "--migration-verify reads this count"); + Assert.That(result.Saved.SkipReasons, Is.EqualTo(new Dictionary<MigrationSkipReason, long> { [MigrationSkipReason.BlankGroupComment] = 3 }), "a harmless skip is still counted, so status and verify can show it"); + Assert.That(comments[kept.Groups.Single().Id], Is.EqualTo("Waiting on the payments team"), "ServicePulse's group list shows the comment on the group it was written for"); + Assert.That(new[] { cleared, neverWritten, spaces }.Select(failure => comments[failure.Groups.Single().Id]), Is.All.Null, "the product shows no comment for a group whose comment was blank"); + } + } + + // The prefix is retyped because this project cannot reference the RavenDB persister that declares it. + Task SeedSourceGroupComments(params (string GroupId, string Comment)[] comments) => + SeedSource([.. comments.Select(comment => ($"GroupComment/{comment.GroupId}", (object)new GroupComment { Id = $"GroupComment/{comment.GroupId}", Comment = comment.Comment }))]); +} diff --git a/src/ServiceControl.Migration.AcceptanceTests/LicensingEndpointsMigrationTests.cs b/src/ServiceControl.Migration.AcceptanceTests/LicensingEndpointsMigrationTests.cs index 614f04792d..6089eb2ea6 100644 --- a/src/ServiceControl.Migration.AcceptanceTests/LicensingEndpointsMigrationTests.cs +++ b/src/ServiceControl.Migration.AcceptanceTests/LicensingEndpointsMigrationTests.cs @@ -89,7 +89,7 @@ public async Task Two_names_differing_only_in_a_trailing_space_never_fail_the_ba using (Assert.EnterMultipleScope()) { Assert.That(result.Copied, Is.EqualTo(keyIgnoresTrailingSpace ? 1 : 2), "SQL Server's key ignores the trailing space and keeps the first row; PostgreSQL's does not and keeps both"); - Assert.That(result.AlreadyPresent, Is.EqualTo(keyIgnoresTrailingSpace ? 1 : 0), "the row SQL Server's key treats as the first's is already present, not lost"); + Assert.That(result.AlreadyPresent, Is.EqualTo(keyIgnoresTrailingSpace ? 1 : 0), "the row SQL Server's key treats as the first's is counted as already present"); Assert.That(result.Skipped, Is.Zero); Assert.That(await Target.Count(category), Is.EqualTo(result.Copied), "every row reported as copied is a row in the table"); diff --git a/src/ServiceControl.Migration.AcceptanceTests/LicensingThroughputMigrationTests.cs b/src/ServiceControl.Migration.AcceptanceTests/LicensingThroughputMigrationTests.cs index a00b710498..b6f2048f0c 100644 --- a/src/ServiceControl.Migration.AcceptanceTests/LicensingThroughputMigrationTests.cs +++ b/src/ServiceControl.Migration.AcceptanceTests/LicensingThroughputMigrationTests.cs @@ -102,14 +102,15 @@ public async Task The_registry_order_blocks_the_counts_until_their_endpoints_are } [Test] - public async Task Running_the_category_twice_produces_the_same_count() + public async Task A_retry_copies_the_category_again_with_the_same_count() { await SeedThroughputSource(new Endpoint("Sales.Orders", ThroughputSource.Broker) { SanitizedName = "sales.orders" }); await SeedThroughputTimeSeries("Sales.Orders/Broker", new DateOnly(2026, 1, 1), 100); await CopyCategory("LicensingEndpoints"); await CopyCategory("LicensingThroughput"); - await DeleteCheckpoint("LicensingThroughput"); + await Target.ResetForRetry(RetryResetMigrationTests.ResetForRetry(await ReadCheckpoint("LicensingThroughput"))); + var afterReset = await QueryTarget(dbContext => dbContext.LicensingEndpointThroughput.CountAsync()); var second = await CopyCategory("LicensingThroughput"); var total = await QueryTarget(dbContext => dbContext.LicensingEndpointThroughput @@ -117,33 +118,26 @@ public async Task Running_the_category_twice_produces_the_same_count() .Select(row => row.MessageCount) .SingleAsync()); - Assert.That(second.Copied, Is.EqualTo(1), "the second run wrote nothing, so the count below cannot tell setting from adding"); - Assert.That(total, Is.EqualTo(100), "running the category twice must set the day's count, not add to it"); + using (Assert.EnterMultipleScope()) + { + Assert.That(afterReset, Is.Zero, "the reset deletes the copied days before the category is read again"); + Assert.That(second.Copied, Is.EqualTo(1), "the second run writes the one day once"); + Assert.That(total, Is.EqualTo(100), "the reset deletes the copied days, so copying again does not add the count a second time"); + } } [Test] - public async Task Two_names_differing_only_in_a_trailing_space_add_their_counts_into_one_day_without_failing_the_batch() + public async Task Two_names_differing_only_in_a_trailing_space_add_their_counts_into_one_day_without_failing_the_copy() { - var endpoints = MigrationCategoryRegistry.All.Single(entry => entry.Id == "LicensingEndpoints"); - var endpointBatch = new MigrationBatch( - [ - new MigrationRow("Sales.Orders /Broker", new Endpoint("Sales.Orders ", ThroughputSource.Broker) { SanitizedName = "sales.orders" }, new Dictionary<string, object>()), - new MigrationRow("Sales.Orders/Broker", new Endpoint("Sales.Orders", ThroughputSource.Broker) { SanitizedName = "sales.orders" }, new Dictionary<string, object>()) - ], - "Sales.Orders/Broker"); - await Target.Write(endpoints, endpointBatch, new MigrationCheckpoint(endpoints.Id, MigrationCategoryState.InProgress, endpointBatch.Cursor, 0, 0, null, null, null, null, null, null)); - - var category = MigrationCategoryRegistry.All.Single(entry => entry.Id == "LicensingThroughput"); var day = new DateOnly(2026, 1, 1); - var batch = new MigrationBatch( - [ - new MigrationRow("Sales.Orders /Broker|2026-01-01", new ThroughputDayRow(new EndpointIdentifier("Sales.Orders ", ThroughputSource.Broker), day, 100), new Dictionary<string, object>()), - new MigrationRow("Sales.Orders/Broker|2026-01-01", new ThroughputDayRow(new EndpointIdentifier("Sales.Orders", ThroughputSource.Broker), day, 250), new Dictionary<string, object>()) - ], - "Sales.Orders/Broker"); - var checkpointToExtend = new MigrationCheckpoint(category.Id, MigrationCategoryState.InProgress, batch.Cursor, 0, 0, null, null, null, null, null, null); + await SeedThroughputSource(new Endpoint("Sales.Orders ", ThroughputSource.Broker) { SanitizedName = "sales.orders" }); + await SeedThroughputSource(new Endpoint("Sales.Orders", ThroughputSource.Broker) { SanitizedName = "sales.orders" }); + await SeedThroughputTimeSeries("Sales.Orders /Broker", day, 100); + await SeedThroughputTimeSeries("Sales.Orders/Broker", day, 250); + await CopyCategory("LicensingEndpoints"); + var category = MigrationCategoryRegistry.All.Single(entry => entry.Id == "LicensingThroughput"); - var result = await Target.Write(category, batch, checkpointToExtend); + var result = await CopyCategory("LicensingThroughput"); var keyIgnoresTrailingSpace = (await QueryTarget(dbContext => Task.FromResult(dbContext.Database.ProviderName))).Contains("SqlServer"); var count = await QueryTarget(dbContext => dbContext.LicensingEndpointThroughput @@ -153,10 +147,10 @@ public async Task Two_names_differing_only_in_a_trailing_space_add_their_counts_ using (Assert.EnterMultipleScope()) { - Assert.That(result.Copied, Is.EqualTo(keyIgnoresTrailingSpace ? 1 : 2), "SQL Server's key ignores the trailing space, so the two days are one row; PostgreSQL's does not, so they are two"); - Assert.That(result.AlreadyPresent, Is.EqualTo(keyIgnoresTrailingSpace ? 1 : 0), "the day SQL Server's key folds into the other is already present, not lost or failed"); + Assert.That(result.Copied, Is.EqualTo(2), "the reader sends each endpoint in its own batch, and a day added to the row an earlier batch stored counts as copied"); + Assert.That(result.AlreadyPresent, Is.Zero); Assert.That(result.Skipped, Is.Zero); - Assert.That(await Target.Count(category), Is.EqualTo(result.Copied), "in a single batch, every row reported as copied is a row in the table"); + Assert.That(await Target.Count(category), Is.EqualTo(keyIgnoresTrailingSpace ? 1 : 2), "SQL Server's key ignores the trailing space, so the two days are one row; PostgreSQL's does not, so they are two"); Assert.That(count, Is.EqualTo(keyIgnoresTrailingSpace ? 350 : 250), "where the key makes the two endpoints one, their counts for the day are added, as the product's own throughput write adds them"); } } diff --git a/src/ServiceControl.Migration.AcceptanceTests/MigrationAcceptanceTest.cs b/src/ServiceControl.Migration.AcceptanceTests/MigrationAcceptanceTest.cs index f93b77210c..35590023e8 100644 --- a/src/ServiceControl.Migration.AcceptanceTests/MigrationAcceptanceTest.cs +++ b/src/ServiceControl.Migration.AcceptanceTests/MigrationAcceptanceTest.cs @@ -28,6 +28,7 @@ namespace ServiceControl.Migration.AcceptanceTests; using Particular.ServiceControl.Hosting; using ServiceBus.Management.Infrastructure.Settings; using ServiceControl.AcceptanceTesting.InfrastructureConfig; +using ServiceControl.Contracts.CustomChecks; using ServiceControl.Hosting.Commands; using ServiceControl.Infrastructure.WebApi; using ServiceControl.MessageFailures; @@ -118,6 +119,7 @@ public async Task SetUp() targetServiceCollection.AddLogging(); targetServiceCollection.AddPersistence(Settings); targetServiceCollection.AddSingleton<IBodyStoragePersistence>(RecordedBodies); + targetServiceCollection.AddSingleton(new NServiceBus.CriticalError((_, _) => Task.CompletedTask)); targetServices = targetServiceCollection.BuildServiceProvider(); Target = targetServices.GetRequiredService<IMigrationTarget>(); @@ -269,6 +271,17 @@ protected Task SeedSourceTrialEndDate(DateOnly endDate) => protected Task SeedSourceRetryHistory(RetryHistory history) => SeedSource(("RetryOperations/History", history)); + // The id is the one the RavenDB persister gives a claim, retyped because this project cannot reference that persister. + protected async Task SeedSourceRetryClaim(string uniqueMessageId) + { + var document = new { FailedMessageId = $"FailedMessages/{uniqueMessageId}", RetryBatchId = "RetryBatches/forwarded" }; + + using var session = SourceStore.OpenAsyncSession(Source.PrimaryDatabase); + await session.StoreAsync(document, $"FailedMessageRetries/{uniqueMessageId}"); + session.Advanced.GetMetadataFor(document)["@collection"] = "FailedMessageRetries"; + await session.SaveChangesAsync(); + } + protected Task SeedSourceWithBody(FailedMessage document, byte[] body, string contentType) => SeedSourceWithBodies((document, body, contentType)); @@ -305,6 +318,42 @@ protected Task SeedSourceKnownEndpoints(params string[] names) => .Select(name => new KnownEndpoint { EndpointDetails = new EndpointDetails { Name = name, HostId = Guid.NewGuid(), Host = "HOST01" } }) .Select(endpoint => ($"KnownEndpoints/{endpoint.EndpointDetails.GetDeterministicId()}", (object)endpoint))]); + // A whole second in UTC, because PostgreSQL refuses an unspecified kind and keeps only microseconds. + protected static readonly DateTime SeededReportedAt = new(2026, 9, 1, 8, 0, 0, DateTimeKind.Utc); + + // Written out by shape, because the RavenDB persister's CustomCheck type is in an assembly this project does not reference. + protected async Task SeedSourceCustomCheck( + EndpointDetails endpoint, string customCheckId, string category = "Ping", bool failed = false, string reason = null) + { + var detail = new CustomCheckDetail + { + OriginatingEndpoint = endpoint, + CustomCheckId = customCheckId, + Category = category, + ReportedAt = SeededReportedAt, + HasFailed = failed, + FailureReason = reason + }; + + var documentId = $"CustomChecks/{detail.GetDeterministicId()}"; + var document = new + { + Id = documentId, + detail.CustomCheckId, + detail.Category, + // The integer, so the document reads back the same whether or not the seeding store saves enums as integers. + Status = (int)(failed ? Status.Fail : Status.Pass), + detail.ReportedAt, + FailureReason = reason, + OriginatingEndpoint = endpoint + }; + + using var session = SourceStore.OpenAsyncSession(Source.PrimaryDatabase); + await session.StoreAsync(document, documentId); + session.Advanced.GetMetadataFor(document)["@collection"] = "CustomChecks"; + await session.SaveChangesAsync(); + } + protected Task<IMigrationSource> OpenSource(CancellationToken cancellationToken = default) => PersistenceFactory.OpenMigrationSource(Settings, cancellationToken); @@ -536,6 +585,10 @@ public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Except protected INotificationsDataStore NotificationsStore => targetServices.GetRequiredService<INotificationsDataStore>(); protected ITrialLicenseDataProvider TrialLicenseDataProvider => targetServices.GetRequiredService<ITrialLicenseDataProvider>(); protected IRetryHistoryDataStore RetryHistoryStore => targetServices.GetRequiredService<IRetryHistoryDataStore>(); + protected ExternalIntegrationRequestsDataStore ExternalIntegrationStore => targetServices.GetRequiredService<ExternalIntegrationRequestsDataStore>(); + protected ICustomChecksDataStore CustomChecks => targetServices.GetRequiredService<ICustomChecksDataStore>(); + protected IFailedErrorImportDataStore FailedImportStore => targetServices.GetRequiredService<IFailedErrorImportDataStore>(); + protected IGroupsDataStore GroupsStore => targetServices.GetRequiredService<IGroupsDataStore>(); // Built by hand, because its DI constructor needs NServiceBus settings the target services do not have. // A zero cache duration makes every read go to the table. diff --git a/src/ServiceControl.Migration.AcceptanceTests/PendingIntegrationEventsMigrationTests.cs b/src/ServiceControl.Migration.AcceptanceTests/PendingIntegrationEventsMigrationTests.cs new file mode 100644 index 0000000000..b390acfd56 --- /dev/null +++ b/src/ServiceControl.Migration.AcceptanceTests/PendingIntegrationEventsMigrationTests.cs @@ -0,0 +1,37 @@ +namespace ServiceControl.Migration.AcceptanceTests; + +using System; +using System.Linq; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.ExternalIntegrations; +using ServiceControl.Recoverability.ExternalIntegration; + +[TestFixture] +[NonParallelizable] +class PendingIntegrationEventsMigrationTests : MigrationAcceptanceTest +{ + [Test] + public async Task A_queued_event_is_copied_and_the_product_dispatches_it_as_the_type_it_was_raised_as() + { + var id = $"ExternalIntegrationDispatchRequests/{Guid.NewGuid()}"; + await SeedSource((id, new ExternalIntegrationDispatchRequest { DispatchContext = new FailedMessageArchivedPublisher.DispatchContext { FailedMessageId = "FailedMessages/1" } })); + + var result = await CopyCategory("PendingIntegrationEvents"); + + object[] dispatched = []; + ExternalIntegrationStore.Subscribe((contexts, _) => + { + dispatched = contexts; + return Task.CompletedTask; + }); + await ExternalIntegrationStore.DispatchNow(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(dispatched.Single(), Is.InstanceOf<FailedMessageArchivedPublisher.DispatchContext>()); + Assert.That(((FailedMessageArchivedPublisher.DispatchContext)dispatched.Single()).FailedMessageId, Is.EqualTo("FailedMessages/1")); + } + } +} diff --git a/src/ServiceControl.Migration.AcceptanceTests/RetryResetMigrationTests.cs b/src/ServiceControl.Migration.AcceptanceTests/RetryResetMigrationTests.cs index 6d07667274..349a2266cb 100644 --- a/src/ServiceControl.Migration.AcceptanceTests/RetryResetMigrationTests.cs +++ b/src/ServiceControl.Migration.AcceptanceTests/RetryResetMigrationTests.cs @@ -3,10 +3,12 @@ namespace ServiceControl.Migration.AcceptanceTests; using System; using System.Threading.Tasks; using NUnit.Framework; +using ServiceControl.ExternalIntegrations; using ServiceControl.Persistence.DataMigration; using ServiceControl.Persistence.EFCore; using ServiceControl.Persistence.MessageRedirects; using ServiceControl.Recoverability; +using ServiceControl.Recoverability.ExternalIntegration; [TestFixture] [NonParallelizable] @@ -65,8 +67,27 @@ public async Task A_stale_reset_deletes_nothing() Assert.That(await Target.Count(RetryOperations), Is.EqualTo(4), "the delete shares the reset's transaction, so a refused reset leaves every copied row"); } + [Test] + public async Task A_retry_of_pending_integration_events_dispatches_each_event_once() + { + await SeedSource( + ($"ExternalIntegrationDispatchRequests/{Guid.NewGuid()}", new ExternalIntegrationDispatchRequest { DispatchContext = new FailedMessageArchivedPublisher.DispatchContext { FailedMessageId = "FailedMessages/1" } }), + ($"ExternalIntegrationDispatchRequests/{Guid.NewGuid()}", new ExternalIntegrationDispatchRequest { DispatchContext = new FailedMessageArchivedPublisher.DispatchContext { FailedMessageId = "FailedMessages/2" } })); + await CopyCategory(MigrationCategoryIds.PendingIntegrationEvents); + var copied = await Target.Count(PendingIntegrationEvents); + + await Target.ResetForRetry(ResetForRetry(await ReadCheckpoint(MigrationCategoryIds.PendingIntegrationEvents))); + await CopyCategory(MigrationCategoryIds.PendingIntegrationEvents); + + using (Assert.EnterMultipleScope()) + { + Assert.That(copied, Is.EqualTo(2), "the first copy writes both requests"); + Assert.That(await Target.Count(PendingIntegrationEvents), Is.EqualTo(copied), "each request gets a new key on every insert, so a retry that kept them would dispatch every event twice"); + } + } + // The same reset --migration-retry makes. It keeps StartedAt and leaves SourceTotal alone, because the engine does not write it. - static MigrationCheckpoint ResetForRetry(MigrationCheckpoint checkpoint) => checkpoint with + internal static MigrationCheckpoint ResetForRetry(MigrationCheckpoint checkpoint) => checkpoint with { State = MigrationCategoryState.NotStarted, Cursor = null, @@ -88,6 +109,8 @@ static RetryHistory TwoHistoricAndTwoPending() return history; } + static MigrationCategory PendingIntegrationEvents => MigrationCategoryRegistry.Find(MigrationCategoryIds.PendingIntegrationEvents)!; + static MigrationCategory RetryOperations => MigrationCategoryRegistry.Find(MigrationCategoryIds.RetryOperations)!; static readonly DateTime ChangedAt = new(2026, 3, 1, 12, 0, 0, DateTimeKind.Utc); diff --git a/src/ServiceControl.Migration.AcceptanceTests/When_the_copy_merges_or_changes_rows.cs b/src/ServiceControl.Migration.AcceptanceTests/When_the_copy_merges_or_changes_rows.cs new file mode 100644 index 0000000000..a553e8cc93 --- /dev/null +++ b/src/ServiceControl.Migration.AcceptanceTests/When_the_copy_merges_or_changes_rows.cs @@ -0,0 +1,63 @@ +namespace ServiceControl.Migration.AcceptanceTests; + +using System; +using System.Collections.Concurrent; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using NUnit.Framework; +using ServiceControl.Hosting.Commands; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.Tests; + +[TestFixture] +[NonParallelizable] +class When_the_copy_merges_or_changes_rows : MigrationAcceptanceTest +{ + [Test] + public async Task The_category_summary_counts_changed_and_merged_rows_apart_from_the_rest() + { + var retried = new IngestedFailure { MessageType = new string('m', 600) }; + await SeedSourceWithBody(retried.ToFailedMessage(numberOfAttempts: 2), retried.Body, retried.ContentType); + await SeedSourceSubscriptions( + new SubscriptionRow("MyMessages.OrderPlaced", new Version(1, 0, 0), "Billing@BILLING01", "Billing"), + new SubscriptionRow("MyMessages.OrderPlaced", new Version(2, 0, 0), "Billing@BILLING01", "Billing")); + var summaries = new SummaryLines(); + + await MigrateCommand.Run(MigrationSettings(), AllowingAnUnreleasedMigration(builder => builder.Services.AddSingleton<ILoggerProvider>(summaries))); + + using (Assert.EnterMultipleScope()) + { + Assert.That(summaries.Lines, Has.Exactly(1).EqualTo((LogLevel.Warning, $"{MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages}: 1 copied (1 changed), 0 already present, 0 merged, nothing skipped")), + "the message type was cut, which the summary has to count even though nothing was skipped, and the dropped older attempt is not a change"); + Assert.That(summaries.Lines, Has.Exactly(1).EqualTo((LogLevel.Warning, $"{MigrationCategoryIds.Subscriptions}: 1 copied (0 changed), 0 already present, 1 merged, nothing skipped")), + "the second version is counted as already present by the checkpoint, and the summary has to show it as a merge instead"); + } + } + + // Keeps the lines MigrationStartup logs, which is where the summary is written. + sealed class SummaryLines : ILoggerProvider + { + readonly ConcurrentQueue<(LogLevel Level, string Message)> lines = new(); + + public (LogLevel Level, string Message)[] Lines => [.. lines]; + + public ILogger CreateLogger(string categoryName) => + categoryName == typeof(MigrationStartup).FullName ? new SummaryLogger(lines) : NullLogger.Instance; + + public void Dispose() + { + } + + sealed class SummaryLogger(ConcurrentQueue<(LogLevel Level, string Message)> lines) : ILogger + { + public IDisposable BeginScope<TState>(TState state) where TState : notnull => null; + + public bool IsEnabled(LogLevel logLevel) => true; + + public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception exception, Func<TState, Exception, string> formatter) => + lines.Enqueue((logLevel, formatter(state, exception))); + } + } +} diff --git a/src/ServiceControl.Migration.AcceptanceTests/When_unresolved_failed_messages_are_migrated.cs b/src/ServiceControl.Migration.AcceptanceTests/When_unresolved_failed_messages_are_migrated.cs index 135c7a213d..25c51dc849 100644 --- a/src/ServiceControl.Migration.AcceptanceTests/When_unresolved_failed_messages_are_migrated.cs +++ b/src/ServiceControl.Migration.AcceptanceTests/When_unresolved_failed_messages_are_migrated.cs @@ -167,6 +167,7 @@ public async Task A_retry_by_id_after_the_copy_stages_the_unresolved_message_and var retryIssued = new IngestedFailure(); await SeedFailure(unresolved); await SeedFailure(retryIssued, FailedMessageStatus.RetryIssued); + await SeedSourceRetryClaim(retryIssued.UniqueMessageIdString); var copy = await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); @@ -187,10 +188,16 @@ public async Task A_retry_issued_message_resent_from_pending_retries_after_the_c { var retryIssued = new IngestedFailure(); await SeedFailure(retryIssued, FailedMessageStatus.RetryIssued); + await SeedSourceRetryClaim(retryIssued.UniqueMessageIdString); var copy = await CopyCategory(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); - Assert.That(copy.Copied, Is.EqualTo(1), "the test proves nothing unless the message reached the target"); + using (Assert.EnterMultipleScope()) + { + Assert.That(copy.Copied, Is.EqualTo(1), "the test proves nothing unless the message reached the target"); + Assert.That(await QueryTarget(db => db.FailedMessageRetries.CountAsync(claim => claim.UniqueMessageId == retryIssued.UniqueMessageId)), Is.EqualTo(1), + "the test proves nothing about releasing a migrated claim unless the copy wrote one"); + } // Pending Retries releases the claim, then sends a retry by id. await FailedMessageRetryDataStore.RemoveFailedMessageRetry(retryIssued.UniqueMessageIdString); diff --git a/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlMigrationSqlDialect.cs b/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlMigrationSqlDialect.cs index 6c4ec5155a..86962c5a44 100644 --- a/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlMigrationSqlDialect.cs +++ b/src/ServiceControl.Persistence.EFCore.PostgreSql/PostgreSqlMigrationSqlDialect.cs @@ -49,22 +49,28 @@ ON CONFLICT ({keyColumns}) DO NOTHING return inserted; } - public async Task<int> SetLicensingEndpointThroughput(ServiceControlDbContext dbContext, IReadOnlyList<LicensingEndpointThroughputEntity> rows, CancellationToken cancellationToken = default) + public async Task<int> AddLicensingEndpointThroughput(ServiceControlDbContext dbContext, IReadOnlyList<LicensingEndpointThroughputEntity> rows, CancellationToken cancellationToken = default) { var insert = MigrationInsert<LicensingEndpointThroughputEntity>.For(dbContext); + var columns = string.Join(", ", insert.Columns); var keyColumns = string.Join(", ", insert.KeyColumns); + var counts = insert.Columns.Except(insert.KeyColumns).ToList(); var written = 0; foreach (var chunk in rows.Chunk(RowsPerStatement(insert.Columns.Count))) { + // Grouped first, because ON CONFLICT refuses a statement that reaches one row twice. written += (await insert.Execute( dbContext, $""" - INSERT INTO {insert.Table} ({string.Join(", ", insert.Columns)}) - VALUES + INSERT INTO {insert.Table} ({columns}) + SELECT {string.Join(", ", insert.Columns.Select(column => counts.Contains(column) ? $"SUM({column})" : column))} + FROM (VALUES {ParameterRows(chunk.Length, insert.Columns.Count)} + ) AS v ({columns}) + GROUP BY {keyColumns} ON CONFLICT ({keyColumns}) - DO UPDATE SET {string.Join(", ", insert.Columns.Except(insert.KeyColumns).Select(column => $"{column} = excluded.{column}"))} + DO UPDATE SET {string.Join(", ", counts.Select(column => $"{column} = {insert.Table}.{column} + excluded.{column}"))} RETURNING {keyColumns} """, chunk, diff --git a/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerMigrationSqlDialect.cs b/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerMigrationSqlDialect.cs index 0ebab54fc0..2ce3a44741 100644 --- a/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerMigrationSqlDialect.cs +++ b/src/ServiceControl.Persistence.EFCore.SqlServer/SqlServerMigrationSqlDialect.cs @@ -1,6 +1,7 @@ namespace ServiceControl.Persistence.EFCore.SqlServer; using System.Data.Common; +using System.Globalization; using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore.Infrastructure; using Microsoft.EntityFrameworkCore.Metadata; @@ -10,14 +11,12 @@ namespace ServiceControl.Persistence.EFCore.SqlServer; using ServiceControl.Persistence.EFCore.Infrastructure; /// <summary> -/// The migration's SQL Server statements. Two keys that differ only in case or trailing spaces can be one row here, because a +/// The migration's SQL Server statements. Two keys that differ only in case, accents, kana type, width or trailing spaces can be one row here, because a /// column's collation decides what counts as equal, so the statements compare keys under the collation the /// schema gives them and the batch is de-duplicated the same way before it is offered to the table. /// </summary> class SqlServerMigrationSqlDialect : SqlServerDialect, IMigrationSqlDialect { - const int IgnoreCaseStyle = 1; - OpenedKeys? OpenedState { get; set; } public async Task Open(ServiceControlDbContext dbContext, CancellationToken cancellationToken = default) @@ -55,7 +54,7 @@ FROM sys.columns AS c ?? throw new InvalidOperationException($"The migration target found no collation for {tableName}.{column}, so the SQL Server schema is missing or not current. Run ServiceControl with --setup against this database first."); collations[(table, sql.DelimitIdentifier(column))] = collation.Name; - comparers[(entityType.ClrType, property.Name)] = new PaddedKeyComparer((collation.ComparisonStyle & IgnoreCaseStyle) == 0 ? StringComparer.Ordinal : StringComparer.OrdinalIgnoreCase); + comparers[(entityType.ClrType, property.Name)] = new PaddedKeyComparer(ComparerFor(collation.ComparisonStyle)); } } @@ -114,12 +113,13 @@ WHEN NOT MATCHED THEN INSERT ({columns}) VALUES ({string.Join(", ", insert.Colum return inserted; } - public async Task<int> SetLicensingEndpointThroughput(ServiceControlDbContext dbContext, IReadOnlyList<LicensingEndpointThroughputEntity> rows, CancellationToken cancellationToken = default) + public async Task<int> AddLicensingEndpointThroughput(ServiceControlDbContext dbContext, IReadOnlyList<LicensingEndpointThroughputEntity> rows, CancellationToken cancellationToken = default) { var insert = MigrationInsert<LicensingEndpointThroughputEntity>.For(dbContext); var columns = string.Join(", ", insert.Columns); - var partitionBy = string.Join(", ", insert.KeyColumns.Select(column => - Keys.Collations.TryGetValue((insert.Table, column), out var collation) ? $"{column} COLLATE {collation}" : column)); + var counts = insert.Columns.Except(insert.KeyColumns).ToList(); + var collatedKeys = insert.KeyColumns.ToDictionary(column => column, column => + Keys.Collations.TryGetValue((insert.Table, column), out var collation) ? $"{column} COLLATE {collation}" : column); var written = 0; foreach (var chunk in rows.Chunk(RowsPerStatement(insert.Columns.Count))) @@ -131,17 +131,14 @@ public async Task<int> SetLicensingEndpointThroughput(ServiceControlDbContext db $""" MERGE {insert.Table} WITH (HOLDLOCK) AS t USING ( - SELECT {columns} - FROM ( - SELECT {columns}, ROW_NUMBER() OVER (PARTITION BY {partitionBy} ORDER BY [MigrationOrdinal] DESC) AS [MigrationDuplicate] - FROM (VALUES - {ParameterRowsWithOrdinal(chunk.Length, insert.Columns.Count)} - ) AS v ({columns}, [MigrationOrdinal]) - ) AS deduplicated - WHERE [MigrationDuplicate] = 1 + SELECT {string.Join(", ", insert.Columns.Select(column => collatedKeys.TryGetValue(column, out var key) ? key : $"SUM({column})"))} + FROM (VALUES + {ParameterRows(chunk.Length, insert.Columns.Count)} + ) AS v ({columns}) + GROUP BY {string.Join(", ", collatedKeys.Values)} ) AS s ({columns}) ON {string.Join(" AND ", insert.KeyColumns.Select(column => $"t.{column} = s.{column}"))} - WHEN MATCHED THEN UPDATE SET {string.Join(", ", insert.Columns.Except(insert.KeyColumns).Select(column => $"{column} = s.{column}"))} + WHEN MATCHED THEN UPDATE SET {string.Join(", ", counts.Select(column => $"{column} = t.{column} + s.{column}"))} WHEN NOT MATCHED THEN INSERT ({columns}) VALUES ({string.Join(", ", insert.Columns.Select(column => $"s.{column}"))}) OUTPUT {string.Join(", ", insert.KeyColumns.Select(column => $"s.{column}"))}; """, @@ -163,6 +160,29 @@ static void DeclareUnboundedTextAsMax(IProperty property, DbParameter parameter) } } + // COLLATIONPROPERTY's ComparisonStyle is 0 for a binary collation and for one that ignores nothing, and otherwise sets one bit for each thing the collation ignores. + static StringComparer ComparerFor(int comparisonStyle) + { + if (comparisonStyle == 0) + { + return StringComparer.Ordinal; + } + + var options = ComparisonStyleOptions + .Where(style => (comparisonStyle & style.Bit) != 0) + .Aggregate(CompareOptions.None, (combined, style) => combined | style.Option); + + return CultureInfo.InvariantCulture.CompareInfo.GetStringComparer(options); + } + + static readonly (int Bit, CompareOptions Option)[] ComparisonStyleOptions = + [ + (1, CompareOptions.IgnoreCase), + (2, CompareOptions.IgnoreNonSpace), + (0x10000, CompareOptions.IgnoreKanaType), + (0x20000, CompareOptions.IgnoreWidth) + ]; + OpenedKeys Keys => OpenedState ?? throw new InvalidOperationException($"The SQL Server migration dialect is not open. Call {nameof(Open)} first."); sealed record OpenedKeys( diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/CustomCheckRowMapper.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/CustomCheckRowMapper.cs new file mode 100644 index 0000000000..ae192efe04 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/CustomCheckRowMapper.cs @@ -0,0 +1,21 @@ +namespace ServiceControl.Persistence.EFCore.DataMigration; + +using ServiceControl.Contracts.CustomChecks; +using ServiceControl.Persistence.EFCore.Entities; + +static class CustomCheckRowMapper +{ + public static CustomCheckEntity Map(CustomCheckDetail detail) => new() + { + // The key a live report of the same check upserts on, so the first report after the host opens replaces this row rather than adding a second. + Id = detail.GetDeterministicId(), + CustomCheckId = detail.CustomCheckId, + Category = detail.Category, + Status = detail.HasFailed ? Status.Fail : Status.Pass, + ReportedAt = detail.ReportedAt, + FailureReason = detail.FailureReason, + OriginatingEndpointName = detail.OriginatingEndpoint.Name, + OriginatingEndpointHostId = detail.OriginatingEndpoint.HostId, + OriginatingEndpointHost = detail.OriginatingEndpoint.Host + }; +} diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/EFCoreMigrationTarget.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/EFCoreMigrationTarget.cs index 066d6e8a2c..6d8297c073 100644 --- a/src/ServiceControl.Persistence.EFCore/DataMigration/EFCoreMigrationTarget.cs +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/EFCoreMigrationTarget.cs @@ -35,7 +35,11 @@ sealed class EFCoreMigrationTarget( new SubscriptionsWriter(migrationDialect), new NotificationSettingsWriter(migrationDialect), new TrialEndDateWriter(migrationDialect), - new RetryOperationsWriter(migrationDialect) + new RetryOperationsWriter(migrationDialect), + new PendingIntegrationEventsWriter(migrationDialect), + new CustomChecksWriter(migrationDialect), + new FailedErrorImportsWriter(migrationDialect, storagePersistence, settings), + new GroupCommentsWriter(migrationDialect) }.ToFrozenDictionary(writer => writer.CategoryId, StringComparer.Ordinal); public Task Open(CancellationToken cancellationToken = default) => @@ -84,13 +88,26 @@ public Task<MigrationWriteResult> Write( logger.LogWarning("Merged {SourceId} in category {CategoryId} into a row with the same key: {Detail}", sourceId, category.Id, detail); } + foreach (var (sourceId, detail) in prepared.Changes) + { + logger.LogWarning("Changed {SourceId} in category {CategoryId} on the way into SQL: {Detail}", sourceId, category.Id, detail); + } + + // Once per copy of the category, because the line describes every batch alike and a retry's reset clears the count. + if (prepared.RenumberedRows is { } renumbered && checkpointToExtend.CopiedCount == 0) + { + logger.LogInformation("Category {CategoryId}: the database gives the copied {RenumberedRows} new ids. Nothing in SQL refers to the ids they had in the source.", category.Id, renumbered); + } + return new MigrationWriteResult( saved, copied, prepared.Skips.Count, [.. prepared.Skips.Select(skip => skip.SourceId)], AlreadyPresent: alreadyPresent, - SkipReasons: skipReasons); + SkipReasons: skipReasons, + Merged: prepared.Merges.Count, + Changed: prepared.Changes.Count); }, cancellationToken); // Checked before the insert, because the subtraction below can only catch a writer that over-reports. @@ -129,6 +146,12 @@ public Task<MigrationCheckpoint> ResetForRetry(MigrationCheckpoint reset, Cancel await using var transaction = await dbContext.Database.BeginTransactionAsync(token); + // Only a reset that deletes is refused, because an optional category is retried after ServiceControl opens and keeps its rows. + if (storeKeyed is not null && await dbContext.GetSetting<DateTime?>(SettingKeys.MigrationHostOpenedOnTarget, token) is not null) + { + throw new InvalidOperationException($"Category {reset.CategoryId} cannot be retried, because ServiceControl has opened on this database since the copy and its tables may now hold live rows. A retry deletes every row in them, so it is only possible before ServiceControl first opens on this database."); + } + var deletedRows = storeKeyed is null ? 0 : await storeKeyed.DeleteCopiedRows(dbContext, token); var stored = await dbContext.UpsertCheckpoint(reset, token); await transaction.CommitAsync(token); @@ -142,7 +165,7 @@ public Task<MigrationCheckpoint> ResetForRetry(MigrationCheckpoint reset, Cancel } else { - logger.LogInformation("Reset category {CategoryId} for a retry and deleted the {DeletedRowCount} rows it copied, because a re-read would add them again under new keys", reset.CategoryId, deleted); + logger.LogInformation("Reset category {CategoryId} for a retry and deleted the {DeletedRowCount} rows it copied, because a re-read would add them a second time", reset.CategoryId, deleted); } return saved; diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/FailedErrorImportRowMapper.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/FailedErrorImportRowMapper.cs new file mode 100644 index 0000000000..942dc4ea71 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/FailedErrorImportRowMapper.cs @@ -0,0 +1,25 @@ +namespace ServiceControl.Persistence.EFCore.DataMigration; + +using ServiceControl.Operations; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Infrastructure; + +static class FailedErrorImportRowMapper +{ + public static FailedErrorImportEntity Map(FailedErrorImport import, DateTime failedAt, bool bodyStoredExternally) + { + var message = import.Message!; + + return new FailedErrorImportEntity + { + // The key a live import of the same message derives, so a later failure of that message on SQL updates this row rather than adding a second. + UniqueMessageId = FailedErrorImport.DeriveKey(message.Headers, message.Id), + FailedAt = failedAt, + MessageId = message.Id, + HeadersJson = MessageHeaders.Write(message.Headers), + Body = bodyStoredExternally ? [] : message.Body ?? [], + BodyStoredExternally = bodyStoredExternally, + ExceptionInfo = import.ExceptionInfo ?? string.Empty + }; + } +} diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/FailedMessageDocumentTranslation.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/FailedMessageDocumentTranslation.cs index b7360c1725..f23837b53e 100644 --- a/src/ServiceControl.Persistence.EFCore/DataMigration/FailedMessageDocumentTranslation.cs +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/FailedMessageDocumentTranslation.cs @@ -14,8 +14,9 @@ abstract record FailedMessageDocumentTranslation /// <summary> /// The document translates to a failed message row and the group rows keyed by it. + /// Changes has one sentence for each way the row differs from the document on purpose, and is empty when it differs in none. /// </summary> - public sealed record Translated(FailedMessageEntity Row, IReadOnlyList<FailedMessageGroupEntity> Groups) : FailedMessageDocumentTranslation; + public sealed record Translated(FailedMessageEntity Row, IReadOnlyList<FailedMessageGroupEntity> Groups, IReadOnlyList<string> Changes) : FailedMessageDocumentTranslation; /// <summary> /// The document cannot become a row, for the reason given, and <paramref name="Detail" /> says which document and why. diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/FailedMessageDocumentTranslator.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/FailedMessageDocumentTranslator.cs index a5f241096e..24d2b6435c 100644 --- a/src/ServiceControl.Persistence.EFCore/DataMigration/FailedMessageDocumentTranslator.cs +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/FailedMessageDocumentTranslator.cs @@ -119,7 +119,33 @@ public static FailedMessageDocumentTranslation Translate(MigrationRow source) }) ]; - return new FailedMessageDocumentTranslation.Translated(row, groups); + return new FailedMessageDocumentTranslation.Translated(row, groups, ChangesMadeTo(document, metadata.GetValueOrDefault("MessageType") as string)); + } + + // One sentence per way the row differs from the document, for the warning the target logs after the commit. + static IReadOnlyList<string> ChangesMadeTo(FailedMessage document, string? messageType) + { + List<string> changes = []; + var groups = document.FailureGroups ?? []; + FailedMessage.FailureGroup?[] withoutId = [.. groups.Where(failureGroup => failureGroup is not { Id: not null })]; + FailedMessage.FailureGroup[] withoutTitleOrType = [.. groups.Where(failureGroup => failureGroup is { Id: not null } && (failureGroup.Title is null || failureGroup.Type is null))]; + + if (messageType is { Length: > ColumnLengths.ShortTextLength }) + { + changes.Add($"Cut the message type from {messageType.Length} characters to its last {ColumnLengths.ShortTextLength}, as ingestion does."); + } + + if (withoutId.Length > 0) + { + changes.Add($"Dropped the failure groups with no id, because SQL keys a group by its id: {string.Join(", ", withoutId.Select(failureGroup => failureGroup is null ? "an empty entry" : $"the group titled '{failureGroup.Title}'"))}."); + } + + if (withoutTitleOrType.Length > 0) + { + changes.Add($"Stored an empty title or type for the failure groups that had none: {string.Join(", ", withoutTitleOrType.Select(failureGroup => $"'{failureGroup.Id}'"))}."); + } + + return changes; } /// <summary> diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/GroupCommentRowMapper.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/GroupCommentRowMapper.cs new file mode 100644 index 0000000000..86fb1ebcdd --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/GroupCommentRowMapper.cs @@ -0,0 +1,16 @@ +namespace ServiceControl.Persistence.EFCore.DataMigration; + +using ServiceControl.MessageFailures; +using ServiceControl.Persistence.EFCore.Entities; + +/// <summary> +/// Turns a group comment from the source into the row the product's own edit would write. The writer skips a null or blank comment before it calls this. +/// </summary> +static class GroupCommentRowMapper +{ + public static GroupCommentEntity Map(GroupComment comment) => new() + { + GroupId = comment.Id, + Comment = comment.Comment! + }; +} diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/IStoreKeyedCategoryWriter.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/IStoreKeyedCategoryWriter.cs index 726c80ad1e..f66dd7d99f 100644 --- a/src/ServiceControl.Persistence.EFCore/DataMigration/IStoreKeyedCategoryWriter.cs +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/IStoreKeyedCategoryWriter.cs @@ -3,7 +3,8 @@ namespace ServiceControl.Persistence.EFCore.DataMigration; using DbContexts; /// <summary> -/// A category writer whose rows get their keys from the database, so a re-read would add every row again. +/// A category writer whose rows a re-read cannot recognise as already copied, so copying the category again would add every row a second time. +/// Rows whose keys the database makes are one kind, and throughput days, whose counts the copy adds to, are another. /// The target deletes those rows when it resets the category for a retry. /// The delete empties whole tables, so it is only safe for a category copied while ServiceControl is stopped. /// </summary> diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/PendingIntegrationEventRowMapper.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/PendingIntegrationEventRowMapper.cs new file mode 100644 index 0000000000..326e2af514 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/PendingIntegrationEventRowMapper.cs @@ -0,0 +1,19 @@ +namespace ServiceControl.Persistence.EFCore.DataMigration; + +using Implementation; +using ServiceControl.ExternalIntegrations; +using ServiceControl.Persistence.EFCore.Entities; + +static class PendingIntegrationEventRowMapper +{ + /// <summary> + /// Builds the row for one queued request, with its context stored the way the product stores it. Throws when the request has no context, so a caller must skip such a request before it calls this. + /// </summary> + public static ExternalIntegrationDispatchRequestEntity Map(ExternalIntegrationDispatchRequest request) + { + // The product's own serializer, so the dispatcher reads a copied row exactly as it reads one it stored. + var (typeName, json) = DispatchContextSerializer.Serialize(request.DispatchContext); + + return new ExternalIntegrationDispatchRequestEntity { DispatchContextTypeName = typeName, DispatchContextJson = json }; + } +} diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/PreparedBatch.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/PreparedBatch.cs index c4e3faebb6..2ffa313724 100644 --- a/src/ServiceControl.Persistence.EFCore/DataMigration/PreparedBatch.cs +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/PreparedBatch.cs @@ -4,15 +4,20 @@ namespace ServiceControl.Persistence.EFCore.DataMigration; using ServiceControl.Persistence.DataMigration; /// <summary> -/// One batch turned into rows, ready for the target to insert inside its own transaction. Nothing here has -/// touched the database yet. +/// A batch a writer has turned into rows, which the target inserts in the same transaction that saves the checkpoint. +/// Skips are rows not written, Merges are rows dropped because another row has the same key, and Changes are rows written with a value that differs from the source's. +/// The target logs each skip, merge and change as a warning after the commit. /// </summary> -/// <param name="Insert">Runs the insert and returns how many rows it added. The target calls it inside the transaction that also saves the checkpoint.</param> -/// <param name="PreparedRowCount">How many rows the writer built, which is more than the insert adds when the table already holds some of their keys.</param> -/// <param name="Skips">One entry per row the writer will not insert, with the reason and a detail line for the log.</param> -/// <param name="Merges">One entry per prepared row whose key the target treats as the same key as another row's, either earlier in the batch or already in the table, so the insert keeps the other row and drops this one. Each carries a detail line naming both keys and the one kept, which the target logs as a warning after the commit. The dropped row is counted as already present.</param> sealed record PreparedBatch( Func<ServiceControlDbContext, CancellationToken, Task<int>> Insert, int PreparedRowCount, IReadOnlyList<(string SourceId, MigrationSkipReason Reason, string Detail)> Skips, - IReadOnlyList<(string SourceId, string Detail)> Merges); + IReadOnlyList<(string SourceId, string Detail)> Merges) +{ + public IReadOnlyList<(string SourceId, string Detail)> Changes { get; init; } = []; + + /// <summary> + /// Names the rows the database gives new ids, or is null when it gives none. + /// </summary> + public string? RenumberedRows { get; init; } +} diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/CustomChecksWriter.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/CustomChecksWriter.cs new file mode 100644 index 0000000000..b13ab708fb --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/CustomChecksWriter.cs @@ -0,0 +1,71 @@ +namespace ServiceControl.Persistence.EFCore.DataMigration.Writers; + +using DbContexts; +using Entities; +using Infrastructure; +using Microsoft.EntityFrameworkCore; +using ServiceControl.Contracts.CustomChecks; +using ServiceControl.Persistence.DataMigration; + +/// <summary> +/// Writes the custom checks under the id the product derives from the endpoint and the check, and never replaces one the target already holds. +/// A row with SourceStorageCheck true is skipped as SourceStorageCheck, which is harmless, and a SourceStorageCheck value that is not a bool throws. +/// A row with a required text value left null is skipped as RequiredValueMissing, and one with a character the database refuses as MalformedValue. +/// </summary> +sealed class CustomChecksWriter(IMigrationSqlDialect migrationDialect) : MigrationCategoryWriter<CustomCheckDetail> +{ + public override string CategoryId => MigrationCategoryIds.CustomChecks; + + public override int BatchSize(ServiceControlDbContext dbContext) => migrationDialect.RowsPerStatementFor<CustomCheckEntity>(dbContext); + + public override Task<long> Count(ServiceControlDbContext dbContext, CancellationToken cancellationToken = default) => + dbContext.CustomChecks.LongCountAsync(cancellationToken); + + protected override Task<PreparedBatch> PrepareDocuments(ServiceControlDbContext dbContext, IReadOnlyList<(MigrationRow Row, CustomCheckDetail Document)> documents, CancellationToken cancellationToken = default) + { + var insert = MigrationInsert<CustomCheckEntity>.For(dbContext); + List<CustomCheckEntity> rows = []; + List<(string SourceId, MigrationSkipReason Reason, string Detail)> skips = []; + + foreach (var (row, detail) in documents) + { + if (IsSourceStorageCheck(row)) + { + skips.Add((row.SourceId, MigrationSkipReason.SourceStorageCheck, $"'{detail.CustomCheckId}' under '{detail.Category}' is a check the source's own storage runs about itself, which nothing reports once the instance runs on the target")); + continue; + } + + var column = detail.CustomCheckId is null ? nameof(CustomCheckEntity.CustomCheckId) + : detail.Category is null ? nameof(CustomCheckEntity.Category) + : detail.OriginatingEndpoint?.Name is null ? nameof(CustomCheckEntity.OriginatingEndpointName) + : detail.OriginatingEndpoint.Host is null ? nameof(CustomCheckEntity.OriginatingEndpointHost) + : null; + + if (column is not null) + { + skips.Add((row.SourceId, MigrationSkipReason.RequiredValueMissing, $"CustomChecks.{column} is NOT NULL and the document has no value for it")); + continue; + } + + var entity = CustomCheckRowMapper.Map(detail); + + if (UnstorableValueSkip(insert, migrationDialect, row, entity) is { } unstorable) + { + skips.Add(unstorable); + continue; + } + + rows.Add(entity); + } + + return Task.FromResult(new PreparedBatch(async (context, token) => (await migrationDialect.InsertMissing(context, rows, token)).Count, rows.Count, skips, Merges: [])); + } + + static bool IsSourceStorageCheck(MigrationRow row) => + row.Metadata.GetValueOrDefault(MigrationMetadataKeys.SourceStorageCheck) switch + { + bool flagged => flagged, + null => false, + var other => throw new InvalidOperationException($"Row '{row.SourceId}' carries a {other.GetType().FullName} as its {MigrationMetadataKeys.SourceStorageCheck} value, where the source sets a {typeof(bool).FullName}") + }; +} diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/EndpointSettingsWriter.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/EndpointSettingsWriter.cs index b13a6e8e43..ee2ef84935 100644 --- a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/EndpointSettingsWriter.cs +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/EndpointSettingsWriter.cs @@ -8,7 +8,7 @@ namespace ServiceControl.Persistence.EFCore.DataMigration.Writers; /// <summary> /// Copies every per-endpoint setting as stored. -/// Where the name column ignores case, names that differ only in case become one row, and the dropped ones are +/// Where the name column's collation ignores case, accents, kana type or width, names that differ only in those become one row, and the dropped ones are /// reported as merges. The row kept is the one whose name matches a known endpoint exactly. /// </summary> sealed class EndpointSettingsWriter(IMigrationSqlDialect migrationDialect) : MigrationCategoryWriter<EndpointSettings> diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/FailedErrorImportsWriter.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/FailedErrorImportsWriter.cs new file mode 100644 index 0000000000..6e2b01156f --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/FailedErrorImportsWriter.cs @@ -0,0 +1,146 @@ +namespace ServiceControl.Persistence.EFCore.DataMigration.Writers; + +using Abstractions; +using DbContexts; +using Entities; +using Infrastructure; +using Microsoft.EntityFrameworkCore; +using ServiceControl.Operations; +using ServiceControl.Persistence.DataMigration; + +/// <summary> +/// Writes the failed error imports under the key a live import of the same message derives, and never replaces one the target already holds. +/// A body larger than the target keeps inline goes to body storage, under the body id the SQL import store uses for that key, before the rows are inserted. +/// A second import of the same message, in the batch or already in the target from another document, is reported as a merge. +/// </summary> +sealed class FailedErrorImportsWriter(IMigrationSqlDialect migrationDialect, IBodyStoragePersistence storagePersistence, EFPersisterSettings persisterSettings) : MigrationCategoryWriter<FailedErrorImport> +{ + public override string CategoryId => MigrationCategoryIds.FailedErrorImports; + + public override int BatchSize(ServiceControlDbContext dbContext) => migrationDialect.RowsPerStatementFor<FailedErrorImportEntity>(dbContext); + + public override Task<long> Count(ServiceControlDbContext dbContext, CancellationToken cancellationToken = default) => + dbContext.FailedErrorImports.LongCountAsync(cancellationToken); + + protected override async Task<PreparedBatch> PrepareDocuments(ServiceControlDbContext dbContext, IReadOnlyList<(MigrationRow Row, FailedErrorImport Document)> documents, CancellationToken cancellationToken = default) + { + var insert = MigrationInsert<FailedErrorImportEntity>.For(dbContext); + List<(string SourceId, FailedErrorImportEntity Row, MigrationBody Body)> rows = []; + List<(string SourceId, MigrationSkipReason Reason, string Detail)> skips = []; + List<(string SourceId, Guid UniqueMessageId, string Detail)> changes = []; + + foreach (var (row, import) in documents) + { + var column = import.Message?.Id is null ? nameof(FailedErrorImportEntity.MessageId) + : import.Message.Headers is null ? nameof(FailedErrorImportEntity.HeadersJson) + : null; + + if (column is not null) + { + skips.Add((row.SourceId, MigrationSkipReason.RequiredValueMissing, $"FailedErrorImports.{column} is NOT NULL and the document has no message, or its message has no value for it")); + continue; + } + + var failedAt = row.Metadata.GetValueOrDefault(MigrationMetadataKeys.FailedAt) as DateTime? + ?? throw new InvalidOperationException($"Row '{row.SourceId}' in category {CategoryId} carries no {MigrationMetadataKeys.FailedAt}, so the import's failure time is unknown."); + var body = row.Body + ?? throw new InvalidOperationException($"Row '{row.SourceId}' in category {CategoryId} reached the writer without a body. The engine attaches a body to every row of a category that carries bodies, or skips the row."); + var entity = FailedErrorImportRowMapper.Map(import, failedAt, body.Content.Length > persisterSettings.BodyStorage.MaxBodySizeToStore); + + if (UnstorableValueSkip(insert, migrationDialect, row, entity) is { } unstorable) + { + skips.Add(unstorable); + continue; + } + + rows.Add((row.SourceId, entity, body)); + + if (EmptiedValues(import) is { } emptied) + { + changes.Add((row.SourceId, entity.UniqueMessageId, emptied)); + } + } + + var stored = await FailureTimesStoredFor(dbContext, rows, cancellationToken); + var (kept, merges) = KeptAndMerged(rows, stored); + + // Every import of one message shares one body id, so only the row the insert keeps may write its body. + await WriteBodies([.. rows.Where(row => row.Row.BodyStoredExternally && !stored.ContainsKey(row.Row.UniqueMessageId) && kept[row.Row.UniqueMessageId] == row.SourceId)], cancellationToken); + + return new PreparedBatch(async (context, token) => (await migrationDialect.InsertMissing(context, [.. rows.Select(row => row.Row)], token)).Count, rows.Count, skips, merges) + { + Changes = [.. changes.Where(change => !stored.ContainsKey(change.UniqueMessageId) && kept[change.UniqueMessageId] == change.SourceId).Select(change => (change.SourceId, change.Detail))] + }; + } + + // FailedErrorImportRowMapper stores a missing exception or body as empty, because both columns are NOT NULL. + static string? EmptiedValues(FailedErrorImport import) => + (import.ExceptionInfo, import.Message!.Body) switch + { + (null, null) => "Stored an empty exception and an empty body, because the document has neither.", + (null, _) => "Stored an empty exception, because the document has none.", + (_, null) => "Stored an empty body, because the document's message has none.", + _ => null + }; + + static async Task<Dictionary<Guid, DateTime>> FailureTimesStoredFor(ServiceControlDbContext dbContext, IReadOnlyList<(string SourceId, FailedErrorImportEntity Row, MigrationBody Body)> rows, CancellationToken cancellationToken) + { + var ids = rows.Select(row => row.Row.UniqueMessageId).Distinct().ToArray(); + + return await dbContext.FailedErrorImports + .AsNoTracking() + .Where(import => ids.Contains(import.UniqueMessageId)) + .ToDictionaryAsync(import => import.UniqueMessageId, import => import.FailedAt, cancellationToken); + } + + // A stored row wins over every row of the batch, and an earlier row over a later one, as in the insert. A stored row with the same failure time is this document's own row, which a retry reads again. + static (Dictionary<Guid, string?> Kept, List<(string SourceId, string Detail)> Merges) KeptAndMerged(IReadOnlyList<(string SourceId, FailedErrorImportEntity Row, MigrationBody Body)> rows, Dictionary<Guid, DateTime> stored) + { + var kept = stored.ToDictionary(entry => entry.Key, entry => (string?)null); + List<(string SourceId, string Detail)> merges = []; + + foreach (var (sourceId, row, _) in rows) + { + if (!kept.TryGetValue(row.UniqueMessageId, out var keeper) || (keeper is null && SameFailureTime(stored[row.UniqueMessageId], row.FailedAt))) + { + kept[row.UniqueMessageId] = sourceId; + } + else if (keeper is null) + { + merges.Add((sourceId, $"FailedErrorImports already holds an import of the same message, '{row.MessageId}', that failed at {stored[row.UniqueMessageId]:O}, while this document failed at {row.FailedAt:O}, so it kept the stored row and dropped this one")); + } + else + { + merges.Add((sourceId, $"'{keeper}' is an import of the same message, '{row.MessageId}', so FailedErrorImports kept the row from '{keeper}' and dropped the row from '{sourceId}'")); + } + } + + return (kept, merges); + } + + // PostgreSQL keeps only whole microseconds, so the stored time of the same document can differ from a re-read by less than one. + static bool SameFailureTime(DateTime stored, DateTime read) => Math.Abs((stored - read).Ticks) < TimeSpan.TicksPerMicrosecond; + + // All bodies are written before the transaction opens, so a crash leaves a body with no row rather than a row pointing at a body that was never stored. + Task WriteBodies(IReadOnlyList<(string SourceId, FailedErrorImportEntity Row, MigrationBody Body)> externalBodies, CancellationToken cancellationToken) => + Task.WhenAll(externalBodies.Select(external => WriteBody(external, cancellationToken))); + + async Task WriteBody((string SourceId, FailedErrorImportEntity Row, MigrationBody Body) external, CancellationToken cancellationToken) + { + var bodyId = FailedErrorImportEntity.ExternalBodyId(external.Row.UniqueMessageId); + + try + { + await storagePersistence.WriteBody(bodyId, external.Body.Content, external.Body.ContentType, cancellationToken); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + // The storage SDK errors do not name the body, so without this the error that stops the copy would not say which import failed. + catch (Exception ex) + { + throw new InvalidOperationException($"Writing the body of document '{external.SourceId}' in category {CategoryId} to external body storage as '{bodyId}' ({external.Body.Content.Length} bytes) failed with {ex.GetType().Name}: {ex.Message}", ex); + } + } +} diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/FailedMessagesWriter.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/FailedMessagesWriter.cs index 9fe40c125e..a1f596b00c 100644 --- a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/FailedMessagesWriter.cs +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/FailedMessagesWriter.cs @@ -9,9 +9,11 @@ namespace ServiceControl.Persistence.EFCore.DataMigration.Writers; using ServiceControl.Persistence.DataMigration; /// <summary> -/// Writes failed messages, their failure group rows and a retry claim for each retry-issued message, inserting only -/// messages the target does not already hold and never updating one it does. Each category that copies failed -/// messages derives from this and names its category id and the statuses it copies. +/// Writes failed messages, their failure group rows and a retry claim for each retry-issued message the source still +/// held one for, inserting only messages the target does not already hold and never updating one it does. The claims +/// mirror the source, so a retry by id behaves as it did before the move. A retry-issued row with no bool RetryClaimHeld +/// value fails the batch. Each category that copies failed messages derives from this and names its category id and the +/// statuses it copies. /// </summary> abstract class FailedMessagesWriterBase(IMigrationSqlDialect migrationDialect, IBodyStoragePersistence storagePersistence, EFPersisterSettings persisterSettings) : MigrationCategoryWriter<FailedMessage> { @@ -39,6 +41,9 @@ protected override async Task<PreparedBatch> PrepareDocuments(ServiceControlDbCo List<FailedMessageDocumentTranslation.Translated> translated = []; HashSet<Guid> translatedIds = []; List<(string SourceId, FailedMessageEntity Row, MigrationBody Body)> externalBodies = []; + List<(string SourceId, Guid UniqueMessageId, string Detail)> changes = []; + HashSet<Guid> heldClaims = []; + var groupIds = migrationDialect.KeyComparer(typeof(FailedMessageGroupEntity), nameof(FailedMessageGroupEntity.GroupId)); foreach (var (row, document) in documents) { @@ -61,10 +66,23 @@ protected override async Task<PreparedBatch> PrepareDocuments(ServiceControlDbCo { translated.Add(translation); - // The insert keeps only the first document with an id, so only its body may be stored under that id. - if (translatedIds.Add(translation.Row.UniqueMessageId) && storeExternally) + // The insert keeps only the first document with an id, so only its body is stored under that id, only its changes are reported and only its claim is copied. + if (translatedIds.Add(translation.Row.UniqueMessageId)) { - externalBodies.Add((row.SourceId, translation.Row, body)); + if (storeExternally) + { + externalBodies.Add((row.SourceId, translation.Row, body)); + } + + if (translation.Row.Status == FailedMessageStatus.RetryIssued && SourceHeldRetryClaim(row)) + { + heldClaims.Add(translation.Row.UniqueMessageId); + } + + if (ChangesIn(translation, groupIds) is { Count: > 0 } changed) + { + changes.Add((row.SourceId, translation.Row.UniqueMessageId, string.Join(" ", changed))); + } } } @@ -77,32 +95,47 @@ protected override async Task<PreparedBatch> PrepareDocuments(ServiceControlDbCo } } - await WriteBodiesTheTargetLacks(dbContext, externalBodies, cancellationToken); + // This only skips the bodies and changes of messages the target already holds; InsertMissing still decides which rows go in. + var present = externalBodies.Count == 0 && changes.Count == 0 + ? [] + : await MessagesTheTargetHolds(dbContext, [.. externalBodies.Select(external => external.Row.UniqueMessageId), .. changes.Select(change => change.UniqueMessageId)], cancellationToken); - return new PreparedBatch((context, token) => InsertFailedMessages(context, translated, token), translated.Count, skips, Merges: []); - } + // All bodies are written before the transaction opens, so a crash leaves a body with no row rather than a row pointing at a body that was never stored. + await Task.WhenAll(externalBodies + .Where(external => !present.Contains(external.Row.UniqueMessageId)) + .Select(external => WriteBody(external, cancellationToken))); - // All bodies are written before the transaction opens, so a crash leaves a body with no row rather than a row pointing at a body that was never stored. - async Task WriteBodiesTheTargetLacks(ServiceControlDbContext dbContext, IReadOnlyList<(string SourceId, FailedMessageEntity Row, MigrationBody Body)> externalBodies, CancellationToken cancellationToken) - { - if (externalBodies.Count == 0) + return new PreparedBatch((context, token) => InsertFailedMessages(context, translated, heldClaims, token), translated.Count, skips, Merges: []) { - return; - } - - var ids = externalBodies.Select(external => external.Row.UniqueMessageId).ToArray(); + Changes = [.. changes.Where(change => !present.Contains(change.UniqueMessageId)).Select(change => (change.SourceId, change.Detail))] + }; + } - // This only skips writing bodies for messages the target already holds; InsertMissing still decides which rows go in. - var present = (await dbContext.FailedMessages + static async Task<HashSet<Guid>> MessagesTheTargetHolds(ServiceControlDbContext dbContext, Guid[] ids, CancellationToken cancellationToken) => + (await dbContext.FailedMessages .AsNoTracking() .Where(message => ids.Contains(message.UniqueMessageId)) .Select(message => message.UniqueMessageId) .ToListAsync(cancellationToken)).ToHashSet(); - await Task.WhenAll(externalBodies - .Where(external => !present.Contains(external.Row.UniqueMessageId)) - .Select(external => WriteBody(external, cancellationToken))); - } + // The group insert keeps the first of the ids its key treats as one, so a later group with such an id loses its title and type. + static IReadOnlyList<string> ChangesIn(FailedMessageDocumentTranslation.Translated translation, IEqualityComparer<string> groupIds) => + [ + .. translation.Changes, + .. translation.Groups + .GroupBy(group => group.GroupId, groupIds) + .Where(sameKey => sameKey.Count() > 1) + .Select(sameKey => $"Kept the failure group '{sameKey.First().GroupId}' titled '{sameKey.First().Title}' and dropped {string.Join(", ", sameKey.Skip(1).Select(group => $"'{group.GroupId}' titled '{group.Title}'"))}, because SQL treats their ids as one key.") + ]; + + // Throws rather than guess, because a wrong guess either blocks a retry by id or lets one send the message twice. + static bool SourceHeldRetryClaim(MigrationRow row) => + row.Metadata.GetValueOrDefault(MigrationMetadataKeys.RetryClaimHeld) switch + { + bool held => held, + null => throw new InvalidOperationException($"Row '{row.SourceId}' is retry-issued but arrived without a {MigrationMetadataKeys.RetryClaimHeld} value, which the source resolves for every retry-issued failed message"), + var other => throw new InvalidOperationException($"Row '{row.SourceId}' carries a {other.GetType().FullName} as its {MigrationMetadataKeys.RetryClaimHeld} value, where the source sets a {typeof(bool).FullName}") + }; async Task WriteBody((string SourceId, FailedMessageEntity Row, MigrationBody Body) external, CancellationToken cancellationToken) { @@ -136,7 +169,7 @@ T NamingTheDocument<T>(MigrationRow row, Func<T> step) } } - async Task<int> InsertFailedMessages(ServiceControlDbContext dbContext, IReadOnlyList<FailedMessageDocumentTranslation.Translated> translated, CancellationToken cancellationToken) + async Task<int> InsertFailedMessages(ServiceControlDbContext dbContext, IReadOnlyList<FailedMessageDocumentTranslation.Translated> translated, IReadOnlySet<Guid> heldClaims, CancellationToken cancellationToken) { var inserted = (await migrationDialect.InsertMissing(dbContext, [.. translated.Select(translation => translation.Row)], cancellationToken)) .ToHashSet<FailedMessageEntity>(ReferenceEqualityComparer.Instance); @@ -145,9 +178,9 @@ async Task<int> InsertFailedMessages(ServiceControlDbContext dbContext, IReadOnl // A group goes in only when the message its foreign key points at went in, so a message already in the target keeps the groups it had. await migrationDialect.InsertMissing(dbContext, [.. translated.Where(translation => inserted.Contains(translation.Row)).SelectMany(translation => translation.Groups)], cancellationToken); - // A retry-issued message needs a claim, or a retry by id sends it again while its first retry is still outstanding. + // The claims mirror the source, so a retry by id stays blocked or goes through exactly as it would have before the move. await migrationDialect.InsertMissing(dbContext, [.. inserted - .Where(row => row.Status == FailedMessageStatus.RetryIssued) + .Where(row => heldClaims.Contains(row.UniqueMessageId)) .Select(row => new FailedMessageRetryEntity { UniqueMessageId = row.UniqueMessageId, RetryBatchId = MigratedRetryClaimBatchId })], cancellationToken); return inserted.Count; diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/GroupCommentsWriter.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/GroupCommentsWriter.cs new file mode 100644 index 0000000000..3179f62c16 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/GroupCommentsWriter.cs @@ -0,0 +1,79 @@ +namespace ServiceControl.Persistence.EFCore.DataMigration.Writers; + +using DbContexts; +using Entities; +using Infrastructure; +using Microsoft.EntityFrameworkCore; +using ServiceControl.MessageFailures; +using ServiceControl.Persistence.DataMigration; + +/// <summary> +/// Copies the group comments, one row per comment keyed on its group id, and never replaces a comment the target already holds. +/// A null or blank comment is skipped as BlankGroupComment, because the product keeps no row for one. +/// A comment whose group id the key column treats as another's is reported as a merge. +/// A copied comment on a group the target holds no failed message in is reported as a change, because the retention sweep deletes it once ServiceControl opens. +/// </summary> +sealed class GroupCommentsWriter(IMigrationSqlDialect migrationDialect) : MigrationCategoryWriter<GroupComment> +{ + public override string CategoryId => MigrationCategoryIds.GroupComments; + + public override int BatchSize(ServiceControlDbContext dbContext) => migrationDialect.RowsPerStatementFor<GroupCommentEntity>(dbContext); + + public override Task<long> Count(ServiceControlDbContext dbContext, CancellationToken cancellationToken = default) => + dbContext.GroupComments.LongCountAsync(cancellationToken); + + protected override async Task<PreparedBatch> PrepareDocuments(ServiceControlDbContext dbContext, IReadOnlyList<(MigrationRow Row, GroupComment Document)> documents, CancellationToken cancellationToken = default) + { + var insert = MigrationInsert<GroupCommentEntity>.For(dbContext); + var rows = new List<(string SourceId, GroupCommentEntity Row)>(documents.Count); + var skips = new List<(string SourceId, MigrationSkipReason Reason, string Detail)>(); + + foreach (var (row, comment) in documents) + { + if (string.IsNullOrWhiteSpace(comment.Comment)) + { + skips.Add((row.SourceId, MigrationSkipReason.BlankGroupComment, $"the comment on group '{comment.Id}' is blank, and the product keeps no row for a blank comment")); + continue; + } + + var entity = GroupCommentRowMapper.Map(comment); + + if (UnstorableValueSkip(insert, migrationDialect, row, entity) is { } unstorable) + { + skips.Add(unstorable); + continue; + } + + rows.Add((row.SourceId, entity)); + } + + var (merges, inserted) = await KeyMerges.Split(dbContext, migrationDialect, rows, cancellationToken: cancellationToken); + + return new PreparedBatch( + async (context, token) => (await migrationDialect.InsertMissing(context, [.. rows.Select(row => row.Row)], token)).Count, + rows.Count, + skips, + merges) + { + Changes = await CommentsTheSweepWillDelete(dbContext, inserted, cancellationToken) + }; + } + + // Group comments copy after the unresolved failed messages, so the groups the target holds now are the groups it holds when ServiceControl opens. + async Task<IReadOnlyList<(string SourceId, string Detail)>> CommentsTheSweepWillDelete(ServiceControlDbContext dbContext, IReadOnlyList<(string SourceId, GroupCommentEntity Row)> inserted, CancellationToken cancellationToken) + { + string[] groupIds = [.. inserted.Select(comment => comment.Row.GroupId).Distinct(StringComparer.Ordinal)]; + var held = (await dbContext.FailedMessageGroups + .AsNoTracking() + .Where(group => groupIds.Contains(group.GroupId)) + .Select(group => group.GroupId) + .ToListAsync(cancellationToken)) + .ToHashSet(migrationDialect.KeyComparer(typeof(FailedMessageGroupEntity), nameof(FailedMessageGroupEntity.GroupId))); + + return [.. inserted + .Where(comment => !held.Contains(comment.Row.GroupId)) + .Select(comment => (comment.SourceId, $"The group '{comment.Row.GroupId}' has no copied failed message, so the retention sweep removes this comment shortly after ServiceControl opens. " + + "Its messages are archived or resolved, are gone from the source, or were skipped by the copy. " + + $"The comment was: '{comment.Row.Comment}'"))]; + } +} diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/KeyMerges.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/KeyMerges.cs index 8116cac289..6650d9b5fb 100644 --- a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/KeyMerges.cs +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/KeyMerges.cs @@ -7,17 +7,22 @@ namespace ServiceControl.Persistence.EFCore.DataMigration.Writers; /// <summary> /// Finds the rows the insert will drop because another row, earlier in the batch or already in the table, has the same /// key under the column's collation. A row the table already holds exactly is not a merge. -/// Returns each dropped row's source id with a line naming the key that was kept. +/// Returns each dropped row's source id with a line naming both keys, the source row whose key was kept, or the stored row, and the source row dropped. +/// Split also returns the rows the insert adds, and takes a describeSource that turns a source id into the words the line uses for it, quoting it by default. /// </summary> static class KeyMerges { - public static async Task<IReadOnlyList<(string SourceId, string Detail)>> In<TEntity>(ServiceControlDbContext dbContext, IMigrationSqlDialect dialect, IReadOnlyList<(string SourceId, TEntity Row)> rows, CancellationToken cancellationToken = default) where TEntity : class + public static async Task<IReadOnlyList<(string SourceId, string Detail)>> In<TEntity>(ServiceControlDbContext dbContext, IMigrationSqlDialect dialect, IReadOnlyList<(string SourceId, TEntity Row)> rows, CancellationToken cancellationToken = default) where TEntity : class => + (await Split(dbContext, dialect, rows, cancellationToken: cancellationToken)).Merges; + + public static async Task<(IReadOnlyList<(string SourceId, string Detail)> Merges, IReadOnlyList<(string SourceId, TEntity Row)> Inserted)> Split<TEntity>(ServiceControlDbContext dbContext, IMigrationSqlDialect dialect, IReadOnlyList<(string SourceId, TEntity Row)> rows, Func<string, string>? describeSource = null, CancellationToken cancellationToken = default) where TEntity : class { + describeSource ??= sourceId => $"'{sourceId}'"; var entityType = dbContext.Model.FindEntityType(typeof(TEntity))!; var keyProperties = entityType.FindPrimaryKey()!.Properties; var queryColumn = keyProperties.FirstOrDefault(property => property.ClrType == typeof(string)) ?? throw new InvalidOperationException($"{typeof(TEntity).Name} has no text key column, so no collation can join two of its keys."); - Func<object?, object?, bool>[] columnMatches = [.. keyProperties.Select(property => property.ClrType == typeof(string) ? TextMatch(dialect.KeyComparer(typeof(TEntity), property.Name)) : object.Equals)]; + Func<object?, object?, bool>[] columnMatches = [.. keyProperties.Select(property => property.ClrType == typeof(string) ? TextMatch(dialect.KeyComparer(typeof(TEntity), property.Name)) : Equals)]; object?[] KeyOf(TEntity entity) => [.. keyProperties.Select(property => property.GetGetter().GetClrValue(entity))]; bool SameKey(object?[] first, object?[] second) => columnMatches.Select((matches, column) => matches(first[column], second[column])).All(same => same); @@ -31,8 +36,10 @@ static class KeyMerges } // A stored key wins over every row of the batch, and an earlier row over a later one, as in the insert. - var kept = stored.Select(entity => (Key: KeyOf(entity), Stored: true)).ToList(); + // SourceId is null for a stored row no row of the batch has claimed. + var kept = stored.Select(entity => (Key: KeyOf(entity), SourceId: (string?)null)).ToList(); var merges = new List<(string SourceId, string Detail)>(); + var inserted = new List<(string SourceId, TEntity Row)>(); foreach (var (sourceId, row) in rows) { @@ -41,20 +48,22 @@ static class KeyMerges if (match < 0) { - kept.Add((key, false)); + kept.Add((key, sourceId)); + inserted.Add((sourceId, row)); } - else if (kept[match].Stored && kept[match].Key.SequenceEqual(key)) + else if (kept[match].SourceId is null && kept[match].Key.SequenceEqual(key)) { // Claimed by this row, so a later row of the batch with the same key is a merge. - kept[match] = (key, false); + kept[match] = (key, sourceId); } else { - merges.Add((sourceId, $"{entityType.GetTableName()} treats {Describe(key)} and {Describe(kept[match].Key)} as one key, so it kept {Describe(kept[match].Key)} and dropped the row for {Describe(key)}")); + var keptRow = kept[match].SourceId is { } keptSourceId ? $"the row from {describeSource(keptSourceId)}" : "the row the table already held"; + merges.Add((sourceId, $"{entityType.GetTableName()} treats {Describe(key)} and {Describe(kept[match].Key)} as one key, so it kept {Describe(kept[match].Key)} and dropped the row for {Describe(key)}. It kept {keptRow} and dropped the row from {describeSource(sourceId)}.")); } } - return merges; + return (merges, inserted); } static Func<object?, object?, bool> TextMatch(IEqualityComparer<string> comparer) => (first, second) => comparer.Equals((string?)first, (string?)second); diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/LicensingEndpointsWriter.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/LicensingEndpointsWriter.cs index cac5d17b6b..fc653b0d12 100644 --- a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/LicensingEndpointsWriter.cs +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/LicensingEndpointsWriter.cs @@ -10,6 +10,7 @@ namespace ServiceControl.Persistence.EFCore.DataMigration.Writers; /// <summary> /// Writes the licensing endpoints. An endpoint the target already holds, or a second one in the same batch whose key /// the column treats as the same, is left as it is and counted as already present. +/// Each endpoint folded into another's key is also reported as a merge, naming both. /// </summary> sealed class LicensingEndpointsWriter(IMigrationSqlDialect migrationDialect) : MigrationCategoryWriter<Endpoint> { @@ -20,10 +21,10 @@ sealed class LicensingEndpointsWriter(IMigrationSqlDialect migrationDialect) : M public override Task<long> Count(ServiceControlDbContext dbContext, CancellationToken cancellationToken = default) => dbContext.LicensingEndpoints.LongCountAsync(cancellationToken); - protected override Task<PreparedBatch> PrepareDocuments(ServiceControlDbContext dbContext, IReadOnlyList<(MigrationRow Row, Endpoint Document)> documents, CancellationToken cancellationToken = default) + protected override async Task<PreparedBatch> PrepareDocuments(ServiceControlDbContext dbContext, IReadOnlyList<(MigrationRow Row, Endpoint Document)> documents, CancellationToken cancellationToken = default) { var insert = MigrationInsert<LicensingEndpointEntity>.For(dbContext); - var rows = new List<LicensingEndpointEntity>(documents.Count); + var rows = new List<(string SourceId, LicensingEndpointEntity Row)>(documents.Count); var skips = new List<(string SourceId, MigrationSkipReason Reason, string Detail)>(); foreach (var (row, endpoint) in documents.OrderBy(document => document.Row.SourceId, StringComparer.Ordinal)) @@ -42,13 +43,13 @@ protected override Task<PreparedBatch> PrepareDocuments(ServiceControlDbContext continue; } - rows.Add(entity); + rows.Add((row.SourceId, entity)); } - return Task.FromResult(new PreparedBatch( - async (context, token) => (await migrationDialect.InsertMissing(context, rows, token)).Count, + return new PreparedBatch( + async (context, token) => (await migrationDialect.InsertMissing(context, [.. rows.Select(row => row.Row)], token)).Count, rows.Count, skips, - Merges: [])); + await KeyMerges.In(dbContext, migrationDialect, rows, cancellationToken)); } } diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/LicensingThroughputWriter.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/LicensingThroughputWriter.cs index 5a03dd5353..eda0e848e0 100644 --- a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/LicensingThroughputWriter.cs +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/LicensingThroughputWriter.cs @@ -9,13 +9,12 @@ namespace ServiceControl.Persistence.EFCore.DataMigration.Writers; using ServiceControl.Persistence.DataMigration; /// <summary> -/// Writes the licensing endpoints' daily message counts, setting each day's count rather than adding to it, so a copy -/// run twice leaves the same numbers. A batch naming an endpoint the target does not hold is refused with an -/// <see cref="InvalidOperationException" /> naming that endpoint and its source. A day repeated for one endpoint keeps -/// its last row. Days of different endpoints whose names the key column treats as the same are added into one row, -/// as the product's own throughput write would add them. Either way the rows folded away are counted as already present. +/// Adds each day's message count to the stored count for that day, and the database sums endpoints whose names its key treats as one. +/// A day repeated for one source endpoint keeps its last row, and a retry deletes every copied day first. +/// It reports no merge, because the endpoints writer reports an endpoint the key folds into another. +/// Throws <see cref="InvalidOperationException" /> when a batch names an endpoint the target does not hold. /// </summary> -sealed class LicensingThroughputWriter(IMigrationSqlDialect migrationDialect) : MigrationCategoryWriter<ThroughputDayRow> +sealed class LicensingThroughputWriter(IMigrationSqlDialect migrationDialect) : MigrationCategoryWriter<ThroughputDayRow>, IStoreKeyedCategoryWriter { public override string CategoryId => MigrationCategoryIds.LicensingThroughput; @@ -24,12 +23,13 @@ sealed class LicensingThroughputWriter(IMigrationSqlDialect migrationDialect) : public override Task<long> Count(ServiceControlDbContext dbContext, CancellationToken cancellationToken = default) => dbContext.LicensingEndpointThroughput.LongCountAsync(cancellationToken); + public Task<int> DeleteCopiedRows(ServiceControlDbContext dbContext, CancellationToken cancellationToken = default) => + dbContext.LicensingEndpointThroughput.ExecuteDeleteAsync(cancellationToken); + protected override async Task<PreparedBatch> PrepareDocuments(ServiceControlDbContext dbContext, IReadOnlyList<(MigrationRow Row, ThroughputDayRow Document)> documents, CancellationToken cancellationToken = default) { var insert = MigrationInsert<LicensingEndpointThroughputEntity>.For(dbContext); - var nameComparer = migrationDialect.KeyComparer(typeof(LicensingEndpointThroughputEntity), nameof(LicensingEndpointThroughputEntity.NormalizedName)); - var mapped = new List<LicensingEndpointThroughputEntity>(documents.Count); - var endpoints = new List<EndpointIdentifier>(documents.Count); + var mapped = new List<(EndpointIdentifier Endpoint, LicensingEndpointThroughputEntity Day)>(documents.Count); var skips = new List<(string SourceId, MigrationSkipReason Reason, string Detail)>(); // Date order is the order the product's own throughput write takes row locks in. @@ -49,11 +49,10 @@ protected override async Task<PreparedBatch> PrepareDocuments(ServiceControlDbCo continue; } - mapped.Add(entity); - endpoints.Add(day.Endpoint); + mapped.Add((day.Endpoint, entity)); } - foreach (var endpoint in endpoints.Distinct()) + foreach (var endpoint in mapped.Select(day => day.Endpoint).Distinct()) { var normalizedName = LicensingDataStore.Normalize(endpoint.Name); @@ -67,19 +66,11 @@ protected override async Task<PreparedBatch> PrepareDocuments(ServiceControlDbCo List<LicensingEndpointThroughputEntity> rows = [ .. mapped - .GroupBy(day => (day.ThroughputSource, day.DateUtc)) - .SelectMany(sameDay => sameDay.GroupBy(day => day.NormalizedName, nameComparer).Select(AddEndpoints)) + .GroupBy(day => (day.Endpoint.Name, day.Endpoint.ThroughputSource, day.Day.DateUtc)) + .Select(sameEndpointDay => sameEndpointDay.Last().Day) ]; - return new PreparedBatch((context, token) => migrationDialect.SetLicensingEndpointThroughput(context, rows, token), mapped.Count, skips, Merges: []); - } - - // The sum is set, not added to the stored count, so the reader must put every endpoint that can share a key in one batch. - static LicensingEndpointThroughputEntity AddEndpoints(IEnumerable<LicensingEndpointThroughputEntity> sameKey) - { - var perEndpoint = sameKey.GroupBy(day => day.NormalizedName, StringComparer.Ordinal).Select(sameEndpoint => sameEndpoint.Last()).ToList(); - var kept = perEndpoint[0]; - kept.MessageCount = perEndpoint.Sum(day => day.MessageCount); - return kept; + // Adding is safe to resume, because the batch and its checkpoint commit in one transaction, and a replayed checkpoint save is refused. + return new PreparedBatch((context, token) => migrationDialect.AddLicensingEndpointThroughput(context, rows, token), mapped.Count, skips, Merges: []); } } diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/PendingIntegrationEventsWriter.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/PendingIntegrationEventsWriter.cs new file mode 100644 index 0000000000..2c21cda8de --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/PendingIntegrationEventsWriter.cs @@ -0,0 +1,59 @@ +namespace ServiceControl.Persistence.EFCore.DataMigration.Writers; + +using DbContexts; +using Entities; +using Infrastructure; +using Microsoft.EntityFrameworkCore; +using ServiceControl.ExternalIntegrations; +using ServiceControl.Persistence.DataMigration; + +/// <summary> +/// Writes the waiting integration events under new keys from the database, so a retry's reset empties the table before they are read again. +/// A request with no context, a type name longer than its column or a character the database refuses is skipped as RequiredValueMissing, ValueTooLong or MalformedValue. +/// The rest of the batch still copies. +/// </summary> +sealed class PendingIntegrationEventsWriter(IMigrationSqlDialect migrationDialect) : MigrationCategoryWriter<ExternalIntegrationDispatchRequest>, IStoreKeyedCategoryWriter +{ + public override string CategoryId => MigrationCategoryIds.PendingIntegrationEvents; + + public override int BatchSize(ServiceControlDbContext dbContext) => StoreKeyedInsert.BatchSize; + + public override Task<long> Count(ServiceControlDbContext dbContext, CancellationToken cancellationToken = default) => + dbContext.ExternalIntegrationDispatchRequests.LongCountAsync(cancellationToken); + + // The whole table, because the target is refused unless it held no ServiceControl data before the first copy. + public Task<int> DeleteCopiedRows(ServiceControlDbContext dbContext, CancellationToken cancellationToken = default) => + dbContext.ExternalIntegrationDispatchRequests.ExecuteDeleteAsync(cancellationToken); + + protected override Task<PreparedBatch> PrepareDocuments(ServiceControlDbContext dbContext, IReadOnlyList<(MigrationRow Row, ExternalIntegrationDispatchRequest Document)> documents, CancellationToken cancellationToken = default) + { + var textColumns = StoreKeyedInsert.TextColumns<ExternalIntegrationDispatchRequestEntity>(dbContext); + List<ExternalIntegrationDispatchRequest> requests = []; + List<(string SourceId, MigrationSkipReason Reason, string Detail)> skips = []; + + foreach (var (row, request) in documents) + { + if (request.DispatchContext is null) + { + skips.Add((row.SourceId, MigrationSkipReason.RequiredValueMissing, "ExternalIntegrationDispatchRequests.DispatchContextTypeName and DispatchContextJson are NOT NULL and the document has no context to serialize")); + } + else if (StoreKeyedInsert.UnstorableValueSkip(textColumns, migrationDialect, row, PendingIntegrationEventRowMapper.Map(request)) is { } unstorable) + { + skips.Add(unstorable); + } + else + { + requests.Add(request); + } + } + + return Task.FromResult(new PreparedBatch( + (context, token) => StoreKeyedInsert.AddAll(context, requests, PendingIntegrationEventRowMapper.Map, token), + requests.Count, + skips, + Merges: []) + { + RenumberedRows = requests.Count == 0 ? null : "pending integration events" + }); + } +} diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/RetryOperationsWriter.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/RetryOperationsWriter.cs index e84dd7d149..63440ed63d 100644 --- a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/RetryOperationsWriter.cs +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/RetryOperationsWriter.cs @@ -4,7 +4,6 @@ namespace ServiceControl.Persistence.EFCore.DataMigration.Writers; using Entities; using Infrastructure; using Microsoft.EntityFrameworkCore; -using Microsoft.EntityFrameworkCore.Metadata; using ServiceControl.Persistence.DataMigration; using ServiceControl.Recoverability; @@ -29,7 +28,7 @@ public async Task<int> DeleteCopiedRows(ServiceControlDbContext dbContext, Cance protected override async Task<PreparedBatch> PrepareDocuments(ServiceControlDbContext dbContext, IReadOnlyList<(MigrationRow Row, RetryOperationRow Document)> documents, CancellationToken cancellationToken = default) { var pendingInsert = MigrationInsert<UnacknowledgedRetryOperationEntity>.For(dbContext); - var historicText = HistoricTextColumns(dbContext); + var historicText = StoreKeyedInsert.TextColumns<HistoricRetryOperationEntity>(dbContext); List<HistoricRetryOperation> historic = []; List<(string SourceId, UnacknowledgedRetryOperationEntity Row)> unacknowledged = []; List<(string SourceId, MigrationSkipReason Reason, string Detail)> skips = []; @@ -42,7 +41,7 @@ protected override async Task<PreparedBatch> PrepareDocuments(ServiceControlDbCo { skips.Add((row.SourceId, MigrationSkipReason.RequiredValueMissing, "HistoricRetryOperations.RequestId is NOT NULL and the document has no value for it")); } - else if (UnstorableHistoricValue(historicText, row, RetryOperationRowMapper.Map(historicOperation)) is { } unstorable) + else if (StoreKeyedInsert.UnstorableValueSkip(historicText, migrationDialect, row, RetryOperationRowMapper.Map(historicOperation)) is { } unstorable) { skips.Add(unstorable); } @@ -81,44 +80,17 @@ protected override async Task<PreparedBatch> PrepareDocuments(ServiceControlDbCo { var inserted = await migrationDialect.InsertMissing(context, [.. unacknowledged.Select(pending => pending.Row)], token); - // Fresh entities per attempt, because a failed save can leave generated ids on the previous ones. - // Reversed, so on a tie in completion time the first in the document still wins, as in RavenDB. - context.HistoricRetryOperations.AddRange(Enumerable.Reverse(historic).Select(RetryOperationRowMapper.Map)); - await context.SaveChangesAsync(token); + // Reversed so the first in the document is added last and gets the highest id, which wins a tie on completion time. + // That holds only while the store hands out ids in add order. + await StoreKeyedInsert.AddAll(context, [.. Enumerable.Reverse(historic)], (HistoricRetryOperation operation) => RetryOperationRowMapper.Map(operation), token); return historic.Count + inserted.Count; }, historic.Count + unacknowledged.Count, skips, - await KeyMerges.In(dbContext, migrationDialect, unacknowledged, cancellationToken)); - } - - // Read from the model, because MigrationInsert<T>.For refuses a table whose key the store generates. - static (IProperty Property, string TableAndColumn)[] HistoricTextColumns(ServiceControlDbContext dbContext) - { - var entityType = dbContext.Model.FindEntityType(typeof(HistoricRetryOperationEntity))!; - - return [.. entityType.GetProperties().Where(property => property.ClrType == typeof(string)).Select(property => (property, $"{entityType.GetTableName()}.{property.GetColumnName()}"))]; - } - - (string SourceId, MigrationSkipReason Reason, string Detail)? UnstorableHistoricValue((IProperty Property, string TableAndColumn)[] text, MigrationRow row, HistoricRetryOperationEntity entity) - { - foreach (var (property, tableAndColumn) in text) - { - if (property.GetGetter().GetClrValue(entity) is string { Length: var units } value && property.GetMaxLength() is { } maxLength && units > maxLength && migrationDialect.TextLength(value) is var length && length > maxLength) - { - return (row.SourceId, MigrationSkipReason.ValueTooLong, $"{tableAndColumn} takes at most {maxLength} characters and the value holds {length}, on document '{row.SourceId}'"); - } - } - - foreach (var (property, tableAndColumn) in text) + await KeyMerges.In(dbContext, migrationDialect, unacknowledged, cancellationToken)) { - if (property.GetGetter().GetClrValue(entity) is string value && migrationDialect.RejectedCharacterIn(value) is { } rejected) - { - return (row.SourceId, MigrationSkipReason.MalformedValue, $"{tableAndColumn} cannot hold the character U+{(int)rejected:X4}, which the value contains, on document '{row.SourceId}'"); - } - } - - return null; + RenumberedRows = historic.Count == 0 ? null : "historic retry operations" + }; } } diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/StoreKeyedInsert.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/StoreKeyedInsert.cs new file mode 100644 index 0000000000..c21ac50312 --- /dev/null +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/StoreKeyedInsert.cs @@ -0,0 +1,66 @@ +namespace ServiceControl.Persistence.EFCore.DataMigration.Writers; + +using DbContexts; +using Infrastructure; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Metadata; +using ServiceControl.Persistence.DataMigration; + +/// <summary> +/// Writes rows whose primary key the database generates. +/// The model-driven insert refuses such rows, so this class adds them through EF in the caller's context, and the database assigns each key. +/// It also holds its own copy of the text check, because the base writer's check needs that insert. +/// </summary> +static class StoreKeyedInsert +{ + // EF splits its own statements under each provider's limits, so this sizes only how many rows one checkpoint covers. + public const int BatchSize = 500; + + /// <summary> + /// Returns every text column of <typeparamref name="TEntity" />, each with its name as "table.column" for a skip's detail. Read it once per batch and pass it to <see cref="UnstorableValueSkip{TEntity}" />. + /// </summary> + public static (IProperty Property, string TableAndColumn)[] TextColumns<TEntity>(ServiceControlDbContext dbContext) where TEntity : class + { + var entityType = dbContext.Model.FindEntityType(typeof(TEntity))!; + + return [.. entityType.GetProperties().Where(property => property.ClrType == typeof(string)).Select(property => (property, $"{entityType.GetTableName()}.{property.GetColumnName()}"))]; + } + + /// <summary> + /// Returns a skip when <paramref name="entity" />, the row mapped from the source document in <paramref name="row" />, holds a text value the database would refuse, or null when every value can be stored. + /// The reason is ValueTooLong for a value longer than its column, or MalformedValue for a value holding a character the provider cannot store. The skip carries the source id of <paramref name="row" />, and its detail names the table, the column and the document. + /// </summary> + public static (string SourceId, MigrationSkipReason Reason, string Detail)? UnstorableValueSkip<TEntity>((IProperty Property, string TableAndColumn)[] textColumns, IMigrationSqlDialect dialect, MigrationRow row, TEntity entity) where TEntity : class + { + foreach (var (property, tableAndColumn) in textColumns) + { + if (property.GetGetter().GetClrValue(entity) is string { Length: var units } value && property.GetMaxLength() is { } maxLength && units > maxLength && dialect.TextLength(value) is var length && length > maxLength) + { + return (row.SourceId, MigrationSkipReason.ValueTooLong, $"{tableAndColumn} takes at most {maxLength} characters and the value holds {length}, on document '{row.SourceId}'"); + } + } + + foreach (var (property, tableAndColumn) in textColumns) + { + if (property.GetGetter().GetClrValue(entity) is string value && dialect.RejectedCharacterIn(value) is { } rejected) + { + return (row.SourceId, MigrationSkipReason.MalformedValue, $"{tableAndColumn} cannot hold the character U+{(int)rejected:X4}, which the value contains, on document '{row.SourceId}'"); + } + } + + return null; + } + + /// <summary> + /// Builds a new entity for each source row with <paramref name="map" />, adds them all to the caller's context in the order given, and saves inside the transaction the target already holds. + /// Returns how many rows it added, which is all of them, because the save writes every row or throws. Call it inside the target's insert, so each attempt of the execution strategy builds fresh entities. + /// </summary> + public static async Task<int> AddAll<TSource, TEntity>(ServiceControlDbContext dbContext, IReadOnlyList<TSource> sources, Func<TSource, TEntity> map, CancellationToken cancellationToken = default) where TEntity : class + { + // Mapped on every call, because a failed save can leave generated ids on the entities it tried. + dbContext.Set<TEntity>().AddRange(sources.Select(map)); + await dbContext.SaveChangesAsync(cancellationToken); + + return sources.Count; + } +} diff --git a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/SubscriptionsWriter.cs b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/SubscriptionsWriter.cs index a748509255..29a57fd6c8 100644 --- a/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/SubscriptionsWriter.cs +++ b/src/ServiceControl.Persistence.EFCore/DataMigration/Writers/SubscriptionsWriter.cs @@ -46,10 +46,14 @@ protected override async Task<PreparedBatch> PrepareDocuments(ServiceControlDbCo rows.Add((row.SourceId, entity)); } + // The key leaves the version out, so a merge line names each row's version and endpoint as well as its source id. + var subscriptions = documents.ToDictionary(document => document.Row.SourceId, document => document.Document, StringComparer.Ordinal); + var (merges, _) = await KeyMerges.Split(dbContext, migrationDialect, rows, sourceId => $"'{sourceId}' (version {subscriptions[sourceId].Version}, endpoint '{subscriptions[sourceId].Endpoint}')", cancellationToken); + return new PreparedBatch( async (context, token) => (await migrationDialect.InsertMissing(context, [.. rows.Select(row => row.Row)], token)).Count, rows.Count, skips, - await KeyMerges.In(dbContext, migrationDialect, rows, cancellationToken)); + merges); } } diff --git a/src/ServiceControl.Persistence.EFCore/Infrastructure/IMigrationSqlDialect.cs b/src/ServiceControl.Persistence.EFCore/Infrastructure/IMigrationSqlDialect.cs index 951088d8cb..fefcc7ee24 100644 --- a/src/ServiceControl.Persistence.EFCore/Infrastructure/IMigrationSqlDialect.cs +++ b/src/ServiceControl.Persistence.EFCore/Infrastructure/IMigrationSqlDialect.cs @@ -4,7 +4,7 @@ namespace ServiceControl.Persistence.EFCore.Infrastructure; using ServiceControl.Persistence.EFCore.Entities; /// <summary> -/// The migration's provider-specific SQL. Statements run inside the caller's transaction and insert only what is absent. +/// The migration's provider-specific SQL. Statements run inside the caller's transaction. /// </summary> public interface IMigrationSqlDialect { @@ -42,10 +42,10 @@ public interface IMigrationSqlDialect Task<IReadOnlyList<TEntity>> InsertMissing<TEntity>(ServiceControlDbContext dbContext, IReadOnlyList<TEntity> rows, CancellationToken cancellationToken = default) where TEntity : class; /// <summary> - /// Sets each day's message count: a day the table does not hold is inserted, and a day it holds gets its count replaced, never added to. + /// Adds each day's message count to the count the table holds for that day, and inserts a day the table does not hold. + /// Rows whose keys the key columns' collation treats as one are summed in the statement, so the database decides which names are one day. /// The rows go to the database in the order given. Every row's endpoint must already be in the table, or the foreign key fails the statement. - /// Two rows with the same key in one call are not allowed on PostgreSQL, which refuses the statement. SQL Server keeps the last of them, which also covers names that differ only by a trailing space. /// Returns how many rows the statements inserted or updated. It runs on the caller's open transaction, and throws <see cref="InvalidOperationException"/> when there is none or when a returned key matches no row given. /// </summary> - Task<int> SetLicensingEndpointThroughput(ServiceControlDbContext dbContext, IReadOnlyList<LicensingEndpointThroughputEntity> rows, CancellationToken cancellationToken = default); + Task<int> AddLicensingEndpointThroughput(ServiceControlDbContext dbContext, IReadOnlyList<LicensingEndpointThroughputEntity> rows, CancellationToken cancellationToken = default); } diff --git a/src/ServiceControl.Persistence.RavenDB/DataMigration/FailedMessageStream.cs b/src/ServiceControl.Persistence.RavenDB/DataMigration/FailedMessageStream.cs index 67def01ba1..3f32f8227f 100644 --- a/src/ServiceControl.Persistence.RavenDB/DataMigration/FailedMessageStream.cs +++ b/src/ServiceControl.Persistence.RavenDB/DataMigration/FailedMessageStream.cs @@ -5,6 +5,8 @@ namespace ServiceControl.Persistence.RavenDB.DataMigration; using System.Collections.Generic; using System.Globalization; using System.Linq; +using System.Threading; +using System.Threading.Tasks; using Raven.Client; using Raven.Client.Documents.Commands; using ServiceControl.MessageFailures; @@ -48,6 +50,37 @@ class FailedMessageStream(RavenReadOnlySourceLifecycle lifecycle) }); }; + /// <summary> + /// Returns the batch with every retry-issued row carrying <see cref="MigrationMetadataKeys.RetryClaimHeld" />, true when + /// RavenDB still holds a retry claim for the message. It looks up every claim in the batch with one request, and returns a + /// batch with no retry-issued row as it is. + /// </summary> + public async Task<MigrationBatch> ResolveRetryClaims(MigrationBatch batch, CancellationToken cancellationToken = default) + { + string[] claimIds = [.. batch.Rows.Where(IsRetryIssued).Select(ClaimId)]; + + if (claimIds.Length == 0) + { + return batch; + } + + using var session = lifecycle.OpenSession(lifecycle.Settings.DatabaseName); + + var claims = await session.LoadAsync<FailedMessageRetry>(claimIds.Distinct(StringComparer.OrdinalIgnoreCase), cancellationToken); + var held = claims.Where(claim => claim.Value is not null).Select(claim => claim.Key).ToHashSet(StringComparer.OrdinalIgnoreCase); + + return batch with + { + Rows = [.. batch.Rows.Select(row => IsRetryIssued(row) + ? row with { Metadata = new Dictionary<string, object?>(row.Metadata) { [MigrationMetadataKeys.RetryClaimHeld] = held.Contains(ClaimId(row)) } } + : row)] + }; + } + + static bool IsRetryIssued(MigrationRow row) => ((FailedMessage)row.Document).Status == FailedMessageStatus.RetryIssued; + + static string ClaimId(MigrationRow row) => RetryDocumentDataStore.MakeFailedMessageRetriesDocumentId(((FailedMessage)row.Document).UniqueMessageId); + // The translator checks TimeSent, SendingEndpoint and ReceivingEndpoint by exact type, but RavenDB can hand them back as text or untyped objects. // Only the newest attempt is fixed because the translator reads no other, and a TimeSent that is not a date is left as it is for the translator to skip. internal static void RestoreMetadataTypes(FailedMessage document) @@ -98,7 +131,7 @@ static void RestoreEndpoint(Dictionary<string, object> metadata, string key, Fun static DateTime AsUtc(DateTime value) => value.Kind == DateTimeKind.Unspecified ? DateTime.SpecifyKind(value, DateTimeKind.Utc) : value.ToUniversalTime(); - static DateTime? ReadMetadataDate(StreamResult<FailedMessage> result, string key) => + internal static DateTime? ReadMetadataDate<TDocument>(StreamResult<TDocument> result, string key) => result.Metadata.TryGetValue(key, out var value) && value != null ? AsUtc(DateTime.Parse(value.ToString()!, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind)) : null; diff --git a/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenMigrationSource.cs b/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenMigrationSource.cs index 18266881e0..a77fe39612 100644 --- a/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenMigrationSource.cs +++ b/src/ServiceControl.Persistence.RavenDB/DataMigration/RavenMigrationSource.cs @@ -36,7 +36,11 @@ sealed class RavenMigrationSource(RavenReadOnlySourceLifecycle lifecycle) : IMig new SubscriptionsReader(lifecycle), new NotificationSettingsReader(lifecycle), new TrialEndDateReader(lifecycle), - new RetryOperationsReader(lifecycle) + new RetryOperationsReader(lifecycle), + new PendingIntegrationEventsReader(lifecycle), + new CustomChecksReader(lifecycle), + new FailedErrorImportsReader(lifecycle), + new GroupCommentsReader(lifecycle) }.ToFrozenDictionary(reader => reader.CategoryId, StringComparer.Ordinal); public Task Open(CancellationToken cancellationToken = default) => lifecycle.Open(cancellationToken); diff --git a/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/CustomChecksReader.cs b/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/CustomChecksReader.cs new file mode 100644 index 0000000000..979be4d817 --- /dev/null +++ b/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/CustomChecksReader.cs @@ -0,0 +1,38 @@ +#nullable enable + +namespace ServiceControl.Persistence.RavenDB.DataMigration.Readers; + +using System.Collections.Generic; +using System.Collections.ObjectModel; +using System.Threading; +using ServiceControl.Contracts.CustomChecks; +using ServiceControl.Persistence.DataMigration; + +/// <summary> +/// Reads the custom check results out of the primary database, each one turned into the shared <see cref="CustomCheckDetail" />, a type the target can reference. +/// A null value in the stored document stays null, so the writer can name the column it would fill. +/// A result of one of the RavenDB persister's own checks carries the SourceStorageCheck metadata value, so the target can skip it. +/// </summary> +sealed class CustomChecksReader(RavenReadOnlySourceLifecycle lifecycle) : MigrationCategoryReader<CustomCheckDetail>(lifecycle) +{ + public override string CategoryId => MigrationCategoryIds.CustomChecks; + + public override IAsyncEnumerable<MigrationBatch> Read(string? resumeAfter, int batchSize, CancellationToken cancellationToken = default) => + RavenDocumentStream.ByPrefix<CustomCheck>(Lifecycle, CategoryId, Lifecycle.Settings.DatabaseName, $"{RavenCustomCheckDataStore.CollectionName}/", resumeAfter, batchSize, RavenDocumentStream.MaxDocumentsWalkedPerBatch, + static result => new MigrationRow(result.Id, ToDetail(result.Document), MetadataFor(result.Document)), cancellationToken); + + static IReadOnlyDictionary<string, object?> MetadataFor(CustomCheck document) => + document.CustomCheckId is { } id && document.Category is { } category && RavenStorageCheckIds.All.Contains((id, category)) + ? new Dictionary<string, object?> { [MigrationMetadataKeys.SourceStorageCheck] = true } + : ReadOnlyDictionary<string, object?>.Empty; + + static CustomCheckDetail ToDetail(CustomCheck document) => new() + { + OriginatingEndpoint = document.OriginatingEndpoint!, + CustomCheckId = document.CustomCheckId!, + ReportedAt = document.ReportedAt, + Category = document.Category!, + HasFailed = document.Status == Status.Fail, + FailureReason = document.FailureReason + }; +} diff --git a/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/FailedErrorImportsReader.cs b/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/FailedErrorImportsReader.cs new file mode 100644 index 0000000000..c45b0e0eac --- /dev/null +++ b/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/FailedErrorImportsReader.cs @@ -0,0 +1,30 @@ +#nullable enable + +namespace ServiceControl.Persistence.RavenDB.DataMigration.Readers; + +using System.Collections.Generic; +using System.Threading; +using Raven.Client; +using Raven.Client.Documents.Commands; +using ServiceControl.Operations; +using ServiceControl.Persistence.DataMigration; + +/// <summary> +/// Reads the messages ServiceControl took off the error queue but could not store, each with its body and the time it failed. +/// </summary> +sealed class FailedErrorImportsReader(RavenReadOnlySourceLifecycle lifecycle) : MigrationCategoryReader<FailedErrorImport>(lifecycle) +{ + public override string CategoryId => MigrationCategoryIds.FailedErrorImports; + + public override IAsyncEnumerable<MigrationBatch> Read(string? resumeAfter, int batchSize, CancellationToken cancellationToken = default) => + Documents($"{FailedErrorImportDataStore.CollectionName}/", resumeAfter, batchSize, ToRow, cancellationToken); + + // A document with no message still gets an empty body, because the engine would otherwise ask ReadBody, which refuses this category. + static MigrationRow ToRow(StreamResult<FailedErrorImport> result) => new( + result.Id, + result.Document, + new Dictionary<string, object?> { [MigrationMetadataKeys.FailedAt] = FailedMessageStream.ReadMetadataDate(result, Constants.Documents.Metadata.LastModified) }, + new MigrationBody( + result.Document.Message?.Body ?? [], + result.Document.Message?.Headers?.GetValueOrDefault(NServiceBus.Headers.ContentType) ?? "application/octet-stream")); +} diff --git a/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/FailedMessagesReader.cs b/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/FailedMessagesReader.cs index 4cc1393ac9..1d32916de9 100644 --- a/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/FailedMessagesReader.cs +++ b/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/FailedMessagesReader.cs @@ -41,7 +41,8 @@ sealed class FailedMessagesReader(RavenReadOnlySourceLifecycle lifecycle) : Migr public override async IAsyncEnumerable<MigrationBatch> Read(string? resumeAfter, int batchSize, [EnumeratorCancellation] CancellationToken cancellationToken = default) { var databaseName = Lifecycle.Settings.DatabaseName; - var project = new FailedMessageStream(Lifecycle).Project(MigrationCategoryRegistry.UnresolvedAndRetryIssuedStatuses); + var stream = new FailedMessageStream(Lifecycle); + var project = stream.Project(MigrationCategoryRegistry.UnresolvedAndRetryIssuedStatuses); var batches = await IndexHoldsEveryCopiedFailedMessage(databaseName, cancellationToken) ? ThroughIndex(databaseName, resumeAfter, batchSize, project, cancellationToken) @@ -49,7 +50,7 @@ public override async IAsyncEnumerable<MigrationBatch> Read(string? resumeAfter, await foreach (var batch in batches) { - yield return batch; + yield return await stream.ResolveRetryClaims(batch, cancellationToken); } } diff --git a/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/GroupCommentsReader.cs b/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/GroupCommentsReader.cs new file mode 100644 index 0000000000..1397a35a4d --- /dev/null +++ b/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/GroupCommentsReader.cs @@ -0,0 +1,23 @@ +#nullable enable + +namespace ServiceControl.Persistence.RavenDB.DataMigration.Readers; + +using System.Collections.Generic; +using System.Collections.ObjectModel; +using System.Threading; +using ServiceControl.MessageFailures; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.RavenDB.Recoverability; + +/// <summary> +/// Reads the comments operators wrote on failure groups. Each row carries the bare group id, because SQL keys a comment on the group id while RavenDB stores it under its own document id. +/// </summary> +sealed class GroupCommentsReader(RavenReadOnlySourceLifecycle lifecycle) : MigrationCategoryReader<GroupComment>(lifecycle) +{ + public override string CategoryId => MigrationCategoryIds.GroupComments; + + public override IAsyncEnumerable<MigrationBatch> Read(string? resumeAfter, int batchSize, CancellationToken cancellationToken = default) => + Documents(GroupsDataStore.CommentIdPrefix, resumeAfter, batchSize, + static result => new MigrationRow(result.Id, new GroupComment { Id = result.Id[GroupsDataStore.CommentIdPrefix.Length..], Comment = result.Document.Comment }, ReadOnlyDictionary<string, object?>.Empty), + cancellationToken); +} diff --git a/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/LicensingThroughputReader.cs b/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/LicensingThroughputReader.cs index 0b87d2ee76..a77b53067c 100644 --- a/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/LicensingThroughputReader.cs +++ b/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/LicensingThroughputReader.cs @@ -15,8 +15,8 @@ namespace ServiceControl.Persistence.RavenDB.DataMigration.Readers; using ServiceControl.Persistence.RavenDB.Throughput; /// <summary> -/// Reads each licensing endpoint's daily message counts out of the throughput database, one batch per group from -/// <see cref="ThroughputDocuments.EndpointGroups" />, with the batch's cursor on the group's last document id. +/// Reads each licensing endpoint's daily message counts out of the throughput database, one batch per endpoint in +/// ordinal document-id order, with the batch's cursor on that endpoint's document id. /// </summary> sealed class LicensingThroughputReader(RavenReadOnlySourceLifecycle lifecycle) : MigrationCategoryReader<ThroughputDayRow>(lifecycle) { @@ -27,13 +27,12 @@ public override IAsyncEnumerable<MigrationBatch> Read(string? resumeAfter, int b async IAsyncEnumerable<MigrationBatch> ReadThroughputDays(string? resumeAfter, [EnumeratorCancellation] CancellationToken cancellationToken) { - // One group of endpoints per batch, because the batch size counts rows and one endpoint can bring a year or more of days, - // and because the writer adds the days of a group's endpoints only when they arrive in one batch. - await foreach (var endpoints in ThroughputDocuments.EndpointGroups(Lifecycle, CategoryId, resumeAfter, cancellationToken)) + // One endpoint per batch, because the batch size counts rows and one endpoint can bring a year or more of days. + await foreach (var endpoints in ThroughputDocuments.Endpoints(Lifecycle, CategoryId, resumeAfter, batchSize: 1, cancellationToken)) { var days = new List<MigrationRow>(); - foreach (var endpoint in endpoints) + foreach (var endpoint in endpoints.Rows) { var id = ((Endpoint)endpoint.Document).Id; @@ -43,7 +42,7 @@ async IAsyncEnumerable<MigrationBatch> ReadThroughputDays(string? resumeAfter, [ } } - yield return new MigrationBatch(days, endpoints[^1].SourceId); + yield return new MigrationBatch(days, endpoints.Cursor); } } diff --git a/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/PendingIntegrationEventsReader.cs b/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/PendingIntegrationEventsReader.cs new file mode 100644 index 0000000000..e1e9775e64 --- /dev/null +++ b/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/PendingIntegrationEventsReader.cs @@ -0,0 +1,19 @@ +#nullable enable + +namespace ServiceControl.Persistence.RavenDB.DataMigration.Readers; + +using System.Collections.Generic; +using System.Threading; +using ServiceControl.ExternalIntegrations; +using ServiceControl.Persistence.DataMigration; + +/// <summary> +/// Reads the integration events the old instance had not yet dispatched, whole, out of the primary database. +/// </summary> +sealed class PendingIntegrationEventsReader(RavenReadOnlySourceLifecycle lifecycle) : MigrationCategoryReader<ExternalIntegrationDispatchRequest>(lifecycle) +{ + public override string CategoryId => MigrationCategoryIds.PendingIntegrationEvents; + + public override IAsyncEnumerable<MigrationBatch> Read(string? resumeAfter, int batchSize, CancellationToken cancellationToken = default) => + WholeDocuments(ExternalIntegrationRequestsDataStore.KeyPrefix + "/", resumeAfter, batchSize, cancellationToken); +} diff --git a/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/RavenStorageCheckIds.cs b/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/RavenStorageCheckIds.cs new file mode 100644 index 0000000000..bd92c28dbf --- /dev/null +++ b/src/ServiceControl.Persistence.RavenDB/DataMigration/Readers/RavenStorageCheckIds.cs @@ -0,0 +1,22 @@ +namespace ServiceControl.Persistence.RavenDB.DataMigration.Readers; + +using System.Collections.Frozen; +using System.Collections.Generic; + +/// <summary> +/// The six custom checks the RavenDB persister runs about its own database, each as its id and the category it reports under. +/// The custom checks reader flags a result only when both match, so a result that reuses one of these ids under another category is not flagged, such as the audit instance's "RavenDB dirty memory" under "ServiceControl.Audit Health". +/// </summary> +static class RavenStorageCheckIds +{ + // Each check passes its id and category to the CustomCheck base constructor as literals, so there is nothing to reference. RavenStorageCheckIdsTests fails if the two lists differ. + public static IReadOnlySet<(string Id, string Category)> All { get; } = new[] + { + ("Error Database Index Errors", "ServiceControl Health"), + ("Error Database Index Lag", "ServiceControl Health"), + ("ServiceControl database", "Storage space"), + ("Message Ingestion Process", "ServiceControl Health"), + ("RavenDB dirty memory", "ServiceControl Health"), + ("Error Database Search Engine", "ServiceControl Health") + }.ToFrozenSet(); +} diff --git a/src/ServiceControl.Persistence.RavenDB/DataMigration/ThroughputDocuments.cs b/src/ServiceControl.Persistence.RavenDB/DataMigration/ThroughputDocuments.cs index 1207f6b3fb..77701e8a65 100644 --- a/src/ServiceControl.Persistence.RavenDB/DataMigration/ThroughputDocuments.cs +++ b/src/ServiceControl.Persistence.RavenDB/DataMigration/ThroughputDocuments.cs @@ -9,7 +9,6 @@ namespace ServiceControl.Persistence.RavenDB.DataMigration; using System.Runtime.CompilerServices; using System.Threading; using System.Threading.Tasks; -using Particular.LicensingComponent.Contracts; using Raven.Client; using Raven.Client.Documents.Commands; using Raven.Client.Documents.Conventions; @@ -46,37 +45,6 @@ public static async IAsyncEnumerable<MigrationBatch> Endpoints( } } - /// <summary> - /// Reads the endpoints in groups whose names can share a key in SQL: the same source, and the same name ignoring - /// case and trailing spaces. Groups come in the ordinal order of their last document id, which is each group's cursor, - /// and only groups after <paramref name="resumeAfter" /> are read. Throws <see cref="InvalidOperationException" /> when - /// the source holds no document with the id in <paramref name="resumeAfter" />. - /// </summary> - public static async IAsyncEnumerable<IReadOnlyList<MigrationRow>> EndpointGroups( - RavenReadOnlySourceLifecycle lifecycle, - string categoryId, - string? resumeAfter, - [EnumeratorCancellation] CancellationToken cancellationToken = default) - { - var groups = (await AllEndpoints(lifecycle, categoryId, resumeAfter, cancellationToken)) - .GroupBy(SharedKey) - .Select(group => group.OrderBy(row => row.SourceId, StringComparer.Ordinal).ToList()) - .Where(group => resumeAfter is null || string.CompareOrdinal(group[^1].SourceId, resumeAfter) > 0) - .OrderBy(group => group[^1].SourceId, StringComparer.Ordinal); - - foreach (var group in groups) - { - yield return group; - } - } - - // SQL Server's default collation ignores case and trailing spaces, and the product lower-cases the name it stores. - static (string? Name, ThroughputSource Source) SharedKey(MigrationRow row) - { - var id = ((Endpoint)row.Document).Id; - return (id.Name?.TrimEnd(' ').ToLowerInvariant(), id.ThroughputSource); - } - // With no id prefix RavenDB streams in reverse etag order and ignores startAfter, so callers order the rows and apply the cursor. static async Task<List<MigrationRow>> AllEndpoints(RavenReadOnlySourceLifecycle lifecycle, string categoryId, string? resumeAfter, CancellationToken cancellationToken) { diff --git a/src/ServiceControl.Persistence.RavenDB/ExternalIntegrationRequestsDataStore.cs b/src/ServiceControl.Persistence.RavenDB/ExternalIntegrationRequestsDataStore.cs index d9848ccb28..c1365095bc 100644 --- a/src/ServiceControl.Persistence.RavenDB/ExternalIntegrationRequestsDataStore.cs +++ b/src/ServiceControl.Persistence.RavenDB/ExternalIntegrationRequestsDataStore.cs @@ -39,7 +39,7 @@ public ExternalIntegrationRequestsDataStore( ); } - const string KeyPrefix = "ExternalIntegrationDispatchRequests"; + internal const string KeyPrefix = "ExternalIntegrationDispatchRequests"; public async Task StoreDispatchRequest(IEnumerable<ExternalIntegrationDispatchRequest> dispatchRequests, CancellationToken cancellationToken = default) { diff --git a/src/ServiceControl.Persistence.RavenDB/RavenCustomCheckDataStore.cs b/src/ServiceControl.Persistence.RavenDB/RavenCustomCheckDataStore.cs index e9cfd8f89d..acbc5081eb 100644 --- a/src/ServiceControl.Persistence.RavenDB/RavenCustomCheckDataStore.cs +++ b/src/ServiceControl.Persistence.RavenDB/RavenCustomCheckDataStore.cs @@ -44,7 +44,9 @@ public async Task<CheckStateChange> UpdateCustomCheckStatus(CustomCheckDetail de return status; } - static string MakeId(Guid id) => $"CustomChecks/{id}"; + internal const string CollectionName = "CustomChecks"; + + static string MakeId(Guid id) => $"{CollectionName}/{id}"; public async Task<QueryResult<IList<CustomCheckView>>> GetStats(PagingInfo paging, string status = null, CancellationToken cancellationToken = default) { diff --git a/src/ServiceControl.Persistence.RavenDB/Recoverability/GroupsDataStore.cs b/src/ServiceControl.Persistence.RavenDB/Recoverability/GroupsDataStore.cs index 92fcfdf898..284700ead9 100644 --- a/src/ServiceControl.Persistence.RavenDB/Recoverability/GroupsDataStore.cs +++ b/src/ServiceControl.Persistence.RavenDB/Recoverability/GroupsDataStore.cs @@ -146,6 +146,8 @@ public async Task DeleteComment(string groupId, CancellationToken cancellationTo await session.SaveChangesAsync(cancellationToken); } - public static string MakeId(string groupId) => $"GroupComment/{groupId}"; + internal const string CommentIdPrefix = "GroupComment/"; + + public static string MakeId(string groupId) => $"{CommentIdPrefix}{groupId}"; } } diff --git a/src/ServiceControl.Persistence.Tests.PostgreSql/MigrationValueLengthTests.cs b/src/ServiceControl.Persistence.Tests.PostgreSql/MigrationValueLengthTests.cs index 07c39bd01b..2c38adfcb1 100644 --- a/src/ServiceControl.Persistence.Tests.PostgreSql/MigrationValueLengthTests.cs +++ b/src/ServiceControl.Persistence.Tests.PostgreSql/MigrationValueLengthTests.cs @@ -6,8 +6,10 @@ namespace ServiceControl.Persistence.Tests; using System.Threading.Tasks; using Microsoft.Extensions.DependencyInjection; using NUnit.Framework; +using ServiceControl.Contracts.CustomChecks; using ServiceControl.Operations; using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.Abstractions; using ServiceControl.Persistence.EFCore.DataMigration.Writers; using ServiceControl.Persistence.EFCore.Entities; using ServiceControl.Persistence.EFCore.Infrastructure; @@ -77,5 +79,85 @@ public async Task A_value_holding_a_NUL_character_is_skipped_as_malformed_naming } } + [Test] + public async Task A_custom_check_whose_failure_reason_holds_a_NUL_character_is_skipped_as_malformed_and_the_rest_copy() + { + var endpoint = new EndpointDetails { Name = "Sales", HostId = Guid.NewGuid(), Host = "SALES01" }; + var batch = new MigrationBatch( + [ + new MigrationRow("CustomChecks/1", CustomCheck(endpoint, "Sales.HealthCheck", "timed out"), new Dictionary<string, object>()), + new MigrationRow("CustomChecks/2", CustomCheck(endpoint, "Sales.DiskCheck", "disk \u0000 full"), new Dictionary<string, object>()), + new MigrationRow("CustomChecks/3", CustomCheck(endpoint, "Sales.QueueCheck", null), new Dictionary<string, object>()) + ], + "CustomChecks/3"); + + var details = await PreparedSkips.DetailsFrom(ServiceProvider, new CustomChecksWriter(ServiceProvider.GetRequiredService<IMigrationSqlDialect>()), batch); + var result = await Target.Write(CustomChecksCategory, batch, new MigrationCheckpoint(MigrationCategoryIds.CustomChecks, MigrationCategoryState.InProgress, "CustomChecks/3", 0, 0, null, null, null, null, null, null)); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(2), "PostgreSQL refuses U+0000 in text, which would fail the whole insert and halt the category"); + Assert.That(result.SkipReasons, Is.EqualTo(new Dictionary<MigrationSkipReason, long> { [MigrationSkipReason.MalformedValue] = 1 })); + Assert.That(details, Is.EqualTo(new[] + { + $"{PreparedSkips.TableAndColumnOf<CustomCheckEntity>(ServiceProvider, nameof(CustomCheckEntity.FailureReason))} cannot hold the character U+0000, which the value contains, on document 'CustomChecks/2'" + })); + Assert.That(await Target.Count(CustomChecksCategory), Is.EqualTo(2)); + } + } + + [Test] + public async Task A_failed_error_import_whose_exception_info_holds_a_NUL_character_is_skipped_as_malformed_and_the_rest_copy() + { + var batch = new MigrationBatch( + [ + FailedErrorImportRow("native-1", "Deserialization failed"), + FailedErrorImportRow("native-2", "Deserialization \u0000 failed"), + FailedErrorImportRow("native-3", "Deserialization failed") + ], + "FailedErrorImports/native-3"); + var writer = new FailedErrorImportsWriter(ServiceProvider.GetRequiredService<IMigrationSqlDialect>(), ServiceProvider.GetRequiredService<IBodyStoragePersistence>(), ServiceProvider.GetRequiredService<EFPersisterSettings>()); + + var details = await PreparedSkips.DetailsFrom(ServiceProvider, writer, batch); + var result = await Target.Write(FailedErrorImportsCategory, batch, new MigrationCheckpoint(MigrationCategoryIds.FailedErrorImports, MigrationCategoryState.InProgress, "FailedErrorImports/native-3", 0, 0, null, null, null, null, null, null)); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(2), "PostgreSQL refuses U+0000 in text, which would fail the whole insert and halt the category"); + Assert.That(result.SkipReasons, Is.EqualTo(new Dictionary<MigrationSkipReason, long> { [MigrationSkipReason.MalformedValue] = 1 })); + Assert.That(details, Is.EqualTo(new[] + { + $"{PreparedSkips.TableAndColumnOf<FailedErrorImportEntity>(ServiceProvider, nameof(FailedErrorImportEntity.ExceptionInfo))} cannot hold the character U+0000, which the value contains, on document 'FailedErrorImports/native-2'" + })); + Assert.That(await Target.Count(FailedErrorImportsCategory), Is.EqualTo(2)); + } + } + + static readonly MigrationCategory FailedErrorImportsCategory = MigrationCategoryRegistry.Find(MigrationCategoryIds.FailedErrorImports)!; + + // The NUL goes in ExceptionInfo, because the headers are written as JSON, which escapes it. + static MigrationRow FailedErrorImportRow(string nativeMessageId, string exceptionInfo) => new( + $"FailedErrorImports/{nativeMessageId}", + new FailedErrorImport + { + Id = $"FailedErrorImports/{nativeMessageId}", + Message = new FailedTransportMessage { Id = nativeMessageId, Headers = new Dictionary<string, string> { ["NServiceBus.MessageId"] = nativeMessageId }, Body = [1, 2, 3] }, + ExceptionInfo = exceptionInfo + }, + new Dictionary<string, object> { [MigrationMetadataKeys.FailedAt] = new DateTime(2026, 9, 1, 8, 0, 0, DateTimeKind.Utc) }, + new MigrationBody(new byte[] { 1, 2, 3 }, "application/octet-stream")); + + static readonly MigrationCategory CustomChecksCategory = MigrationCategoryRegistry.Find(MigrationCategoryIds.CustomChecks)!; + + static CustomCheckDetail CustomCheck(EndpointDetails endpoint, string customCheckId, string failureReason) => new() + { + OriginatingEndpoint = endpoint, + CustomCheckId = customCheckId, + Category = "Ping", + ReportedAt = new DateTime(2026, 9, 1, 8, 0, 0, DateTimeKind.Utc), + HasFailed = failureReason is not null, + FailureReason = failureReason + }; + static string Repeat(int codePoints) => string.Concat(Enumerable.Repeat(OutsideTheBasicPlane, codePoints)); } diff --git a/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/CustomCheckSourceTests.cs b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/CustomCheckSourceTests.cs new file mode 100644 index 0000000000..75d1b0bc09 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/CustomCheckSourceTests.cs @@ -0,0 +1,76 @@ +namespace ServiceControl.Persistence.Tests.RavenDB.DataMigration; + +using System; +using System.Linq; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.Contracts.CustomChecks; +using ServiceControl.Operations; +using ServiceControl.Persistence.DataMigration; + +class CustomCheckSourceTests : RavenMigrationSourceTestBase +{ + [Test] + public async Task Reads_a_custom_check_as_the_neutral_detail_type_with_its_id_recoverable() + { + var endpoint = new EndpointDetails { Name = "Sales", Host = "MACHINE", HostId = Guid.NewGuid() }; + var expectedId = await SeedCustomCheck(endpoint, "Sales.HealthCheck", "Ping", failed: true, reason: "timed out"); + await CompleteDatabaseOperation(); + + var category = MigrationCategoryRegistry.All.Single(entry => entry.Id == "CustomChecks"); + + await using var source = await OpenMigrationSource(); + + var detail = (CustomCheckDetail)(await CollectBatches(source, category)).SelectMany(batch => batch.Rows).Single().Document; + + using (Assert.EnterMultipleScope()) + { + Assert.That(detail.GetDeterministicId(), Is.EqualTo(expectedId), "the id must be recoverable from the same fields RavenDB used to generate it"); + Assert.That(detail.CustomCheckId, Is.EqualTo("Sales.HealthCheck")); + Assert.That(detail.HasFailed, Is.True); + Assert.That(detail.FailureReason, Is.EqualTo("timed out")); + } + } + + [Test] + public async Task Flags_only_a_row_whose_id_and_category_are_one_of_the_persisters_own_checks() + { + var endpoint = new EndpointDetails { Name = "Sales", Host = "MACHINE", HostId = Guid.NewGuid() }; + await SeedCustomCheck(endpoint, "Error Database Index Lag", "ServiceControl Health", failed: false, reason: null); + await SeedCustomCheck(new EndpointDetails { Name = "Billing", Host = "MACHINE", HostId = Guid.NewGuid() }, "Error Database Index Lag", "Another category", failed: false, reason: null); + await SeedCustomCheck(endpoint, "Sales.HealthCheck", "Ping", failed: false, reason: null); + await SeedCustomCheck(endpoint, "Error Message Ingestion Process", "ServiceControl Health", failed: false, reason: null); + await CompleteDatabaseOperation(); + + var category = MigrationCategoryRegistry.All.Single(entry => entry.Id == "CustomChecks"); + + await using var source = await OpenMigrationSource(); + + var rows = (await CollectBatches(source, category)).SelectMany(batch => batch.Rows).ToDictionary(row => ((CustomCheckDetail)row.Document).CustomCheckId + "|" + ((CustomCheckDetail)row.Document).Category); + + using (Assert.EnterMultipleScope()) + { + Assert.That(rows["Error Database Index Lag|ServiceControl Health"].Metadata[MigrationMetadataKeys.SourceStorageCheck], Is.True); + Assert.That(rows["Error Database Index Lag|Another category"].Metadata, Does.Not.ContainKey(MigrationMetadataKeys.SourceStorageCheck), "the same id under another category is not the persister's check"); + Assert.That(rows["Error Message Ingestion Process|ServiceControl Health"].Metadata, Does.Not.ContainKey(MigrationMetadataKeys.SourceStorageCheck), "another id under the persister's category is not the persister's check"); + Assert.That(rows["Sales.HealthCheck|Ping"].Metadata, Does.Not.ContainKey(MigrationMetadataKeys.SourceStorageCheck)); + } + } + + async Task<Guid> SeedCustomCheck(EndpointDetails endpoint, string customCheckId, string category, bool failed, string reason) + { + var detail = new CustomCheckDetail + { + OriginatingEndpoint = endpoint, + CustomCheckId = customCheckId, + Category = category, + ReportedAt = Now, + HasFailed = failed, + FailureReason = reason + }; + + await CustomChecks.UpdateCustomCheckStatus(detail); + + return detail.GetDeterministicId(); + } +} diff --git a/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/FailedErrorImportSourceTests.cs b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/FailedErrorImportSourceTests.cs new file mode 100644 index 0000000000..27a23e01bf --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/FailedErrorImportSourceTests.cs @@ -0,0 +1,48 @@ +namespace ServiceControl.Persistence.Tests.RavenDB.DataMigration; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.Operations; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.RavenDB; + +class FailedErrorImportSourceTests : RavenMigrationSourceTestBase +{ + [Test] + public async Task Every_import_arrives_with_its_body_attached_and_the_time_RavenDB_last_modified_it() + { + await FailedImportStore.StoreFailedErrorImport(new FailedErrorImport + { + Id = "native-1", + Message = new FailedTransportMessage { Id = "native-1", Headers = new Dictionary<string, string> { ["NServiceBus.MessageId"] = "native-1" }, Body = [1, 2, 3] }, + ExceptionInfo = "Deserialization failed" + }); + await FailedImportStore.StoreFailedErrorImport(new FailedErrorImport { Id = "native-2", Message = null, ExceptionInfo = "Deserialization failed" }); + await CompleteDatabaseOperation(); + + DateTime lastModified; + + using (var session = await SessionProvider.OpenSession()) + { + lastModified = session.Advanced.GetLastModifiedFor(await session.LoadAsync<FailedErrorImport>(FailedErrorImportDataStore.MakeDocumentId("native-1"))).Value; + } + + await using var source = await OpenMigrationSource(); + + var rows = (await CollectBatches(source, MigrationCategoryRegistry.Find(MigrationCategoryIds.FailedErrorImports))).SelectMany(batch => batch.Rows).ToDictionary(row => row.SourceId); + var withMessage = rows[FailedErrorImportDataStore.MakeDocumentId("native-1")]; + var withoutMessage = rows[FailedErrorImportDataStore.MakeDocumentId("native-2")]; + + using (Assert.EnterMultipleScope()) + { + Assert.That(withMessage.Body!.Content.ToArray(), Is.EqualTo(new byte[] { 1, 2, 3 }), "attached in Read, so the engine never asks ReadBody, which refuses this category"); + Assert.That(withMessage.Body.ContentType, Is.EqualTo("application/octet-stream"), "the default the SQL import store gives a message with no content type header"); + Assert.That(withMessage.Metadata[MigrationMetadataKeys.FailedAt], Is.EqualTo(lastModified.ToUniversalTime()), "RavenDB keeps no failure time, only the stamp it wrote when the import was stored"); + Assert.That(((DateTime)withMessage.Metadata[MigrationMetadataKeys.FailedAt]!).Kind, Is.EqualTo(DateTimeKind.Utc)); + Assert.That(withoutMessage.Body?.Content.Length, Is.Zero, "an empty body rather than none, so the writer skips the row instead of the engine stopping on ReadBody"); + } + } +} diff --git a/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/GroupCommentsSourceTests.cs b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/GroupCommentsSourceTests.cs new file mode 100644 index 0000000000..af5f2d510d --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/GroupCommentsSourceTests.cs @@ -0,0 +1,31 @@ +namespace ServiceControl.Persistence.Tests.RavenDB.DataMigration; + +using System.Linq; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.MessageFailures; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.RavenDB.Recoverability; + +class GroupCommentsSourceTests : RavenMigrationSourceTestBase +{ + static readonly MigrationCategory Category = MigrationCategoryRegistry.Find(MigrationCategoryIds.GroupComments); + + [Test] + public async Task Reads_each_stored_comment_keyed_by_its_group_id_a_cleared_one_included() + { + await GroupsStore.EditComment("group-1", "Known issue, ignore until Thursday"); + await GroupsStore.EditComment("group-2", ""); + await CompleteDatabaseOperation(); + + await using var source = await OpenMigrationSource(); + var rows = (await CollectBatches(source, Category)).SelectMany(batch => batch.Rows).ToList(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(rows.Select(row => row.SourceId), Is.EqualTo(new[] { GroupsDataStore.MakeId("group-1"), GroupsDataStore.MakeId("group-2") }), "the cursor and every skip name the RavenDB document id"); + Assert.That(rows.Select(row => ((GroupComment)row.Document).Id), Is.EqualTo(new[] { "group-1", "group-2" }), "SQL keys a comment on the bare group id, so that is what crosses"); + Assert.That(rows.Select(row => ((GroupComment)row.Document).Comment), Is.EqualTo(new[] { "Known issue, ignore until Thursday", "" }), "a cleared comment is read, so the writer counts its skip instead of it vanishing uncounted"); + } + } +} diff --git a/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/LicensingThroughputSourceTests.cs b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/LicensingThroughputSourceTests.cs index 065b0fa62e..7497fef6f8 100644 --- a/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/LicensingThroughputSourceTests.cs +++ b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/LicensingThroughputSourceTests.cs @@ -72,7 +72,7 @@ public async Task A_resume_reads_only_the_endpoints_after_the_cursor_whatever_or } [Test] - public async Task Endpoints_whose_names_differ_only_in_case_and_trailing_spaces_come_in_one_batch() + public async Task Endpoints_whose_names_differ_only_in_case_and_trailing_spaces_come_in_separate_batches() { foreach (var name in new[] { "Sales.Orders", "Sales.Orders.Archive", "SALES.ORDERS " }) { @@ -93,9 +93,9 @@ public async Task Endpoints_whose_names_differ_only_in_case_and_trailing_spaces_ using (Assert.EnterMultipleScope()) { - Assert.That(batches.Select(batch => batch.Cursor), Is.EqualTo(new[] { "Sales.Orders.Archive/Broker", "Sales.Orders/Broker" }), "each group cursors on its last id in ordinal order, so a resume never splits a group"); - Assert.That(batches[1].Rows.Select(row => ((ThroughputDayRow)row.Document).Endpoint.Name), Is.EquivalentTo(new[] { "Sales.Orders", "SALES.ORDERS " }), "the writer can add the two endpoints' days only when they arrive in one batch"); - Assert.That(resumed.SelectMany(batch => batch.Rows).Select(row => ((ThroughputDayRow)row.Document).Endpoint.Name), Is.EquivalentTo(new[] { "Sales.Orders", "SALES.ORDERS " }), "a resume after the first group reads the whole second group"); + Assert.That(batches.Select(batch => batch.Cursor), Is.EqualTo(new[] { "SALES.ORDERS /Broker", "Sales.Orders.Archive/Broker", "Sales.Orders/Broker" }), "the source groups by its own document id only, and which names are one key is the target's to decide"); + Assert.That(batches.Select(batch => batch.Rows.Select(row => ((ThroughputDayRow)row.Document).Endpoint.Name).Single()), Is.EqualTo(new[] { "SALES.ORDERS ", "Sales.Orders.Archive", "Sales.Orders" }), "each batch holds the days of the one endpoint its cursor names"); + Assert.That(resumed.SelectMany(batch => batch.Rows).Select(row => ((ThroughputDayRow)row.Document).Endpoint.Name), Is.EqualTo(new[] { "Sales.Orders.Archive", "Sales.Orders" }), "a resume reads only the endpoints after the cursor, even one whose name differs from the cursor's only in case and spaces"); } } diff --git a/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/PendingIntegrationEventsSourceTests.cs b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/PendingIntegrationEventsSourceTests.cs new file mode 100644 index 0000000000..e5afca8d87 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/PendingIntegrationEventsSourceTests.cs @@ -0,0 +1,38 @@ +namespace ServiceControl.Persistence.Tests.RavenDB.DataMigration; + +using System.Linq; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.ExternalIntegrations; +using ServiceControl.Persistence.DataMigration; + +class PendingIntegrationEventsSourceTests : RavenMigrationSourceTestBase +{ + [Test] + public async Task Reads_each_queued_request_with_its_context_as_the_concrete_type() + { + await ExternalIntegrationStore.StoreDispatchRequest( + [ + new ExternalIntegrationDispatchRequest { DispatchContext = new TestDispatchContext { Value = "one" } }, + new ExternalIntegrationDispatchRequest { DispatchContext = new TestDispatchContext { Value = "two" } } + ]); + await CompleteDatabaseOperation(); + + var category = MigrationCategoryRegistry.All.Single(entry => entry.Id == MigrationCategoryIds.PendingIntegrationEvents); + + await using var source = await OpenMigrationSource(); + var contexts = (await CollectBatches(source, category)).SelectMany(batch => batch.Rows).Select(row => ((ExternalIntegrationDispatchRequest)row.Document).DispatchContext).ToArray(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(contexts, Has.All.InstanceOf<TestDispatchContext>(), "an untyped context would serialize as the JSON library's own type and never dispatch"); + Assert.That(contexts.Cast<TestDispatchContext>().Select(context => context.Value), Is.EquivalentTo(new[] { "one", "two" })); + Assert.That(await source.Count(category), Is.EqualTo(2)); + } + } + + public class TestDispatchContext + { + public string Value { get; set; } + } +} diff --git a/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/RavenMigrationSourceTests.cs b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/RavenMigrationSourceTests.cs index 67a81ea63f..21502d2d23 100644 --- a/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/RavenMigrationSourceTests.cs +++ b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/RavenMigrationSourceTests.cs @@ -9,6 +9,8 @@ namespace ServiceControl.Persistence.Tests.RavenDB.DataMigration; using NUnit.Framework; using Particular.LicensingComponent.Contracts; using Raven.Client.Documents.Operations.Indexes; +using ServiceControl.Contracts.CustomChecks; +using ServiceControl.ExternalIntegrations; using ServiceControl.Infrastructure.RavenDB.Subscriptions; using ServiceControl.Notifications; using ServiceControl.Operations; @@ -143,7 +145,11 @@ public async Task Reading_every_category_creates_no_index() [MigrationCategoryIds.Subscriptions] = tests => tests.SeedSubscription(), [MigrationCategoryIds.NotificationSettings] = tests => tests.NotificationsStore.SaveSettings(new NotificationsSettings { Email = new EmailNotifications { SmtpServer = "smtp.contoso.com" } }), [MigrationCategoryIds.TrialEndDate] = tests => tests.ServiceProvider.GetRequiredService<ITrialLicenseDataProvider>().StoreTrialEndDate(new DateOnly(2027, 3, 15)), - [MigrationCategoryIds.RetryOperations] = tests => tests.RetryHistoryStore.RecordRetryOperationCompleted("group-1", RetryType.FailureGroup, DateTime.UtcNow, DateTime.UtcNow, "Sales.Orders", "Exception Type", false, 1, DateTime.UtcNow, 10) + [MigrationCategoryIds.RetryOperations] = tests => tests.RetryHistoryStore.RecordRetryOperationCompleted("group-1", RetryType.FailureGroup, DateTime.UtcNow, DateTime.UtcNow, "Sales.Orders", "Exception Type", false, 1, DateTime.UtcNow, 10), + [MigrationCategoryIds.PendingIntegrationEvents] = tests => tests.ExternalIntegrationStore.StoreDispatchRequest([new ExternalIntegrationDispatchRequest { DispatchContext = new TestDispatchContext { Value = "seed" } }]), + [MigrationCategoryIds.CustomChecks] = tests => tests.CustomChecks.UpdateCustomCheckStatus(new CustomCheckDetail { OriginatingEndpoint = new EndpointDetails { Name = "Sales.Orders", HostId = Guid.NewGuid(), Host = "HOST01" }, CustomCheckId = "Sales.HealthCheck", Category = "Ping" }), + [MigrationCategoryIds.FailedErrorImports] = tests => tests.FailedImportStore.StoreFailedErrorImport(new FailedErrorImport { Id = "native-1", Message = new FailedTransportMessage { Id = "native-1", Headers = new Dictionary<string, string> { ["NServiceBus.MessageId"] = "native-1" }, Body = [1, 2, 3] }, ExceptionInfo = "Deserialization failed" }), + [MigrationCategoryIds.GroupComments] = tests => tests.GroupsStore.EditComment("group-1", "Known issue, ignore until Thursday") }; string DatabaseName => ((RavenPersisterSettings)PersistenceSettings).DatabaseName; @@ -188,4 +194,9 @@ async Task SeedEndpointSettings(int count) await EndpointSettingsStore.UpdateEndpointSettings(new EndpointSettings { Name = $"Endpoint{index}", TrackInstances = true }); } } + + public class TestDispatchContext + { + public string Value { get; set; } + } } diff --git a/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/RavenStorageCheckIdsTests.cs b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/RavenStorageCheckIdsTests.cs new file mode 100644 index 0000000000..eefcf35846 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/RavenStorageCheckIdsTests.cs @@ -0,0 +1,15 @@ +namespace ServiceControl.Persistence.Tests.RavenDB.DataMigration; + +using System.Linq; +using Microsoft.Extensions.DependencyInjection; +using NServiceBus.CustomChecks; +using NUnit.Framework; +using ServiceControl.Persistence.RavenDB.DataMigration.Readers; + +class RavenStorageCheckIdsTests : RavenPersistenceTestBase +{ + [Test] + public void Lists_exactly_the_checks_the_RavenDB_persister_registers_with_their_categories() => + Assert.That(ServiceProvider.GetServices<ICustomCheck>().Select(check => (check.Id, check.Category)), Is.EquivalentTo(RavenStorageCheckIds.All), + "the custom checks reader flags exactly these pairs, so a check the persister adds, renames or moves to another category would be copied to SQL, where nothing reports it again"); +} diff --git a/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/UnresolvedFailedMessageSourceTests.cs b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/UnresolvedFailedMessageSourceTests.cs index f2b5f8bc11..9faec057e1 100644 --- a/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/UnresolvedFailedMessageSourceTests.cs +++ b/src/ServiceControl.Persistence.Tests.RavenDB/DataMigration/UnresolvedFailedMessageSourceTests.cs @@ -13,6 +13,7 @@ namespace ServiceControl.Persistence.Tests.RavenDB.DataMigration; using ServiceControl.MessageFailures.Api; using ServiceControl.Operations; using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.RavenDB; using ServiceControl.Persistence.RavenDB.DataMigration; [TestFixture] @@ -99,6 +100,32 @@ public async Task Sends_only_the_migration_metadata_keys_across_the_seam() Assert.That(row.Metadata.Keys, Is.EquivalentTo(new[] { MigrationMetadataKeys.StatusChangedAt, MigrationMetadataKeys.LastModified }), "a store's own metadata, such as @expires, never travels"); } + [Test] + public async Task A_retry_issued_message_says_whether_the_source_still_holds_its_retry_claim() + { + var claimed = await SeedFailedMessage(new IngestedFailure().ToFailedMessage(FailedMessageStatus.RetryIssued)); + var released = await SeedFailedMessage(new IngestedFailure().ToFailedMessage(FailedMessageStatus.RetryIssued)); + var unresolved = await SeedFailedMessage(new IngestedFailure().ToFailedMessage(FailedMessageStatus.Unresolved)); + + using (var session = DocumentStore.OpenAsyncSession()) + { + await session.StoreAsync(new FailedMessageRetry { FailedMessageId = claimed.Id, RetryBatchId = "RetryBatches/forwarded" }, RetryDocumentDataStore.MakeFailedMessageRetriesDocumentId(claimed.UniqueMessageId)); + await session.StoreAsync(new FailedMessageRetry { FailedMessageId = unresolved.Id, RetryBatchId = "RetryBatches/staging" }, RetryDocumentDataStore.MakeFailedMessageRetriesDocumentId(unresolved.UniqueMessageId)); + await session.SaveChangesAsync(); + } + + await CompleteDatabaseOperation(); + + var rows = (await ReadAll()).ToDictionary(row => ((FailedMessage)row.Document).UniqueMessageId); + + using (Assert.EnterMultipleScope()) + { + Assert.That(rows[claimed.UniqueMessageId].Metadata[MigrationMetadataKeys.RetryClaimHeld], Is.True); + Assert.That(rows[released.UniqueMessageId].Metadata[MigrationMetadataKeys.RetryClaimHeld], Is.False); + Assert.That(rows[unresolved.UniqueMessageId].Metadata, Does.Not.ContainKey(MigrationMetadataKeys.RetryClaimHeld), "only a retry-issued message gets a claim in SQL, even when the source holds one for another"); + } + } + [Test] public async Task Carries_the_documents_own_last_modified() { @@ -258,10 +285,21 @@ public async Task Reads_the_whole_collection_when_the_index_is_missing() { await DocumentStore.Maintenance.SendAsync(new DeleteIndexOperation(IndexName), TestContext.CurrentContext.CancellationToken); var stored = await SeedFailedMessage(new IngestedFailure().ToFailedMessage(FailedMessageStatus.Unresolved)); + var claimed = await SeedFailedMessage(new IngestedFailure().ToFailedMessage(FailedMessageStatus.RetryIssued)); + + using (var session = DocumentStore.OpenAsyncSession()) + { + await session.StoreAsync(new FailedMessageRetry { FailedMessageId = claimed.Id, RetryBatchId = "RetryBatches/forwarded" }, RetryDocumentDataStore.MakeFailedMessageRetriesDocumentId(claimed.UniqueMessageId)); + await session.SaveChangesAsync(); + } var rows = await ReadAll().WaitAsync(TimeSpan.FromSeconds(30), TestContext.CurrentContext.CancellationToken); - Assert.That(rows.Select(row => row.SourceId), Is.EqualTo(new[] { stored.Id }), "the copy cannot build the index on a source it only reads, so reading everything is the only way through"); + using (Assert.EnterMultipleScope()) + { + Assert.That(rows.Select(row => row.SourceId), Is.EquivalentTo(new[] { stored.Id, claimed.Id }), "the copy cannot build the index on a source it only reads, so reading everything is the only way through"); + Assert.That(rows.Single(row => row.SourceId == claimed.Id).Metadata[MigrationMetadataKeys.RetryClaimHeld], Is.True, "the whole-collection read must resolve claims too, or the writer stops on the first retry-issued row"); + } } [Test] diff --git a/src/ServiceControl.Persistence.Tests.SqlServer/EndpointSettingsKeyCollationTests.cs b/src/ServiceControl.Persistence.Tests.SqlServer/EndpointSettingsKeyCollationTests.cs index 9a081a0b2b..b611b22fc6 100644 --- a/src/ServiceControl.Persistence.Tests.SqlServer/EndpointSettingsKeyCollationTests.cs +++ b/src/ServiceControl.Persistence.Tests.SqlServer/EndpointSettingsKeyCollationTests.cs @@ -166,34 +166,4 @@ public async Task The_key_columns_own_collation_decides_a_merge_when_the_databas "the dry run predicts merges with this comparer, so it follows the same column the statement does"); } } - - // RecordingLoggerProvider lives in ServiceControl.Infrastructure.Tests, which this project does not reference. - sealed class TargetWarnings : ILoggerProvider - { - readonly System.Collections.Concurrent.ConcurrentQueue<string> messages = new(); - - public IReadOnlyList<string> Messages => [.. messages]; - - public ILogger CreateLogger(string categoryName) => - categoryName == typeof(EFCoreMigrationTarget).FullName ? new WarningLogger(messages) : Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance; - - public void Dispose() - { - } - - sealed class WarningLogger(System.Collections.Concurrent.ConcurrentQueue<string> messages) : ILogger - { - public IDisposable BeginScope<TState>(TState state) where TState : notnull => null; - - public bool IsEnabled(LogLevel logLevel) => logLevel == LogLevel.Warning; - - public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception exception, Func<TState, Exception, string> formatter) - { - if (logLevel == LogLevel.Warning) - { - messages.Enqueue(formatter(state, exception)); - } - } - } - } } diff --git a/src/ServiceControl.Persistence.Tests.SqlServer/LicensingThroughputKeyCollationTests.cs b/src/ServiceControl.Persistence.Tests.SqlServer/LicensingThroughputKeyCollationTests.cs new file mode 100644 index 0000000000..c10838be08 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests.SqlServer/LicensingThroughputKeyCollationTests.cs @@ -0,0 +1,138 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Storage; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using Particular.LicensingComponent.Contracts; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.DataMigration.Writers; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Infrastructure; + +class LicensingThroughputKeyCollationTests : PersistenceTestBase +{ + [Test] + public async Task Days_of_endpoints_an_accent_insensitive_key_treats_as_one_are_added_within_a_batch_and_across_batches() + { + await OpenAnAccentInsensitiveTarget(); + await LicensingDataStore.SaveEndpoint(new Endpoint("Café", ThroughputSource.Broker) { SanitizedName = "cafe" }); + var day = new DateOnly(2026, 1, 1); + + var first = await Write( + new MigrationCheckpoint(MigrationCategoryIds.LicensingThroughput, MigrationCategoryState.InProgress, null, 0, 0, null, null, null, null, null, null), + ("Café/Broker|2026-01-01", "Café", day, 100), + ("Cafe/Broker|2026-01-01", "Cafe", day, 250)); + await Write(first.Saved, ("CAFE/Broker|2026-01-01", "CAFE", day, 50)); + + long[] stored; + await using (var scope = ServiceProvider.CreateAsyncScope()) + { + stored = await scope.ServiceProvider.GetRequiredService<ServiceControlDbContext>().LicensingEndpointThroughput.Select(row => row.MessageCount).ToArrayAsync(); + } + + using (Assert.EnterMultipleScope()) + { + Assert.That(first.Copied, Is.EqualTo(1), "the column's collation makes the two names one day, which the statement writes once"); + Assert.That(first.AlreadyPresent, Is.EqualTo(1), "the day folded into the other is counted as already present and summed into the kept row"); + Assert.That(stored, Is.EqualTo(new[] { 400L }), "every count for the day is added under the key column's own collation, whichever batch it came in"); + } + } + + [Test] + public async Task Endpoints_an_accent_insensitive_key_treats_as_one_are_reported_as_a_merge() + { + await OpenAnAccentInsensitiveTarget(); + var batch = new MigrationBatch( + [ + new MigrationRow("Café/Broker", new Endpoint("Café", ThroughputSource.Broker) { SanitizedName = "cafe" }, new Dictionary<string, object>()), + new MigrationRow("Cafe/Broker", new Endpoint("Cafe", ThroughputSource.Broker) { SanitizedName = "cafe" }, new Dictionary<string, object>()) + ], + "Cafe/Broker"); + + var merges = await PreparedSkips.MergesFrom(ServiceProvider, new LicensingEndpointsWriter(ServiceProvider.GetRequiredService<IMigrationSqlDialect>()), batch); + var result = await ServiceProvider.GetRequiredService<IMigrationTarget>().Write( + MigrationCategoryRegistry.Find(MigrationCategoryIds.LicensingEndpoints)!, + batch, + new MigrationCheckpoint(MigrationCategoryIds.LicensingEndpoints, MigrationCategoryState.InProgress, null, 0, 0, null, null, null, null, null, null)); + + using (Assert.EnterMultipleScope()) + { + Assert.That(merges.Select(merge => merge.SourceId), Is.EqualTo(new[] { "Café/Broker" }), "the copy keeps the endpoint whose source id sorts first, and the column folds the accent, so the other is a merge"); + Assert.That(merges.Single().Detail, Does.Contain("kept 'cafe', 'Broker' and dropped the row for 'café', 'Broker'")); + Assert.That((result.Copied, result.AlreadyPresent, result.Merged), Is.EqualTo((1, 1, 1)), "every row the insert drops on a first copy is a merge the log names"); + } + } + + [TestCase("Café", "Cafe")] + [TestCase("cafe", "cafe")] + [TestCase("CAFE", "cafe")] + [TestCase("cafe ", "cafe")] + [TestCase("cafe", "cafes")] + public async Task The_key_comparer_folds_exactly_what_the_columns_collation_folds(string first, string second) + { + await OpenAnAccentInsensitiveTarget(); + bool sqlServerFolds; + + await using (var scope = ServiceProvider.CreateAsyncScope()) + { + sqlServerFolds = await scope.ServiceProvider.GetRequiredService<ServiceControlDbContext>().Database + .SqlQuery<int>($"SELECT CASE WHEN {first} = {second} COLLATE Latin1_General_CI_AI THEN 1 ELSE 0 END AS [Value]") + .SingleAsync() == 1; + } + + var comparer = ServiceProvider.GetRequiredService<IMigrationSqlDialect>().KeyComparer(typeof(LicensingEndpointEntity), nameof(LicensingEndpointEntity.NormalizedName)); + + Assert.That(comparer.Equals(first, second), Is.EqualTo(sqlServerFolds), $"the merge lines and the dry run predict with this comparer, so it must agree with SQL Server on '{first}' and '{second}'"); + } + + // Set on both tables, because the writer looks up each day's endpoint by name before it writes, and the test server's default respects accents. + async Task OpenAnAccentInsensitiveTarget() + { + await using (var scope = ServiceProvider.CreateAsyncScope()) + { + var dbContext = scope.ServiceProvider.GetRequiredService<ServiceControlDbContext>(); + var endpoints = TableOf<LicensingEndpointEntity>(dbContext); + var throughput = TableOf<LicensingEndpointThroughputEntity>(dbContext); + + var ignoreAccents = $""" + ALTER TABLE {throughput} DROP CONSTRAINT [FK_LicensingEndpointThroughput_LicensingEndpoints_NormalizedName_ThroughputSource]; + ALTER TABLE {throughput} DROP CONSTRAINT [PK_LicensingEndpointThroughput]; + ALTER TABLE {endpoints} DROP CONSTRAINT [PK_LicensingEndpoints]; + ALTER TABLE {endpoints} ALTER COLUMN [NormalizedName] nvarchar(300) COLLATE Latin1_General_CI_AI NOT NULL; + ALTER TABLE {throughput} ALTER COLUMN [NormalizedName] nvarchar(300) COLLATE Latin1_General_CI_AI NOT NULL; + ALTER TABLE {endpoints} ADD CONSTRAINT [PK_LicensingEndpoints] PRIMARY KEY ([NormalizedName], [ThroughputSource]); + ALTER TABLE {throughput} ADD CONSTRAINT [PK_LicensingEndpointThroughput] PRIMARY KEY ([NormalizedName], [ThroughputSource], [DateUtc]); + ALTER TABLE {throughput} ADD CONSTRAINT [FK_LicensingEndpointThroughput_LicensingEndpoints_NormalizedName_ThroughputSource] + FOREIGN KEY ([NormalizedName], [ThroughputSource]) REFERENCES {endpoints} ([NormalizedName], [ThroughputSource]) ON DELETE CASCADE; + """; + + await dbContext.Database.ExecuteSqlRawAsync(ignoreAccents); + } + + await ServiceProvider.GetRequiredService<IMigrationTarget>().Open(); + } + + Task<MigrationWriteResult> Write(MigrationCheckpoint checkpointToExtend, params (string SourceId, string Name, DateOnly Day, long Count)[] days) + { + var category = MigrationCategoryRegistry.All.Single(entry => entry.Id == MigrationCategoryIds.LicensingThroughput); + var batch = new MigrationBatch( + [.. days.Select(day => new MigrationRow(day.SourceId, new ThroughputDayRow(new EndpointIdentifier(day.Name, ThroughputSource.Broker), day.Day, day.Count), new Dictionary<string, object>()))], + days[^1].SourceId); + + return ServiceProvider.GetRequiredService<IMigrationTarget>().Write(category, batch, checkpointToExtend); + } + + // Named from the model, because the table sits in the configured schema when the persister has one. + static string TableOf<TEntity>(ServiceControlDbContext dbContext) + { + var entityType = dbContext.Model.FindEntityType(typeof(TEntity)); + return dbContext.GetService<ISqlGenerationHelper>().DelimitIdentifier(entityType.GetTableName(), entityType.GetSchema()); + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/CustomCheckRowMapperTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/CustomCheckRowMapperTests.cs new file mode 100644 index 0000000000..b23deff4cf --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/CustomCheckRowMapperTests.cs @@ -0,0 +1,30 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.Contracts.CustomChecks; +using ServiceControl.Operations; +using ServiceControl.Persistence.EFCore.DataMigration; +using ServiceControl.Persistence.EFCore.DbContexts; + +class CustomCheckRowMapperTests : PersistenceTestBase +{ + [Test] + public void Sets_every_mapped_property_of_a_custom_check() + { + var entity = CustomCheckRowMapper.Map(new CustomCheckDetail + { + OriginatingEndpoint = new EndpointDetails { Name = "Sales", Host = "MACHINE", HostId = Guid.NewGuid() }, + CustomCheckId = "Sales.HealthCheck", + Category = "Ping", + ReportedAt = new DateTime(2026, 9, 1, 8, 0, 0, DateTimeKind.Utc), + // Passing, because Status.Fail is the enum's zero and would read as unset. + HasFailed = false, + FailureReason = "recovered after a timeout" + }); + + using var scope = ServiceProvider.CreateScope(); + MigrationEntityCoverage.AssertEveryMappedPropertyIsSet(scope.ServiceProvider.GetRequiredService<ServiceControlDbContext>().Model, entity); + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/CustomChecksMigrationTargetTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/CustomChecksMigrationTargetTests.cs new file mode 100644 index 0000000000..4927d099d0 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/CustomChecksMigrationTargetTests.cs @@ -0,0 +1,142 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.Contracts.CustomChecks; +using ServiceControl.Operations; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.DataMigration.Writers; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Infrastructure; +using ServiceControl.Persistence.Infrastructure; + +class CustomChecksMigrationTargetTests : PersistenceTestBase +{ + [SetUp] + public Task OpenTarget() => Target.Open(); + + IMigrationTarget Target => ServiceProvider.GetRequiredService<IMigrationTarget>(); + + static readonly MigrationCategory Category = MigrationCategoryRegistry.Find(MigrationCategoryIds.CustomChecks)!; + + static readonly EndpointDetails Endpoint = new() { Name = "Sales", Host = "MACHINE", HostId = Guid.NewGuid() }; + + [Test] + public async Task A_check_the_target_already_holds_is_counted_as_already_present_and_keeps_its_stored_status() + { + var first = await Target.Write(Category, BatchOf(("CustomChecks/1", Detail(failed: false))), CheckpointAfter("CustomChecks/1")); + var second = await Target.Write(Category, BatchOf(("CustomChecks/1", Detail(failed: true))), first.Saved); + + var stored = (await CustomChecks.GetStats(new PagingInfo())).Results!; + + using (Assert.EnterMultipleScope()) + { + Assert.That(second.Copied, Is.Zero); + Assert.That(second.AlreadyPresent, Is.EqualTo(1), "a replayed batch finds its row already there, where a plain insert would fail the batch on the duplicate key"); + Assert.That(stored.Single().Status, Is.EqualTo(Status.Pass), "the copy never replaces a row the target already holds"); + Assert.That(await Target.Count(Category), Is.EqualTo(1)); + } + } + + [TestCase(nameof(CustomCheckEntity.CustomCheckId))] + [TestCase(nameof(CustomCheckEntity.Category))] + [TestCase(nameof(CustomCheckEntity.OriginatingEndpointHost))] + public async Task A_check_with_no_value_for_a_required_column_is_skipped_as_a_required_value_missing_and_the_rest_copy(string column) + { + var missing = column switch + { + nameof(CustomCheckEntity.CustomCheckId) => Detail(failed: true, checkId: null), + nameof(CustomCheckEntity.Category) => Detail(failed: true, category: null), + _ => Detail(failed: true, endpoint: new EndpointDetails { Name = "Sales", Host = null, HostId = Guid.NewGuid() }) + }; + var batch = BatchOf(("CustomChecks/1", missing), ("CustomChecks/2", Detail(failed: false, checkId: "Sales.DiskCheck"))); + + var details = await PreparedSkips.DetailsFrom(ServiceProvider, new CustomChecksWriter(ServiceProvider.GetRequiredService<IMigrationSqlDialect>()), batch); + var result = await Target.Write(Category, batch, CheckpointAfter("CustomChecks/2")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1), "a skip copies the rest of the batch, where a NOT NULL failure would fail the whole insert"); + Assert.That(result.SkipReasons, Is.EqualTo(new Dictionary<MigrationSkipReason, long> { [MigrationSkipReason.RequiredValueMissing] = 1 })); + Assert.That(details, Is.EqualTo(new[] { $"CustomChecks.{column} is NOT NULL and the document has no value for it" })); + } + } + + [Test] + public async Task A_row_the_source_flags_as_a_storage_check_is_skipped_as_harmless_whatever_its_id() + { + var batch = new MigrationBatch( + [ + new MigrationRow("CustomChecks/1", Detail(failed: false, checkId: "Some.Other.Source.Check"), Flag(true)), + new MigrationRow("CustomChecks/2", Detail(failed: false, checkId: "Sales.DiskCheck"), new Dictionary<string, object>()) + ], "CustomChecks/2"); + + var details = await PreparedSkips.DetailsFrom(ServiceProvider, new CustomChecksWriter(ServiceProvider.GetRequiredService<IMigrationSqlDialect>()), batch); + var result = await Target.Write(Category, batch, CheckpointAfter("CustomChecks/2")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(result.SkipReasons, Is.EqualTo(new Dictionary<MigrationSkipReason, long> { [MigrationSkipReason.SourceStorageCheck] = 1 })); + Assert.That(result.SkipReasons.Keys.All(reason => reason.IsBenign()), Is.True); + Assert.That(details, Is.EqualTo(new[] { "'Some.Other.Source.Check' under 'Ping' is a check the source's own storage runs about itself, which nothing reports once the instance runs on the target" })); + } + } + + [Test] + public async Task A_row_with_a_RavenDB_persister_id_and_category_but_no_flag_is_copied() + { + var result = await Target.Write(Category, BatchOf(("CustomChecks/1", Detail(failed: false, checkId: "Error Database Index Lag", category: "ServiceControl Health"))), CheckpointAfter("CustomChecks/1")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1), "the target holds no list of ids, so only the source's flag can skip a row"); + Assert.That(result.SkipReasons, Is.Null); + } + } + + [Test] + public void A_flag_that_is_not_a_bool_fails_the_batch_naming_the_row() + { + var batch = new MigrationBatch([new MigrationRow("CustomChecks/1", Detail(failed: false), Flag("true"))], "CustomChecks/1"); + + var exception = Assert.ThrowsAsync<InvalidOperationException>(() => Target.Write(Category, batch, CheckpointAfter("CustomChecks/1"))); + + Assert.That(exception.Message, Does.Contain("'CustomChecks/1'").And.Contain("System.String")); + } + + [Test] + public async Task A_row_whose_storage_check_flag_is_false_is_copied() + { + var batch = new MigrationBatch([new MigrationRow("CustomChecks/1", Detail(failed: false), Flag(false))], "CustomChecks/1"); + + var result = await Target.Write(Category, batch, CheckpointAfter("CustomChecks/1")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(result.SkipReasons, Is.Null); + } + } + + static Dictionary<string, object> Flag(object value) => new() { [MigrationMetadataKeys.SourceStorageCheck] = value }; + + static CustomCheckDetail Detail(bool failed, string checkId = "Sales.HealthCheck", string category = "Ping", EndpointDetails endpoint = null) => new() + { + OriginatingEndpoint = endpoint ?? Endpoint, + CustomCheckId = checkId, + Category = category, + ReportedAt = new DateTime(2026, 9, 1, 8, 0, 0, DateTimeKind.Utc), + HasFailed = failed + }; + + static MigrationBatch BatchOf(params (string Id, CustomCheckDetail Document)[] rows) => + new([.. rows.Select(row => new MigrationRow(row.Id, row.Document, new Dictionary<string, object>()))], rows[^1].Id); + + static MigrationCheckpoint CheckpointAfter(string cursor) => + new(MigrationCategoryIds.CustomChecks, MigrationCategoryState.InProgress, cursor, 0, 0, null, null, null, null, null, null); +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedErrorImportRowMapperTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedErrorImportRowMapperTests.cs new file mode 100644 index 0000000000..eb0bcbfab4 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedErrorImportRowMapperTests.cs @@ -0,0 +1,33 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.Operations; +using ServiceControl.Persistence.EFCore.DataMigration; +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Entities; + +class FailedErrorImportRowMapperTests : PersistenceTestBase +{ + [Test] + public async Task Sets_every_mapped_property_of_a_failed_error_import() + { + var import = new FailedErrorImport + { + Id = "native-1", + Message = new FailedTransportMessage { Id = "native-1", Headers = new Dictionary<string, string> { ["NServiceBus.MessageId"] = "native-1" }, Body = [1, 2, 3] }, + ExceptionInfo = "Deserialization failed" + }; + var failedAt = new DateTime(2026, 9, 1, 8, 0, 0, DateTimeKind.Utc); + + await using var scope = ServiceProvider.CreateAsyncScope(); + var model = scope.ServiceProvider.GetRequiredService<ServiceControlDbContext>().Model; + + // Inline leaves the flag false and external leaves the column empty, so each call covers the one the other cannot. + MigrationEntityCoverage.AssertEveryMappedPropertyIsSet(model, FailedErrorImportRowMapper.Map(import, failedAt, bodyStoredExternally: false), nameof(FailedErrorImportEntity.BodyStoredExternally)); + MigrationEntityCoverage.AssertEveryMappedPropertyIsSet(model, FailedErrorImportRowMapper.Map(import, failedAt, bodyStoredExternally: true), nameof(FailedErrorImportEntity.Body)); + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedErrorImportsMigrationTargetTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedErrorImportsMigrationTargetTests.cs new file mode 100644 index 0000000000..06024a2453 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedErrorImportsMigrationTargetTests.cs @@ -0,0 +1,133 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.Operations; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.Abstractions; +using ServiceControl.Persistence.EFCore.DataMigration.Writers; +using ServiceControl.Persistence.EFCore.Infrastructure; + +class FailedErrorImportsMigrationTargetTests : PersistenceTestBase +{ + [SetUp] + public Task OpenTarget() => Target.Open(); + + IMigrationTarget Target => ServiceProvider.GetRequiredService<IMigrationTarget>(); + + static readonly MigrationCategory Category = MigrationCategoryRegistry.Find(MigrationCategoryIds.FailedErrorImports)!; + + // Not a whole microsecond, because PostgreSQL keeps only microseconds and a re-read must still match what it stored. + static readonly DateTime FirstFailure = new DateTime(2026, 9, 1, 8, 0, 0, DateTimeKind.Utc).AddTicks(1234567); + + [Test] + public async Task A_second_import_of_the_same_message_in_a_batch_is_reported_as_a_merge_naming_the_document_kept() + { + var batch = BatchOf(Row("FailedErrorImports/a", FirstFailure), Row("FailedErrorImports/b", FirstFailure.AddMinutes(5))); + + var merges = await PreparedSkips.MergesFrom(ServiceProvider, Writer(), batch); + var result = await Target.Write(Category, batch, CheckpointAfter("FailedErrorImports/b")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(merges.Select(merge => merge.SourceId), Is.EqualTo(new[] { "FailedErrorImports/b" }), "6.18 and earlier stored each failed import attempt under its own document, and both derive one key"); + Assert.That(merges.Single().Detail, Does.Contain("kept the row from 'FailedErrorImports/a'"), "the target logs this line, so it has to name the document it kept"); + Assert.That((result.Copied, result.AlreadyPresent), Is.EqualTo((1, 1))); + } + } + + [Test] + public async Task An_import_of_a_message_an_earlier_batch_copied_from_another_document_is_reported_as_a_merge() + { + var first = await Target.Write(Category, BatchOf(Row("FailedErrorImports/a", FirstFailure)), CheckpointAfter("FailedErrorImports/a")); + var later = BatchOf(Row("FailedErrorImports/b", FirstFailure.AddMinutes(5))); + + var merges = await PreparedSkips.MergesFrom(ServiceProvider, Writer(), later); + var result = await Target.Write(Category, later, first.Saved); + + using (Assert.EnterMultipleScope()) + { + Assert.That(merges.Select(merge => merge.SourceId), Is.EqualTo(new[] { "FailedErrorImports/b" }), "random document ids spread the imports of one message over different batches"); + Assert.That(merges.Single().Detail, Does.Contain("already holds an import of the same message")); + Assert.That(result.AlreadyPresent, Is.EqualTo(1)); + } + } + + [Test] + public async Task A_document_an_earlier_pass_copied_is_already_present_and_not_a_merge() + { + var batch = BatchOf(Row("FailedErrorImports/a", FirstFailure)); + var first = await Target.Write(Category, batch, CheckpointAfter("FailedErrorImports/a")); + + var merges = await PreparedSkips.MergesFrom(ServiceProvider, Writer(), batch); + var again = await Target.Write(Category, batch, first.Saved); + + using (Assert.EnterMultipleScope()) + { + Assert.That(merges, Is.Empty, "--migration-retry re-reads the same documents, and finding its own rows again is not a merge"); + Assert.That(again.AlreadyPresent, Is.EqualTo(1)); + } + } + + [Test] + public async Task An_import_with_no_exception_or_no_body_is_reported_as_a_change_once() + { + var batch = BatchOf( + ImportRow("FailedErrorImports/no-exception", "native-1", exceptionInfo: null, [1]), + ImportRow("FailedErrorImports/no-body", "native-2", "Deserialization failed", body: null), + ImportRow("FailedErrorImports/neither", "native-3", exceptionInfo: null, body: null), + ImportRow("FailedErrorImports/complete", "native-4", "Deserialization failed", [1]), + ImportRow("FailedErrorImports/no-exception-again", "native-1", exceptionInfo: null, [1])); + + var changes = await PreparedSkips.ChangesFrom(ServiceProvider, Writer(), batch); + var first = await Target.Write(Category, batch, CheckpointAfter("FailedErrorImports/no-exception-again")); + var again = await Target.Write(Category, batch, first.Saved); + + using (Assert.EnterMultipleScope()) + { + Assert.That(changes, Is.EqualTo(new[] + { + ("FailedErrorImports/no-exception", "Stored an empty exception, because the document has none."), + ("FailedErrorImports/no-body", "Stored an empty body, because the document's message has none."), + ("FailedErrorImports/neither", "Stored an empty exception and an empty body, because the document has neither.") + }), "a complete import changes nothing, and the second import of one message is a merge, not a change"); + Assert.That((first.Changed, again.Changed), Is.EqualTo((3, 0)), "a re-read finds the rows an earlier pass copied and changes nothing"); + } + } + + static MigrationRow ImportRow(string documentId, string messageId, string exceptionInfo, byte[] body) => new( + documentId, + new FailedErrorImport + { + Id = documentId, + Message = new FailedTransportMessage { Id = messageId, Headers = new Dictionary<string, string> { ["NServiceBus.MessageId"] = messageId }, Body = body }, + ExceptionInfo = exceptionInfo + }, + new Dictionary<string, object> { [MigrationMetadataKeys.FailedAt] = FirstFailure }, + new MigrationBody(body ?? [], "application/octet-stream")); + + FailedErrorImportsWriter Writer() => new( + ServiceProvider.GetRequiredService<IMigrationSqlDialect>(), + ServiceProvider.GetRequiredService<IBodyStoragePersistence>(), + ServiceProvider.GetRequiredService<EFPersisterSettings>()); + + static MigrationRow Row(string documentId, DateTime failedAt) => new( + documentId, + new FailedErrorImport + { + Id = documentId, + Message = new FailedTransportMessage { Id = "native-8", Headers = new Dictionary<string, string> { ["NServiceBus.MessageId"] = "native-8" }, Body = [1, 2, 3] }, + ExceptionInfo = "Deserialization failed" + }, + new Dictionary<string, object> { [MigrationMetadataKeys.FailedAt] = failedAt }, + new MigrationBody(new byte[] { 1, 2, 3 }, "application/octet-stream")); + + static MigrationBatch BatchOf(params MigrationRow[] rows) => new([.. rows], rows[^1].SourceId); + + static MigrationCheckpoint CheckpointAfter(string cursor) => + new(MigrationCategoryIds.FailedErrorImports, MigrationCategoryState.InProgress, cursor, 0, 0, null, null, null, null, null, null); +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedMessageMigrationCoverageTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedMessageMigrationCoverageTests.cs index 59afd305ea..b273ece28a 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedMessageMigrationCoverageTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedMessageMigrationCoverageTests.cs @@ -45,7 +45,12 @@ public async Task Every_retry_claim_column_is_set_for_a_copied_retry_issued_mess { var target = ServiceProvider.GetRequiredService<IMigrationTarget>(); var category = MigrationCategoryRegistry.All.Single(entry => entry.Id == MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages); - var source = FullyPopulatedRow() with { Body = new MigrationBody(Encoding.UTF8.GetBytes("<order>1</order>"), "text/xml") }; + var row = FullyPopulatedRow(); + var source = row with + { + Metadata = new Dictionary<string, object?>(row.Metadata) { [MigrationMetadataKeys.RetryClaimHeld] = true }, + Body = new MigrationBody(Encoding.UTF8.GetBytes("<order>1</order>"), "text/xml") + }; ((FailedMessage)source.Document).Status = FailedMessageStatus.RetryIssued; await target.Open(); diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedMessagesMigrationTargetTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedMessagesMigrationTargetTests.cs index 576d0e0cdb..814f5565a3 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedMessagesMigrationTargetTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/FailedMessagesMigrationTargetTests.cs @@ -22,6 +22,10 @@ namespace ServiceControl.Persistence.Tests; class FailedMessagesMigrationTargetTests : PersistenceTestBase { + readonly TargetWarnings targetWarnings = new(); + + public FailedMessagesMigrationTargetTests() => RegisterServices = services => services.AddSingleton<ILoggerProvider>(targetWarnings); + [SetUp] public Task OpenTarget() => Target.Open(); @@ -260,6 +264,221 @@ public async Task A_message_or_group_value_longer_than_its_column_skips_the_mess } } + [Test] + public async Task A_change_is_not_logged_when_the_batch_does_not_commit() + { + var longType = new IngestedFailure { MessageType = new string('m', 600) }; + await Target.Write(FailedMessagesCategory, BatchOf(new IngestedFailure().ToFailedMessage()), Checkpoint()); + + Assert.ThrowsAsync<MigrationCheckpointConflictException>(() => Target.Write(FailedMessagesCategory, BatchOf(longType.ToFailedMessage()), Checkpoint()), + "the checkpoint it extends is older than the stored one, so the save in the batch's transaction is refused"); + Assert.That(targetWarnings.Messages, Has.None.StartsWith("Changed "), "a change line written before the commit would name a row SQL never stored"); + } + + [Test] + public async Task A_second_document_with_the_same_message_id_reports_no_change_of_its_own() + { + var first = new IngestedFailure(); + var second = new IngestedFailure { MessageType = new string('m', 600) }.ToFailedMessage(); + second.UniqueMessageId = first.UniqueMessageIdString; + var batch = BatchOf(first.ToFailedMessage(), second); + + var changes = await PreparedSkips.ChangesFrom(ServiceProvider, Writer(), batch); + + Assert.That(changes, Is.Empty, "the insert keeps the first document, so the second's cut message type never reaches SQL"); + } + + [Test] + public async Task A_message_type_longer_than_its_column_is_copied_with_a_change_naming_its_length() + { + var longType = new IngestedFailure { MessageType = new string('m', 600) }; + var batch = BatchOf(longType.ToFailedMessage()); + + var changes = await PreparedSkips.ChangesFrom(ServiceProvider, Writer(), batch); + var result = await Target.Write(FailedMessagesCategory, batch, Checkpoint()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(changes, Is.EqualTo(new[] { ($"FailedMessages/{longType.UniqueMessageIdString}", "Cut the message type from 600 characters to its last 450, as ingestion does.") })); + Assert.That(result.Changed, Is.EqualTo(1)); + } + } + + [Test] + public async Task A_failure_group_with_no_id_or_no_title_is_reported_as_a_change_naming_the_group() + { + var failure = new IngestedFailure + { + Groups = + [ + new() { Id = "group-kept", Title = null, Type = "Exception Type" }, + new() { Id = null!, Title = "Has no id", Type = "Message Type" }, + null! + ] + }; + var batch = BatchOf(failure.ToFailedMessage()); + + var changes = await PreparedSkips.ChangesFrom(ServiceProvider, Writer(), batch); + var result = await Target.Write(FailedMessagesCategory, batch, Checkpoint()); + var groups = await Query(db => db.FailedMessageGroups.AsNoTracking().Select(group => new { group.GroupId, group.Title }).ToListAsync()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(changes, Is.EqualTo(new[] + { + ($"FailedMessages/{failure.UniqueMessageIdString}", + "Dropped the failure groups with no id, because SQL keys a group by its id: the group titled 'Has no id', an empty entry. Stored an empty title or type for the failure groups that had none: 'group-kept'.") + })); + Assert.That(result.Changed, Is.EqualTo(1)); + Assert.That(groups.Select(group => (group.GroupId, group.Title)), Is.EqualTo(new[] { ("group-kept", "") }), "the message is missing from the group with no id, which the line has to say"); + } + } + + [Test] + public async Task Two_failure_groups_of_one_message_with_one_key_are_reported_as_a_change_naming_both() + { + var failure = new IngestedFailure + { + Groups = + [ + new() { Id = "group-a", Title = "First", Type = "Exception Type" }, + new() { Id = "group-a", Title = "Second", Type = "Exception Type" } + ] + }; + var batch = BatchOf(failure.ToFailedMessage()); + + var changes = await PreparedSkips.ChangesFrom(ServiceProvider, Writer(), batch); + var result = await Target.Write(FailedMessagesCategory, batch, Checkpoint()); + var titles = await Query(db => db.FailedMessageGroups.AsNoTracking().Select(group => group.Title).ToListAsync()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(changes, Is.EqualTo(new[] + { + ($"FailedMessages/{failure.UniqueMessageIdString}", "Kept the failure group 'group-a' titled 'First' and dropped 'group-a' titled 'Second', because SQL treats their ids as one key.") + })); + Assert.That(result.Changed, Is.EqualTo(1)); + Assert.That(titles, Is.EqualTo(new[] { "First" }), "the line names the group the insert kept"); + } + } + + [Test] + public async Task A_retry_issued_message_gets_a_retry_claim_only_when_the_source_still_held_one() + { + var released = new IngestedFailure(); + var held = new IngestedFailure(); + var batch = BatchOf( + WithClaimHeld(RowOf(released.ToFailedMessage(FailedMessageStatus.RetryIssued), SmallTextBody), false), + WithClaimHeld(RowOf(held.ToFailedMessage(FailedMessageStatus.RetryIssued), SmallTextBody), true)); + + var result = await Target.Write(FailedMessagesCategory, batch, Checkpoint()); + var claims = await Query(db => db.FailedMessageRetries.AsNoTracking().Select(claim => claim.UniqueMessageId).ToListAsync()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(2), "the test proves nothing unless both messages reached the target"); + Assert.That(claims, Is.EqualTo(new[] { held.UniqueMessageId }), "a claim the source had released would block a retry by id that RavenDB would have sent"); + Assert.That(result.Changed, Is.Zero, "the claims match the source, so nothing changed on the way in"); + } + } + + [Test] + public async Task A_retry_by_id_after_the_copy_stages_the_message_whose_claim_the_source_had_released() + { + var released = new IngestedFailure(); + var held = new IngestedFailure(); + await Target.Write(FailedMessagesCategory, BatchOf( + WithClaimHeld(RowOf(released.ToFailedMessage(FailedMessageStatus.RetryIssued), SmallTextBody), false), + WithClaimHeld(RowOf(held.ToFailedMessage(FailedMessageStatus.RetryIssued), SmallTextBody), true)), Checkpoint()); + + string[] ids = [released.UniqueMessageIdString, held.UniqueMessageIdString]; + var batchId = await RetryBatchStore.CreateBatch("migration-test-session", "retry-by-id", RetryType.MultipleMessages, ids, null, DateTime.UtcNow); + await RetryBatchStore.AssignMessagesToBatch(batchId, ids); + + var staged = await RetryStagingStore.GetMessagesToStage(batchId); + + Assert.That(staged.Select(message => message.UniqueMessageId), Is.EqualTo(new[] { released.UniqueMessageIdString }), + "RavenDB would send the released message and hold back the claimed one, so after the move a retry by id must do the same"); + } + + [Test] + public void A_retry_issued_message_without_its_claim_flag_fails_the_write_naming_the_row() + { + var document = new IngestedFailure().ToFailedMessage(FailedMessageStatus.RetryIssued); + + var exception = Assert.ThrowsAsync<InvalidOperationException>(() => Target.Write(FailedMessagesCategory, BatchOf(WithClaimHeld(RowOf(document, SmallTextBody), null)), Checkpoint())); + + Assert.That(exception.Message, Does.Contain($"'FailedMessages/{document.UniqueMessageId}'").And.Contain(MigrationMetadataKeys.RetryClaimHeld), + "the source resolves the flag for every retry-issued message, so a missing one is a defect that must not pass as a held claim"); + } + + [Test] + public async Task A_retry_issued_message_whose_claim_flag_is_not_a_bool_fails_the_write_and_stores_nothing() + { + var document = new IngestedFailure().ToFailedMessage(FailedMessageStatus.RetryIssued); + var row = WithClaimHeld(RowOf(document, SmallTextBody), null); + row = row with { Metadata = new Dictionary<string, object>(row.Metadata) { [MigrationMetadataKeys.RetryClaimHeld] = "true" } }; + + var exception = Assert.ThrowsAsync<InvalidOperationException>(() => Target.Write(FailedMessagesCategory, BatchOf(row), Checkpoint())); + var claims = await Query(db => db.FailedMessageRetries.AsNoTracking().CountAsync()); + var messages = await Query(db => db.FailedMessages.AsNoTracking().CountAsync()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(exception.Message, Does.Contain($"'FailedMessages/{document.UniqueMessageId}'").And.Contain(MigrationMetadataKeys.RetryClaimHeld).And.Contain(typeof(string).FullName)); + Assert.That((claims, messages), Is.EqualTo((0, 0)), "a flag the writer cannot read must stop the batch before anything is stored, not be read as either answer"); + } + } + + [Test] + public async Task Two_retry_issued_documents_sharing_a_unique_message_id_take_the_claim_flag_of_the_first() + { + var first = new IngestedFailure(); + var second = first.ToFailedMessage(FailedMessageStatus.RetryIssued); + var batch = BatchOf( + WithClaimHeld(RowOf(first.ToFailedMessage(FailedMessageStatus.RetryIssued), SmallTextBody), false), + WithClaimHeld(RowOf(second, SmallTextBody) with { SourceId = "FailedMessages/second" }, true)); + + var result = await Target.Write(FailedMessagesCategory, batch, Checkpoint()); + var claims = await Query(db => db.FailedMessageRetries.AsNoTracking().CountAsync()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1), "the test proves nothing unless the insert kept one of the two"); + Assert.That(claims, Is.Zero, "the insert keeps the first document, whose claim the source had released, so the second's held flag must not add one"); + } + } + + [Test] + public async Task Copying_a_batch_with_a_held_claim_again_leaves_one_claim() + { + var held = new IngestedFailure(); + var batch = BatchOf(held.ToFailedMessage(FailedMessageStatus.RetryIssued)); + + var first = await Target.Write(FailedMessagesCategory, batch, Checkpoint()); + await Target.Write(FailedMessagesCategory, batch, first.Saved); + var claims = await Query(db => db.FailedMessageRetries.AsNoTracking().Select(claim => claim.UniqueMessageId).ToListAsync()); + + Assert.That(claims, Is.EqualTo(new[] { held.UniqueMessageId }), "a retry re-reads rows an earlier pass copied, and must not add a second claim or fail on the first"); + } + + [Test] + public async Task A_changed_message_the_target_already_holds_is_not_reported_again() + { + var changed = new IngestedFailure { MessageType = new string('m', 600) }; + var batch = BatchOf(changed.ToFailedMessage()); + + var first = await Target.Write(FailedMessagesCategory, batch, Checkpoint()); + var again = await Target.Write(FailedMessagesCategory, batch, first.Saved); + + using (Assert.EnterMultipleScope()) + { + Assert.That((first.Changed, again.Changed), Is.EqualTo((1, 0)), "a retry re-reads rows an earlier pass copied, and this write changed nothing"); + Assert.That(again.AlreadyPresent, Is.EqualTo(1)); + Assert.That(targetWarnings.Messages.Count(message => message.StartsWith("Changed ")), Is.EqualTo(1)); + } + } + [Test] public async Task Count_takes_in_only_unresolved_and_retry_issued_messages() { @@ -443,13 +662,36 @@ async Task<T> Query<T>(Func<ServiceControlDbContext, Task<T>> query) static MigrationBatch BatchOf(params MigrationRow[] rows) => new(rows, rows[^1].SourceId); - // Every row carries the two dates the source resolves, as a real source's rows do. - static MigrationRow RowOf(FailedMessage document, MigrationBody body) => - new($"FailedMessages/{document.UniqueMessageId}", document, new Dictionary<string, object> + // Every row carries the values the source resolves, as a real source's rows do, and a retry-issued one finds its claim still held. + static MigrationRow RowOf(FailedMessage document, MigrationBody body) + { + var metadata = new Dictionary<string, object> { [MigrationMetadataKeys.StatusChangedAt] = new DateTime(2026, 9, 1, 8, 0, 0, DateTimeKind.Utc), [MigrationMetadataKeys.LastModified] = new DateTime(2026, 9, 2, 8, 0, 0, DateTimeKind.Utc) - }, body); + }; + + if (document.Status == FailedMessageStatus.RetryIssued) + { + metadata[MigrationMetadataKeys.RetryClaimHeld] = true; + } + + return new($"FailedMessages/{document.UniqueMessageId}", document, metadata, body); + } + + // A null removes the flag, as a source that never resolved it would send the row. + static MigrationRow WithClaimHeld(MigrationRow row, bool? held) + { + var metadata = new Dictionary<string, object>(row.Metadata); + metadata.Remove(MigrationMetadataKeys.RetryClaimHeld); + + if (held is { } value) + { + metadata[MigrationMetadataKeys.RetryClaimHeld] = value; + } + + return row with { Metadata = metadata }; + } sealed class UnwritableBodyStore : IBodyStoragePersistence { @@ -462,6 +704,8 @@ sealed class UnwritableBodyStore : IBodyStoragePersistence public Task DeleteBodyIfExists(string bodyId, CancellationToken cancellationToken = default) => throw new NotSupportedException(); } + FailedMessagesWriter Writer() => new(ServiceProvider.GetRequiredService<IMigrationSqlDialect>(), BodyStore, ServiceProvider.GetRequiredService<EFPersisterSettings>()); + static MigrationCheckpoint Checkpoint() => new(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages, MigrationCategoryState.InProgress, null, 0, 0, null, null, null, null, null, null); } diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/GroupCommentRowMapperTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/GroupCommentRowMapperTests.cs new file mode 100644 index 0000000000..c4b017a7af --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/GroupCommentRowMapperTests.cs @@ -0,0 +1,22 @@ +namespace ServiceControl.Persistence.Tests; + +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.MessageFailures; +using ServiceControl.Persistence.EFCore.DataMigration; +using ServiceControl.Persistence.EFCore.DbContexts; + +class GroupCommentRowMapperTests : PersistenceTestBase +{ + [Test] + public async Task Sets_every_mapped_property() + { + await using var scope = ServiceProvider.CreateAsyncScope(); + var model = scope.ServiceProvider.GetRequiredService<ServiceControlDbContext>().Model; + + var entity = GroupCommentRowMapper.Map(new GroupComment { Id = "group-1", Comment = "Waiting on the payments team" }); + + MigrationEntityCoverage.AssertEveryMappedPropertyIsSet(model, entity); + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/GroupCommentsMigrationTargetTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/GroupCommentsMigrationTargetTests.cs new file mode 100644 index 0000000000..9f50b13692 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/GroupCommentsMigrationTargetTests.cs @@ -0,0 +1,115 @@ +namespace ServiceControl.Persistence.Tests; + +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.MessageFailures; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.DataMigration.Writers; +using ServiceControl.Persistence.EFCore.Entities; +using ServiceControl.Persistence.EFCore.Infrastructure; +using ServiceControl.Persistence.Infrastructure; + +class GroupCommentsMigrationTargetTests : ErrorIngestionTestBase +{ + [SetUp] + public Task OpenTarget() => Target.Open(); + + IMigrationTarget Target => ServiceProvider.GetRequiredService<IMigrationTarget>(); + + static readonly MigrationCategory GroupCommentsCategory = MigrationCategoryRegistry.Find(MigrationCategoryIds.GroupComments)!; + + [Test] + public async Task A_comment_whose_group_id_is_longer_than_its_column_is_skipped_as_value_too_long_and_the_rest_copy() + { + var fits = new string('g', 64); + var tooLong = new string('g', 65); + var batch = BatchOf((fits, "Waiting on the payments team"), (tooLong, "Raised with the shipping team")); + + var details = await PreparedSkips.DetailsFrom(ServiceProvider, new GroupCommentsWriter(ServiceProvider.GetRequiredService<IMigrationSqlDialect>()), batch); + var result = await Target.Write(GroupCommentsCategory, batch, CheckpointAfter(tooLong)); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1), "a group id exactly as long as its column fits"); + Assert.That(result.SkippedIds, Is.EqualTo(new[] { $"GroupComment/{tooLong}" })); + Assert.That(result.SkipReasons, Is.EqualTo(new Dictionary<MigrationSkipReason, long> { [MigrationSkipReason.ValueTooLong] = 1 }), "an id too long for its column would fail the whole insert and stop the category"); + Assert.That(await Target.Count(GroupCommentsCategory), Is.EqualTo(1), "--migration-verify reads this count"); + Assert.That(details.Single(), Does.EndWith($"takes at most 64 characters and the value holds 65, on document 'GroupComment/{tooLong}'")); + } + } + + [Test] + public async Task A_comment_an_earlier_pass_copied_keeps_its_text_and_is_counted_as_already_present() + { + var failure = new IngestedFailure(); + var group = failure.Groups.Single(); + await Ingest(failure); + + var first = await Target.Write(GroupCommentsCategory, BatchOf((group.Id, "Waiting on the payments team")), CheckpointAfter(group.Id)); + var again = await Target.Write(GroupCommentsCategory, BatchOf((group.Id, "Edited in RavenDB since")), first.Saved); + var view = (await GroupsStore.GetUnresolvedGroupsByClassifier(group.Type, null, new PagingInfo())).Single(); + + using (Assert.EnterMultipleScope()) + { + Assert.That(again.Copied, Is.Zero); + Assert.That(again.AlreadyPresent, Is.EqualTo(1), "a retry re-reads the category from the start and finds what the first pass copied"); + Assert.That((view.Id, view.Comment), Is.EqualTo((group.Id, "Waiting on the payments team")), "the copy only inserts what is missing and never changes a row"); + } + } + + [Test] + public async Task A_comment_whose_group_id_the_key_treats_as_an_earlier_ones_is_reported_as_a_merge() + { + var dialect = ServiceProvider.GetRequiredService<IMigrationSqlDialect>(); + var batch = BatchOf(("group-1", "Waiting on the payments team"), ("group-1 ", "Raised with the shipping team"), ("group-1", "Edited in RavenDB since")); + var keyIgnoresTrailingSpaces = dialect.KeyComparer(typeof(GroupCommentEntity), nameof(GroupCommentEntity.GroupId)).Equals("group-1", "group-1 "); + + var merges = await PreparedSkips.MergesFrom(ServiceProvider, new GroupCommentsWriter(dialect), batch); + var result = await Target.Write(GroupCommentsCategory, batch, CheckpointAfter("group-1")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(merges.Select(merge => merge.SourceId), Is.EqualTo(keyIgnoresTrailingSpaces ? new[] { "GroupComment/group-1 ", "GroupComment/group-1" } : ["GroupComment/group-1"]), + "an id that differs only in trailing spaces is dropped only where the key column ignores them, an exact repeat on every provider, and the log has to say so"); + Assert.That(merges.Select(merge => merge.Detail), Has.All.Contains("kept 'group-1'"), "the target logs this line, so it has to name the group id it kept"); + Assert.That((result.Copied, result.AlreadyPresent), Is.EqualTo(keyIgnoresTrailingSpaces ? (1, 2) : (2, 1)), + "on an empty target every row the insert drops is a merge, so the merges must account for every row counted as already present"); + } + } + + [Test] + public async Task A_comment_on_a_group_with_no_copied_message_is_reported_as_a_change_the_sweep_will_delete() + { + var failure = new IngestedFailure(); + var group = failure.Groups.Single(); + await Ingest(failure); + var longComment = new string('c', 150); + var batch = BatchOf((group.Id, "Waiting on the payments team"), ("group-of-archived-messages", longComment)); + + var changes = await PreparedSkips.ChangesFrom(ServiceProvider, new GroupCommentsWriter(ServiceProvider.GetRequiredService<IMigrationSqlDialect>()), batch); + var first = await Target.Write(GroupCommentsCategory, batch, CheckpointAfter("group-of-archived-messages")); + var again = await Target.Write(GroupCommentsCategory, batch, first.Saved); + + using (Assert.EnterMultipleScope()) + { + Assert.That(changes, Is.EqualTo(new[] + { + ("GroupComment/group-of-archived-messages", + "The group 'group-of-archived-messages' has no copied failed message, so the retention sweep removes this comment shortly after ServiceControl opens. " + + "Its messages are archived or resolved, are gone from the source, or were skipped by the copy. " + + $"The comment was: '{longComment}'") + }), "a comment on a group with a copied message stays, so only the other is named, with the whole comment because the sweep deletes it"); + Assert.That((first.Copied, first.Changed), Is.EqualTo((2, 1))); + Assert.That(again.Changed, Is.Zero, "a re-read finds the comment already present, and this write copied nothing"); + } + } + + static MigrationBatch BatchOf(params (string GroupId, string Comment)[] comments) => + new([.. comments.Select(comment => new MigrationRow($"GroupComment/{comment.GroupId}", new GroupComment { Id = comment.GroupId, Comment = comment.Comment }, new Dictionary<string, object>()))], $"GroupComment/{comments[^1].GroupId}"); + + static MigrationCheckpoint CheckpointAfter(string groupId) => + new(MigrationCategoryIds.GroupComments, MigrationCategoryState.InProgress, $"GroupComment/{groupId}", 0, 0, null, null, null, null, null, null); +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/LicensingEndpointsMigrationTargetTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/LicensingEndpointsMigrationTargetTests.cs index 2eb2c74e25..7b0661a1b2 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/Migration/LicensingEndpointsMigrationTargetTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/LicensingEndpointsMigrationTargetTests.cs @@ -9,6 +9,7 @@ namespace ServiceControl.Persistence.Tests; using Particular.LicensingComponent.Contracts; using ServiceControl.Persistence.DataMigration; using ServiceControl.Persistence.EFCore.DataMigration.Writers; +using ServiceControl.Persistence.EFCore.Entities; using ServiceControl.Persistence.EFCore.Infrastructure; class LicensingEndpointsMigrationTargetTests : PersistenceTestBase @@ -55,6 +56,89 @@ public async Task A_licensing_endpoint_with_a_value_longer_than_its_column_is_sk } } + [Test] + public async Task Two_names_differing_only_in_a_trailing_space_in_one_batch_are_reported_as_one_merge_where_the_key_folds_them() + { + var dialect = ServiceProvider.GetRequiredService<IMigrationSqlDialect>(); + var batch = BatchOf( + ("Sales.Orders/Broker", new Endpoint("Sales.Orders", ThroughputSource.Broker) { SanitizedName = "sales.orders" }), + ("Sales.Orders /Broker", new Endpoint("Sales.Orders ", ThroughputSource.Broker) { SanitizedName = "sales.orders" })); + var keyIgnoresTrailingSpace = KeyIgnoresTrailingSpace(dialect); + + var merges = await PreparedSkips.MergesFrom(ServiceProvider, new LicensingEndpointsWriter(dialect), batch); + + using (Assert.EnterMultipleScope()) + { + Assert.That(merges.Select(merge => merge.SourceId), Is.EqualTo(keyIgnoresTrailingSpace ? new[] { "Sales.Orders/Broker" } : Array.Empty<string>()), + "the row whose id sorts first is kept, and the other is reported only where the key folds a trailing space"); + + if (keyIgnoresTrailingSpace) + { + Assert.That(merges[0].Detail, Does.Contain("kept 'sales.orders ', 'Broker' and dropped the row for 'sales.orders', 'Broker'"), "the line names both endpoints and the one kept"); + } + } + } + + [Test] + public async Task A_name_differing_only_in_a_trailing_space_from_a_stored_endpoint_is_reported_as_a_merge_where_the_key_folds_them() + { + var dialect = ServiceProvider.GetRequiredService<IMigrationSqlDialect>(); + var first = await Target.Write(LicensingEndpointsCategory, BatchOf(("Sales.Orders/Broker", new Endpoint("Sales.Orders", ThroughputSource.Broker) { SanitizedName = "sales.orders" })), CheckpointAfter("Sales.Orders/Broker")); + var second = BatchOf(("Sales.Orders /Broker", new Endpoint("Sales.Orders ", ThroughputSource.Broker) { SanitizedName = "sales.orders" })); + var keyIgnoresTrailingSpace = KeyIgnoresTrailingSpace(dialect); + + var merges = await PreparedSkips.MergesFrom(ServiceProvider, new LicensingEndpointsWriter(dialect), second); + var result = await Target.Write(LicensingEndpointsCategory, second, first.Saved with { Cursor = second.Cursor }); + + using (Assert.EnterMultipleScope()) + { + Assert.That(merges.Select(merge => merge.SourceId), Is.EqualTo(keyIgnoresTrailingSpace ? new[] { "Sales.Orders /Broker" } : Array.Empty<string>())); + + if (keyIgnoresTrailingSpace) + { + Assert.That(merges[0].Detail, Does.Contain("kept 'sales.orders', 'Broker' and dropped the row for 'sales.orders ', 'Broker'"), "the stored endpoint is the one kept"); + } + + Assert.That(result.AlreadyPresent, Is.EqualTo(keyIgnoresTrailingSpace ? 1 : 0), "the folded endpoint is counted as already present"); + } + } + + [Test] + public async Task An_exact_replay_of_a_stored_endpoint_is_already_present_and_not_a_merge() + { + var batch = BatchOf(("Sales.Orders/Broker", new Endpoint("Sales.Orders", ThroughputSource.Broker) { SanitizedName = "sales.orders" })); + var first = await Target.Write(LicensingEndpointsCategory, batch, CheckpointAfter("Sales.Orders/Broker")); + + var merges = await PreparedSkips.MergesFrom(ServiceProvider, new LicensingEndpointsWriter(ServiceProvider.GetRequiredService<IMigrationSqlDialect>()), batch); + var again = await Target.Write(LicensingEndpointsCategory, batch, first.Saved); + + using (Assert.EnterMultipleScope()) + { + Assert.That(merges, Is.Empty, "a retry re-reads the rows an earlier pass copied, and the stored row is that same endpoint"); + Assert.That((again.Copied, again.AlreadyPresent, again.Merged), Is.EqualTo((0, 1, 0))); + } + } + + [Test] + public async Task Two_names_differing_only_in_case_are_reported_as_one_merge_on_every_provider() + { + var dialect = ServiceProvider.GetRequiredService<IMigrationSqlDialect>(); + var batch = BatchOf( + ("SALES.ORDERS/Broker", new Endpoint("SALES.ORDERS", ThroughputSource.Broker) { SanitizedName = "sales.orders" }), + ("Sales.Orders/Broker", new Endpoint("Sales.Orders", ThroughputSource.Broker) { SanitizedName = "sales.orders" })); + + var merges = await PreparedSkips.MergesFrom(ServiceProvider, new LicensingEndpointsWriter(dialect), batch); + + using (Assert.EnterMultipleScope()) + { + Assert.That(merges.Select(merge => merge.SourceId), Is.EqualTo(new[] { "Sales.Orders/Broker" }), "the product lower-cases the name, so both providers store one row"); + Assert.That(merges[0].Detail, Does.Contain("kept 'sales.orders', 'Broker' and dropped the row for 'sales.orders', 'Broker'")); + } + } + + static bool KeyIgnoresTrailingSpace(IMigrationSqlDialect dialect) => + dialect.KeyComparer(typeof(LicensingEndpointEntity), nameof(LicensingEndpointEntity.NormalizedName)).Equals("a", "a "); + static MigrationBatch BatchOf(params (string Id, object Document)[] rows) => new([.. rows.Select(row => new MigrationRow(row.Id, row.Document, new Dictionary<string, object>()))], rows[^1].Id); diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/LicensingThroughputMigrationTargetTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/LicensingThroughputMigrationTargetTests.cs index bcae1c840f..56edd2b44f 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/Migration/LicensingThroughputMigrationTargetTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/LicensingThroughputMigrationTargetTests.cs @@ -75,8 +75,8 @@ public async Task A_day_repeated_in_one_batch_keeps_its_last_row_and_counts_the_ using (Assert.EnterMultipleScope()) { Assert.That(result.Copied, Is.EqualTo(1)); - Assert.That(result.AlreadyPresent, Is.EqualTo(1), "PostgreSQL refuses a statement that updates one row twice, so the repeat is dropped before it and counted rather than failing the batch"); - Assert.That(stored, Is.EqualTo(250), "the row whose id sorts last by ordinal order sets the day, whatever order the batch arrives in"); + Assert.That(result.AlreadyPresent, Is.EqualTo(1), "the repeat is the same day of the same endpoint, so it is dropped before the statement and counted"); + Assert.That(stored, Is.EqualTo(250), "the row whose id sorts last by ordinal order sets the day rather than adding to it, whatever order the batch arrives in"); } } @@ -94,7 +94,6 @@ public async Task Days_of_two_endpoints_the_key_treats_as_one_are_added_when_the ("Sales.Orders/Broker|2026-01-01", new ThroughputDayRow(new EndpointIdentifier("Sales.Orders", ThroughputSource.Broker), day, 250))); var first = await Target.Write(LicensingThroughputCategory, batch, CheckpointAfter("Sales.Orders/Broker")); - var again = await Target.Write(LicensingThroughputCategory, batch, first.Saved); await using var scope = ServiceProvider.CreateAsyncScope(); var keyIgnoresTrailingSpace = scope.ServiceProvider.GetRequiredService<ServiceControlDbContext>().Database.ProviderName!.Contains("SqlServer"); @@ -102,28 +101,120 @@ public async Task Days_of_two_endpoints_the_key_treats_as_one_are_added_when_the using (Assert.EnterMultipleScope()) { Assert.That(first.Copied, Is.EqualTo(keyIgnoresTrailingSpace ? 1 : 2)); - Assert.That(first.AlreadyPresent, Is.EqualTo(keyIgnoresTrailingSpace ? 1 : 0), "the endpoint folded into the other's row is counted, not lost"); + Assert.That(first.AlreadyPresent, Is.EqualTo(keyIgnoresTrailingSpace ? 1 : 0), "the folded endpoint's days are summed into the kept row, and the endpoint writer reports the fold"); Assert.That(await StoredCounts(), Is.EqualTo(keyIgnoresTrailingSpace ? new[] { 350L } : new[] { 100L, 250L }), "SQL Server's key makes the two one day, which holds both counts, as the product's own throughput write would"); - Assert.That(again.Copied, Is.EqualTo(first.Copied), "copying the batch again sets the same sum rather than adding to it"); } } [Test] - public async Task A_later_batch_replaces_the_count_an_earlier_batch_stored_for_the_same_day() + public async Task Days_of_two_endpoints_differing_only_in_case_are_added_when_they_arrive_in_one_batch() + { + await LicensingDataStore.SaveEndpoint(new Endpoint("Sales.Orders", ThroughputSource.Broker) { SanitizedName = "sales.orders" }); + var day = new DateOnly(2026, 1, 1); + var batch = BatchOf( + ("Sales.Orders/Broker|2026-01-01", new ThroughputDayRow(new EndpointIdentifier("Sales.Orders", ThroughputSource.Broker), day, 100)), + ("SALES.ORDERS/Broker|2026-01-01", new ThroughputDayRow(new EndpointIdentifier("SALES.ORDERS", ThroughputSource.Broker), day, 250))); + + var result = await Target.Write(LicensingThroughputCategory, batch, CheckpointAfter("SALES.ORDERS/Broker")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(result.AlreadyPresent, Is.EqualTo(1), "the folded endpoint's days are summed into the kept row, and the endpoint writer reports the fold"); + Assert.That(await StoredCounts(), Is.EqualTo(new[] { 350L }), "two source endpoints store the same lower-cased name, so their days are added, not one kept as a repeat of the other"); + } + } + + [Test] + public async Task A_later_batch_adds_its_count_to_the_day_an_earlier_batch_stored_for_a_name_differing_only_in_case() { await LicensingDataStore.SaveEndpoint(new Endpoint("Sales.Orders", ThroughputSource.Broker) { SanitizedName = "sales.orders" }); var day = new DateOnly(2026, 1, 1); var first = await Target.Write(LicensingThroughputCategory, BatchOf(("Sales.Orders/Broker|2026-01-01", new ThroughputDayRow(new EndpointIdentifier("Sales.Orders", ThroughputSource.Broker), day, 100))), CheckpointAfter("Sales.Orders/Broker")); - var second = await Target.Write(LicensingThroughputCategory, BatchOf(("Sales.Orders/Broker|2026-01-01", new ThroughputDayRow(new EndpointIdentifier("Sales.Orders", ThroughputSource.Broker), day, 250))), first.Saved); + var second = await Target.Write(LicensingThroughputCategory, BatchOf(("SALES.ORDERS/Broker|2026-01-01", new ThroughputDayRow(new EndpointIdentifier("SALES.ORDERS", ThroughputSource.Broker), day, 250))), first.Saved); using (Assert.EnterMultipleScope()) { Assert.That(second.Copied, Is.EqualTo(1), "an update is a row the statement wrote"); - Assert.That(await StoredCounts(), Is.EqualTo(new[] { 250L }), "the count RavenDB holds now replaces the stored one, and is never added to it or ignored"); + Assert.That(await StoredCounts(), Is.EqualTo(new[] { 350L }), "both endpoints store the same lower-cased name, so the second batch adds its count to the day rather than replacing it"); } } + [Test] + public async Task A_batch_written_again_from_the_checkpoint_it_already_extended_is_refused_and_adds_nothing() + { + await LicensingDataStore.SaveEndpoint(new Endpoint("Sales.Orders", ThroughputSource.Broker) { SanitizedName = "sales.orders" }); + var batch = BatchOf(("Sales.Orders/Broker|2026-01-01", new ThroughputDayRow(new EndpointIdentifier("Sales.Orders", ThroughputSource.Broker), new DateOnly(2026, 1, 1), 100))); + var checkpointToExtend = CheckpointAfter("Sales.Orders/Broker"); + + await Target.Write(LicensingThroughputCategory, batch, checkpointToExtend); + + Assert.ThrowsAsync<MigrationCheckpointConflictException>(() => Target.Write(LicensingThroughputCategory, batch, checkpointToExtend)); + Assert.That(await StoredCounts(), Is.EqualTo(new[] { 100L }), "the add and the checkpoint commit together, so a replay of a committed batch rolls back instead of counting the day twice"); + } + + [Test] + public async Task A_retry_deletes_the_copied_days_so_copying_again_gives_the_same_totals() + { + await LicensingDataStore.SaveEndpoint(new Endpoint("Sales.Orders", ThroughputSource.Broker) { SanitizedName = "sales.orders" }); + var batch = BatchOf(("Sales.Orders/Broker|2026-01-01", new ThroughputDayRow(new EndpointIdentifier("Sales.Orders", ThroughputSource.Broker), new DateOnly(2026, 1, 1), 100))); + var first = await Target.Write(LicensingThroughputCategory, batch, CheckpointAfter("Sales.Orders/Broker")); + + var reset = await Target.ResetForRetry(ResetForRetry(first.Saved)); + var afterReset = await StoredCounts(); + await Target.Write(LicensingThroughputCategory, batch, reset); + + using (Assert.EnterMultipleScope()) + { + Assert.That(afterReset, Is.Empty, "the reset deletes the days the first copy added, because copying them again adds each count a second time"); + Assert.That(await StoredCounts(), Is.EqualTo(new[] { 100L }), "copying again after the retry gives the totals one copy gave, not double"); + } + } + + [Test] + public async Task A_stale_retry_reset_deletes_no_copied_day() + { + await LicensingDataStore.SaveEndpoint(new Endpoint("Sales.Orders", ThroughputSource.Broker) { SanitizedName = "sales.orders" }); + var batch = BatchOf(("Sales.Orders/Broker|2026-01-01", new ThroughputDayRow(new EndpointIdentifier("Sales.Orders", ThroughputSource.Broker), new DateOnly(2026, 1, 1), 100))); + var stale = (await Target.Write(LicensingThroughputCategory, batch, CheckpointAfter("Sales.Orders/Broker"))).Saved; + await ServiceProvider.GetRequiredService<IMigrationCheckpointStore>().Upsert(stale); + + Assert.ThrowsAsync<MigrationCheckpointConflictException>(() => Target.ResetForRetry(ResetForRetry(stale))); + Assert.That(await StoredCounts(), Is.EqualTo(new[] { 100L }), "the delete shares the reset's transaction, so a refused reset leaves every copied day"); + } + + [Test] + public async Task A_retry_reset_after_ServiceControl_opened_on_the_target_is_refused_and_deletes_nothing() + { + await LicensingDataStore.SaveEndpoint(new Endpoint("Sales.Orders", ThroughputSource.Broker) { SanitizedName = "sales.orders" }); + var batch = BatchOf(("Sales.Orders/Broker|2026-01-01", new ThroughputDayRow(new EndpointIdentifier("Sales.Orders", ThroughputSource.Broker), new DateOnly(2026, 1, 1), 100))); + var copied = (await Target.Write(LicensingThroughputCategory, batch, CheckpointAfter("Sales.Orders/Broker"))).Saved; + await ServiceProvider.GetRequiredService<IMigrationTargetReadiness>().RecordHostOpened(); + + var refused = Assert.ThrowsAsync<InvalidOperationException>(() => Target.ResetForRetry(ResetForRetry(copied))); + var stored = await ServiceProvider.GetRequiredService<IMigrationCheckpointStore>().Read(MigrationCategoryIds.LicensingThroughput); + + using (Assert.EnterMultipleScope()) + { + Assert.That(refused.Message, Does.Contain(MigrationCategoryIds.LicensingThroughput).And.Contain("only possible before ServiceControl first opens on this database")); + Assert.That(await StoredCounts(), Is.EqualTo(new[] { 100L }), "the days may now include counts the running instance collected, so deleting them would lose live throughput"); + Assert.That((stored!.Version, stored.State, stored.Cursor, stored.CopiedCount), Is.EqualTo((copied.Version, copied.State, copied.Cursor, copied.CopiedCount)), "a refused reset saves nothing"); + } + } + + [Test] + public async Task A_retry_reset_that_deletes_nothing_still_succeeds_after_ServiceControl_opened_on_the_target() + { + var checkpoints = ServiceProvider.GetRequiredService<IMigrationCheckpointStore>(); + var stopped = await checkpoints.Upsert(new MigrationCheckpoint(MigrationCategoryIds.ArchivedAndResolvedFailedMessages, MigrationCategoryState.InProgress, "FailedMessages/1", 1, 0, null, null, null, null, null, "RavenDB is unreachable")); + await ServiceProvider.GetRequiredService<IMigrationTargetReadiness>().RecordHostOpened(); + + var reset = await Target.ResetForRetry(ResetForRetry(stopped)); + + Assert.That((reset.State, reset.Cursor, reset.LastError), Is.EqualTo((MigrationCategoryState.NotStarted, (string)null, (string)null)), "an optional category copies after ServiceControl opens, and its reset deletes nothing, so it is still allowed"); + } + [Test] public async Task One_endpoint_with_more_days_than_one_statement_takes_stores_every_day_with_its_own_count() { @@ -157,7 +248,7 @@ public async Task One_endpoint_with_more_days_than_one_statement_takes_stores_ev } [Test] - public async Task A_later_batch_sets_a_day_stored_under_a_name_that_differs_only_by_a_trailing_space() + public async Task A_later_batch_adds_to_a_day_stored_under_a_name_that_differs_only_by_a_trailing_space() { var endpoints = MigrationCategoryRegistry.All.Single(category => category.Id == MigrationCategoryIds.LicensingEndpoints); var endpointBatch = BatchOf( @@ -175,7 +266,7 @@ public async Task A_later_batch_sets_a_day_stored_under_a_name_that_differs_only using (Assert.EnterMultipleScope()) { Assert.That(second.Copied, Is.EqualTo(1), "the second batch's day is matched back to the row that produced it, not refused because the stored name keeps its trailing space"); - Assert.That(await StoredCounts(), Is.EqualTo(keyIgnoresTrailingSpace ? new[] { 250L } : new[] { 100L, 250L }), "SQL Server's key makes the two names one day, which the later count sets; PostgreSQL's keeps two"); + Assert.That(await StoredCounts(), Is.EqualTo(keyIgnoresTrailingSpace ? new[] { 350L } : new[] { 100L, 250L }), "SQL Server's key makes the two names one day, which the later count adds to; PostgreSQL's keeps two"); } } @@ -188,6 +279,19 @@ async Task<long[]> StoredCounts() static MigrationBatch BatchOf(params (string Id, object Document)[] rows) => new([.. rows.Select(row => new MigrationRow(row.Id, row.Document, new Dictionary<string, object>()))], rows[^1].Id); + // The reset --migration-retry makes. + static MigrationCheckpoint ResetForRetry(MigrationCheckpoint checkpoint) => checkpoint with + { + State = MigrationCategoryState.NotStarted, + Cursor = null, + CopiedCount = 0, + SkippedCount = 0, + AlreadyPresentCount = 0, + SkipReasons = null, + SettledAt = null, + LastError = null + }; + static MigrationCheckpoint CheckpointAfter(string cursor) => new(MigrationCategoryIds.LicensingThroughput, MigrationCategoryState.InProgress, cursor, 0, 0, null, null, null, null, null, null); } diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/MigrationTargetCoverageTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/MigrationTargetCoverageTests.cs index 3237b5700e..3e942eace6 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/Migration/MigrationTargetCoverageTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/MigrationTargetCoverageTests.cs @@ -51,8 +51,21 @@ public async Task The_settings_value_column_has_no_maximum_length_so_no_setting_ Assert.That(model.FindEntityType(typeof(SettingEntity))!.FindProperty(nameof(SettingEntity.Value))!.GetMaxLength(), Is.Null, "NoValueCanBeTooLong exempts the settings categories because of this; give them a value-too-long test if the column gains a limit"); } + [TestCase(typeof(CustomCheckEntity))] + [TestCase(typeof(FailedErrorImportEntity))] + public async Task No_text_column_of_the_entity_has_a_maximum_length_so_no_value_can_be_too_long(Type entity) + { + await using var scope = ServiceProvider.CreateAsyncScope(); + var entityType = scope.ServiceProvider.GetRequiredService<ServiceControlDbContext>().Model.FindEntityType(entity)!; + + Assert.That( + entityType.GetProperties().Where(property => property.ClrType == typeof(string) && property.GetMaxLength() is not null).Select(property => $"{entityType.GetTableName()}.{property.GetColumnName()}"), + Is.Empty, + "NoValueCanBeTooLong exempts this entity's category because of this; give it a value-too-long test if a column gains a limit"); + } + [Test] - public async Task Every_writer_whose_rows_get_their_key_from_the_store_deletes_them_for_a_retry_and_no_other_writer_does() + public async Task Every_writer_whose_rows_a_re_read_cannot_recognise_deletes_them_for_a_retry_and_no_other_writer_does() { await using var scope = ServiceProvider.CreateAsyncScope(); var model = scope.ServiceProvider.GetRequiredService<ServiceControlDbContext>().Model; @@ -67,14 +80,14 @@ public async Task Every_writer_whose_rows_get_their_key_from_the_store_deletes_t { var storeKeyed = entities.Any(entity => model.FindEntityType(entity)!.FindPrimaryKey()!.Properties.Any(property => property.ValueGenerated != ValueGenerated.Never)); - Assert.That(typeof(IStoreKeyedCategoryWriter).IsAssignableFrom(writer), Is.EqualTo(storeKeyed), - $"{writer.Name}: a retry keeps the rows of a writer that is not an IStoreKeyedCategoryWriter, so a re-read adds rows with store keys a second time, and a writer whose keys the copy supplies must not delete"); + Assert.That(typeof(IStoreKeyedCategoryWriter).IsAssignableFrom(writer), Is.EqualTo(storeKeyed || WritersThatAddToRowsTheyCopied.Contains(writer)), + $"{writer.Name}: a retry keeps the rows of a writer that is not an IStoreKeyedCategoryWriter, so a re-read adds rows with store keys or added counts a second time, and a writer whose rows a re-read finds already present must not delete"); } } } - // Each is one Settings row under a constant key whose value column has no maximum length, so no source value can be too long for it. - static readonly string[] NoValueCanBeTooLong = [MigrationCategoryIds.LicensingReportMasks, MigrationCategoryIds.LicensedEndpointDetails, MigrationCategoryIds.NotificationSettings, MigrationCategoryIds.TrialEndDate]; + // None of these copies a source value into a text column with a maximum length, which the two no-maximum-length tests above pin. + static readonly string[] NoValueCanBeTooLong = [MigrationCategoryIds.LicensingReportMasks, MigrationCategoryIds.LicensedEndpointDetails, MigrationCategoryIds.NotificationSettings, MigrationCategoryIds.TrialEndDate, MigrationCategoryIds.CustomChecks, MigrationCategoryIds.FailedErrorImports]; // Uses nameof, so renaming or deleting one of these tests fails the build instead of leaving this list out of date. static readonly Dictionary<string, string> ValueTooLongTests = new() @@ -86,9 +99,14 @@ public async Task Every_writer_whose_rows_get_their_key_from_the_store_deletes_t [MigrationCategoryIds.LicensingThroughput] = nameof(LicensingThroughputMigrationTargetTests.A_day_whose_endpoint_name_is_longer_than_its_column_is_skipped_as_value_too_long_and_the_rest_copy), [MigrationCategoryIds.MessageRedirects] = nameof(MessageRedirectsMigrationTargetTests.A_redirect_with_an_address_longer_than_its_column_is_skipped_as_value_too_long_and_the_rest_copy), [MigrationCategoryIds.Subscriptions] = nameof(SubscriptionsMigrationTargetTests.A_subscription_with_a_value_longer_than_its_column_is_skipped_under_its_reason_and_the_rest_copy), - [MigrationCategoryIds.RetryOperations] = nameof(RetryOperationsMigrationTargetTests.A_retry_operation_with_a_value_longer_than_its_column_is_skipped_as_value_too_long_and_the_rest_copy) + [MigrationCategoryIds.RetryOperations] = nameof(RetryOperationsMigrationTargetTests.A_retry_operation_with_a_value_longer_than_its_column_is_skipped_as_value_too_long_and_the_rest_copy), + [MigrationCategoryIds.PendingIntegrationEvents] = nameof(PendingIntegrationEventsMigrationTargetTests.A_request_whose_context_type_name_is_longer_than_its_column_is_skipped_as_value_too_long_and_the_rest_copy), + [MigrationCategoryIds.GroupComments] = nameof(GroupCommentsMigrationTargetTests.A_comment_whose_group_id_is_longer_than_its_column_is_skipped_as_value_too_long_and_the_rest_copy) }; + // Their keys come from the source, but a re-read adds its counts to the stored ones, so a re-read cannot find them already present. + static readonly Type[] WritersThatAddToRowsTheyCopied = [typeof(LicensingThroughputWriter)]; + // The writers do not say which entities they write, so this list does. static readonly Dictionary<Type, Type[]> EntitiesEachWriterWrites = new() { @@ -103,6 +121,10 @@ public async Task Every_writer_whose_rows_get_their_key_from_the_store_deletes_t [typeof(SubscriptionsWriter)] = [typeof(SubscriptionEntity)], [typeof(NotificationSettingsWriter)] = [typeof(SettingEntity)], [typeof(TrialEndDateWriter)] = [typeof(SettingEntity)], - [typeof(RetryOperationsWriter)] = [typeof(HistoricRetryOperationEntity), typeof(UnacknowledgedRetryOperationEntity)] + [typeof(RetryOperationsWriter)] = [typeof(HistoricRetryOperationEntity), typeof(UnacknowledgedRetryOperationEntity)], + [typeof(PendingIntegrationEventsWriter)] = [typeof(ExternalIntegrationDispatchRequestEntity)], + [typeof(CustomChecksWriter)] = [typeof(CustomCheckEntity)], + [typeof(FailedErrorImportsWriter)] = [typeof(FailedErrorImportEntity)], + [typeof(GroupCommentsWriter)] = [typeof(GroupCommentEntity)] }; } diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/PendingIntegrationEventRowMapperTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/PendingIntegrationEventRowMapperTests.cs new file mode 100644 index 0000000000..9dfa1dd395 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/PendingIntegrationEventRowMapperTests.cs @@ -0,0 +1,44 @@ +namespace ServiceControl.Persistence.Tests; + +using System.Threading.Tasks; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using ServiceControl.ExternalIntegrations; +using ServiceControl.Persistence.EFCore.DataMigration; +using ServiceControl.Persistence.EFCore.DbContexts; + +class PendingIntegrationEventRowMapperTests : PersistenceTestBase +{ + [Test] + public void Sets_every_mapped_property_of_a_dispatch_request() + { + var entity = PendingIntegrationEventRowMapper.Map(Request("one")); + + using var scope = ServiceProvider.CreateScope(); + MigrationEntityCoverage.AssertEveryMappedPropertyIsSet(scope.ServiceProvider.GetRequiredService<ServiceControlDbContext>().Model, entity); + } + + [Test] + public async Task Writes_exactly_the_type_name_and_json_the_product_writes_for_the_same_context() + { + await ExternalIntegrationStore.StoreDispatchRequest([Request("one")]); + + using var scope = ServiceProvider.CreateScope(); + var stored = await scope.ServiceProvider.GetRequiredService<ServiceControlDbContext>().ExternalIntegrationDispatchRequests.AsNoTracking().SingleAsync(); + var mapped = PendingIntegrationEventRowMapper.Map(Request("one")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(mapped.DispatchContextTypeName, Is.EqualTo(stored.DispatchContextTypeName)); + Assert.That(mapped.DispatchContextJson, Is.EqualTo(stored.DispatchContextJson)); + } + } + + static ExternalIntegrationDispatchRequest Request(string value) => new() { DispatchContext = new TestDispatchContext { Value = value } }; + + public class TestDispatchContext + { + public string Value { get; set; } + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/PendingIntegrationEventsMigrationTargetTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/PendingIntegrationEventsMigrationTargetTests.cs new file mode 100644 index 0000000000..f3ca3ea132 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/PendingIntegrationEventsMigrationTargetTests.cs @@ -0,0 +1,120 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Reflection; +using System.Reflection.Emit; +using System.Threading.Tasks; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using NUnit.Framework; +using ServiceControl.ExternalIntegrations; +using ServiceControl.Persistence.DataMigration; +using ServiceControl.Persistence.EFCore.DbContexts; +using ServiceControl.Persistence.EFCore.Implementation; + +class PendingIntegrationEventsMigrationTargetTests : PersistenceTestBase +{ + readonly TargetWarnings targetLog = new(); + + public PendingIntegrationEventsMigrationTargetTests() => RegisterServices = services => services.AddSingleton<ILoggerProvider>(targetLog); + + [SetUp] + public Task OpenTarget() => Target.Open(); + + IMigrationTarget Target => ServiceProvider.GetRequiredService<IMigrationTarget>(); + + ExternalIntegrationRequestsDataStore Dispatcher => ServiceProvider.GetServices<IHostedService>().OfType<ExternalIntegrationRequestsDataStore>().Single(); + + static readonly MigrationCategory Category = MigrationCategoryRegistry.Find(MigrationCategoryIds.PendingIntegrationEvents)!; + + [Test] + public async Task A_copied_request_is_dispatched_by_the_product_as_its_concrete_type_and_then_removed() + { + var dispatched = Array.Empty<object>(); + ExternalIntegrationStore.Subscribe((contexts, _) => + { + dispatched = contexts; + return Task.CompletedTask; + }); + + var result = await Write(("ExternalIntegrationDispatchRequests/1", new ExternalIntegrationDispatchRequest { DispatchContext = new TestDispatchContext { Value = "copied" } })); + + await Dispatcher.DispatchNow(TestContext.CurrentContext.CancellationToken); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(((TestDispatchContext)dispatched.Single()).Value, Is.EqualTo("copied")); + Assert.That(await Rows(), Is.Zero, "the dispatcher deletes what it delivered, which is delivery rather than loss"); + } + } + + [Test] + public async Task A_request_with_no_context_is_skipped_as_a_required_value_missing() + { + var result = await Write( + ("ExternalIntegrationDispatchRequests/1", new ExternalIntegrationDispatchRequest { DispatchContext = null! }), + ("ExternalIntegrationDispatchRequests/2", new ExternalIntegrationDispatchRequest { DispatchContext = new TestDispatchContext { Value = "kept" } })); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(result.SkipReasons[MigrationSkipReason.RequiredValueMissing], Is.EqualTo(1), "both columns are NOT NULL, and a throw would stop the category at this batch"); + Assert.That(await Target.Count(Category), Is.EqualTo(1), "Count is what --migration-verify reads, so it must see the row the copy wrote"); + } + } + + [Test] + public async Task A_request_whose_context_type_name_is_longer_than_its_column_is_skipped_as_value_too_long_and_the_rest_copy() + { + // The stored type name ends with the assembly name, so a long assembly name makes it longer than the column. + var assembly = AssemblyBuilder.DefineDynamicAssembly(new AssemblyName(new string('a', 460)), AssemblyBuilderAccess.Run); + var longNamedContext = Activator.CreateInstance(assembly.DefineDynamicModule("Contexts").DefineType("Context", TypeAttributes.Public).CreateType())!; + + var result = await Write( + ("ExternalIntegrationDispatchRequests/1", new ExternalIntegrationDispatchRequest { DispatchContext = longNamedContext }), + ("ExternalIntegrationDispatchRequests/2", new ExternalIntegrationDispatchRequest { DispatchContext = new TestDispatchContext { Value = "kept" } })); + + using (Assert.EnterMultipleScope()) + { + Assert.That(result.Copied, Is.EqualTo(1)); + Assert.That(result.SkippedIds, Is.EqualTo(new[] { "ExternalIntegrationDispatchRequests/1" })); + Assert.That(result.SkipReasons[MigrationSkipReason.ValueTooLong], Is.EqualTo(1), "one value longer than its column fails the whole statement, so it is skipped instead"); + Assert.That(await Target.Count(Category), Is.EqualTo(1)); + } + } + + [Test] + public async Task The_first_batch_that_copies_events_says_once_that_the_database_gives_them_new_ids() + { + var first = await Write(("ExternalIntegrationDispatchRequests/1", new ExternalIntegrationDispatchRequest { DispatchContext = new TestDispatchContext { Value = "first" } })); + await Target.Write( + Category, + new MigrationBatch([new MigrationRow("ExternalIntegrationDispatchRequests/2", new ExternalIntegrationDispatchRequest { DispatchContext = new TestDispatchContext { Value = "second" } }, new Dictionary<string, object>())], "ExternalIntegrationDispatchRequests/2"), + first.Saved); + + Assert.That(targetLog.Information, Has.Exactly(1).EqualTo($"Category {MigrationCategoryIds.PendingIntegrationEvents}: the database gives the copied pending integration events new ids. Nothing in SQL refers to the ids they had in the source."), + "one line per category says the ids changed, rather than one per row or per batch"); + } + + Task<MigrationWriteResult> Write(params (string Id, ExternalIntegrationDispatchRequest Request)[] rows) => + Target.Write( + Category, + new MigrationBatch([.. rows.Select(row => new MigrationRow(row.Id, row.Request, new Dictionary<string, object>()))], rows[^1].Id), + new MigrationCheckpoint(Category.Id, MigrationCategoryState.InProgress, rows[^1].Id, 0, 0, null, null, null, null, null, null)); + + async Task<int> Rows() + { + using var scope = ServiceProvider.CreateScope(); + return await scope.ServiceProvider.GetRequiredService<ServiceControlDbContext>().ExternalIntegrationDispatchRequests.CountAsync(); + } + + public class TestDispatchContext + { + public string Value { get; set; } + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/PreparedSkips.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/PreparedSkips.cs index 8e43f3c55f..98e6b8c93f 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/Migration/PreparedSkips.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/PreparedSkips.cs @@ -34,6 +34,17 @@ public static async Task<IReadOnlyList<string>> DetailsFrom(IServiceProvider ser return prepared.Merges; } + /// <summary> + /// Prepares a batch without writing it, and returns the rows the writer reports as changed on the way in with their log detail. + /// </summary> + public static async Task<IReadOnlyList<(string SourceId, string Detail)>> ChangesFrom(IServiceProvider services, IMigrationCategoryWriter writer, MigrationBatch batch) + { + using var scope = services.CreateScope(); + var prepared = await writer.Prepare(scope.ServiceProvider.GetRequiredService<ServiceControlDbContext>(), batch); + + return prepared.Changes; + } + /// <summary> /// The table and column a property is stored in, as this provider names them. /// </summary> diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/RetryOperationsMigrationTargetTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/RetryOperationsMigrationTargetTests.cs index 58adff0d22..ded7804dac 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/Migration/RetryOperationsMigrationTargetTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/RetryOperationsMigrationTargetTests.cs @@ -5,6 +5,7 @@ namespace ServiceControl.Persistence.Tests; using System.Linq; using System.Threading.Tasks; using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; using NUnit.Framework; using ServiceControl.Persistence.DataMigration; using ServiceControl.Persistence.EFCore.DataMigration.Writers; @@ -14,6 +15,10 @@ namespace ServiceControl.Persistence.Tests; class RetryOperationsMigrationTargetTests : PersistenceTestBase { + readonly TargetWarnings targetLog = new(); + + public RetryOperationsMigrationTargetTests() => RegisterServices = services => services.AddSingleton<ILoggerProvider>(targetLog); + [SetUp] public Task OpenTarget() => Target.Open(); @@ -151,6 +156,24 @@ public async Task A_second_pending_operation_with_the_same_key_is_reported_as_a_ } } + [Test] + public async Task A_merged_pending_operation_names_the_source_row_kept_and_the_row_dropped() + { + var dialect = ServiceProvider.GetRequiredService<IMigrationSqlDialect>(); + var batch = BatchOf( + ("unacknowledged|group-2|FailureGroup|older", new RetryOperationRow(null, Pending("group-2", "Shipping-Failures"))), + ("unacknowledged|group-2|FailureGroup|newer", new RetryOperationRow(null, Pending("group-2", "Shipping-Failures")))); + + var merges = await PreparedSkips.MergesFrom(ServiceProvider, new RetryOperationsWriter(dialect), batch); + + using (Assert.EnterMultipleScope()) + { + Assert.That(merges.Select(merge => merge.SourceId), Is.EqualTo(new[] { "unacknowledged|group-2|FailureGroup|older" }), "the latest retry wins"); + Assert.That(merges.Single().Detail, Does.EndWith("It kept the row from 'unacknowledged|group-2|FailureGroup|newer' and dropped the row from 'unacknowledged|group-2|FailureGroup|older'."), + "both keys print the same, so only the source rows, whose ids carry the completion time, tell the operator which run survived"); + } + } + [Test] public async Task A_pending_key_the_target_already_holds_merges_only_a_variant_the_column_treats_as_the_same() { @@ -214,6 +237,24 @@ public async Task A_retry_operation_time_of_local_or_unspecified_kind_is_copied_ } } + [Test] + public async Task Copied_historic_operations_get_one_line_saying_the_database_gives_them_new_ids() + { + var first = await Target.Write(RetryOperationsCategory, BatchOf(("historic/1", new RetryOperationRow(Historic("group-1"), null)), ("pending/1", new RetryOperationRow(null, Pending("group-2", "Shipping-Failures")))), CheckpointAfter("pending/1")); + await Target.Write(RetryOperationsCategory, BatchOf(("historic/2", new RetryOperationRow(Historic("group-3"), null))), first.Saved with { Cursor = "historic/2" }); + + Assert.That(targetLog.Information, Is.EqualTo(new[] { $"Category {MigrationCategoryIds.RetryOperations}: the database gives the copied historic retry operations new ids. Nothing in SQL refers to the ids they had in the source." }), + "one line per category says the ids changed, rather than one per row or per batch"); + } + + [Test] + public async Task Copied_pending_operations_alone_get_no_line_about_new_ids() + { + await Target.Write(RetryOperationsCategory, BatchOf(("pending/1", new RetryOperationRow(null, Pending("group-2", "Shipping-Failures")))), CheckpointAfter("pending/1")); + + Assert.That(targetLog.Information, Has.None.Contains("new ids"), "a pending operation keeps its key, so nothing about it is renumbered"); + } + static HistoricRetryOperation Historic(string requestId) => new() { RequestId = requestId, diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/SubscriptionsMigrationTargetTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/SubscriptionsMigrationTargetTests.cs index 0b437aece1..535520537d 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/Migration/SubscriptionsMigrationTargetTests.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/SubscriptionsMigrationTargetTests.cs @@ -79,6 +79,44 @@ public async Task A_second_version_of_a_type_for_one_subscriber_is_reported_as_a } } + [Test] + public async Task A_merged_subscription_names_the_version_and_endpoint_of_the_row_kept_and_the_row_dropped() + { + var dialect = ServiceProvider.GetRequiredService<IMigrationSqlDialect>(); + var batch = BatchOf( + ("s/1", new SubscriptionRow("MyMessages.OrderPlaced", new Version(1, 0, 0), "Billing@BILLING01", "Billing")), + ("s/2", new SubscriptionRow("MyMessages.OrderPlaced", new Version(2, 0, 0), "Billing@BILLING01", "Billing.V2"))); + + var merges = await PreparedSkips.MergesFrom(ServiceProvider, new SubscriptionsWriter(dialect), batch); + var result = await Target.Write(SubscriptionsCategory, batch, CheckpointAfter("s/2")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(merges.Single().Detail, Does.EndWith("It kept the row from 's/1' (version 1.0.0, endpoint 'Billing') and dropped the row from 's/2' (version 2.0.0, endpoint 'Billing.V2')."), + "both keys print the same, so only the source rows tell the operator which version survived"); + Assert.That((result.AlreadyPresent, result.Merged), Is.EqualTo((1, 1)), "the summary counts the merge apart from the other rows already present"); + } + } + + [Test] + public async Task A_subscription_merged_on_a_re_read_names_the_row_an_earlier_pass_copied() + { + var dialect = ServiceProvider.GetRequiredService<IMigrationSqlDialect>(); + var kept = ("s/1", new SubscriptionRow("MyMessages.OrderPlaced", new Version(1, 0, 0), "Billing@BILLING01", "Billing")); + var first = await Target.Write(SubscriptionsCategory, BatchOf(kept), CheckpointAfter("s/1")); + var reRead = BatchOf(kept, ("s/2", new SubscriptionRow("MyMessages.OrderPlaced", new Version(2, 0, 0), "Billing@BILLING01", "Billing"))); + + var merges = await PreparedSkips.MergesFrom(ServiceProvider, new SubscriptionsWriter(dialect), reRead); + var result = await Target.Write(SubscriptionsCategory, reRead, first.Saved); + + using (Assert.EnterMultipleScope()) + { + Assert.That(merges.Single().Detail, Does.EndWith("It kept the row from 's/1' (version 1.0.0, endpoint 'Billing') and dropped the row from 's/2' (version 2.0.0, endpoint 'Billing')."), + "the stored row is the one 's/1' copied on the earlier pass, so the line names that source row"); + Assert.That((result.AlreadyPresent, result.Merged), Is.EqualTo((2, 1))); + } + } + [Test] public async Task A_key_the_target_already_holds_merges_only_a_variant_the_column_treats_as_the_same() { @@ -103,6 +141,7 @@ public async Task A_key_the_target_already_holds_merges_only_a_variant_the_colum if (addressIgnoresCase) { Assert.That(merges[0].Detail, Does.Contain("kept 'MyMessages.OrderPlaced', 'Billing@BILLING01' and dropped the row for 'MyMessages.OrderPlaced', 'billing@billing01'")); + Assert.That(merges[0].Detail, Does.EndWith("It kept the row the table already held and dropped the row from 's/2' (version 1.0.0, endpoint 'Billing')."), "no row of this batch claimed the stored row"); } Assert.That(result.Copied, Is.EqualTo(addressIgnoresCase ? 1 : 2), "the other address is copied, and so is the case variant where the column tells it apart"); diff --git a/src/ServiceControl.Persistence.Tests/EFCore/Migration/TargetWarnings.cs b/src/ServiceControl.Persistence.Tests/EFCore/Migration/TargetWarnings.cs new file mode 100644 index 0000000000..e84ab5f282 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/Migration/TargetWarnings.cs @@ -0,0 +1,49 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using ServiceControl.Persistence.EFCore.DataMigration; + +// RecordingLoggerProvider lives in ServiceControl.Infrastructure.Tests, which the persistence test projects do not reference. +/// <summary> +/// Keeps the lines the migration target logs, so a test can read what the operator sees. Messages holds the warnings and Information the information lines. +/// Register it as an ILoggerProvider through RegisterServices. +/// </summary> +sealed class TargetWarnings : ILoggerProvider +{ + readonly ConcurrentQueue<string> messages = new(); + readonly ConcurrentQueue<string> information = new(); + + public IReadOnlyList<string> Messages => [.. messages]; + + public IReadOnlyList<string> Information => [.. information]; + + public ILogger CreateLogger(string categoryName) => + categoryName == typeof(EFCoreMigrationTarget).FullName ? new TargetLogger(this) : NullLogger.Instance; + + public void Dispose() + { + } + + sealed class TargetLogger(TargetWarnings lines) : ILogger + { + public IDisposable BeginScope<TState>(TState state) where TState : notnull => null; + + public bool IsEnabled(LogLevel logLevel) => logLevel is LogLevel.Warning or LogLevel.Information; + + public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception exception, Func<TState, Exception, string> formatter) + { + if (logLevel == LogLevel.Warning) + { + lines.messages.Enqueue(formatter(state, exception)); + } + else if (logLevel == LogLevel.Information) + { + lines.information.Enqueue(formatter(state, exception)); + } + } + } +} diff --git a/src/ServiceControl.Persistence/DataMigration/HaltThreshold.cs b/src/ServiceControl.Persistence/DataMigration/HaltThreshold.cs index 534e120b7d..08f6365e87 100644 --- a/src/ServiceControl.Persistence/DataMigration/HaltThreshold.cs +++ b/src/ServiceControl.Persistence/DataMigration/HaltThreshold.cs @@ -6,14 +6,11 @@ namespace ServiceControl.Persistence.DataMigration; public static class HaltThreshold { /// <summary> - /// Whether the skipped rows are enough to stop the category. Both limits have to be passed, so the floor - /// ignores a few bad rows in a small category and the percentage ignores a small share of a large one. + /// Returns true, which stops the category, when skippedCount passes minimumFloor and its share of totalCount passes percentThreshold percent. + /// The floor ignores a few bad rows in a small category, and the percentage ignores a small share of a large one. + /// skippedCount holds fault skips only, never a harmless skip that <see cref="MigrationSkipReasonExtensions.IsBenign" /> accepts, and + /// totalCount holds every row dealt with over the same stretch, copied, skipped and already present alike. /// </summary> - /// <param name="skippedCount">Rows skipped as faults. A skip the product would have dropped anyway does not belong here.</param> - /// <param name="totalCount">Rows dealt with over the same stretch as <paramref name="skippedCount" />, copied, skipped and already present alike.</param> - /// <param name="percentThreshold">The share of skipped rows, as a percentage, that has to be passed.</param> - /// <param name="minimumFloor">The number of skipped rows that has to be passed before the percentage counts at all.</param> - /// <returns>True when both limits are passed, which means the category stops.</returns> public static bool Exceeded(long skippedCount, long totalCount, int percentThreshold, int minimumFloor) { if (skippedCount <= minimumFloor || totalCount == 0) diff --git a/src/ServiceControl.Persistence/DataMigration/IMigrationTarget.cs b/src/ServiceControl.Persistence/DataMigration/IMigrationTarget.cs index 449c4cf85d..3dd50089d9 100644 --- a/src/ServiceControl.Persistence/DataMigration/IMigrationTarget.cs +++ b/src/ServiceControl.Persistence/DataMigration/IMigrationTarget.cs @@ -57,5 +57,7 @@ Task<MigrationWriteResult> Write( /// <summary> /// What the target did with one batch, and the checkpoint it saved with it. Every skipped row has a reason in SkipReasons. +/// Merged counts the rows inside AlreadyPresent that the target dropped because their key matched another row's. +/// Changed counts the rows inside Copied whose stored value differs from the source's on purpose. The target logs each merged and changed row. /// </summary> -public sealed record MigrationWriteResult(MigrationCheckpoint Saved, int Copied, int Skipped, IReadOnlyList<string> SkippedIds, int AlreadyPresent = 0, IReadOnlyDictionary<MigrationSkipReason, long>? SkipReasons = null); +public sealed record MigrationWriteResult(MigrationCheckpoint Saved, int Copied, int Skipped, IReadOnlyList<string> SkippedIds, int AlreadyPresent = 0, IReadOnlyDictionary<MigrationSkipReason, long>? SkipReasons = null, int Merged = 0, int Changed = 0); diff --git a/src/ServiceControl.Persistence/DataMigration/MigrationCategoryIds.cs b/src/ServiceControl.Persistence/DataMigration/MigrationCategoryIds.cs index fadc585355..6670fcfbcb 100644 --- a/src/ServiceControl.Persistence/DataMigration/MigrationCategoryIds.cs +++ b/src/ServiceControl.Persistence/DataMigration/MigrationCategoryIds.cs @@ -18,6 +18,7 @@ public static class MigrationCategoryIds public const string LicensingReportMasks = nameof(LicensingReportMasks); public const string LicensedEndpointDetails = nameof(LicensedEndpointDetails); public const string UnresolvedAndRetryIssuedFailedMessages = nameof(UnresolvedAndRetryIssuedFailedMessages); + public const string PendingIntegrationEvents = nameof(PendingIntegrationEvents); public const string EventLog = nameof(EventLog); public const string CustomChecks = nameof(CustomChecks); diff --git a/src/ServiceControl.Persistence/DataMigration/MigrationCategoryRegistry.cs b/src/ServiceControl.Persistence/DataMigration/MigrationCategoryRegistry.cs index 13e8a0fbca..e3db874daf 100644 --- a/src/ServiceControl.Persistence/DataMigration/MigrationCategoryRegistry.cs +++ b/src/ServiceControl.Persistence/DataMigration/MigrationCategoryRegistry.cs @@ -38,6 +38,7 @@ public static class MigrationCategoryRegistry new(MigrationCategoryIds.LicensingReportMasks, MigrationCategoryKind.Required, CarriesBodies: false, Order: 10), new(MigrationCategoryIds.LicensedEndpointDetails, MigrationCategoryKind.Required, CarriesBodies: false, Order: 11), new(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages, MigrationCategoryKind.Required, CarriesBodies: true, Order: 12), + new(MigrationCategoryIds.PendingIntegrationEvents, MigrationCategoryKind.Required, CarriesBodies: false, Order: 13), new(MigrationCategoryIds.CustomChecks, MigrationCategoryKind.Required, CarriesBodies: false, Order: 14), new(MigrationCategoryIds.FailedErrorImports, MigrationCategoryKind.Required, CarriesBodies: true, Order: 15), new(MigrationCategoryIds.GroupComments, MigrationCategoryKind.Required, CarriesBodies: false, Order: 16, MustFollow: MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages), diff --git a/src/ServiceControl.Persistence/DataMigration/MigrationCategoryStateExtensions.cs b/src/ServiceControl.Persistence/DataMigration/MigrationCategoryStateExtensions.cs index 8462c1e454..d68b8e6e61 100644 --- a/src/ServiceControl.Persistence/DataMigration/MigrationCategoryStateExtensions.cs +++ b/src/ServiceControl.Persistence/DataMigration/MigrationCategoryStateExtensions.cs @@ -23,14 +23,16 @@ state is MigrationCategoryState.Halted public static class MigrationSkipReasonExtensions { /// <summary> - /// Whether the running product would have dropped this row anyway, which makes the skip no loss. The halt - /// threshold and the settle rule never count a harmless skip, so no number of them can stop a category or - /// leave it Failed. The list is fixed. <see cref="MigrationSkipReason.Unknown" /> is not on it, because a - /// newer build's reason read back here may be a real loss. + /// Whether the skip is no loss, because the running product would have dropped the row anyway, or the row + /// is a result of a check the source's own storage runs about itself, which nothing reports on the target. The halt threshold and the settle rule never count a + /// harmless skip, so no number of them can stop a category or leave it Failed. The list is fixed, and + /// <see cref="MigrationSkipReason.Unknown" /> is not on it, because a newer build's reason read back here + /// may be a real loss. /// </summary> public static bool IsBenign(this MigrationSkipReason reason) => reason is MigrationSkipReason.PastRetention - or MigrationSkipReason.BlankGroupComment; + or MigrationSkipReason.BlankGroupComment + or MigrationSkipReason.SourceStorageCheck; /// <summary> /// Whether no retry can fix a row skipped for this reason. diff --git a/src/ServiceControl.Persistence/DataMigration/MigrationEngine.cs b/src/ServiceControl.Persistence/DataMigration/MigrationEngine.cs index e13538cdf7..77c3fcbd4e 100644 --- a/src/ServiceControl.Persistence/DataMigration/MigrationEngine.cs +++ b/src/ServiceControl.Persistence/DataMigration/MigrationEngine.cs @@ -1,6 +1,7 @@ namespace ServiceControl.Persistence.DataMigration; using System; +using System.Collections.Concurrent; using System.Collections.Generic; using System.Linq; using System.Threading; @@ -31,6 +32,14 @@ public sealed class MigrationEngine( readonly MigrationBodyLoader bodyLoader = new(source, timeProvider, options.BodyRetryBackoff, logger); + readonly ConcurrentDictionary<string, (long Merged, long Changed)> mergedAndChangedThisRun = new(StringComparer.Ordinal); + + /// <summary> + /// For each category this engine wrote to, how many rows the target merged into another row and how many it changed on the way in. + /// The counts are not saved, so they cover only the batches this engine wrote, and a category it resumed counts none from earlier runs. + /// </summary> + public IReadOnlyDictionary<string, (long Merged, long Changed)> MergedAndChangedThisRun => mergedAndChangedThisRun; + /// <summary> /// The categories of one kind to copy, in the order to copy them. Optional categories the operator did not /// ask for are left out. @@ -167,6 +176,7 @@ public async Task<MigrationCheckpoint> RunCategoryAsync(MigrationCategory catego // The target has committed this row, so every halt below settles from it. A halt settling from the // older version would be refused by the store as a conflict and lose its reason. checkpoint = result.Saved; + mergedAndChangedThisRun.AddOrUpdate(category.Id, (result.Merged, result.Changed), (_, sum) => (sum.Merged + result.Merged, sum.Changed + result.Changed)); // The result states the batch's outcome twice, as its own counts and as deltas on the checkpoint // it committed. The halt threshold reads the first and status and verify read the second. @@ -179,11 +189,6 @@ public async Task<MigrationCheckpoint> RunCategoryAsync(MigrationCategory catego break; } - foreach (var id in result.SkippedIds) - { - logger.LogWarning("Skipped {SourceId} in category {CategoryId}", id, category.Id); - } - processedThisRun += bodySkips + result.Copied + result.Skipped + result.AlreadyPresent; skippedThisRun += bodySkips + FaultSkips(result.SkipReasons); @@ -239,7 +244,7 @@ public async Task<MigrationCheckpoint> RunCategoryAsync(MigrationCategory catego return await Settle(checkpoint with { State = FaultSkips(checkpoint.SkipReasons) > 0 ? MigrationCategoryState.CompleteWithErrors : MigrationCategoryState.Complete }, cancellationToken); } - // Harmless skips are rows the product would have removed anyway, so they neither stop a category nor leave it Failed. + // A harmless skip loses nothing, as IsBenign says, so it neither stops a category nor leaves it Failed. static long FaultSkips(IReadOnlyDictionary<MigrationSkipReason, long>? skipReasons) => skipReasons is null ? 0 : skipReasons.Where(reason => !reason.Key.IsBenign()).Sum(reason => reason.Value); diff --git a/src/ServiceControl.Persistence/DataMigration/MigrationMetadataKeys.cs b/src/ServiceControl.Persistence/DataMigration/MigrationMetadataKeys.cs index 808d5090f6..2ea004b4bf 100644 --- a/src/ServiceControl.Persistence/DataMigration/MigrationMetadataKeys.cs +++ b/src/ServiceControl.Persistence/DataMigration/MigrationMetadataKeys.cs @@ -16,4 +16,22 @@ public static class MigrationMetadataKeys /// The time the source last changed the row's document, as a UTC <see cref="System.DateTime" />. /// </summary> public const string LastModified = "LastModified"; + + /// <summary> + /// The time the source last stored a failed error import, as a UTC <see cref="System.DateTime" />. + /// RavenDB keeps no failure time, so the target stores this one as the failure time, and replay runs the oldest first by it. + /// </summary> + public const string FailedAt = "FailedAt"; + + /// <summary> + /// A <see cref="bool" /> that is true when the row is a result of a check the source's own storage runs about itself. + /// Rows that are not such a result carry no value. + /// </summary> + public const string SourceStorageCheck = "SourceStorageCheck"; + + /// <summary> + /// A <see cref="bool" /> on every retry-issued failed message. It is true when the source still holds a retry claim + /// for the message, and false when the source had already released it. Other rows carry no value. + /// </summary> + public const string RetryClaimHeld = "RetryClaimHeld"; } diff --git a/src/ServiceControl.Persistence/DataMigration/MigrationSkipReason.cs b/src/ServiceControl.Persistence/DataMigration/MigrationSkipReason.cs index 6c4d0fe806..4c848b4f89 100644 --- a/src/ServiceControl.Persistence/DataMigration/MigrationSkipReason.cs +++ b/src/ServiceControl.Persistence/DataMigration/MigrationSkipReason.cs @@ -28,6 +28,12 @@ public enum MigrationSkipReason /// </summary> BlankGroupComment, + /// <summary> + /// The source marked the row as a result of a check its own storage runs about itself. + /// Nothing reports such a check once the instance runs on the target, so a copied result would never change. + /// </summary> + SourceStorageCheck, + /// <summary> /// The document's unique message id is not a GUID, and the row's key has to be one. /// </summary> diff --git a/src/ServiceControl.UnitTests/ApprovalFiles/MigrationEngineCategorySelectionTests.Every_category_runs_in_a_fixed_order.approved.txt b/src/ServiceControl.UnitTests/ApprovalFiles/MigrationEngineCategorySelectionTests.Every_category_runs_in_a_fixed_order.approved.txt index 1e54a214bc..6c99c6b3e0 100644 --- a/src/ServiceControl.UnitTests/ApprovalFiles/MigrationEngineCategorySelectionTests.Every_category_runs_in_a_fixed_order.approved.txt +++ b/src/ServiceControl.UnitTests/ApprovalFiles/MigrationEngineCategorySelectionTests.Every_category_runs_in_a_fixed_order.approved.txt @@ -10,6 +10,7 @@ Required 9: LicensingThroughput, after LicensingEndpoints Required 10: LicensingReportMasks Required 11: LicensedEndpointDetails Required 12: UnresolvedAndRetryIssuedFailedMessages, with bodies +Required 13: PendingIntegrationEvents Required 14: CustomChecks Required 15: FailedErrorImports, with bodies Required 16: GroupComments, after UnresolvedAndRetryIssuedFailedMessages diff --git a/src/ServiceControl.UnitTests/Migration/CustomCheckStatusTests.cs b/src/ServiceControl.UnitTests/Migration/CustomCheckStatusTests.cs new file mode 100644 index 0000000000..d8bdb1d464 --- /dev/null +++ b/src/ServiceControl.UnitTests/Migration/CustomCheckStatusTests.cs @@ -0,0 +1,14 @@ +namespace ServiceControl.UnitTests.Migration; + +using System; +using NUnit.Framework; +using ServiceControl.Persistence; + +[TestFixture] +class CustomCheckStatusTests +{ + // CustomCheckDetail carries the status as a bool, so a third member would be copied as Pass. + [Test] + public void The_status_enum_still_has_exactly_the_two_values_a_bool_can_carry() => + Assert.That(Enum.GetValues<Status>(), Is.EqualTo(new[] { Status.Fail, Status.Pass })); +} diff --git a/src/ServiceControl.UnitTests/Migration/MigrationCategoryRegistryTests.cs b/src/ServiceControl.UnitTests/Migration/MigrationCategoryRegistryTests.cs index c82e610f09..214b13a2d2 100644 --- a/src/ServiceControl.UnitTests/Migration/MigrationCategoryRegistryTests.cs +++ b/src/ServiceControl.UnitTests/Migration/MigrationCategoryRegistryTests.cs @@ -12,12 +12,12 @@ namespace ServiceControl.UnitTests.Migration; class MigrationCategoryRegistryTests { [Test] - public void Contains_all_seventeen_categories_with_unique_ids() + public void Contains_all_eighteen_categories_with_unique_ids() { using (Assert.EnterMultipleScope()) { - Assert.That(MigrationCategoryRegistry.All, Has.Count.EqualTo(17)); - Assert.That(MigrationCategoryRegistry.All.Select(c => c.Id).Distinct().Count(), Is.EqualTo(17)); + Assert.That(MigrationCategoryRegistry.All, Has.Count.EqualTo(18)); + Assert.That(MigrationCategoryRegistry.All.Select(c => c.Id).Distinct().Count(), Is.EqualTo(18)); } } @@ -36,7 +36,7 @@ public void Required_categories_are_ordered_exactly_as_the_contract_lists_them() "NotificationSettings", "TrialEndDate", "RetryOperations", "LicensingEndpoints", "LicensingThroughput", "LicensingReportMasks", "LicensedEndpointDetails", "UnresolvedAndRetryIssuedFailedMessages", - "CustomChecks", "FailedErrorImports", "GroupComments" + "PendingIntegrationEvents", "CustomChecks", "FailedErrorImports", "GroupComments" })); } diff --git a/src/ServiceControl.UnitTests/Migration/MigrationCategoryStateExtensionsTests.cs b/src/ServiceControl.UnitTests/Migration/MigrationCategoryStateExtensionsTests.cs index 8843923d12..23c4080bfb 100644 --- a/src/ServiceControl.UnitTests/Migration/MigrationCategoryStateExtensionsTests.cs +++ b/src/ServiceControl.UnitTests/Migration/MigrationCategoryStateExtensionsTests.cs @@ -2,7 +2,7 @@ namespace ServiceControl.UnitTests.Migration; using System; -using System.Linq; +using System.Collections.Generic; using NUnit.Framework; using ServiceControl.Persistence.DataMigration; @@ -10,9 +10,38 @@ namespace ServiceControl.UnitTests.Migration; class MigrationCategoryStateExtensionsTests { [Test] - public void Exactly_the_two_harmless_reasons_are_harmless_and_every_other_skip_reason_is_a_fault() => - Assert.That( - Enum.GetValues<MigrationSkipReason>().Where(reason => reason.IsBenign()), - Is.EquivalentTo(new[] { MigrationSkipReason.PastRetention, MigrationSkipReason.BlankGroupComment }), - "a harmless reason is exempt from the halt threshold and the settle rule, so any number of rows lost to one settles the category Done"); + public void Every_skip_reason_is_exactly_one_of_harmless_permanent_or_retryable() + { + using (Assert.EnterMultipleScope()) + { + Assert.That(Kinds.Keys, Is.EquivalentTo(Enum.GetValues<MigrationSkipReason>()), "a reason on neither list is treated as a fault a retry can fix, so each new reason has to be placed here on purpose"); + + foreach (var (reason, kind) in Kinds) + { + Assert.That((reason.IsBenign(), reason.IsPermanent()), Is.EqualTo((kind == SkipKind.Harmless, kind == SkipKind.Permanent)), $"{reason} is {kind}"); + } + } + } + + enum SkipKind + { + Harmless, + Permanent, + Retryable + } + + static readonly Dictionary<MigrationSkipReason, SkipKind> Kinds = new() + { + [MigrationSkipReason.BodyUnreadable] = SkipKind.Retryable, + [MigrationSkipReason.PastRetention] = SkipKind.Harmless, + [MigrationSkipReason.RequiredValueMissing] = SkipKind.Permanent, + [MigrationSkipReason.BlankGroupComment] = SkipKind.Harmless, + [MigrationSkipReason.SourceStorageCheck] = SkipKind.Harmless, + [MigrationSkipReason.UnparseableUniqueMessageId] = SkipKind.Permanent, + [MigrationSkipReason.NoProcessingAttempts] = SkipKind.Permanent, + [MigrationSkipReason.ValueTooLong] = SkipKind.Permanent, + [MigrationSkipReason.MalformedValue] = SkipKind.Permanent, + [MigrationSkipReason.BodyMissing] = SkipKind.Permanent, + [MigrationSkipReason.Unknown] = SkipKind.Retryable + }; } diff --git a/src/ServiceControl.UnitTests/Migration/MigrationEngineCategorySelectionTests.cs b/src/ServiceControl.UnitTests/Migration/MigrationEngineCategorySelectionTests.cs index 15cf13aaad..ec774e7a09 100644 --- a/src/ServiceControl.UnitTests/Migration/MigrationEngineCategorySelectionTests.cs +++ b/src/ServiceControl.UnitTests/Migration/MigrationEngineCategorySelectionTests.cs @@ -75,7 +75,7 @@ public async Task A_category_removed_from_configuration_leaves_its_checkpoint_ro var target = new InMemoryMigrationTarget(checkpointStore); var options = new MigrationEngineOptions(TimeSpan.Zero, 5, 100, [stillConfigured]); var engine = new MigrationEngine(source, target, checkpointStore, new FakeTimeProvider(), options, NullLogger<MigrationEngine>.Instance); - var previousRun = new MigrationCheckpoint(MigrationCategoryIds.ArchivedAndResolvedFailedMessages, MigrationCategoryState.CompleteWithErrors, "cursor-99", 40, 2, 42, null, DateTime.UtcNow, DateTime.UtcNow, DateTime.UtcNow, null); + var previousRun = new MigrationCheckpoint(MigrationCategoryIds.ArchivedAndResolvedFailedMessages, MigrationCategoryState.InProgress, "cursor-99", 40, 2, 42, null, DateTime.UtcNow, DateTime.UtcNow, null, null); await checkpointStore.Upsert(previousRun); var results = await engine.RunCategories(engine.SelectCategories(MigrationCategoryKind.Optional)); diff --git a/src/ServiceControl.UnitTests/Migration/MigrationEngineCopyTests.cs b/src/ServiceControl.UnitTests/Migration/MigrationEngineCopyTests.cs index 6135b96aa5..484a171e1a 100644 --- a/src/ServiceControl.UnitTests/Migration/MigrationEngineCopyTests.cs +++ b/src/ServiceControl.UnitTests/Migration/MigrationEngineCopyTests.cs @@ -3,6 +3,8 @@ namespace ServiceControl.UnitTests.Migration; using System; using System.Collections.Generic; +using System.Linq; +using System.Threading; using System.Threading.Tasks; using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Time.Testing; @@ -248,4 +250,42 @@ public async Task A_category_already_Complete_is_left_alone_on_a_second_run() Assert.That(target.WrittenRows(category.Id), Is.Empty); } } + + [Test] + public async Task The_merged_and_changed_rows_the_target_reports_add_up_for_each_category_in_this_run() + { + var category = MigrationCategoryRegistry.Find("KnownEndpoints")!; + var source = new InMemoryMigrationSource(); + source.Seed(category.Id, Row("a"), Row("b"), Row("c"), Row("d"), Row("e"), Row("f")); + var checkpointStore = new InMemoryMigrationCheckpointStore(); + var options = new MigrationEngineOptions(TimeSpan.Zero, 5, 100, []); + var engine = new MigrationEngine(source, new MergingTarget(checkpointStore), checkpointStore, new FakeTimeProvider(), options, NullLogger<MigrationEngine>.Instance); + + await engine.RunCategoryAsync(category); + + Assert.That(engine.MergedAndChangedThisRun, Is.EquivalentTo(new Dictionary<string, (long Merged, long Changed)> { [category.Id] = (2, 4) }), + "the checkpoint does not hold these counts, so the summary after the copy has only what the engine adds up"); + } + + // Reports one row of every batch merged and two of the rest changed. + sealed class MergingTarget(IMigrationCheckpointStore checkpointStore) : IMigrationTarget + { + public Task Open(CancellationToken cancellationToken = default) => Task.CompletedTask; + + public Task<int> BatchSizeFor(MigrationCategory category, CancellationToken cancellationToken = default) => Task.FromResult(3); + + public async Task<MigrationWriteResult> Write(MigrationCategory category, MigrationBatch batch, MigrationCheckpoint checkpointToExtend, CancellationToken cancellationToken = default) + { + var saved = await checkpointStore.Upsert(checkpointToExtend.Extend(batch.Rows.Count - 1, 0, 1, null), cancellationToken); + return new MigrationWriteResult(saved, batch.Rows.Count - 1, 0, [], AlreadyPresent: 1, Merged: 1, Changed: 2); + } + + public Task<long> Count(MigrationCategory category, CancellationToken cancellationToken = default) => Task.FromResult(0L); + + public Task<MigrationCheckpoint> ResetForRetry(MigrationCheckpoint reset, CancellationToken cancellationToken = default) => checkpointStore.Upsert(reset, cancellationToken); + + public IReadOnlyCollection<string> SupportedCategoryIds => [.. MigrationCategoryRegistry.All.Select(category => category.Id)]; + + public IReadOnlyDictionary<string, Type> DocumentTypes => MigrationCategoryRegistry.All.ToDictionary(category => category.Id, _ => typeof(object)); + } } diff --git a/src/ServiceControl.UnitTests/Migration/MigrationEngineFailurePathTests.cs b/src/ServiceControl.UnitTests/Migration/MigrationEngineFailurePathTests.cs index 9d02e7bb4d..39ce6653c2 100644 --- a/src/ServiceControl.UnitTests/Migration/MigrationEngineFailurePathTests.cs +++ b/src/ServiceControl.UnitTests/Migration/MigrationEngineFailurePathTests.cs @@ -377,25 +377,6 @@ public async Task The_halt_reason_says_at_the_start_when_the_first_batch_never_c Assert.That(checkpoint.LastError, Does.Contain("at the start")); } - [Test] - public async Task Every_row_the_target_skips_is_named_in_the_log() - { - // The counts say how much was left behind. Only the log says which rows, and it is the way back - // to them while the RavenDB database still exists. - var category = MigrationCategoryRegistry.Find("KnownEndpoints")!; - var source = new InMemoryMigrationSource(); - source.Seed(category.Id, Row("a"), Row("b")); - var checkpointStore = new InMemoryMigrationCheckpointStore(); - var target = new InMemoryMigrationTarget(checkpointStore); - target.RejectKey("b", MigrationSkipReason.PastRetention); - var logger = new CapturingLogger(); - var engine = new MigrationEngine(source, target, checkpointStore, new FakeTimeProvider(), new MigrationEngineOptions(TimeSpan.Zero, 5, 100, []), logger); - - await engine.RunCategoryAsync(category); - - Assert.That(logger.Entries.Select(entry => entry.Message), Has.Some.EqualTo("Skipped b in category KnownEndpoints")); - } - [Test] public async Task A_stop_between_batches_leaves_the_row_in_progress_at_the_batch_that_committed() { diff --git a/src/ServiceControl.UnitTests/Migration/RequiredCopyGateTests.cs b/src/ServiceControl.UnitTests/Migration/RequiredCopyGateTests.cs index 91945ae6e9..0bd83978d7 100644 --- a/src/ServiceControl.UnitTests/Migration/RequiredCopyGateTests.cs +++ b/src/ServiceControl.UnitTests/Migration/RequiredCopyGateTests.cs @@ -197,7 +197,8 @@ public void A_failed_category_that_skipped_rows_is_told_a_retry_re_reads_them() [MigrationSkipReason.BodyUnreadable] = 1 })], logger, - RunStartedAt); + RunStartedAt, + NothingMergedOrChanged); var entry = logger.Entries.Single(); @@ -224,7 +225,8 @@ public void A_failed_category_whose_faults_no_retry_can_fix_is_pointed_at_abando [MigrationSkipReason.RequiredValueMissing] = 2 })], logger, - RunStartedAt); + RunStartedAt, + NothingMergedOrChanged); var entry = logger.Entries.Single(); @@ -245,7 +247,8 @@ public void A_halted_category_whose_faults_no_retry_can_fix_is_told_abandon_give [Settled(MigrationCategoryState.Halted, copied: 7, skipped: 120, new Dictionary<MigrationSkipReason, long> { [MigrationSkipReason.MalformedValue] = 120 }) with { LastError = "Halted: 120 of 127 rows skipped in this run exceeds the configured threshold of 5% and 100 rows." }], logger, - RunStartedAt); + RunStartedAt, + NothingMergedOrChanged); var entry = logger.Entries.Single(); @@ -268,7 +271,8 @@ public void A_halted_category_with_faults_a_retry_can_fix_is_also_told_what_aban MigrationStartup.ReportWhatTheCopyLeftBehind( [Settled(MigrationCategoryState.Halted, copied: 7, skipped: 120, new Dictionary<MigrationSkipReason, long> { [MigrationSkipReason.BodyUnreadable] = 120 })], logger, - RunStartedAt); + RunStartedAt, + NothingMergedOrChanged); var entry = logger.Entries.Single(); @@ -288,7 +292,7 @@ public void A_category_that_failed_with_only_harmless_skips_is_still_offered_a_r with { LastError = "TimeoutException at cursor r-7: the target did not answer" }; - MigrationStartup.ReportWhatTheCopyLeftBehind([stoppedByAnException], logger, RunStartedAt); + MigrationStartup.ReportWhatTheCopyLeftBehind([stoppedByAnException], logger, RunStartedAt, NothingMergedOrChanged); Assert.That(logger.Entries.Single().Message, Does.Contain($"--migration-retry {MigrationCategoryIds.EndpointSettings} re-reads them").And.Not.Contain("--migration-abandon"), "no fault skip stands in the way of a retry, so the exception's cause is what the operator fixes"); @@ -302,7 +306,8 @@ public void A_done_category_with_harmless_skips_is_not_offered_a_retry() MigrationStartup.ReportWhatTheCopyLeftBehind( [Settled(MigrationCategoryState.Complete, copied: 7, skipped: 3, new Dictionary<MigrationSkipReason, long> { [MigrationSkipReason.PastRetention] = 3 })], logger, - RunStartedAt); + RunStartedAt, + NothingMergedOrChanged); var entry = logger.Entries.Single(); @@ -314,12 +319,59 @@ public void A_done_category_with_harmless_skips_is_not_offered_a_retry() }); } + [Test] + public void Every_harmless_skip_reason_has_its_own_explanation_and_no_other_reason_has_one() => + Assert.That(MigrationStartup.HarmlessSkipExplanations.Keys, Is.EquivalentTo(Enum.GetValues<MigrationSkipReason>().Where(reason => reason.IsBenign())), + "the warning for a Done category explains each harmless reason it skipped, so a harmless reason with no explanation would be left out unexplained"); + + [Test] + public void A_done_category_is_told_why_each_of_its_harmless_reasons_is_harmless_and_nothing_about_the_others() + { + var logger = new CapturingLogger(); + + MigrationStartup.ReportWhatTheCopyLeftBehind( + [Settled(MigrationCategoryState.Complete, copied: 7, skipped: 8, new Dictionary<MigrationSkipReason, long> + { + [MigrationSkipReason.SourceStorageCheck] = 6, + [MigrationSkipReason.BlankGroupComment] = 2 + })], + logger, + RunStartedAt, + NothingMergedOrChanged); + + var message = logger.Entries.Single().Message; + + Assert.Multiple(() => + { + Assert.That(message, Does.Contain(MigrationStartup.HarmlessSkipExplanations[MigrationSkipReason.SourceStorageCheck]).And.Contain(MigrationStartup.HarmlessSkipExplanations[MigrationSkipReason.BlankGroupComment])); + Assert.That(message, Does.Not.Contain(MigrationStartup.HarmlessSkipExplanations[MigrationSkipReason.PastRetention]), "a reason the category did not skip for would give the operator a false reason"); + }); + } + + [Test] + public void The_refusal_marks_a_harmless_reason_so_it_does_not_read_as_rows_a_retry_could_copy() + { + var failed = Checkpoint(MigrationCategoryState.CompleteWithErrors, MigrationCategoryIds.CustomChecks) with + { + SkippedCount = 7, + SkipReasons = new Dictionary<MigrationSkipReason, long> + { + [MigrationSkipReason.SourceStorageCheck] = 6, + [MigrationSkipReason.MalformedValue] = 1 + } + }; + + var exception = Assert.Throws<Exception>(() => MigrationStartup.RefuseIfAnyCategoryDidNotComplete([CustomChecks], [failed], NewSettings())); + + Assert.That(exception.Message, Does.Contain("SourceStorageCheck 6, harmless").And.Contain("MalformedValue 1, no retry can fix"), "a retry skips the six harmless rows again"); + } + [Test] public void A_blocked_category_is_left_to_the_refusal_rather_than_reported_as_a_clean_copy() { var logger = new CapturingLogger(); - MigrationStartup.ReportWhatTheCopyLeftBehind([Settled(MigrationCategoryState.Blocked, copied: 0, skipped: 0, null)], logger, RunStartedAt); + MigrationStartup.ReportWhatTheCopyLeftBehind([Settled(MigrationCategoryState.Blocked, copied: 0, skipped: 0, null)], logger, RunStartedAt, NothingMergedOrChanged); Assert.That(logger.Entries, Is.Empty); } @@ -332,7 +384,7 @@ public void A_failed_category_that_skipped_nothing_is_not_reported_as_a_clean_co var stalled = Settled(MigrationCategoryState.Halted, copied: 10, skipped: 0, null) with { LastError = MigrationStartup.StallExplanation(MigrationCategoryIds.EndpointSettings) }; - MigrationStartup.ReportWhatTheCopyLeftBehind([stalled], logger, RunStartedAt); + MigrationStartup.ReportWhatTheCopyLeftBehind([stalled], logger, RunStartedAt, NothingMergedOrChanged); var entry = logger.Entries.Single(); @@ -348,7 +400,7 @@ public void A_category_that_skipped_nothing_says_so_without_raising_a_warning() { var logger = new CapturingLogger(); - MigrationStartup.ReportWhatTheCopyLeftBehind([Settled(MigrationCategoryState.Complete, copied: 7, skipped: 0, null)], logger, RunStartedAt); + MigrationStartup.ReportWhatTheCopyLeftBehind([Settled(MigrationCategoryState.Complete, copied: 7, skipped: 0, null)], logger, RunStartedAt, NothingMergedOrChanged); var entry = logger.Entries.Single(); @@ -370,7 +422,7 @@ public void A_category_an_earlier_run_finished_is_not_reported_as_copied_again() with { SettledAt = RunStartedAt.AddMinutes(-5) }; - MigrationStartup.ReportWhatTheCopyLeftBehind([alreadyDone], logger, RunStartedAt); + MigrationStartup.ReportWhatTheCopyLeftBehind([alreadyDone], logger, RunStartedAt, NothingMergedOrChanged); var entry = logger.Entries.Single(); @@ -393,7 +445,7 @@ public void A_category_this_run_finished_is_still_reported_as_copied() with { SettledAt = RunStartedAt.AddSeconds(2) }; - MigrationStartup.ReportWhatTheCopyLeftBehind([justDone], logger, RunStartedAt); + MigrationStartup.ReportWhatTheCopyLeftBehind([justDone], logger, RunStartedAt, NothingMergedOrChanged); Assert.That(logger.Entries.Single().Message, Does.Contain("7 copied").And.Not.Contain("already finished")); } @@ -408,7 +460,7 @@ public void A_category_an_earlier_run_left_failed_is_not_reported_as_finished() with { SettledAt = RunStartedAt.AddMinutes(-5) }; - MigrationStartup.ReportWhatTheCopyLeftBehind([failedEarlier], logger, RunStartedAt); + MigrationStartup.ReportWhatTheCopyLeftBehind([failedEarlier], logger, RunStartedAt, NothingMergedOrChanged); var entry = logger.Entries.Single(); @@ -420,6 +472,68 @@ public void A_category_an_earlier_run_left_failed_is_not_reported_as_finished() }); } + [Test] + public void A_category_that_merged_or_changed_rows_says_how_many_apart_from_already_present_and_warns() + { + var logger = new CapturingLogger(); + var done = Settled(MigrationCategoryState.Complete, copied: 15, skipped: 0, null) with { AlreadyPresentCount = 3, SettledAt = RunStartedAt.AddSeconds(2) }; + + MigrationStartup.ReportWhatTheCopyLeftBehind([done], logger, RunStartedAt, new Dictionary<string, (long Merged, long Changed)> { [MigrationCategoryIds.EndpointSettings] = (1, 4) }); + + var entry = logger.Entries.Single(); + + Assert.Multiple(() => + { + Assert.That(entry.Message, Is.EqualTo($"{MigrationCategoryIds.EndpointSettings}: 15 copied (4 changed), 2 already present, 1 merged, nothing skipped"), + "a merged row is counted as already present, so without taking it out the summary says nothing changed"); + Assert.That(entry.Level, Is.EqualTo(LogLevel.Warning), "a merged or changed row is one the operator did not expect, and the per-row warnings above name each one"); + }); + } + + [Test] + public void A_category_that_merged_rows_and_skipped_some_counts_both() + { + var logger = new CapturingLogger(); + var failed = Settled(MigrationCategoryState.CompleteWithErrors, copied: 7, skipped: 2, new Dictionary<MigrationSkipReason, long> { [MigrationSkipReason.RequiredValueMissing] = 2 }) + with + { AlreadyPresentCount = 5 }; + + MigrationStartup.ReportWhatTheCopyLeftBehind([failed], logger, RunStartedAt, new Dictionary<string, (long Merged, long Changed)> { [MigrationCategoryIds.EndpointSettings] = (2, 0) }); + + Assert.That(logger.Entries.Single().Message, Does.StartWith($"{MigrationCategoryIds.EndpointSettings}: 7 copied (0 changed), 3 already present, 2 merged, 2 skipped (RequiredValueMissing 2).")); + } + + [Test] + public void More_merged_rows_than_rows_already_present_is_a_warning_naming_both_counts() + { + var logger = new CapturingLogger(); + var done = Settled(MigrationCategoryState.Complete, copied: 5, skipped: 0, null) with { AlreadyPresentCount = 1, SettledAt = RunStartedAt.AddSeconds(2) }; + + MigrationStartup.ReportWhatTheCopyLeftBehind([done], logger, RunStartedAt, new Dictionary<string, (long Merged, long Changed)> { [MigrationCategoryIds.EndpointSettings] = (3, 0) }); + + var messages = logger.Entries.Select(entry => entry.Message).ToArray(); + + Assert.Multiple(() => + { + Assert.That(messages, Has.Member($"{MigrationCategoryIds.EndpointSettings}: the target reported 3 merged rows but only 1 rows already present. Its key comparer disagrees with the database about which keys are equal, so the merge lines for this category may name the wrong rows."), + "every merged row is also already present, so more merges than that is the comparer predicting wrongly"); + Assert.That(messages, Has.Member($"{MigrationCategoryIds.EndpointSettings}: 5 copied (0 changed), 0 already present, 3 merged, nothing skipped"), "never a negative count"); + }); + } + + [Test] + public void A_failed_category_that_skipped_nothing_still_counts_its_merged_and_changed_rows() + { + var logger = new CapturingLogger(); + var stalled = Settled(MigrationCategoryState.Halted, copied: 10, skipped: 0, null) with { AlreadyPresentCount = 3, LastError = MigrationStartup.StallExplanation(MigrationCategoryIds.EndpointSettings) }; + + MigrationStartup.ReportWhatTheCopyLeftBehind([stalled], logger, RunStartedAt, new Dictionary<string, (long Merged, long Changed)> { [MigrationCategoryIds.EndpointSettings] = (1, 2) }); + + Assert.That(logger.Entries.Single().Message, Does.StartWith($"{MigrationCategoryIds.EndpointSettings} is Failed (Halted) after 10 copied (2 changed), 2 already present and 1 merged.")); + } + + static readonly IReadOnlyDictionary<string, (long Merged, long Changed)> NothingMergedOrChanged = new Dictionary<string, (long Merged, long Changed)>(); + static readonly DateTime RunStartedAt = new(2026, 9, 20, 12, 0, 0, DateTimeKind.Utc); static MigrationCheckpoint Settled(MigrationCategoryState state, long copied, long skipped, IReadOnlyDictionary<MigrationSkipReason, long>? skipReasons) => @@ -428,6 +542,7 @@ static MigrationCheckpoint Settled(MigrationCategoryState state, long copied, lo static readonly MigrationCategory EndpointSettings = MigrationCategoryRegistry.Find(MigrationCategoryIds.EndpointSettings)!; static readonly MigrationCategory KnownEndpoints = MigrationCategoryRegistry.Find(MigrationCategoryIds.KnownEndpoints)!; static readonly MigrationCategory MessageRedirects = MigrationCategoryRegistry.Find(MigrationCategoryIds.MessageRedirects)!; + static readonly MigrationCategory CustomChecks = MigrationCategoryRegistry.Find(MigrationCategoryIds.CustomChecks)!; static readonly MigrationCategory UnresolvedAndRetryIssuedFailedMessages = MigrationCategoryRegistry.Find(MigrationCategoryIds.UnresolvedAndRetryIssuedFailedMessages)!; const string UnknownCategoryId = "SomeCategoryFromANewerBuild"; diff --git a/src/ServiceControl/Migration/MigrationStartup.cs b/src/ServiceControl/Migration/MigrationStartup.cs index f36d874995..9c0626bd7e 100644 --- a/src/ServiceControl/Migration/MigrationStartup.cs +++ b/src/ServiceControl/Migration/MigrationStartup.cs @@ -117,7 +117,7 @@ await MigrationStartupCheckRunner.Run( RunRequiredCategories(engine, toCopy, checkpointStore, timeProvider, logger, cancellationToken), settings); - ReportWhatTheCopyLeftBehind(finished, logger, runStartedAt); + ReportWhatTheCopyLeftBehind(finished, logger, runStartedAt, engine.MergedAndChangedThisRun); RefuseIfAnyCategoryDidNotComplete(toCopy, finished, unfinishedOutsideTheCopy, settings); } @@ -355,7 +355,7 @@ [.. checkpoints internal static string FailedDetail(MigrationCheckpoint checkpoint) { var reasons = checkpoint.SkipReasons is { Count: > 0 } counts - ? $" ({string.Join("; ", counts.Select(reason => $"{reason.Key} {reason.Value}{(reason.Key.IsPermanent() ? ", no retry can fix" : "")}"))})" + ? $" ({string.Join("; ", counts.Select(reason => $"{reason.Key} {reason.Value}{(reason.Key.IsPermanent() ? ", no retry can fix" : reason.Key.IsBenign() ? ", harmless" : "")}"))})" : ""; return $"{checkpoint.CategoryId} is Failed ({checkpoint.State}) after copying {checkpoint.CopiedCount} and skipping {checkpoint.SkippedCount}{reasons}{LastErrorClause(checkpoint)}; " + @@ -372,8 +372,11 @@ static string LastErrorClause(MigrationCheckpoint checkpoint) => /// <summary> /// Logs one line per category saying what it copied and what it left behind. Skipped rows are warned about /// one at a time while the copy runs, and nothing else states the total or says what becomes of them. + /// The merged and changed counts are this run's, by category id, and the line counts merged rows apart from the other rows already present. + /// A category that merged or changed a row is reported as a warning even when it skipped nothing. + /// A category with more merged rows than rows already present gets a warning naming both counts, because the target's key comparer then disagrees with the database. /// </summary> - internal static void ReportWhatTheCopyLeftBehind(IReadOnlyList<MigrationCheckpoint> finished, ILogger logger, DateTime runStartedAt) + internal static void ReportWhatTheCopyLeftBehind(IReadOnlyList<MigrationCheckpoint> finished, ILogger logger, DateTime runStartedAt, IReadOnlyDictionary<string, (long Merged, long Changed)> mergedAndChanged) { foreach (var checkpoint in finished) { @@ -393,18 +396,29 @@ internal static void ReportWhatTheCopyLeftBehind(IReadOnlyList<MigrationCheckpoi continue; } + var (merged, changed) = mergedAndChanged.GetValueOrDefault(checkpoint.CategoryId); + + // Every merged row is also counted as already present, so more merges than that means the merge lines were predicted wrongly. + if (merged > checkpoint.AlreadyPresentCount) + { + logger.LogWarning("{CategoryId}: the target reported {Merged} merged rows but only {AlreadyPresent} rows already present. Its key comparer disagrees with the database about which keys are equal, so the merge lines for this category may name the wrong rows.", + checkpoint.CategoryId, merged, checkpoint.AlreadyPresentCount); + } + + var alreadyPresent = Math.Max(0, checkpoint.AlreadyPresentCount - merged); + // An exception, a stall or unbalanced counts fail a category without a skip, and that is not a clean copy. if (checkpoint.State.IsFailed() && checkpoint.SkippedCount == 0) { - logger.LogWarning("{CategoryId} is Failed ({State}) after {Copied} copied and {AlreadyPresent} already present. The refusal that follows says why and what to run.", - checkpoint.CategoryId, checkpoint.State, checkpoint.CopiedCount, checkpoint.AlreadyPresentCount); + logger.LogWarning("{CategoryId} is Failed ({State}) after {Copied} copied ({Changed} changed), {AlreadyPresent} already present and {Merged} merged. The refusal that follows says why and what to run.", + checkpoint.CategoryId, checkpoint.State, checkpoint.CopiedCount, changed, alreadyPresent, merged); continue; } if (checkpoint.SkippedCount == 0) { - logger.LogInformation("{CategoryId}: {Copied} copied, {AlreadyPresent} already present, nothing skipped", - checkpoint.CategoryId, checkpoint.CopiedCount, checkpoint.AlreadyPresentCount); + logger.Log(merged + changed == 0 ? LogLevel.Information : LogLevel.Warning, "{CategoryId}: {Copied} copied ({Changed} changed), {AlreadyPresent} already present, {Merged} merged, nothing skipped", + checkpoint.CategoryId, checkpoint.CopiedCount, changed, alreadyPresent, merged); continue; } @@ -416,15 +430,28 @@ internal static void ReportWhatTheCopyLeftBehind(IReadOnlyList<MigrationCheckpoi var whatBecomesOfThem = checkpoint.State.IsFailed() ? WhatARetryCanDo(checkpoint) : checkpoint.State == MigrationCategoryState.Complete - ? "They were left out as harmless, because ServiceControl would have removed them anyway, and they stay only in the source database." + ? string.Join(" ", ["They were left out as harmless.", .. HarmlessReasonsIn(checkpoint), "They stay only in the source database."]) : "They stay only in the source database."; logger.LogWarning( - "{CategoryId}: {Copied} copied, {AlreadyPresent} already present, {Skipped} skipped ({Reasons}). {WhatBecomesOfThem}", - checkpoint.CategoryId, checkpoint.CopiedCount, checkpoint.AlreadyPresentCount, checkpoint.SkippedCount, reasons, whatBecomesOfThem); + "{CategoryId}: {Copied} copied ({Changed} changed), {AlreadyPresent} already present, {Merged} merged, {Skipped} skipped ({Reasons}). {WhatBecomesOfThem}", + checkpoint.CategoryId, checkpoint.CopiedCount, changed, alreadyPresent, merged, checkpoint.SkippedCount, reasons, whatBecomesOfThem); } } + /// <summary> + /// Why each harmless skip reason loses nothing, in one sentence or two that the operator reads in the log. Every reason <see cref="MigrationSkipReasonExtensions.IsBenign" /> accepts has one. + /// </summary> + internal static readonly IReadOnlyDictionary<MigrationSkipReason, string> HarmlessSkipExplanations = new Dictionary<MigrationSkipReason, string> + { + [MigrationSkipReason.PastRetention] = "PastRetention rows were past the retention period, so ServiceControl would have deleted them soon anyway.", + [MigrationSkipReason.BlankGroupComment] = "BlankGroupComment rows are blank group comments, and ServiceControl keeps no row for a blank comment.", + [MigrationSkipReason.SourceStorageCheck] = "SourceStorageCheck rows are results of the checks the source's own storage runs about itself, such as the RavenDB persister's checks on its database. Nothing reports those checks once the primary instance runs on SQL, so a copied result would never change." + }; + + static IEnumerable<string> HarmlessReasonsIn(MigrationCheckpoint checkpoint) => + checkpoint.SkipReasons?.Keys.Where(reason => reason.IsBenign()).Order().Select(reason => HarmlessSkipExplanations[reason]) ?? []; + // A retry reads the whole category again, so it is offered when a retry could copy some of the skipped rows, or when a halt left rows unread. static string WhatARetryCanDo(MigrationCheckpoint checkpoint) {