diff --git a/README.md b/README.md index 49f1a2a..4b6846b 100644 --- a/README.md +++ b/README.md @@ -2,4 +2,48 @@ A Node.js/TypeScript CLI (`github-backup`) that backs up every repository you can reach through the GitHub CLI (`gh`): your personal repositories and those of all organizations you belong to, with wikis and Git LFS objects. Backups go to a local directory and/or any S3-compatible bucket (AWS S3, Cloudflare R2, Backblaze B2, MinIO) as git bundle snapshots, and a new snapshot is created only when refs change. -Status: under development. `github-backup list` works; `backup` is not implemented yet. See the plan (Turkish) in [plans/README.md](plans/README.md). +Status: under development. `list` and `backup` to local targets work; S3 targets and Git LFS are not implemented yet. See the plan (Turkish) in [plans/README.md](plans/README.md). + +## Requirements + +Node.js 22 or newer, `git`, and the GitHub CLI (`gh`) signed in with at least the `repo` and `read:org` scopes. Linux and macOS. + +## Build + +```sh +npm ci && npm run build +node dist/cli.js list +``` + +## Usage + +```sh +github-backup list [--include ] [--exclude ] [--no-archived] [--no-forks] [--json] +github-backup backup --local ~/git-backup [--keep 7] [--concurrency 4] [--no-wiki] [--json] +``` + +`backup` mirrors every repository (and its wiki) into the cache directory (`~/.cache/github-backup-cli` by default), and writes a new snapshot to each target when the branches or tags changed since the last one. Mirrors only ever grow: branches deleted on GitHub stay in the backup. Each target keeps the newest `keep` snapshots per repository (default 7; a target's own `keep` wins over `--keep`). + +`--target ` runs only the named targets from the config file; `--local ` adds a temporary `local` target. Without any target the command exits with code 2. Log lines go to stderr; `--json` prints a machine-readable summary to stdout. + +Exit codes: `0` success, `1` at least one repository failed (the others are still backed up), `2` configuration or prerequisite error, `3` another run is in progress, `130` interrupted. + +## Snapshot layout + +```text +////.bundle +////.json # manifest, written last +///.wiki/... # wiki, same layout +``` + +`` is the UTC time as `YYYYMMDDTHHMMSSZ`. A snapshot without its `.json` manifest is incomplete and is removed on the next run. + +## Restoring + +A bundle is a normal git repository history: + +```sh +git clone ~/git-backup/github/acme/app/20260101T020000Z.bundle app +``` + +To get every branch and tag as local refs instead, use `git clone --mirror app.git`. diff --git a/developer-todos.md b/developer-todos.md index 548a1ae..dc20341 100644 --- a/developer-todos.md +++ b/developer-todos.md @@ -19,7 +19,7 @@ Sıradaki fazı engelleyen işler. ### Faz 2 — Mirror, snapshot ve `local` hedefi -- [ ] 💻 Kendi repolarınla `backup --local` çalıştırıp bir bundle'ı `git clone` ile doğrula (private repo dahil): gerçek `gh` oturumu gerekir. +- [ ] 💻 `npm ci && npm run build && node dist/cli.js backup --local ~/git-backup-test --include '/*'` çalıştır, bir bundle'ı `git clone /tmp/x` ile aç ve private repo dahil olduğunu kontrol et; komutu tekrar çalıştırınca özetin `unchanged` göstermesini doğrula: gerçek `gh` oturumu gerekir, bulutta yok. ### Faz 3 — `s3` hedefi diff --git a/plans/README.md b/plans/README.md index cc525c6..d000a57 100644 --- a/plans/README.md +++ b/plans/README.md @@ -45,7 +45,7 @@ Fazlar tablodaki sırayla uygulanır; sıranın tek kaynağı bu tablodur. Her f | Faz | Başlık | Durum | Dosya | | --- | --- | --- | --- | | 1 | İskelet, config ve `list` | Tamamlandı | [faz-01-iskelet-ve-list.md](faz-01-iskelet-ve-list.md) | -| 2 | Mirror, snapshot ve `local` hedefi | Bekliyor | [faz-02-mirror-ve-local.md](faz-02-mirror-ve-local.md) | +| 2 | Mirror, snapshot ve `local` hedefi | Tamamlandı | [faz-02-mirror-ve-local.md](faz-02-mirror-ve-local.md) | | 3 | `s3` hedefi | Bekliyor | [faz-03-s3.md](faz-03-s3.md) | | 4 | Git LFS | Bekliyor | [faz-04-lfs.md](faz-04-lfs.md) | | 5 | Dokümantasyon ve son kontrol | Bekliyor | [faz-05-dokuman-ve-son-kontrol.md](faz-05-dokuman-ve-son-kontrol.md) | diff --git a/plans/faz-02-mirror-ve-local.md b/plans/faz-02-mirror-ve-local.md index 0f81cb6..8ed785f 100644 --- a/plans/faz-02-mirror-ve-local.md +++ b/plans/faz-02-mirror-ve-local.md @@ -1,6 +1,6 @@ # Faz 2 — Mirror, snapshot ve `local` hedefi -**Durum:** Bekliyor +**Durum:** Tamamlandı ## Amaç @@ -30,19 +30,19 @@ ## Görevler -- [ ] Test kaynağı (`test/` altında): verilen `file://` bare repolarını `RepoInfo` olarak döndüren sahte `Source`. -- [ ] `core/mirror.ts` + entegrasyon testi: ilk clone, yeni commit sonrası fetch, uzakta silinen branch'in mirror'da kalması, `refs/pull/*`'ın alınmaması. -- [ ] `core/snapshot.ts` + testler: aynı ref'ler → aynı özet; bundle'dan `git clone` ile geri yüklenen repoda tüm branch ve tag'ler. -- [ ] `targets/local.ts` + testler: yazma, `latestManifest`, listeleme, silme, yarım dosya temizliği. -- [ ] Saklama mantığı + birim testi. -- [ ] `core/lock.ts` + test. -- [ ] `core/run.ts` + entegrasyon testi: 3 repo, biri bozuk URL → diğer ikisi yedeklenir, çıkış 1, özet doğru; ikinci çalıştırmada değişiklik yoksa yeni snapshot yok (`unchanged`). -- [ ] Wiki akışı + test (wiki repo var / yok). -- [ ] `commands/backup.ts` bağlanır; `--json` özeti. +- [x] Test kaynağı (`test/helpers/git.ts`: `TestRemote`, `repoInfo`, `fakeSource`, `isolatedExec`): verilen `file://` bare repolarını `RepoInfo` olarak döndüren sahte `Source`. +- [x] `core/mirror.ts` + entegrasyon testi: ilk clone, yeni commit sonrası fetch, uzakta silinen branch'in mirror'da kalması, `refs/pull/*`'ın alınmaması. +- [x] `core/snapshot.ts` + testler: aynı ref'ler → aynı özet; bundle'dan `git clone` ile geri yüklenen repoda tüm branch ve tag'ler. +- [x] `targets/local.ts` + testler: yazma, `latestManifest`, listeleme, silme, yarım dosya temizliği. +- [x] Saklama mantığı + birim testi. +- [x] `core/lock.ts` + test. +- [x] `core/run.ts` + entegrasyon testi: 3 repo, biri bozuk URL → diğer ikisi yedeklenir, çıkış 1, özet doğru; ikinci çalıştırmada değişiklik yoksa yeni snapshot yok (`unchanged`). +- [x] Wiki akışı + test (wiki repo var / yok). +- [x] `commands/backup.ts` bağlanır; `--json` özeti. ## 👤 Kullanıcı aksiyonları -- [ ] 💻 Kendi makinende `node dist/cli.js backup --local ~/git-backup-test --include '/*'` çalıştır; bir bundle'ı `git clone /tmp/x` ile açıp doğrula. Private bir repo da dahil olsun (gh kimlik doğrulamasının çalıştığını kanıtlar). +- [ ] 💻 Kendi makinende `node dist/cli.js backup --local ~/git-backup-test --include '/*'` çalıştır; bir bundle'ı `git clone /tmp/x` ile açıp doğrula. Private bir repo da dahil olsun (gh kimlik doğrulamasının çalıştığını kanıtlar). İkinci çalıştırmada özetin `unchanged` göstermesini kontrol et. ## Kabul kriterleri @@ -59,4 +59,19 @@ ## Devam notları -_Henüz yok._ +Kod tarafı bitti. Lint, typecheck, test (178) ve build temiz. Kabul kriterlerinin tamamı otomatik testlerle doğrulandı (`test/integration/backup.test.ts`, `mirror-snapshot.test.ts`, `local-target.test.ts`; `test/unit/lock|retention|async.test.ts`). Açık kalan tek madde gerçek `gh` oturumuyla elle deneme (yukarıdaki 💻; bulutta `gh` oturumu yok). Bu denemede `refs/pull/*` yüzünden beklenmedik bir şey çıkarsa `core/mirror.ts` refspec'lerine bak. + +Faz 3 (s3) için: + +- `Target` arayüzü `targets/types.ts`'te: `latestManifest`, `putSnapshot` (bundle/lfsTar yerel yol, manifest nesne; manifest en son yazılır), `listSnapshots` (yalnızca manifest'i olanlar, eskiden yeniye), `deleteSnapshot` (manifest önce), `cleanupIncomplete`. S3 uygulaması aynısını yapmalı; saklama mantığı `core/retention.ts` `applyRetention`'da ve hedeften bağımsız. +- `targets/index.ts` `createTarget`'ta `s3` dalı şimdilik `ConfigError("... s3 targets are not available yet")` fırlatır; Faz 3'te `createS3Target` ile değiştirilecek. `HeadBucket` ön kontrolü için `commands/backup.ts`'te `createTarget`'tan sonra, `checkGit` yanına (kilitten önce, çıkış 2) bir `checkAccess` adımı eklenmeli; arayüze isteğe bağlı `checkAccess?()` eklenebilir. +- `local` hedefinin `.partial` + `rename` mantığının S3 karşılığı: `Upload` tamamlanana kadar nesne görünmez; sıra aynı (bundle, lfs.tar, manifest). `cleanupIncomplete` S3'te yalnızca manifest'siz bundle/lfs.tar nesnelerini silmeli (tamamlanmamış multipart yüklemeler ayrıca `AbortMultipartUpload` ister). +- Test yardımcıları: `test/helpers/git.ts` (`TestRemote`, `fakeSource`, `isolatedExec`); `test/integration/backup.test.ts` içindeki `setup()` `--local` kullanıyor, S3 testleri aynı kalıbı config dosyası veya env ile kurabilir. Her `now()` çağrısı sahte saatte 1 sn ilerler; gerçek saatte aynı saniyede iki snapshot aynı `stamp`'i alırdı (repo başına çalıştırma başına tek snapshot olduğundan sorun değil). + +Faz 4 (LFS) için: + +- Takılacak yerler `core/run.ts` `processUnit`: `syncMirror`'dan sonra `git lfs fetch --all origin` (git zaten kaynağın `gitEnv()`'iyle çalışıyor); `createBundle` çağrısının yanında `lfs.tar` üretimi ve `SnapshotFiles.lfsTar` / `ManifestInput.lfsTarBytes` (ikisi de hazır). `BackupInput`'a `lfs: boolean` eklenmeli; `commands/backup.ts` `config.lfs`'i geçirmeli ve `checkGit` yanına `git lfs version` ön koşulunu eklemeli (çıkış 2, `--no-lfs` önerisi). +- `local` hedefi `.lfs.tar`'ı bundle ile birlikte yazar, siler ve manifest'siz bırakılanı temizler (testli); LFS tarafında yalnızca üretim kalıyor. Wiki repolarında LFS adımı atlanmalı (`unit.kind === "wiki"`). +- Bu ortamda `git-lfs` 3.4.1 kurulu; Faz 4 entegrasyon testi yerelde çalıştırılabilir. + +Genel notlar: `--keep` yalnızca genel değeri belirler, hedefteki `keep` onu ezer (K-032). `exec`'in `signal` seçeneği Node'un `spawn({signal})`'ini kullanır; iptalde hata kodu `ABORT_ERR`'dir. Özet, repo ve wiki'yi ayrı birim sayar. Mirror dizini `///.git`, wiki `.wiki.git`; geçici bundle'lar `/tmp/unit-*` altındadır ve her çalıştırma sonunda silinir. diff --git a/plans/faz-03-s3.md b/plans/faz-03-s3.md index 0ec73ea..ec360de 100644 --- a/plans/faz-03-s3.md +++ b/plans/faz-03-s3.md @@ -12,7 +12,7 @@ Snapshot'lar S3 uyumlu bir bucket'a (AWS S3, R2, B2, MinIO) `local` ile aynı d ## Kapsam -- `targets/s3.ts`: `@aws-sdk/client-s3` + `@aws-sdk/lib-storage` `Upload` (stream). Config: `bucket`, `prefix`, `endpoint`, `region` (varsayılan `us-east-1`), `forcePathStyle`, `accessKeyId`, `secretAccessKey`, `keep`. +- `targets/s3.ts` (`targets/index.ts`'teki yer tutucunun yerine; `Target` arayüzü `cleanupIncomplete` dahil): `@aws-sdk/client-s3` + `@aws-sdk/lib-storage` `Upload` (stream). Config: `bucket`, `prefix`, `endpoint`, `region` (varsayılan `us-east-1`), `forcePathStyle`, `accessKeyId`, `secretAccessKey`, `keep`. - Kimlik önceliği: config → `GHBACKUP_S3_*` env → SDK varsayılan zinciri. - `latestManifest`: `ListObjectsV2` (sayfalı) ile `//` altındaki `.json`'lar, en yenisi `GetObject`. - Yükleme sırası: bundle, lfs.tar (varsa), manifest. @@ -51,4 +51,4 @@ Snapshot'lar S3 uyumlu bir bucket'a (AWS S3, R2, B2, MinIO) `local` ile aynı d ## Devam notları -_Henüz yok._ +_Henüz başlanmadı._ Faz 2'den devralınanlar için [faz-02-mirror-ve-local.md](faz-02-mirror-ve-local.md) **Devam notları**'na bak: `Target` arayüzü (`targets/types.ts`, `cleanupIncomplete` dahil), `createTarget`'taki `s3` yer tutucusu, `applyRetention`, `HeadBucket` ön kontrolünün takılacağı yer ve test yardımcıları. diff --git a/plans/faz-04-lfs.md b/plans/faz-04-lfs.md index 5b88c5b..1f7798a 100644 --- a/plans/faz-04-lfs.md +++ b/plans/faz-04-lfs.md @@ -49,4 +49,4 @@ LFS kullanan repoların LFS nesneleri de yedeklenir: her yeni snapshot'ın yanı ## Devam notları -_Henüz yok._ +_Henüz başlanmadı._ Faz 2'den devralınanlar için [faz-02-mirror-ve-local.md](faz-02-mirror-ve-local.md) **Devam notları**'na bak: LFS adımının `core/run.ts` içinde takılacağı yer, `lfsTar`/`lfsTarBytes` için hazır alanlar ve ön koşulun `commands/backup.ts`'teki yeri. diff --git a/plans/kararlar.md b/plans/kararlar.md index 7bd4b4d..48b9810 100644 --- a/plans/kararlar.md +++ b/plans/kararlar.md @@ -35,3 +35,6 @@ Projenin güncel kararları. Bir karar değişirse satırı güncellenir ve tari | K-029 | 2026-10-14 | Desteklenen OS: Linux ve macOS. | Hedef kullanım cron/launchd; Windows test edilmez. | | K-030 | 2026-10-14 | Glob eşleşmesi `owner/repo` üzerindedir: `*` ve `?` `/` geçmez, `**` geçer; büyük/küçük harf duyarsız; boş `include` her şeyi kabul eder. | Beklenen `acme/*` davranışı; `*/*` her repoyu yakalar; ayrıntı mimari.md'de. | | K-031 | 2026-10-14 | Config şeması katıdır (bilinmeyen alan hata verir) ve hedef adları benzersizdir. S3 anahtarı config'te yoksa `GHBACKUP_S3_ACCESS_KEY_ID`/`_SECRET_ACCESS_KEY` env'i kullanılır. | Yazım hatası sessizce yok sayılmasın; `--target` adla seçtiği için ad çakışmasın; K-010'un tek istisnası mimari.md'de belirtilen S3 anahtarı sırası. | +| K-032 | 2026-10-14 | `keep` çözümü: hedefte `keep` varsa o, yoksa genel `keep` (bayrak > env > dosya ile belirlenen). Hedefteki değer `--keep` bayrağını ezer. | Hedef başına saklama süresi config'te bilinçli verilir (ör. S3 için 14); tek `resolveKeep` fonksiyonu. | +| K-033 | 2026-10-14 | Mirror fetch'i `git fetch --no-tags --no-prune origin`; her fetch'ten sonra `HEAD`, `git ls-remote --symref` ile uzağın varsayılan branch'ine ayarlanır. | Tag'ler refspec'ten gelir, prune hiçbir koşulda yapılmaz (K-008); bundle'dan klonlanan repo checkout ile açılır. | +| K-034 | 2026-10-14 | Wiki ayrı birimdir (`.wiki` anahtarı, kendi mirror'ı ve snapshot'ları). Wiki'de "repository not found" `skipped: no wiki`; ref'siz mirror `skipped: empty`. | Wiki açık ama sayfasız repolar hata üretmesin; wiki yedeği ana repodan bağımsız saklanır. | diff --git a/plans/mimari.md b/plans/mimari.md index 020821e..1346ff8 100644 --- a/plans/mimari.md +++ b/plans/mimari.md @@ -28,8 +28,10 @@ src/ load.ts # dosya + env + bayrak birleştirme core/ discover.ts # kaynaklardan repo listesi + filtreler + git.ts # Exec üzerinden git çalıştırıcı (kaynağın gitEnv'i, sinyal, verbose log) mirror.ts # bare mirror clone/fetch, wiki, LFS fetch snapshot.ts # ref özeti, bundle ve lfs.tar üretimi, manifest + retention.ts # keep çözümü, süresi dolan snapshot seçimi, saklama uygulaması run.ts # paralel iş akışı, yeniden deneme, özet lock.ts # eşzamanlı çalıştırma kilidi sources/ @@ -37,7 +39,7 @@ src/ github.ts # gh tabanlı GitHub kaynağı index.ts # tür → uygulama kaydı targets/ - types.ts # Target arayüzü + types.ts # Target arayüzü, Manifest, SnapshotFiles local.ts s3.ts index.ts @@ -45,7 +47,9 @@ src/ exec.ts # child_process sarmalayıcısı (timeout, stderr yakalama) log.ts # stderr logger (normal/verbose/quiet) filter.ts # glob eşleştirme - time.ts # snapshot zaman damgası + time.ts # snapshot zaman damgası, sleep + async.ts # withRetry, runPool + tool.ts # package.json'dan araç adı ve sürümü errors.ts # ExitCode, CliError, ConfigError, PrerequisiteError test/ unit/ @@ -60,11 +64,12 @@ test/ - `listRepos(): Promise<{ repos: RepoInfo[]; failures: { owner, message }[] }>`: `RepoInfo` = `{ source, owner, name, fullName, cloneUrl, wikiUrl?, hasWiki, isArchived, isFork, isEmpty }`. Bir sahibin (ör. SSO'lu org) repoları listelenemezse hata fırlatılmaz, `failures`'a eklenir; çağıran uyarı verir ve çıkış kodu 1 olur. - `gitEnv(): Record`: git alt süreçlerine verilecek kimlik doğrulama env'i. - `Target` arayüzü: - - `name`, `type`, `keep`. - - `latestManifest(repoKey): Promise` - - `putSnapshot(repoKey, stamp, files: { bundle, manifest, lfsTar? })` - - `listSnapshots(repoKey): Promise` (zaman damgaları) - - `deleteSnapshot(repoKey, stamp)` + - `name`, `type`, `keep` (çözülmüş değer; `createTarget(config, { keep })` ile verilir). + - `latestManifest(repoKey): Promise`: manifest'i olan en yeni snapshot. + - `putSnapshot(repoKey, stamp, files: { bundle, manifest, lfsTar? })`: `bundle` ve `lfsTar` yerel dosya yoludur, `manifest` nesnedir; hedef en son manifest'i yazar. + - `listSnapshots(repoKey): Promise`: manifest'i olan (tamamlanmış) snapshot'ların zaman damgaları, eskiden yeniye. + - `deleteSnapshot(repoKey, stamp)`: bundle, lfs.tar ve manifest silinir (manifest ilk). + - `cleanupIncomplete(repoKey)`: manifest'i olmayan yarım dosyaları ve `.partial` dosyalarını siler. - `repoKey` = `//` (wiki için `//.wiki`). - Kaynak türü → uygulama eşlemesi `sources/index.ts` içindeki `createSource` fonksiyonundadır; `core/discover.ts` her kaynak yapılandırması için ön koşulları kontrol eder, listeler ve o kaynağın filtrelerini uygular. - Dış komut çağıran modüller `Exec` fonksiyon tipini (`util/exec.ts`) bağımlılık olarak alır; testlerde sahte `exec` verilir. Komutlar `CommandDeps` ile `exec`, `env`, `stdout`, `stderr` alır. @@ -83,22 +88,22 @@ test/ 1. Config yüklenir. En az bir hedef yoksa hata (çıkış 2). 2. Ön koşullar: `git`, `gh` kurulu ve `gh auth status` başarılı; LFS açıksa `git lfs version` başarılı. Biri eksikse kurulum/oturum talimatıyla hata (çıkış 2). `git-lfs` yoksa mesaj `--no-lfs` seçeneğini de söyler. -3. Önbellek dizininde kilit alınır (`/.lock`, içinde PID). Kilit canlı bir süreçteyse çıkış 3. Ölü PID'ye ait kilit devralınır. +3. Önbellek dizininde kilit alınır (`/.lock`, içinde PID; tam içerikli dosya `link` ile atomik oluşturulur). Kilit canlı bir süreçteyse çıkış 3. Ölü PID'ye ait kilit devralınır. Kilit alındıktan sonra önceki bir çalıştırmadan kalan `/tmp` silinir. 4. Repolar listelenir ve filtrelenir. 5. Her repo (ve wiki'si) `concurrency` sınırıyla paralel işlenir: - 1. **Mirror:** `///.git` yoksa `git init --bare`, `git remote add origin ` ve fetch refspec'leri `+refs/heads/*:refs/heads/*` ile `+refs/tags/*:refs/tags/*` olarak ayarlanır (`refs/pull/*` alınmaz). Sonra her çalıştırmada `git fetch origin` (prune yok). `isEmpty` olan repo atlanır (`skipped: empty`). + 1. **Mirror:** `///.git` yoksa `git init --bare`, `git remote add origin ` ve fetch refspec'leri `+refs/heads/*:refs/heads/*` ile `+refs/tags/*:refs/tags/*` olarak ayarlanır (`refs/pull/*` alınmaz). Sonra her çalıştırmada `remote.origin.url` güncellenir ve `git fetch --no-tags --no-prune origin` çalışır (tag'ler refspec'ten gelir). Ardından `git ls-remote --symref origin HEAD` ile mirror'ın `HEAD`'i uzağın varsayılan branch'ine ayarlanır (bundle `git clone` ile açıldığında checkout olsun diye; başarısız olursa sessizce geçilir). `isEmpty` olan repo atlanır (`skipped: empty`); fetch sonrası hiç ref yoksa da `skipped: empty`. İlk fetch başarısız olursa yeni oluşturulan boş mirror dizini silinir. 2. **LFS:** LFS açıksa `git lfs fetch --all origin`. Repoda LFS nesnesi yoksa adım no-op'tur. - 3. **Ref özeti:** `git for-each-ref --format='%(objectname) %(refname)'` çıktısının sha256'sı. + 3. **Ref özeti:** `git for-each-ref --format='%(objectname) %(refname)'` çıktısı ref adına göre sıralanır; satırların (` \n`) sha256'sı `refsHash` olur. 4. **Hedef karşılaştırma:** her hedef için `latestManifest`; `refsHash` aynıysa o hedef için `unchanged`. - 5. **Snapshot:** en az bir hedef değişiklik istiyorsa `/tmp/` altında `git bundle create --all` ve LFS nesnesi varsa `lfs/objects` klasörünün tar'ı oluşturulur. + 5. **Snapshot:** en az bir hedef değişiklik istiyorsa `/tmp/unit-*/` altında `git bundle create --all` ve LFS nesnesi varsa `lfs/objects` klasörünün tar'ı oluşturulur. Zaman damgası bu adımda alınır ve tüm hedeflere aynıdır. 6. **Yükleme:** değişiklik isteyen her hedefe `putSnapshot`. Sıra: önce `.bundle` ve `.lfs.tar`, en son `.json` manifest (manifest, snapshot'ın tamamlandığının işaretidir). - 7. **Saklama:** hedefte manifest'i olan snapshot'lar zamana göre sıralanır, `keep`'ten eskiler (bundle, lfs.tar, json) silinir. Manifest'i olmayan yarım kalmış dosyalar da silinir. + 7. **Saklama:** yeni snapshot yazılan her hedefte `applyRetention`: manifest'i olan snapshot'lar zamana göre sıralanır, `keep`'ten eskiler (bundle, lfs.tar, json) silinir, ardından `cleanupIncomplete` manifest'i olmayan yarım dosyaları siler. Bir hedefin yazma veya saklama adımı hata verirse o hedef `failed` olur, diğer hedefler etkilenmez. 8. Geçici dosyalar silinir. -6. Her adım hata alırsa 2 kez yeniden denenir (1 sn, 5 sn bekleme). Yine başarısızsa repo `failed` olur, diğerleri devam eder. -7. Wiki: `hasWiki` true ise `.wiki.git` aynı adımlardan geçer. Wiki klonu "repository not found" ile başarısız olursa (wiki açık ama sayfa yok) `skipped: no wiki`, hata sayılmaz. -8. Sonda özet stderr'e: başarılı / değişmedi / atlanan / başarısız sayıları ve başarısızların listesi. `--json` ile özet JSON olarak stdout'a. +6. Her adım (mirror, hedef sorgusu, bundle, yükleme, saklama) hata alırsa 2 kez yeniden denenir (1 sn, 5 sn bekleme; `withRetry`). Yine başarısızsa o birim `failed` olur, diğerleri devam eder. +7. Wiki: `wiki` açıkken `hasWiki` true ise `.wiki.git` ayrı bir birim olarak aynı adımlardan geçer (repoKey `//.wiki`, manifest `fullName` değeri `/.wiki`). Wiki fetch'i "repository not found" ile başarısız olursa (wiki açık ama sayfa yok) `skipped: no wiki`, hata sayılmaz; aynı hata ana repoda gerçek hatadır. Özet birim (repo veya wiki) başına sayar. +8. Sonda özet stderr'e: başarılı / değişmedi / atlanan / başarısız sayıları ve başarısızların listesi (listelenemeyen sahipler dahil). `--json` ile stdout'a `{ summary: { ok, unchanged, skipped, failed, interrupted }, failures: [{ repo, error, targets }], listFailures, results }` yazılır. 9. Çıkış kodu: `0` hepsi başarılı veya atlandı/değişmedi; `1` en az bir repo veya hedef başarısız; `2` config/ön koşul hatası; `3` başka bir çalıştırma sürüyor; `130` SIGINT. -10. SIGINT/SIGTERM: devam eden alt süreçler sonlandırılır, geçici dosyalar ve kilit silinir. +10. SIGINT/SIGTERM: `AbortSignal` ile devam eden alt süreçler sonlandırılır, yeni birim başlatılmaz, geçici dosyalar ve kilit silinir, çıkış 130. İkinci sinyal varsayılan davranışla süreci hemen öldürür. ## Kaynak: `github` @@ -169,7 +174,7 @@ Config dosyası: varsayılan `~/.config/github-backup-cli/config.json` (`XDG_CON ``` - `sources` verilmezse varsayılan tek `github` kaynağıdır. -- Hedef başına `keep` genel `keep`'i ezer. +- Hedef başına `keep` genel `keep`'i ezer; `--keep`, `GHBACKUP_KEEP` ve dosyadaki `keep` genel değeri belirler (`resolveKeep`). - Şema katı (`strict`): bilinmeyen alan hata verir; hedef adları benzersiz olmalıdır. Hata mesajı alan yolunu içerir (ör. `targets.0.bucket`). - Bayrak veya env ile verilen `include`/`exclude`/`archived`/`forks` her `github` kaynağının değerini ezer. - S3 anahtarı için tek istisna: config'te yoksa `GHBACKUP_S3_ACCESS_KEY_ID`/`GHBACKUP_S3_SECRET_ACCESS_KEY` kullanılır (config önce gelir). diff --git a/src/commands/backup.ts b/src/commands/backup.ts index cd7f75e..052cb6c 100644 --- a/src/commands/backup.ts +++ b/src/commands/backup.ts @@ -1,9 +1,115 @@ -import { CliError, ExitCode } from "../util/errors.js"; -import type { CommandDeps, CommonOptions } from "./context.js"; +import path from "node:path"; +import { discoverRepos } from "../core/discover.js"; +import { acquireLock, type Lock } from "../core/lock.js"; +import { resolveKeep } from "../core/retention.js"; +import { backupRepos, type Summary } from "../core/run.js"; +import type { ListFailure } from "../sources/types.js"; +import { createTarget } from "../targets/index.js"; +import { ConfigError, ExitCode, PrerequisiteError } from "../util/errors.js"; +import type { Exec } from "../util/exec.js"; +import { readToolInfo } from "../util/tool.js"; +import { type CommandDeps, type CommonOptions, prepare } from "./context.js"; export interface BackupOptions extends CommonOptions {} -/** `github-backup backup`. Mirroring and snapshots arrive in a later phase. */ -export async function runBackup(_options: BackupOptions, _deps: CommandDeps): Promise { - throw new CliError("backup is not implemented yet", ExitCode.Usage); +async function checkGit(exec: Exec): Promise { + try { + await exec("git", ["--version"]); + } catch { + throw new PrerequisiteError("git is not installed or not in PATH; install git and try again"); + } +} + +/** Lines printed to stderr at the end of a run. */ +export function renderSummary(summary: Summary, listFailures: ListFailure[]): string[] { + const lines = [ + `Backup finished: ${summary.ok} backed up, ${summary.unchanged} unchanged, ` + + `${summary.skipped} skipped, ${summary.failed} failed`, + ]; + const failed = summary.results.filter((r) => r.status === "failed"); + for (const result of failed) lines.push(` failed: ${result.key}: ${result.error}`); + for (const failure of listFailures) { + lines.push(` not listed: ${failure.owner}: ${failure.message}`); + } + if (summary.interrupted) lines.push("Interrupted before all repositories were processed."); + return lines; +} + +export function renderJson(summary: Summary, listFailures: ListFailure[]): string { + const { results, ...counts } = summary; + return JSON.stringify( + { + summary: counts, + failures: results + .filter((r) => r.status === "failed") + .map((r) => ({ repo: r.key, error: r.error, targets: r.targets })), + listFailures, + results, + }, + null, + 2, + ); +} + +/** `github-backup backup`. Returns the process exit code. */ +export async function runBackup(options: BackupOptions, deps: CommandDeps): Promise { + const { loaded, log } = prepare(options, deps); + const { config } = loaded; + if (config.targets.length === 0) { + throw new ConfigError("no targets configured; use --local or add targets to config"); + } + const targets = config.targets.map((target) => + createTarget(target, { keep: resolveKeep(target.keep, config.keep) }), + ); + await checkGit(deps.exec); + + const controller = new AbortController(); + const abort = () => controller.abort(); + if (deps.signal?.aborted) abort(); + deps.signal?.addEventListener("abort", abort, { once: true }); + process.once("SIGINT", abort); + process.once("SIGTERM", abort); + let lock: Lock | undefined; + try { + lock = acquireLock(path.join(config.cacheDir, ".lock")); + const discovery = await discoverRepos(config.sources, { + exec: deps.exec, + log, + createSource: deps.createSource, + }); + if (controller.signal.aborted) return ExitCode.Interrupted; + + const summary = await backupRepos( + { + repos: discovery.repos, + sources: discovery.sources, + targets, + cacheDir: config.cacheDir, + concurrency: config.concurrency, + wiki: config.wiki, + }, + { + exec: deps.exec, + log, + tool: readToolInfo(), + signal: controller.signal, + now: deps.now, + sleep: deps.sleep, + retryDelaysMs: deps.retryDelaysMs, + }, + ); + + for (const line of renderSummary(summary, discovery.failures)) log.summary(line); + if (options.json) deps.stdout(`${renderJson(summary, discovery.failures)}\n`); + + if (summary.interrupted) return ExitCode.Interrupted; + return summary.failed > 0 || discovery.failures.length > 0 + ? ExitCode.PartialFailure + : ExitCode.Success; + } finally { + lock?.release(); + deps.signal?.removeEventListener("abort", abort); + process.removeListener("SIGINT", abort); + process.removeListener("SIGTERM", abort); + } } diff --git a/src/commands/context.ts b/src/commands/context.ts index 514872b..ab8c10c 100644 --- a/src/commands/context.ts +++ b/src/commands/context.ts @@ -16,6 +16,12 @@ export interface CommandDeps { homeDir?: string; cwd?: string; createSource?: (config: SourceConfig, deps: { exec: Exec }) => Source; + /** Aborts a running backup, like SIGINT does. */ + signal?: AbortSignal; + /** Clock and waiting, replaced in tests. */ + now?: () => Date; + sleep?: (ms: number, signal?: AbortSignal) => Promise; + retryDelaysMs?: readonly number[]; } export interface CommonOptions extends ConfigFlags { diff --git a/src/core/git.ts b/src/core/git.ts new file mode 100644 index 0000000..96c1fa2 --- /dev/null +++ b/src/core/git.ts @@ -0,0 +1,20 @@ +import { type Exec, type ExecResult, redactUrls } from "../util/exec.js"; +import type { Logger } from "../util/log.js"; + +/** Runs `git` with the source's authentication environment. */ +export type Git = (args: string[], opts?: { cwd?: string }) => Promise; + +export interface GitOptions { + exec: Exec; + env: Record; + signal?: AbortSignal; + log?: Logger; + scope?: string; +} + +export function createGit(options: GitOptions): Git { + return (args, opts = {}) => { + options.log?.debug(redactUrls(`git ${args.join(" ")}`), options.scope); + return options.exec("git", args, { env: options.env, cwd: opts.cwd, signal: options.signal }); + }; +} diff --git a/src/core/lock.ts b/src/core/lock.ts new file mode 100644 index 0000000..f1c27be --- /dev/null +++ b/src/core/lock.ts @@ -0,0 +1,68 @@ +import { linkSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { CliError, ExitCode } from "../util/errors.js"; + +export interface Lock { + release(): void; +} + +export interface LockOptions { + pid?: number; + isAlive?: (pid: number) => boolean; +} + +export function isProcessAlive(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch (err) { + return (err as NodeJS.ErrnoException).code === "EPERM"; + } +} + +function readPid(file: string): number | undefined { + try { + const pid = Number.parseInt(readFileSync(file, "utf8").trim(), 10); + return Number.isInteger(pid) && pid > 0 ? pid : undefined; + } catch { + return undefined; + } +} + +/** + * Takes the run lock (`/.lock` holding our PID). A lock whose PID is gone is taken + * over; a live one ends the run with exit code 3 (K-027). The PID file is created complete + * via a hard link, so a concurrent reader never sees a half-written lock. + */ +export function acquireLock(file: string, options: LockOptions = {}): Lock { + const pid = options.pid ?? process.pid; + const isAlive = options.isAlive ?? isProcessAlive; + mkdirSync(path.dirname(file), { recursive: true }); + const staging = `${file}.${pid}.tmp`; + writeFileSync(staging, `${pid}\n`); + try { + for (let attempt = 0; attempt < 3; attempt++) { + try { + linkSync(staging, file); + return { + release() { + if (readPid(file) === pid) rmSync(file, { force: true }); + }, + }; + } catch (err) { + if ((err as NodeJS.ErrnoException).code !== "EEXIST") throw err; + const holder = readPid(file); + if (holder !== undefined && isAlive(holder)) { + throw new CliError( + `another backup run is in progress (pid ${holder}, lock ${file})`, + ExitCode.Locked, + ); + } + rmSync(file, { force: true }); + } + } + throw new CliError(`could not acquire the lock ${file}`, ExitCode.Locked); + } finally { + rmSync(staging, { force: true }); + } +} diff --git a/src/core/mirror.ts b/src/core/mirror.ts new file mode 100644 index 0000000..b3f66fe --- /dev/null +++ b/src/core/mirror.ts @@ -0,0 +1,61 @@ +import { existsSync } from "node:fs"; +import { mkdir, rm } from "node:fs/promises"; +import { ExecError } from "../util/exec.js"; +import type { Git } from "./git.js"; + +export type MirrorResult = { status: "ok" } | { status: "skipped"; reason: string }; + +export interface MirrorInput { + /** Bare repository directory inside the cache. */ + mirrorDir: string; + url: string; + /** A missing wiki repository is expected (wiki enabled, no pages yet), not an error. */ + kind: "repo" | "wiki"; +} + +const FETCH_REFSPECS = ["+refs/heads/*:refs/heads/*", "+refs/tags/*:refs/tags/*"]; +const NOT_FOUND = /repository .*not found|does not appear to be a git repository/i; + +async function initMirror(git: Git, input: MirrorInput): Promise { + await mkdir(input.mirrorDir, { recursive: true }); + await git(["init", "--bare", input.mirrorDir]); + const inRepo = { cwd: input.mirrorDir }; + await git(["remote", "add", "origin", input.url], inRepo); + // Only branches and tags: refs/pull/* is never fetched, and nothing is pruned (K-008, K-028). + await git( + ["config", "--replace-all", "remote.origin.fetch", FETCH_REFSPECS[0] as string], + inRepo, + ); + await git(["config", "--add", "remote.origin.fetch", FETCH_REFSPECS[1] as string], inRepo); +} + +/** Points the mirror's HEAD at the remote's default branch so bundles clone with a checkout. */ +async function syncHead(git: Git, mirrorDir: string): Promise { + try { + const { stdout } = await git(["ls-remote", "--symref", "origin", "HEAD"], { cwd: mirrorDir }); + const branch = /^ref:\s+(refs\/heads\/\S+)\s+HEAD$/m.exec(stdout)?.[1]; + if (branch) await git(["symbolic-ref", "HEAD", branch], { cwd: mirrorDir }); + } catch (err) { + if (err instanceof ExecError && err.code === "ABORT_ERR") throw err; + // The default branch is cosmetic; the refs themselves are already fetched. + } +} + +/** Creates the bare mirror when missing and fetches new refs into it (no pruning). */ +export async function syncMirror(git: Git, input: MirrorInput): Promise { + const created = !existsSync(`${input.mirrorDir}/HEAD`); + if (created) await initMirror(git, input); + const inRepo = { cwd: input.mirrorDir }; + try { + await git(["config", "remote.origin.url", input.url], inRepo); + await git(["fetch", "--no-tags", "--no-prune", "origin"], inRepo); + } catch (err) { + if (created) await rm(input.mirrorDir, { recursive: true, force: true }); + if (input.kind === "wiki" && err instanceof ExecError && NOT_FOUND.test(err.stderr)) { + return { status: "skipped", reason: "no wiki" }; + } + throw err; + } + await syncHead(git, input.mirrorDir); + return { status: "ok" }; +} diff --git a/src/core/retention.ts b/src/core/retention.ts new file mode 100644 index 0000000..3d68fa7 --- /dev/null +++ b/src/core/retention.ts @@ -0,0 +1,21 @@ +import type { Target } from "../targets/types.js"; + +/** Hedef `keep` > genel `keep` (K-009). */ +export function resolveKeep(targetKeep: number | undefined, globalKeep: number): number { + return targetKeep ?? globalKeep; +} + +/** Stamps older than the newest `keep` ones. Stamps sort chronologically as plain strings. */ +export function selectExpired(stamps: readonly string[], keep: number): string[] { + const newestFirst = [...stamps].sort().reverse(); + return newestFirst.slice(Math.max(keep, 0)).reverse(); +} + +/** Deletes snapshots beyond `keep` and files of interrupted uploads for one repository. */ +export async function applyRetention(target: Target, repoKey: string): Promise { + const stamps = await target.listSnapshots(repoKey); + for (const stamp of selectExpired(stamps, target.keep)) { + await target.deleteSnapshot(repoKey, stamp); + } + await target.cleanupIncomplete(repoKey); +} diff --git a/src/core/run.ts b/src/core/run.ts new file mode 100644 index 0000000..637db8b --- /dev/null +++ b/src/core/run.ts @@ -0,0 +1,247 @@ +import { mkdir, mkdtemp, rm } from "node:fs/promises"; +import path from "node:path"; +import type { RepoInfo, Source } from "../sources/types.js"; +import type { Target } from "../targets/types.js"; +import { runPool, withRetry } from "../util/async.js"; +import type { Exec } from "../util/exec.js"; +import type { Logger } from "../util/log.js"; +import { formatStamp, sleep } from "../util/time.js"; +import type { ToolInfo } from "../util/tool.js"; +import { createGit } from "./git.js"; +import { syncMirror } from "./mirror.js"; +import { applyRetention } from "./retention.js"; +import { buildManifest, createBundle, hashRefs, listRefs } from "./snapshot.js"; + +export type UnitKind = "repo" | "wiki"; +export type UnitStatus = "ok" | "unchanged" | "skipped" | "failed"; + +export interface TargetResult { + target: string; + status: "created" | "unchanged" | "failed"; + stamp?: string; + error?: string; +} + +/** Outcome for one repository or one wiki. */ +export interface UnitResult { + /** `//` or `//.wiki`. */ + key: string; + kind: UnitKind; + status: UnitStatus; + reason?: string; + error?: string; + targets: TargetResult[]; +} + +export interface Summary { + ok: number; + unchanged: number; + skipped: number; + failed: number; + results: UnitResult[]; + /** True when the run was aborted by a signal. */ + interrupted: boolean; +} + +export interface BackupInput { + repos: RepoInfo[]; + /** Sources by id, used for the git authentication environment. */ + sources: Map; + targets: Target[]; + cacheDir: string; + concurrency: number; + wiki: boolean; +} + +export interface BackupContext { + exec: Exec; + log: Logger; + tool: ToolInfo; + signal?: AbortSignal; + now?: () => Date; + sleep?: (ms: number, signal?: AbortSignal) => Promise; + /** Waits before the retries of a failed step (K-020). */ + retryDelaysMs?: readonly number[]; +} + +export const RETRY_DELAYS_MS = [1000, 5000] as const; + +interface Unit { + key: string; + kind: UnitKind; + repo: RepoInfo; + url?: string; + mirrorDir: string; + /** Set when there is nothing to do for this unit. */ + skip?: string; +} + +function planUnits(input: BackupInput): Unit[] { + const units: Unit[] = []; + for (const repo of input.repos) { + const key = `${repo.source}/${repo.owner}/${repo.name}`; + const mirrorBase = path.join(input.cacheDir, repo.source, repo.owner); + units.push({ + key, + kind: "repo", + repo, + url: repo.cloneUrl, + mirrorDir: path.join(mirrorBase, `${repo.name}.git`), + ...(repo.isEmpty ? { skip: "empty" } : {}), + }); + if (input.wiki && repo.hasWiki) { + units.push({ + key: `${key}.wiki`, + kind: "wiki", + repo, + url: repo.wikiUrl, + mirrorDir: path.join(mirrorBase, `${repo.name}.wiki.git`), + ...(repo.wikiUrl ? {} : { skip: "no wiki" }), + }); + } + } + return units; +} + +const message = (err: unknown): string => (err instanceof Error ? err.message : String(err)); + +function finish(unit: Unit, status: UnitStatus, targets: TargetResult[], extra = {}): UnitResult { + return { key: unit.key, kind: unit.kind, status, targets, ...extra }; +} + +async function processUnit( + unit: Unit, + input: BackupInput, + ctx: BackupContext, +): Promise { + const { log } = ctx; + if (unit.skip !== undefined) { + log.info(`skipped: ${unit.skip}`, unit.key); + return finish(unit, "skipped", [], { reason: unit.skip }); + } + const source = input.sources.get(unit.repo.source); + if (!source || !unit.url) { + return finish(unit, "failed", [], { error: `no source "${unit.repo.source}" for ${unit.key}` }); + } + + const git = createGit({ + exec: ctx.exec, + env: source.gitEnv(), + signal: ctx.signal, + log, + scope: unit.key, + }); + const retry = (fn: () => Promise) => + withRetry(fn, { + delaysMs: ctx.retryDelaysMs ?? RETRY_DELAYS_MS, + sleep: ctx.sleep ?? sleep, + signal: ctx.signal, + onRetry: (err, attempt) => log.warn(`retry ${attempt}: ${message(err)}`, unit.key), + }); + const url = unit.url; + let tmpDir: string | undefined; + + try { + const mirror = await retry(() => + syncMirror(git, { mirrorDir: unit.mirrorDir, url, kind: unit.kind }), + ); + if (mirror.status === "skipped") { + log.info(`skipped: ${mirror.reason}`, unit.key); + return finish(unit, "skipped", [], { reason: mirror.reason }); + } + + const refs = await listRefs(git, unit.mirrorDir); + if (refs.length === 0) { + log.info("skipped: empty", unit.key); + return finish(unit, "skipped", [], { reason: "empty" }); + } + const refsHash = hashRefs(refs); + + const results: TargetResult[] = []; + const pending: Target[] = []; + for (const target of input.targets) { + try { + const latest = await retry(() => target.latestManifest(unit.key)); + if (latest?.refsHash === refsHash) { + results.push({ target: target.name, status: "unchanged" }); + } else { + pending.push(target); + } + } catch (err) { + results.push({ target: target.name, status: "failed", error: message(err) }); + } + } + + if (pending.length > 0) { + tmpDir = await mkdtemp(path.join(input.cacheDir, "tmp", "unit-")); + const bundle = path.join(tmpDir, "snapshot.bundle"); + const bundleBytes = await retry(() => createBundle(git, unit.mirrorDir, bundle)); + const stamp = formatStamp((ctx.now ?? (() => new Date()))()); + const manifest = buildManifest({ + source: unit.repo.source, + fullName: unit.kind === "wiki" ? `${unit.repo.fullName}.wiki` : unit.repo.fullName, + stamp, + refs, + bundleBytes, + tool: ctx.tool, + }); + for (const target of pending) { + try { + await retry(() => target.putSnapshot(unit.key, stamp, { bundle, manifest })); + await retry(() => applyRetention(target, unit.key)); + results.push({ target: target.name, status: "created", stamp }); + log.info(`snapshot ${stamp} written to ${target.name}`, unit.key); + } catch (err) { + results.push({ target: target.name, status: "failed", error: message(err) }); + } + } + } + + const failed = results.filter((r) => r.status === "failed"); + if (failed.length > 0) { + const error = failed.map((r) => `${r.target}: ${r.error}`).join("; "); + log.error(error, unit.key); + return finish(unit, "failed", results, { error }); + } + const created = results.some((r) => r.status === "created"); + if (!created) log.info("unchanged", unit.key); + return finish(unit, created ? "ok" : "unchanged", results); + } catch (err) { + const error = ctx.signal?.aborted ? "interrupted" : message(err); + log.error(error, unit.key); + return finish(unit, "failed", [], { error }); + } finally { + if (tmpDir) await rm(tmpDir, { recursive: true, force: true }); + } +} + +export function summarize(results: UnitResult[], interrupted: boolean): Summary { + const count = (status: UnitStatus) => results.filter((r) => r.status === status).length; + return { + ok: count("ok"), + unchanged: count("unchanged"), + skipped: count("skipped"), + failed: count("failed"), + results, + interrupted, + }; +} + +/** Mirrors, snapshots and stores every repository (and wiki) with bounded concurrency. */ +export async function backupRepos(input: BackupInput, ctx: BackupContext): Promise { + const tmpRoot = path.join(input.cacheDir, "tmp"); + // The run lock is held, so leftovers in tmp/ belong to a run that was killed. + await rm(tmpRoot, { recursive: true, force: true }); + await mkdir(tmpRoot, { recursive: true }); + try { + const results = await runPool( + planUnits(input), + input.concurrency, + (unit) => processUnit(unit, input, ctx), + ctx.signal, + ); + return summarize(results, ctx.signal?.aborted ?? false); + } finally { + await rm(tmpRoot, { recursive: true, force: true }); + } +} diff --git a/src/core/snapshot.ts b/src/core/snapshot.ts new file mode 100644 index 0000000..97075d5 --- /dev/null +++ b/src/core/snapshot.ts @@ -0,0 +1,55 @@ +import { createHash } from "node:crypto"; +import { stat } from "node:fs/promises"; +import type { Manifest, RefEntry } from "../targets/types.js"; +import type { ToolInfo } from "../util/tool.js"; +import type { Git } from "./git.js"; + +/** Branches and tags of a mirror, sorted by ref name. */ +export async function listRefs(git: Git, mirrorDir: string): Promise { + const { stdout } = await git(["for-each-ref", "--format=%(objectname) %(refname)"], { + cwd: mirrorDir, + }); + return stdout + .split("\n") + .filter((line) => line.length > 0) + .map((line) => { + const [sha = "", name = ""] = line.split(" "); + return { name, sha }; + }); +} + +/** sha256 over the sorted `sha refname` lines; equal digests mean unchanged refs. */ +export function hashRefs(refs: RefEntry[]): string { + const sorted = [...refs].sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0)); + const text = sorted.map((ref) => `${ref.sha} ${ref.name}\n`).join(""); + return createHash("sha256").update(text).digest("hex"); +} + +export async function createBundle(git: Git, mirrorDir: string, file: string): Promise { + await git(["bundle", "create", file, "--all"], { cwd: mirrorDir }); + return (await stat(file)).size; +} + +export interface ManifestInput { + source: string; + fullName: string; + stamp: string; + refs: RefEntry[]; + bundleBytes: number; + lfsTarBytes?: number; + tool: ToolInfo; +} + +export function buildManifest(input: ManifestInput): Manifest { + return { + version: 1, + source: input.source, + fullName: input.fullName, + stamp: input.stamp, + refsHash: hashRefs(input.refs), + refs: input.refs, + bundleBytes: input.bundleBytes, + ...(input.lfsTarBytes === undefined ? {} : { lfsTarBytes: input.lfsTarBytes }), + tool: input.tool, + }; +} diff --git a/src/program.ts b/src/program.ts index 1bd0251..d669931 100644 --- a/src/program.ts +++ b/src/program.ts @@ -1,10 +1,10 @@ -import { readFileSync } from "node:fs"; import { Command, CommanderError, Option } from "commander"; import { type BackupOptions, runBackup } from "./commands/backup.js"; import type { CommandDeps, CommonOptions } from "./commands/context.js"; import { runList } from "./commands/list.js"; import { CliError, ExitCode } from "./util/errors.js"; import { exec } from "./util/exec.js"; +import { readToolInfo } from "./util/tool.js"; /** Flags whose negated form (`--no-x`) is the only one users can pass. */ const NEGATABLE = ["archived", "forks", "wiki", "lfs"] as const; @@ -43,14 +43,11 @@ export interface Program { } export function createProgram(deps: CommandDeps): Program { - const pkg = JSON.parse(readFileSync(new URL("../package.json", import.meta.url), "utf8")) as { - version: string; - }; let exitCode: number = ExitCode.Success; const program = new Command("github-backup") .description("Back up GitHub repositories to local disk and S3-compatible storage") - .version(pkg.version) + .version(readToolInfo().version) .exitOverride() .configureOutput({ writeOut: (text) => deps.stdout(text), diff --git a/src/targets/index.ts b/src/targets/index.ts new file mode 100644 index 0000000..758e24a --- /dev/null +++ b/src/targets/index.ts @@ -0,0 +1,16 @@ +import type { TargetConfig } from "../config/schema.js"; +import { ConfigError } from "../util/errors.js"; +import { createLocalTarget } from "./local.js"; +import type { Target } from "./types.js"; + +export type { Manifest, SnapshotFiles, Target } from "./types.js"; + +/** Registry: maps a configured target `type` to its implementation. */ +export function createTarget(config: TargetConfig, options: { keep: number }): Target { + switch (config.type) { + case "local": + return createLocalTarget({ name: config.name, path: config.path, keep: options.keep }); + case "s3": + throw new ConfigError(`target "${config.name}": s3 targets are not available yet`); + } +} diff --git a/src/targets/local.ts b/src/targets/local.ts new file mode 100644 index 0000000..566bb3e --- /dev/null +++ b/src/targets/local.ts @@ -0,0 +1,112 @@ +import { copyFile, mkdir, readdir, readFile, rename, rm, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { isStamp } from "../util/time.js"; +import type { Manifest, SnapshotFiles, Target } from "./types.js"; + +export interface LocalTargetOptions { + name: string; + /** Root directory snapshots are written under. */ + path: string; + keep: number; +} + +const PARTIAL = ".partial"; + +/** Splits `20260101T000000Z.bundle` into stamp and extension; null for other names. */ +function parseName(file: string): { stamp: string; ext: string } | null { + const dot = file.indexOf("."); + if (dot < 0) return null; + const stamp = file.slice(0, dot); + return isStamp(stamp) ? { stamp, ext: file.slice(dot) } : null; +} + +async function listDir(dir: string): Promise { + try { + return await readdir(dir); + } catch (err) { + if ((err as NodeJS.ErrnoException).code === "ENOENT") return []; + throw err; + } +} + +async function store(from: string, to: string): Promise { + await copyFile(from, `${to}${PARTIAL}`); + await rename(`${to}${PARTIAL}`, to); +} + +export function createLocalTarget(options: LocalTargetOptions): Target { + const root = path.resolve(options.path); + + const dirOf = (repoKey: string): string => { + const dir = path.resolve(root, repoKey); + if (!dir.startsWith(root + path.sep)) throw new Error(`invalid repository key "${repoKey}"`); + return dir; + }; + + async function completeStamps(dir: string): Promise { + const stamps = (await listDir(dir)) + .map(parseName) + .filter((p): p is { stamp: string; ext: string } => p?.ext === ".json") + .map((p) => p.stamp); + return stamps.sort(); + } + + return { + name: options.name, + type: "local", + keep: options.keep, + + async latestManifest(repoKey) { + const dir = dirOf(repoKey); + const stamps = await completeStamps(dir); + const latest = stamps.at(-1); + if (latest === undefined) return null; + try { + return JSON.parse(await readFile(path.join(dir, `${latest}.json`), "utf8")) as Manifest; + } catch { + return null; + } + }, + + async putSnapshot(repoKey, stamp, files: SnapshotFiles) { + const dir = dirOf(repoKey); + await mkdir(dir, { recursive: true }); + try { + await store(files.bundle, path.join(dir, `${stamp}.bundle`)); + if (files.lfsTar) await store(files.lfsTar, path.join(dir, `${stamp}.lfs.tar`)); + const manifest = path.join(dir, `${stamp}.json`); + await writeFile(`${manifest}${PARTIAL}`, `${JSON.stringify(files.manifest, null, 2)}\n`); + await rename(`${manifest}${PARTIAL}`, manifest); + } catch (err) { + await rm(path.join(dir, `${stamp}.json${PARTIAL}`), { force: true }); + throw err; + } + }, + + async listSnapshots(repoKey) { + return completeStamps(dirOf(repoKey)); + }, + + async deleteSnapshot(repoKey, stamp) { + const dir = dirOf(repoKey); + // The manifest goes first so a half-deleted snapshot never looks complete. + await rm(path.join(dir, `${stamp}.json`), { force: true }); + for (const file of await listDir(dir)) { + if (parseName(file)?.stamp === stamp) await rm(path.join(dir, file), { force: true }); + } + }, + + async cleanupIncomplete(repoKey) { + const dir = dirOf(repoKey); + const files = await listDir(dir); + const complete = new Set(await completeStamps(dir)); + for (const file of files) { + const parsed = parseName(file); + const partial = file.endsWith(PARTIAL); + if (partial || (parsed && !complete.has(parsed.stamp))) { + await rm(path.join(dir, file), { force: true }); + } + } + }, + }; +} diff --git a/src/targets/types.ts b/src/targets/types.ts new file mode 100644 index 0000000..d3b28d2 --- /dev/null +++ b/src/targets/types.ts @@ -0,0 +1,43 @@ +export interface RefEntry { + name: string; + sha: string; +} + +/** Written last for every snapshot; its presence marks the snapshot as complete (K-009). */ +export interface Manifest { + version: 1; + source: string; + fullName: string; + stamp: string; + refsHash: string; + refs: RefEntry[]; + bundleBytes: number; + lfsTarBytes?: number; + tool: { name: string; version: string }; +} + +export interface SnapshotFiles { + /** Local path of the bundle to store. */ + bundle: string; + /** Local path of the LFS object archive, when the repository has LFS objects. */ + lfsTar?: string; + manifest: Manifest; +} + +/** Storage a snapshot is written to. `repoKey` is `//[.wiki]`. */ +export interface Target { + readonly name: string; + readonly type: string; + /** Snapshots kept per repository. */ + readonly keep: number; + /** Manifest of the newest complete snapshot, or null. */ + latestManifest(repoKey: string): Promise; + /** Stores the files; the manifest is stored last. */ + putSnapshot(repoKey: string, stamp: string, files: SnapshotFiles): Promise; + /** Stamps of complete snapshots (those with a manifest), oldest first. */ + listSnapshots(repoKey: string): Promise; + /** Removes bundle, lfs.tar and manifest of one snapshot. */ + deleteSnapshot(repoKey: string, stamp: string): Promise; + /** Removes leftovers of interrupted uploads (files without a manifest). */ + cleanupIncomplete(repoKey: string): Promise; +} diff --git a/src/util/async.ts b/src/util/async.ts new file mode 100644 index 0000000..9e74f74 --- /dev/null +++ b/src/util/async.ts @@ -0,0 +1,46 @@ +import { sleep as defaultSleep } from "./time.js"; + +export interface RetryOptions { + /** Wait before each retry; the number of entries is the number of retries. */ + delaysMs: readonly number[]; + sleep?: (ms: number, signal?: AbortSignal) => Promise; + signal?: AbortSignal; + onRetry?: (error: unknown, attempt: number) => void; +} + +/** Runs `fn`, retrying after each configured delay. Aborting stops further attempts. */ +export async function withRetry(fn: () => Promise, options: RetryOptions): Promise { + const sleep = options.sleep ?? defaultSleep; + for (let attempt = 0; ; attempt++) { + try { + return await fn(); + } catch (err) { + const delay = options.delaysMs[attempt]; + if (delay === undefined || options.signal?.aborted) throw err; + options.onRetry?.(err, attempt + 1); + await sleep(delay, options.signal); + } + } +} + +/** + * Runs `fn` over `items` with at most `limit` in flight. Results keep the input order. + * After `signal` aborts no new item starts, and only started items appear in the result. + */ +export async function runPool( + items: readonly T[], + limit: number, + fn: (item: T) => Promise, + signal?: AbortSignal, +): Promise { + const done: Array<{ index: number; result: R }> = []; + let next = 0; + const worker = async () => { + while (next < items.length && !signal?.aborted) { + const index = next++; + done.push({ index, result: await fn(items[index] as T) }); + } + }; + await Promise.all(Array.from({ length: Math.min(Math.max(limit, 1), items.length) }, worker)); + return done.sort((a, b) => a.index - b.index).map((entry) => entry.result); +} diff --git a/src/util/exec.ts b/src/util/exec.ts index 47db494..329cb63 100644 --- a/src/util/exec.ts +++ b/src/util/exec.ts @@ -4,6 +4,8 @@ export interface ExecOptions { env?: Record; cwd?: string; timeoutMs?: number; + /** Aborting kills the child process and rejects with code `ABORT_ERR`. */ + signal?: AbortSignal; } export interface ExecResult { @@ -18,7 +20,7 @@ export class ExecError extends Error { constructor( message: string, readonly cmd: string, - /** Exit code, or a string such as `ENOENT` / `ETIMEDOUT` when no exit code exists. */ + /** Exit code, or a string such as `ENOENT` / `ETIMEDOUT` / `ABORT_ERR` when no exit code exists. */ readonly code: number | string | null, readonly stderr: string, ) { @@ -41,6 +43,7 @@ export const exec: Exec = (cmd, args, opts = {}) => const child = spawn(cmd, args, { cwd: opts.cwd, env: { ...process.env, ...opts.env }, + signal: opts.signal, stdio: ["ignore", "pipe", "pipe"], }); const stdout: Buffer[] = []; diff --git a/src/util/time.ts b/src/util/time.ts new file mode 100644 index 0000000..f3e7532 --- /dev/null +++ b/src/util/time.ts @@ -0,0 +1,29 @@ +const STAMP_PATTERN = /^\d{8}T\d{6}Z$/; + +function pad(n: number, width = 2): string { + return String(n).padStart(width, "0"); +} + +/** Snapshot timestamp: UTC `YYYYMMDDTHHMMSSZ`, sortable and safe in object keys. */ +export function formatStamp(date: Date): string { + return ( + `${pad(date.getUTCFullYear(), 4)}${pad(date.getUTCMonth() + 1)}${pad(date.getUTCDate())}` + + `T${pad(date.getUTCHours())}${pad(date.getUTCMinutes())}${pad(date.getUTCSeconds())}Z` + ); +} + +export function isStamp(value: string): boolean { + return STAMP_PATTERN.test(value); +} + +export function sleep(ms: number, signal?: AbortSignal): Promise { + return new Promise((resolve) => { + const timer = setTimeout(done, ms); + function done() { + clearTimeout(timer); + signal?.removeEventListener("abort", done); + resolve(); + } + signal?.addEventListener("abort", done, { once: true }); + }); +} diff --git a/src/util/tool.ts b/src/util/tool.ts new file mode 100644 index 0000000..d02c110 --- /dev/null +++ b/src/util/tool.ts @@ -0,0 +1,14 @@ +import { readFileSync } from "node:fs"; + +export interface ToolInfo { + name: string; + version: string; +} + +/** Name and version from package.json (same relative path from `src/` and `dist/`). */ +export function readToolInfo(): ToolInfo { + const pkg = JSON.parse( + readFileSync(new URL("../../package.json", import.meta.url), "utf8"), + ) as ToolInfo; + return { name: pkg.name, version: pkg.version }; +} diff --git a/test/helpers/git.ts b/test/helpers/git.ts new file mode 100644 index 0000000..1b323de --- /dev/null +++ b/test/helpers/git.ts @@ -0,0 +1,121 @@ +import { execFileSync } from "node:child_process"; +import { mkdirSync, mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import type { RepoInfo, Source } from "../../src/sources/types.js"; +import { type Exec, exec } from "../../src/util/exec.js"; + +/** Environment that keeps git away from the real global config and makes commits possible. */ +export function gitTestEnv(home: string): Record { + return { + HOME: home, + XDG_CONFIG_HOME: path.join(home, ".config"), + GIT_CONFIG_NOSYSTEM: "1", + GIT_TERMINAL_PROMPT: "0", + GIT_AUTHOR_NAME: "Test", + GIT_AUTHOR_EMAIL: "test@example.com", + GIT_COMMITTER_NAME: "Test", + GIT_COMMITTER_EMAIL: "test@example.com", + }; +} + +/** `Exec` that always runs with `gitTestEnv(home)`. */ +export function isolatedExec(home: string): Exec { + return (cmd, args, opts) => + exec(cmd, args, { ...opts, env: { ...gitTestEnv(home), ...opts?.env } }); +} + +export function makeTempDir(prefix = "ghb-"): string { + return mkdtempSync(path.join(tmpdir(), prefix)); +} + +/** A bare repository acting as the "remote", plus a work tree to create history in it. */ +export class TestRemote { + readonly bare: string; + readonly work: string; + private readonly env: Record; + + constructor(root: string, name: string, home: string) { + this.env = gitTestEnv(home); + this.bare = path.join(root, `${name}.git`); + this.work = path.join(root, `${name}-work`); + mkdirSync(this.work, { recursive: true }); + this.git(["init", "--bare", "--initial-branch=main", this.bare], root); + this.git(["init", "--initial-branch=main"], this.work); + this.git(["remote", "add", "origin", this.bare], this.work); + } + + get url(): string { + return `file://${this.bare}`; + } + + git(args: string[], cwd = this.work): string { + return execFileSync("git", args, { + cwd, + env: { ...process.env, ...this.env }, + encoding: "utf8", + stdio: ["ignore", "pipe", "pipe"], + }); + } + + /** Commits a file on `branch` (created from the current HEAD when new) and pushes it. */ + commit(file: string, content: string, branch = "main"): void { + const current = this.git(["branch", "--show-current"]).trim(); + if (current !== branch) { + const exists = this.git(["branch", "--list", branch]).trim() !== ""; + this.git(exists ? ["checkout", branch] : ["checkout", "-b", branch]); + } + writeFileSync(path.join(this.work, file), content); + this.git(["add", file]); + this.git(["commit", "-m", `update ${file}`]); + this.git(["push", "origin", branch]); + } + + tag(name: string): void { + this.git(["tag", "-a", name, "-m", name]); + this.git(["push", "origin", name]); + } + + deleteBranch(name: string): void { + this.git(["push", "origin", "--delete", name]); + } + + /** Creates a ref such as `refs/pull/1/head` that GitHub exposes but backups must skip. */ + pushRef(ref: string): void { + this.git(["push", "origin", `HEAD:${ref}`]); + } + + /** `git for-each-ref` output limited to branches and tags. */ + refs(): string { + return this.git(["for-each-ref", "refs/heads", "refs/tags"], this.bare); + } +} + +export function repoInfo( + fullName: string, + remote: Pick, + extra: Partial = {}, +): RepoInfo { + const [owner = "", name = ""] = fullName.split("/"); + return { + source: "test", + owner, + name, + fullName, + cloneUrl: remote.url, + hasWiki: false, + isArchived: false, + isFork: false, + isEmpty: false, + ...extra, + }; +} + +export function fakeSource(repos: RepoInfo[] = [], env: Record = {}): Source { + return { + id: "test", + checkPrerequisites: async () => {}, + listRepos: async () => ({ repos, failures: [] }), + gitEnv: () => env, + }; +} diff --git a/test/integration/backup.test.ts b/test/integration/backup.test.ts new file mode 100644 index 0000000..d4fa6ff --- /dev/null +++ b/test/integration/backup.test.ts @@ -0,0 +1,252 @@ +import { existsSync, readdirSync, rmSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import type { CommandDeps } from "../../src/commands/context.js"; +import { createProgram } from "../../src/program.js"; +import type { RepoInfo } from "../../src/sources/types.js"; +import { ExitCode } from "../../src/util/errors.js"; +import { fakeSource, isolatedExec, makeTempDir, repoInfo, TestRemote } from "../helpers/git.js"; + +interface Json { + summary: { ok: number; unchanged: number; skipped: number; failed: number }; + failures: Array<{ repo: string; error: string }>; + results: Array<{ key: string; status: string; reason?: string }>; +} + +let root: string; +let home: string; +let clock: number; + +beforeEach(() => { + root = makeTempDir("ghb-run-"); + home = path.join(root, "home"); + clock = Date.UTC(2026, 0, 1); +}); +afterEach(() => rmSync(root, { recursive: true, force: true })); + +function setup(repos: RepoInfo[], extra: Partial = {}) { + const out: string[] = []; + const err: string[] = []; + const deps: CommandDeps = { + exec: isolatedExec(home), + env: {}, + stdout: (text) => out.push(text), + stderr: (line) => err.push(line), + homeDir: home, + cwd: root, + createSource: () => fakeSource(repos), + // Every call to now() is one second later so snapshots never share a stamp. + now: () => { + clock += 1000; + return new Date(clock); + }, + sleep: async () => {}, + ...extra, + }; + const backup = async (...flags: string[]) => { + out.length = 0; + const code = await createProgram(deps).run([ + "backup", + "--local", + path.join(root, "out"), + "--cache-dir", + path.join(root, "cache"), + ...flags, + ]); + return { code, json: out.length ? (JSON.parse(out.join("")) as Json) : undefined }; + }; + return { backup, err }; +} + +const snapshotDir = (key: string) => path.join(root, "out", key); +const list = (key: string) => + existsSync(snapshotDir(key)) ? readdirSync(snapshotDir(key)).sort() : []; + +describe("backup", () => { + it("backs up good repositories, fails the broken one and exits 1", async () => { + const a = new TestRemote(root, "a", home); + a.commit("a.txt", "1"); + const b = new TestRemote(root, "b", home); + b.commit("b.txt", "1"); + const broken = { url: `file://${path.join(root, "missing.git")}` }; + const { backup, err } = setup([ + repoInfo("o/a", a), + repoInfo("o/broken", broken), + repoInfo("o/b", b), + ]); + + const { code, json } = await backup("--json"); + expect(code).toBe(ExitCode.PartialFailure); + expect(json?.summary).toEqual({ + ok: 2, + unchanged: 0, + skipped: 0, + failed: 1, + interrupted: false, + }); + expect(json?.failures.map((f) => f.repo)).toEqual(["test/o/broken"]); + expect(list("test/o/a")).toHaveLength(2); + expect(list("test/o/b")).toHaveLength(2); + expect(err.join("\n")).toContain("1 failed"); + expect(err.join("\n")).toContain("failed: test/o/broken"); + }); + + it("creates no new snapshot when refs are unchanged, and one when they change", async () => { + const a = new TestRemote(root, "a", home); + a.commit("a.txt", "1"); + const { backup } = setup([repoInfo("o/a", a)]); + + expect((await backup()).code).toBe(0); + const first = list("test/o/a"); + const second = await backup("--json"); + expect(second.code).toBe(0); + expect(second.json?.summary).toMatchObject({ ok: 0, unchanged: 1 }); + expect(list("test/o/a")).toEqual(first); + + a.commit("a.txt", "2"); + expect((await backup()).code).toBe(0); + expect(list("test/o/a")).toHaveLength(4); + }); + + it("keeps only the newest snapshots (--keep 2 after 3 changes)", async () => { + const a = new TestRemote(root, "a", home); + const { backup } = setup([repoInfo("o/a", a)]); + for (const n of [1, 2, 3]) { + a.commit("a.txt", String(n)); + expect((await backup("--keep", "2")).code).toBe(0); + } + const files = list("test/o/a"); + expect(files.filter((f) => f.endsWith(".json"))).toHaveLength(2); + expect(files.filter((f) => f.endsWith(".bundle"))).toHaveLength(2); + }); + + it("restores a repository from the written bundle", async () => { + const a = new TestRemote(root, "a", home); + a.commit("a.txt", "1"); + a.commit("b.txt", "2", "feature"); + a.tag("v1"); + const { backup } = setup([repoInfo("o/a", a)]); + await backup(); + + const bundle = list("test/o/a").find((f) => f.endsWith(".bundle")) as string; + const restored = path.join(root, "restored"); + const exec = isolatedExec(home); + await exec("git", ["clone", "--mirror", path.join(snapshotDir("test/o/a"), bundle), restored]); + const { stdout } = await exec("git", ["for-each-ref", "refs/heads", "refs/tags"], { + cwd: restored, + }); + expect(stdout).toBe(a.refs()); + }); + + it("backs up the wiki and skips repositories without one or without content", async () => { + const a = new TestRemote(root, "a", home); + a.commit("a.txt", "1"); + const wiki = new TestRemote(root, "a.wiki-src", home); + wiki.commit("Home.md", "hi"); + const missingWiki = new TestRemote(root, "b", home); + missingWiki.commit("b.txt", "1"); + const { backup } = setup([ + repoInfo("o/a", a, { hasWiki: true, wikiUrl: wiki.url }), + repoInfo("o/b", missingWiki, { + hasWiki: true, + wikiUrl: `file://${path.join(root, "gone.wiki.git")}`, + }), + repoInfo("o/empty", a, { isEmpty: true }), + ]); + + const { code, json } = await backup("--json"); + expect(code).toBe(0); + const status = Object.fromEntries(json?.results.map((r) => [r.key, r.status]) ?? []); + expect(status).toEqual({ + "test/o/a": "ok", + "test/o/a.wiki": "ok", + "test/o/b": "ok", + "test/o/b.wiki": "skipped", + "test/o/empty": "skipped", + }); + expect(json?.results.find((r) => r.key === "test/o/b.wiki")?.reason).toBe("no wiki"); + expect(list("test/o/a.wiki")).toHaveLength(2); + + expect((await backup("--no-wiki", "--json")).json?.results.map((r) => r.key)).not.toContain( + "test/o/a.wiki", + ); + }); + + it("exits 3 while another run holds the lock", async () => { + const a = new TestRemote(root, "a", home); + a.commit("a.txt", "1"); + const { backup, err } = setup([repoInfo("o/a", a)]); + const cache = path.join(root, "cache"); + // The test process itself is alive, so its PID counts as a running backup. + const { mkdirSync } = await import("node:fs"); + mkdirSync(cache, { recursive: true }); + writeFileSync(path.join(cache, ".lock"), `${process.pid}\n`); + + expect((await backup()).code).toBe(ExitCode.Locked); + expect(err.join("\n")).toContain("another backup run is in progress"); + expect(existsSync(path.join(cache, ".lock"))).toBe(true); + expect(list("test/o/a")).toEqual([]); + }); + + it("removes the lock and temporary files after a run", async () => { + const a = new TestRemote(root, "a", home); + a.commit("a.txt", "1"); + const { backup } = setup([repoInfo("o/a", a)]); + await backup(); + expect(existsSync(path.join(root, "cache", ".lock"))).toBe(false); + expect(existsSync(path.join(root, "cache", "tmp"))).toBe(false); + }); + + it("exits 130 and starts nothing when aborted", async () => { + const a = new TestRemote(root, "a", home); + a.commit("a.txt", "1"); + const controller = new AbortController(); + controller.abort(); + const { backup } = setup([repoInfo("o/a", a)], { signal: controller.signal }); + expect((await backup()).code).toBe(ExitCode.Interrupted); + expect(list("test/o/a")).toEqual([]); + expect(existsSync(path.join(root, "cache", ".lock"))).toBe(false); + }); + + it("exits 2 without targets", async () => { + const { err } = setup([]); + const code = await createProgram({ + exec: isolatedExec(home), + env: {}, + stdout: () => {}, + stderr: (line) => err.push(line), + homeDir: home, + cwd: root, + createSource: () => fakeSource([]), + }).run(["backup", "--cache-dir", path.join(root, "cache")]); + expect(code).toBe(ExitCode.Usage); + expect(err.join("\n")).toContain("no targets configured; use --local "); + }); + + it("never touches the global git config", async () => { + const a = new TestRemote(root, "a", home); + a.commit("a.txt", "1"); + const { backup } = setup([repoInfo("o/a", a)]); + await backup(); + expect(existsSync(path.join(home, ".gitconfig"))).toBe(false); + expect(existsSync(path.join(home, ".config", "git", "config"))).toBe(false); + }); + + it("retries a failing step and succeeds once the remote appears", async () => { + const a = new TestRemote(root, "a", home); + a.commit("a.txt", "1"); + const waits: number[] = []; + let attempts = 0; + const real = isolatedExec(home); + const { backup } = setup([repoInfo("o/a", a)], { + sleep: async (ms) => void waits.push(ms), + exec: async (cmd, args, opts) => { + if (args.includes("fetch") && ++attempts < 3) throw new Error("network down"); + return real(cmd, args, opts); + }, + }); + expect((await backup()).code).toBe(0); + expect(waits).toEqual([1000, 5000]); + expect(list("test/o/a")).toHaveLength(2); + }); +}); diff --git a/test/integration/local-target.test.ts b/test/integration/local-target.test.ts new file mode 100644 index 0000000..83c0b60 --- /dev/null +++ b/test/integration/local-target.test.ts @@ -0,0 +1,109 @@ +import { existsSync, readdirSync, rmSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { createTarget } from "../../src/targets/index.js"; +import type { Manifest, Target } from "../../src/targets/types.js"; +import { makeTempDir } from "../helpers/git.js"; + +let root: string; +let target: Target; +const KEY = "test/owner/repo"; + +const manifest = (stamp: string, refsHash = "h"): Manifest => ({ + version: 1, + source: "test", + fullName: "owner/repo", + stamp, + refsHash, + refs: [], + bundleBytes: 3, + tool: { name: "t", version: "1" }, +}); + +const files = () => readdirSync(path.join(root, "out", KEY)).sort(); + +beforeEach(() => { + root = makeTempDir("ghb-local-"); + writeFileSync(path.join(root, "a.bundle"), "abc"); + writeFileSync(path.join(root, "a.tar"), "tar"); + target = createTarget({ type: "local", name: "disk", path: path.join(root, "out") }, { keep: 2 }); +}); +afterEach(() => rmSync(root, { recursive: true, force: true })); + +describe("local target", () => { + it("is empty before the first snapshot", async () => { + expect(await target.latestManifest(KEY)).toBeNull(); + expect(await target.listSnapshots(KEY)).toEqual([]); + await target.cleanupIncomplete(KEY); + }); + + it("writes bundle, lfs.tar and manifest without leaving partial files", async () => { + await target.putSnapshot(KEY, "20260101T000000Z", { + bundle: path.join(root, "a.bundle"), + lfsTar: path.join(root, "a.tar"), + manifest: manifest("20260101T000000Z"), + }); + expect(files()).toEqual([ + "20260101T000000Z.bundle", + "20260101T000000Z.json", + "20260101T000000Z.lfs.tar", + ]); + expect((await target.latestManifest(KEY))?.stamp).toBe("20260101T000000Z"); + }); + + it("returns the newest manifest and lists stamps oldest first", async () => { + for (const [stamp, hash] of [ + ["20260102T000000Z", "two"], + ["20260101T000000Z", "one"], + ] as const) { + await target.putSnapshot(KEY, stamp, { + bundle: path.join(root, "a.bundle"), + manifest: manifest(stamp, hash), + }); + } + expect(await target.listSnapshots(KEY)).toEqual(["20260101T000000Z", "20260102T000000Z"]); + expect((await target.latestManifest(KEY))?.refsHash).toBe("two"); + }); + + it("deletes every file of a snapshot", async () => { + for (const stamp of ["20260101T000000Z", "20260102T000000Z"]) { + await target.putSnapshot(KEY, stamp, { + bundle: path.join(root, "a.bundle"), + lfsTar: path.join(root, "a.tar"), + manifest: manifest(stamp), + }); + } + await target.deleteSnapshot(KEY, "20260101T000000Z"); + expect(files().every((f) => f.startsWith("20260102T000000Z"))).toBe(true); + expect(await target.listSnapshots(KEY)).toEqual(["20260102T000000Z"]); + }); + + it("removes files of interrupted snapshots but keeps complete ones", async () => { + await target.putSnapshot(KEY, "20260101T000000Z", { + bundle: path.join(root, "a.bundle"), + manifest: manifest("20260101T000000Z"), + }); + const dir = path.join(root, "out", KEY); + writeFileSync(path.join(dir, "20260102T000000Z.bundle"), "half"); + writeFileSync(path.join(dir, "20260102T000000Z.lfs.tar.partial"), "half"); + writeFileSync(path.join(dir, "notes.txt"), "keep me"); + await target.cleanupIncomplete(KEY); + expect(files()).toEqual(["20260101T000000Z.bundle", "20260101T000000Z.json", "notes.txt"]); + }); + + it("does not report a snapshot without manifest as complete", async () => { + const dir = path.join(root, "out", KEY); + await target.putSnapshot(KEY, "20260101T000000Z", { + bundle: path.join(root, "a.bundle"), + manifest: manifest("20260101T000000Z"), + }); + writeFileSync(path.join(dir, "20260105T000000Z.bundle"), "half"); + expect(await target.listSnapshots(KEY)).toEqual(["20260101T000000Z"]); + expect((await target.latestManifest(KEY))?.stamp).toBe("20260101T000000Z"); + }); + + it("rejects repository keys that escape the root", async () => { + await expect(target.listSnapshots("../outside")).rejects.toThrow(/invalid repository key/); + expect(existsSync(path.join(root, "outside"))).toBe(false); + }); +}); diff --git a/test/integration/mirror-snapshot.test.ts b/test/integration/mirror-snapshot.test.ts new file mode 100644 index 0000000..233a444 --- /dev/null +++ b/test/integration/mirror-snapshot.test.ts @@ -0,0 +1,128 @@ +import { existsSync, rmSync } from "node:fs"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { createGit } from "../../src/core/git.js"; +import { syncMirror } from "../../src/core/mirror.js"; +import { buildManifest, createBundle, hashRefs, listRefs } from "../../src/core/snapshot.js"; +import { isolatedExec, makeTempDir, TestRemote } from "../helpers/git.js"; + +let root: string; +let home: string; +let remote: TestRemote; + +const git = () => createGit({ exec: isolatedExec(home), env: {} }); +const mirrorDir = () => path.join(root, "cache", "test", "o", "r.git"); + +beforeEach(() => { + root = makeTempDir(); + home = path.join(root, "home"); + remote = new TestRemote(root, "remote", home); + remote.commit("a.txt", "one"); +}); + +afterEach(() => { + rmSync(root, { recursive: true, force: true }); +}); + +const mirrorInput = (kind: "repo" | "wiki" = "repo", url = remote.url) => ({ + mirrorDir: mirrorDir(), + url, + kind, +}); + +async function refLines(): Promise { + return (await listRefs(git(), mirrorDir())).map((r) => `${r.sha} ${r.name}`); +} + +describe("syncMirror", () => { + it("clones on the first run and fetches new commits afterwards", async () => { + expect(await syncMirror(git(), mirrorInput())).toEqual({ status: "ok" }); + expect(await refLines()).toHaveLength(1); + + remote.commit("b.txt", "two"); + remote.commit("c.txt", "three", "feature"); + remote.tag("v1"); + await syncMirror(git(), mirrorInput()); + const names = (await listRefs(git(), mirrorDir())).map((r) => r.name); + expect(names).toEqual(["refs/heads/feature", "refs/heads/main", "refs/tags/v1"]); + expect(existsSync(`${mirrorDir()}/HEAD`)).toBe(true); + }); + + it("keeps a branch that was deleted on the remote", async () => { + remote.commit("c.txt", "three", "feature"); + await syncMirror(git(), mirrorInput()); + remote.git(["checkout", "main"]); + remote.deleteBranch("feature"); + await syncMirror(git(), mirrorInput()); + const names = (await listRefs(git(), mirrorDir())).map((r) => r.name); + expect(names).toContain("refs/heads/feature"); + }); + + it("does not fetch refs/pull/*", async () => { + remote.pushRef("refs/pull/1/head"); + await syncMirror(git(), mirrorInput()); + const names = (await listRefs(git(), mirrorDir())).map((r) => r.name); + expect(names).toEqual(["refs/heads/main"]); + }); + + it("points HEAD at the remote default branch", async () => { + remote.commit("c.txt", "three", "feature"); + await syncMirror(git(), mirrorInput()); + const head = await git()(["symbolic-ref", "HEAD"], { cwd: mirrorDir() }); + expect(head.stdout.trim()).toBe("refs/heads/main"); + }); + + it("reports a missing wiki as skipped and leaves no mirror behind", async () => { + const missing = `file://${path.join(root, "nope.wiki.git")}`; + expect(await syncMirror(git(), mirrorInput("wiki", missing))).toEqual({ + status: "skipped", + reason: "no wiki", + }); + expect(existsSync(mirrorDir())).toBe(false); + }); + + it("fails for a missing repository", async () => { + const missing = `file://${path.join(root, "nope.git")}`; + await expect(syncMirror(git(), mirrorInput("repo", missing))).rejects.toThrow(); + }); +}); + +describe("snapshot", () => { + it("hashes equal refs to the same digest regardless of order", () => { + const a = { name: "refs/heads/a", sha: "1".repeat(40) }; + const b = { name: "refs/heads/b", sha: "2".repeat(40) }; + expect(hashRefs([a, b])).toBe(hashRefs([b, a])); + expect(hashRefs([a, b])).not.toBe(hashRefs([a, { ...b, sha: "3".repeat(40) }])); + expect(hashRefs([])).toMatch(/^[0-9a-f]{64}$/); + }); + + it("restores every branch and tag from the bundle", async () => { + remote.commit("c.txt", "three", "feature"); + remote.tag("v1"); + remote.commit("d.txt", "four", "main"); + await syncMirror(git(), mirrorInput()); + + const refs = await listRefs(git(), mirrorDir()); + const bundle = path.join(root, "x.bundle"); + const bytes = await createBundle(git(), mirrorDir(), bundle); + expect(bytes).toBeGreaterThan(0); + + const restored = path.join(root, "restored"); + await git()(["clone", "--mirror", bundle, restored]); + const restoredRefs = await git()(["for-each-ref", "refs/heads", "refs/tags"], { + cwd: restored, + }); + expect(restoredRefs.stdout).toBe(remote.refs()); + + const manifest = buildManifest({ + source: "test", + fullName: "o/r", + stamp: "20260101T000000Z", + refs, + bundleBytes: bytes, + tool: { name: "t", version: "1" }, + }); + expect(manifest).toMatchObject({ version: 1, refsHash: hashRefs(refs), bundleBytes: bytes }); + expect(manifest).not.toHaveProperty("lfsTarBytes"); + }); +}); diff --git a/test/unit/async.test.ts b/test/unit/async.test.ts new file mode 100644 index 0000000..3e4c091 --- /dev/null +++ b/test/unit/async.test.ts @@ -0,0 +1,93 @@ +import { describe, expect, it } from "vitest"; +import { runPool, withRetry } from "../../src/util/async.js"; + +describe("withRetry", () => { + it("returns the first success without waiting", async () => { + const waits: number[] = []; + const result = await withRetry(async () => "ok", { + delaysMs: [1000, 5000], + sleep: async (ms) => void waits.push(ms), + }); + expect(result).toBe("ok"); + expect(waits).toEqual([]); + }); + + it("retries twice with the configured delays, then succeeds", async () => { + const waits: number[] = []; + let calls = 0; + const result = await withRetry( + async () => { + if (++calls < 3) throw new Error("boom"); + return calls; + }, + { delaysMs: [1000, 5000], sleep: async (ms) => void waits.push(ms) }, + ); + expect(result).toBe(3); + expect(waits).toEqual([1000, 5000]); + }); + + it("throws the last error after the retries are used up", async () => { + let calls = 0; + await expect( + withRetry( + async () => { + throw new Error(`fail ${++calls}`); + }, + { delaysMs: [1, 1], sleep: async () => {} }, + ), + ).rejects.toThrow("fail 3"); + }); + + it("does not retry once aborted", async () => { + const controller = new AbortController(); + controller.abort(); + let calls = 0; + await expect( + withRetry( + async () => { + calls++; + throw new Error("x"); + }, + { delaysMs: [1, 1], sleep: async () => {}, signal: controller.signal }, + ), + ).rejects.toThrow("x"); + expect(calls).toBe(1); + }); +}); + +describe("runPool", () => { + it("never exceeds the limit and keeps input order", async () => { + let active = 0; + let peak = 0; + const results = await runPool([5, 1, 4, 2, 3, 6], 3, async (n) => { + active++; + peak = Math.max(peak, active); + await new Promise((resolve) => setTimeout(resolve, n)); + active--; + return n * 2; + }); + expect(results).toEqual([10, 2, 8, 4, 6, 12]); + expect(peak).toBe(3); + }); + + it("starts no new items after the signal aborts", async () => { + const controller = new AbortController(); + const started: number[] = []; + const results = await runPool( + [1, 2, 3, 4], + 1, + async (n) => { + started.push(n); + if (n === 2) controller.abort(); + return n; + }, + controller.signal, + ); + expect(started).toEqual([1, 2]); + expect(results).toEqual([1, 2]); + }); + + it("handles an empty list", async () => { + expect(await runPool([], 4, async (n: number) => n)).toEqual([]); + }); +}); diff --git a/test/unit/commands.test.ts b/test/unit/commands.test.ts index b9d81bb..d194062 100644 --- a/test/unit/commands.test.ts +++ b/test/unit/commands.test.ts @@ -186,10 +186,10 @@ describe("list prerequisites (real github source, fake exec)", () => { }); describe("backup", () => { - it("is not implemented yet and exits 2", async () => { + it("exits 2 when no target is configured", async () => { const { run, err } = setup(); expect(await run(["backup"])).toBe(2); - expect(err.join("\n")).toContain("not implemented"); + expect(err.join("\n")).toContain("no targets configured; use --local "); }); it("accepts every documented flag", async () => { @@ -219,9 +219,9 @@ describe("backup", () => { "--local", "/tmp/out", ]); - // Flags parse; the command itself is the stub. + // Every flag parses; the missing config file stops the command before any work. expect(code).toBe(2); - expect(err.join("\n")).toContain("not implemented"); + expect(err.join("\n")).toContain("config file not found"); }); it("rejects unknown flags with exit 2", async () => { diff --git a/test/unit/lock.test.ts b/test/unit/lock.test.ts new file mode 100644 index 0000000..d6ce683 --- /dev/null +++ b/test/unit/lock.test.ts @@ -0,0 +1,54 @@ +import { existsSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { acquireLock } from "../../src/core/lock.js"; +import { ExitCode } from "../../src/util/errors.js"; +import { makeTempDir } from "../helpers/git.js"; + +let dir: string; +let file: string; + +beforeEach(() => { + dir = makeTempDir("ghb-lock-"); + file = path.join(dir, "cache", ".lock"); +}); +afterEach(() => rmSync(dir, { recursive: true, force: true })); + +describe("acquireLock", () => { + it("writes the pid and removes the file on release", () => { + const lock = acquireLock(file, { pid: 4242 }); + expect(readFileSync(file, "utf8").trim()).toBe("4242"); + lock.release(); + expect(existsSync(file)).toBe(false); + expect(existsSync(`${file}.4242.tmp`)).toBe(false); + }); + + it("refuses a lock held by a live process with exit code 3", () => { + const first = acquireLock(file, { pid: 1 }); + expect(() => acquireLock(file, { pid: 2, isAlive: () => true })).toThrowError( + expect.objectContaining({ exitCode: ExitCode.Locked }), + ); + first.release(); + }); + + it("takes over a lock whose pid is dead", () => { + acquireLock(file, { pid: 1 }); + const second = acquireLock(file, { pid: 2, isAlive: () => false }); + expect(readFileSync(file, "utf8").trim()).toBe("2"); + second.release(); + expect(existsSync(file)).toBe(false); + }); + + it("takes over an unreadable lock file", () => { + acquireLock(file, { pid: 1 }); + writeFileSync(file, "garbage"); + acquireLock(file, { pid: 2, isAlive: () => true }).release(); + }); + + it("does not remove a lock that now belongs to someone else", () => { + const lock = acquireLock(file, { pid: 1 }); + writeFileSync(file, "99\n"); + lock.release(); + expect(existsSync(file)).toBe(true); + }); +}); diff --git a/test/unit/retention.test.ts b/test/unit/retention.test.ts new file mode 100644 index 0000000..0f79d80 --- /dev/null +++ b/test/unit/retention.test.ts @@ -0,0 +1,29 @@ +import { describe, expect, it } from "vitest"; +import { resolveKeep, selectExpired } from "../../src/core/retention.js"; + +describe("resolveKeep", () => { + it.each([ + [undefined, 7, 7], + [14, 7, 14], + [1, 7, 1], + ])("target %s, global %s -> %s", (target, global, expected) => { + expect(resolveKeep(target, global)).toBe(expected); + }); +}); + +describe("selectExpired", () => { + const stamps = ["20260103T000000Z", "20260101T000000Z", "20260102T000000Z", "20260104T000000Z"]; + it.each([ + [2, ["20260101T000000Z", "20260102T000000Z"]], + [3, ["20260101T000000Z"]], + [4, []], + [10, []], + [1, ["20260101T000000Z", "20260102T000000Z", "20260103T000000Z"]], + ])("keep %s", (keep, expected) => { + expect(selectExpired(stamps, keep)).toEqual(expected); + }); + + it("handles an empty list", () => { + expect(selectExpired([], 3)).toEqual([]); + }); +});