Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,6 @@ __pycache__/

# OpenCode local state
.opencode/

# Rust build output
secure/samples/audience-segments-rust/target/
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ The human user is the technical lead. The agent is a supporting engineer that he

## Project Context

This repo is a PoC of the Secure Computing Environment (SCE), running on AWS Nitro Enclaves. It covers the end-to-end workflow: a build phase (Registry Service, part of the CSAV, turns a Data Controller's OCI image into a single EIF, layering the SCE Unit and nsjail on top, stores the artifact in S3, records `AlgorithmId` -> S3 path in DynamoDB) and an execution phase (enclave provisioning, remote attestation against the Verifier Service, data fetch from the Personal Data Management Service and KMS, computation over decrypted records files inside the enclave (`RECORDS_PATH` / `RESULT_PATH`, same contract as rads), async result delivery through SQS and a result-consumer Lambda, polled via `GET /run/{runId}`). See the architecture section in `README.md` and the docs it links before making assumptions about scope or design.
This repo is a PoC of the Secure Computing Environment (SCE), running on AWS Nitro Enclaves. It covers the end-to-end workflow: a build phase (Registry Service, part of the CSAV, turns a Data Controller's OCI image into a single EIF, layering the SCE Unit and nsjail on top, stores the artifact in S3, records `AlgorithmId` -> S3 path in DynamoDB) and an execution phase (enclave provisioning, remote attestation against the Verifier Service, data fetch from the Personal Data Management Service and KMS, computation over decrypted records streamed to the DC app over an AF_UNIX gRPC socket (`FetchRecords` / `SubmitResult`), async result delivery through SQS and a result-consumer Lambda, polled via `GET /run/{runId}`). See the architecture section in `README.md` and the docs it links before making assumptions about scope or design.

## Operating Rules

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Every rads AWS resource is prefixed `rads-`; every secure-rads resource is prefi

**Build phase (CSAV).** A Data Controller uploads an OCI image and assigns it an `algorithmId`. registry-service unpacks the image (daemonless: skopeo + umoci), layers the SCE Unit and nsjail on top, and builds a single Nitro Enclave Image File (EIF). It registers the EIF's PCR measurements against that `algorithmId` with the Verifier Service (over mTLS) and stores the EIF in S3, recording `algorithmId -> S3 path` in DynamoDB.

**Execution phase (SCE).** A caller POSTs an `algorithmId` to the API Gateway and Lambda execution-service control plane, which launches a per-run Nitro Enclave parent EC2 instance. Inside, the SCE Unit seeds enclave entropy from the NSM, attests to the Verifier Service, and - once attestation passes - fetches that algorithm's records from the PDM Service, gets the data key unwrapped by the KMS Service, decrypts them, and writes `records.json` for the jailed Data Controller app (`RECORDS_PATH` / `RESULT_PATH`, same file contract as rads). Results come back asynchronously: `/run` returns a `runId` immediately, the enclave host publishes the result to SQS, a consumer Lambda persists it, and the caller polls `GET /run/{runId}`.
**Execution phase (SCE).** A caller POSTs an `algorithmId` to the API Gateway and Lambda execution-service control plane, which launches a per-run Nitro Enclave parent EC2 instance. Inside, the SCE Unit seeds enclave entropy from the NSM, starts the jailed Data Controller app, and serves it over an AF_UNIX gRPC socket (`/data/sce.sock`). The app's `FetchRecords` call triggers attestation to the Verifier Service, the fetch from the PDM Service, and decryption with the data key unwrapped by the KMS Service. The app streams records, computes, and submits one result message that the SCE Unit converts back to JSON. Results come back asynchronously: `/run` returns a `runId` immediately, the enclave host publishes the result to SQS, a consumer Lambda persists it, and the caller polls `GET /run/{runId}`.

Throughout both phases every component posts timestamped events to the telemetry-service, so a run's full timeline - build to execution - can be exported for benchmarking. The log-service records only the subject-access trail. Components also report selected, attributable AWS operations to the cost-service, keyed by the same `(AlgorithmId, runId)`, so their estimated costs can be summed per AWS service ([`docs/cost-service.md`](docs/cost-service.md)). Event-by-event plaintext flows: [`docs/e2e-rads.txt`](docs/e2e-rads.txt), [`docs/e2e-secure-rads.txt`](docs/e2e-secure-rads.txt).

Expand Down
3 changes: 1 addition & 2 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,7 @@ If the report involves a credential, revoke it first; do not send the secret its
## What is in scope

- Attestation bypass: an enclave or process that should fail verification and does not.
- Trust boundary breaks: the Data Controller app reaching data outside its `RECORDS_PATH`, or
reading decrypted records after the run.
- Trust boundary breaks: the Data Controller app reaching data outside the records streamed over its `SCE_RPC_SOCKET`, or reading decrypted records after the run.
- Enclave escape or unintended host access from `sce-unit` or the jailed app.
- Secret exposure in the repository or in build artifacts.
- IAM or resource policies that grant more than the documented flow requires.
Expand Down
2 changes: 1 addition & 1 deletion benchmark/src/plot.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
COST_COLORS = ["#1F4E79", "#4C9BE8", "#C0392B", "#6FBF73", "#F2C14E", "#8E44AD"]
NOTE = (
"host provisioning = per-run instance boot + tooling install (size-independent); "
"data retrieval is fetch through records.json write (secure-rads includes decrypt).\n"
"data retrieval is fetch through decrypted record delivery (secure-rads includes decrypt).\n"
"rads has no enclave, so enclave boot / sce unit / attestation are blank for it."
)

Expand Down
42 changes: 22 additions & 20 deletions docs/e2e-secure-rads.txt
Original file line number Diff line number Diff line change
Expand Up @@ -70,26 +70,28 @@ Input:
3 emit enclave-started // no token yet, auth none
4 SEED-ENTROPY from NSM // no AWS cost
5 emit unit-ready // no token yet, auth none
6 challenge ← REQUEST-CHALLENGE(Verifier, a, runId) // single-use nonce, verifier API GW/Lambda and DynamoDB metered
7 token ← ATTEST(Verifier, runId, challenge) // NSM nonce ← challenge; claims the run so only one attestation succeeds; verifier API GW/Lambda and DynamoDB metered
8 emit attested
9 records ← FETCH-PDM(a, token, limit ← n)
10 cost apigateway:request + lambda:invocation + lambda:compute + dynamodb:read-capacity // PDM /records
11 emit records-fetched
12 ciphertext ← KMS-SERVICE-DECRYPT(token) // unwraps the data key under this enclave's measurements
13 cost apigateway:request + lambda:invocation + lambda:compute + dynamodb:read-capacity // kms-service /decrypt
14 DEK ← UNWRAP(ciphertext, enclaveKey) // KMS request itself is excluded from cost
15 plaintext ← DECRYPT(records, DEK)
16 emit records-decrypted
17 write {algorithmId, records} to /data/records.json
18 emit records-stored-locally
19 START nsjail // RECORDS_PATH, RESULT_PATH
20 wait for nsjail exit
21 output ← read /data/result.json
22 emit dc-app-completed // on success, else run-failed
23 STREAM output to parent // vsock :5000
24 emit output-sent
25 // on failure: emit run-failed
6 START nsjail // SCE_RPC_SOCKET=/data/sce.sock, ALGORITHM_ID
7 wait for app FetchRecords RPC // first call triggers the steps below; retries reuse the outcome
8 challenge ← REQUEST-CHALLENGE(Verifier, a, runId) // single-use nonce, verifier API GW/Lambda and DynamoDB metered
9 token ← ATTEST(Verifier, runId, challenge) // NSM nonce ← challenge; claims the run so only one attestation succeeds; verifier API GW/Lambda and DynamoDB metered
10 emit attested
11 records ← FETCH-PDM(a, token, limit ← n)
12 cost apigateway:request + lambda:invocation + lambda:compute + dynamodb:read-capacity // PDM /records
13 emit records-fetched
14 ciphertext ← KMS-SERVICE-DECRYPT(token) // unwraps the data key under this enclave's measurements
15 cost apigateway:request + lambda:invocation + lambda:compute + dynamodb:read-capacity // kms-service /decrypt
16 DEK ← UNWRAP(ciphertext, enclaveKey) // KMS request itself is excluded from cost
17 plaintext ← DECRYPT(records, DEK) // RecordEnvelope -> sce.v1.Record
18 emit records-decrypted
19 emit records-stored-locally
20 STREAM records to app // gRPC FetchRecords, one sce.v1.Record per message
21 wait for app SubmitResult RPC // google.protobuf.Struct
22 wait for nsjail exit
23 emit dc-app-completed // on success, else run-failed
24 output ← JSON(result)
25 STREAM output to parent // vsock :5000
26 emit output-sent
27 // on failure: emit run-failed



Expand Down
74 changes: 74 additions & 0 deletions proto/generate.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
#!/usr/bin/env bash
# Regenerates the checked-in gRPC stubs from proto/. Run after editing any .proto.
# Each language gets its own output tree; the .proto files stay the single source of truth.
set -euo pipefail

repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
proto_dir="$repo_root/proto"
wkt_include="/usr/include"

protoc_version="$(protoc --version)"
echo "using $protoc_version"

gen_go() {
local module="$1" out="$2" go_import="$3"
mkdir -p "$out"
protoc -I "$proto_dir" -I "$wkt_include" \
--plugin=protoc-gen-go="${PROTOC_GEN_GO:-protoc-gen-go}" \
--plugin=protoc-gen-go-grpc="${PROTOC_GEN_GO_GRPC:-protoc-gen-go-grpc}" \
--go_out="$out" --go_opt=module="$module" \
--go_opt=Msce/v1/records.proto="$go_import" \
--go_opt=Msce/v1/envelope.proto="$go_import" \
--go-grpc_out="$out" --go-grpc_opt=module="$module" \
--go-grpc_opt=Msce/v1/records.proto="$go_import" \
--go-grpc_opt=Msce/v1/envelope.proto="$go_import" \
sce/v1/records.proto sce/v1/envelope.proto
}

# sce-unit (Go server)
gen_go "sce-unit" "$repo_root/secure/sce-unit" "sce-unit/gen/scev1"

# spend-analysis (Go client)
gen_go "spend-analysis" "$repo_root/secure/samples/spend-analysis-go" "spend-analysis/gen/scev1"

# Python clients: generate at the app root so `from sce.v1 import ...` resolves with the app dir on sys.path.
for app in health-risk-python trial-eligibility-python; do
out="$repo_root/secure/samples/$app"
mkdir -p "$out/sce/v1"
python -m grpc_tools.protoc -I "$proto_dir" -I "$wkt_include" \
--python_out="$out" --grpc_python_out="$out" \
sce/v1/records.proto sce/v1/envelope.proto
for pkg in sce sce/v1; do
touch "$out/$pkg/__init__.py"
done
done
rm -f "$repo_root/secure/samples/health-risk-python"/{records,envelope}_pb2*.py \
"$repo_root/secure/samples/trial-eligibility-python"/{records,envelope}_pb2*.py

# Seeding (TypeScript / protobuf-es)
mkdir -p "$repo_root/seeding/gen"
"${PROTOC_GEN_ES:-protoc-gen-es}" --version >/dev/null 2>&1 || true
protoc -I "$proto_dir" -I "$wkt_include" \
--plugin=protoc-gen-es="${PROTOC_GEN_ES:-protoc-gen-es}" \
--es_out="$repo_root/seeding/gen" --es_opt=target=ts \
sce/v1/records.proto sce/v1/envelope.proto

# Node client (protobuf-es, JS + d.ts so the sample stays plain JS)
node_out="$repo_root/secure/samples/product-trends-node/gen"
mkdir -p "$node_out"
protoc -I "$proto_dir" -I "$wkt_include" \
--plugin=protoc-gen-es="${PROTOC_GEN_ES:-protoc-gen-es}" \
--es_out="$node_out" --es_opt=target=js+dts \
sce/v1/records.proto sce/v1/envelope.proto

# Rust client (prost + tonic)
rust_out="$repo_root/secure/samples/audience-segments-rust/src/gen"
mkdir -p "$rust_out/sce/v1"
protoc -I "$proto_dir" -I "$wkt_include" \
--plugin=protoc-gen-prost="${PROTOC_GEN_PROST:-protoc-gen-prost}" \
--plugin=protoc-gen-tonic="${PROTOC_GEN_TONIC:-protoc-gen-tonic}" \
--prost_out="$rust_out" \
--tonic_out="$rust_out" \
sce/v1/records.proto sce/v1/envelope.proto

echo "generated stubs"
11 changes: 11 additions & 0 deletions proto/sce/v1/envelope.proto
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
syntax = "proto3";

package sce.v1;

// RecordEnvelope is what pdm-service stores in DynamoDB Data. ciphertext is AES-256-GCM over a serialized
// sce.v1.Record; nonce is the 96-bit GCM nonce, and the auth tag is appended to ciphertext (Go GCM layout).
message RecordEnvelope {
uint32 version = 1;
bytes nonce = 2;
bytes ciphertext = 3;
}
28 changes: 28 additions & 0 deletions proto/sce/v1/records.proto
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
syntax = "proto3";

package sce.v1;

import "google/protobuf/struct.proto";

// Record is one personal-data record handed to the DC application. record_id stays separate so the DC app
// never sees data subject ids; data is schemaless per dataset, so it travels as a Struct.
message Record {
string record_id = 1;
google.protobuf.Struct data = 2;
}

message FetchRecordsRequest {}

message SubmitResultRequest {
google.protobuf.Struct result = 1;
}

message SubmitResultResponse {}

// DataAccess runs over an AF_UNIX socket inside the enclave, between sce-unit (server) and the DC app.
service DataAccess {
// Streams this run's decrypted records. sce-unit applies the run-info limit; the request carries no filters.
rpc FetchRecords(FetchRecordsRequest) returns (stream Record);
// The DC app submits its result once, then exits. sce-unit converts it back to JSON for the host.
rpc SubmitResult(SubmitResultRequest) returns (SubmitResultResponse);
}
2 changes: 1 addition & 1 deletion secure/csav/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ flowchart TD
C --> G["opt/controller/rootfs/ (the DC app)"]
```

- **nsjail**: enforces the privilege boundary at runtime — DC algorithm gets its rootfs and `RECORDS_PATH` / `RESULT_PATH` files under `/data`, nothing else.
- **nsjail**: enforces the privilege boundary at runtime — the DC algorithm gets its rootfs, a `/data` scratch dir, and the AF_UNIX gRPC socket sce-unit serves at `/data/sce.sock`, nothing else.
- **Trust material**: public key pin or CA cert injected into the EIF at build time, so the enclave can verify the Verifier Service, Personal Data Management Service, KMS Service, and Log Service later.
- **Storage**: final EIF goes to S3, `AlgorithmId` -> S3 path goes to DynamoDB. Registry Service also generates PCR values and sends them to the Verifier Service.
- **KMS keys**: one PCR-bound key per algorithm, created on first build and reused on rebuild (its policy is rewritten to the new measurements), so a build never orphans the previous key. The policy grants `Decrypt` to the kms-service role under `kms:RecipientAttestation` conditions; the enclave and its host never decrypt directly.
Expand Down
20 changes: 11 additions & 9 deletions secure/csav/registry-service/nsjail.cfg.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,8 @@
# exec_bin is generated per-build from the uploaded image's own
# ENTRYPOINT/CMD - no universal binary path across arbitrary DC images.
#
# Records/results live under /data in the DC rootfs (sce-unit writes
# records.json there before start; the app writes result.json). Same
# RECORDS_PATH / RESULT_PATH contract as rads.
# The DC app pulls records and submits its result over the gRPC socket
# sce-unit serves at /data/sce.sock inside the jail (see sce-unit/rpc.go).

name: "data-controller-application-jail"

Expand All @@ -16,13 +15,13 @@ mount_proc: true

# nsjail strips env vars by default; PATH here is sourced from the DC's own image (see pathFromEnv in oci.go).
envar: "PATH={{.PathEnv}}"
envar: "RECORDS_PATH=/data/records.json"
envar: "RESULT_PATH=/data/result.json"
# Without HOME, glibc NSS calls getpwuid for ~ expansion, which opens an AF_UNIX socket the jail now blocks.
envar: "SCE_RPC_SOCKET=/data/sce.sock"
envar: "ALGORITHM_ID={{.AlgorithmId}}"
# Without HOME, glibc NSS calls getpwuid for ~ expansion.
envar: "HOME=/data"

# rlimits left deliberately loose — lockdown is the seccomp allowlist below (see csav/README.md).
# fsize raised so large records.json dumps (size sweeps) fit on disk inside the jail.
# fsize raised so app scratch files (size sweeps) fit on disk inside the jail.
rlimit_as: 2048
rlimit_cpu: 3600
rlimit_fsize: 8192
Expand All @@ -47,8 +46,8 @@ mount {
nodev: true
}

# Allowlist for the shipped samples: file I/O and process control only. No socket
# syscalls are allowed; the DC app only reads/writes files under /data.
# Allowlist for the shipped samples: file I/O, process control, and AF_UNIX
# sockets for the sce-unit gRPC channel. The jail stays loopback-free.
seccomp_string: "ALLOW {"
seccomp_string: " read, write, close, lseek, pread64, pwrite64, readv, writev,"
seccomp_string: " openat, openat2, newfstatat, newfstat, statx, getdents64, faccessat, faccessat2,"
Expand All @@ -64,6 +63,9 @@ seccomp_string: " getcwd, ioctl, fcntl, pipe2, dup, dup3,"
seccomp_string: " gettimeofday, clock_gettime, clock_nanosleep, nanosleep, getrandom,"
seccomp_string: " prctl, prlimit64, getrlimit, newuname, sysinfo, sched_yield, sched_getaffinity,"
seccomp_string: " epoll_create1, epoll_ctl, epoll_pwait, epoll_pwait2, ppoll, pselect6,"
seccomp_string: " socket, socketpair, connect, bind, listen, accept, accept4,"
seccomp_string: " sendto, recvfrom, sendmsg, recvmsg, shutdown, getsockopt, setsockopt,"
seccomp_string: " getpeername, getsockname,"
seccomp_string: " eventfd2"
seccomp_string: "}"
seccomp_string: "DEFAULT KILL_PROCESS"
Expand Down
13 changes: 7 additions & 6 deletions secure/csav/registry-service/rootfs.go
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ func composeFinalRootfs(bundleDir, execPath, pathEnv string, execArgs []string,
if err := os.MkdirAll(cfgDir, 0o755); err != nil {
return err
}
if err := writeNsjailCfg(filepath.Join(cfgDir, "nsjail.cfg"), execPath, pathEnv, execArgs); err != nil {
if err := writeNsjailCfg(filepath.Join(cfgDir, "nsjail.cfg"), execPath, pathEnv, execArgs, cfg.AlgorithmId); err != nil {
return err
}
if err := writeRunConfig(filepath.Join(cfgDir, "run-config.json"), cfg); err != nil {
Expand All @@ -85,7 +85,7 @@ func composeFinalRootfs(bundleDir, execPath, pathEnv string, execArgs []string,
return os.Rename(tmpFinal, dcRootfs)
}

func writeNsjailCfg(path, execPath, pathEnv string, args []string) error {
func writeNsjailCfg(path, execPath, pathEnv string, args []string, algorithmID string) error {
tmpl, err := template.New("nsjail.cfg").Parse(nsjailCfgTmpl)
if err != nil {
return err
Expand All @@ -98,10 +98,11 @@ func writeNsjailCfg(path, execPath, pathEnv string, args []string) error {
defer out.Close()

return tmpl.Execute(out, struct {
Path string
PathEnv string
Args []string
}{Path: execPath, PathEnv: pathEnv, Args: args})
Path string
PathEnv string
Args []string
AlgorithmId string
}{Path: execPath, PathEnv: pathEnv, Args: args, AlgorithmId: algorithmID})
}

func writeRunConfig(path string, cfg runConfig) error {
Expand Down
6 changes: 5 additions & 1 deletion secure/pdm-service/src/handlers/records.ts
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,11 @@ export const handler: APIGatewayProxyHandlerV2 = async (event) => {
// A standalone list aligned with `records`; the enclave assembles the DSAR trail from it and posts after a
// successful DC app run. Record objects stay {recordId, data}, so the DC app never sees subject ids.
const items = result.Items ?? [];
const records = items.map((item) => ({ recordId: item.RecordId, data: item.Data }));
const records = items.map((item) => ({
recordId: item.RecordId,
// Data is a DynamoDB Binary holding a serialized RecordEnvelope; JSON transport needs it base64.
data: Buffer.from(item.Data as Uint8Array).toString("base64"),
}));
const dataSubjectIds = items.map((item) => (typeof item.DataSubjectId === "string" ? item.DataSubjectId : ""));
const nextCursor = result.LastEvaluatedKey
? Buffer.from(JSON.stringify(result.LastEvaluatedKey)).toString("base64url")
Expand Down
Loading
Loading