diff --git a/.agents/docs/2026-11-05-cmake-contract-argv-and-link-inputs.md b/.agents/docs/2026-11-05-cmake-contract-argv-and-link-inputs.md new file mode 100644 index 0000000..cf4e434 --- /dev/null +++ b/.agents/docs/2026-11-05-cmake-contract-argv-and-link-inputs.md @@ -0,0 +1,145 @@ +--- +subject: cmake-contract +status: active +--- + +# CMake contract arguments and consumer link inputs + +Audience: maintainers implementing and reviewing the cross-build adapter. + +This record explains the failure boundaries, the selected representation and +the evidence needed before releasing the changes in PR #52. It does not claim +qualification of every CMake generator or every foreign executable platform. + +## Confirmed failures and ownership + +| Failure | Evidence | Owner | +|---|---|---| +| Joining JSON arguments loses literal quotes, backslashes and tabs | The previous reader changes values and argument boundaries under the native parser | Plugin adapter | +| LLVM's GNU response parser drops a quoted empty argument | A native `-D` followed by an empty word is invalid; the corresponding response-file invocation consumes the next valid definition and succeeds | Compiler-parser limitation; adapter rejects the lossy representation | +| A single raw flags string needs different dollar escaping in Make compilation and its direct link script | Actual compiler controls fail despite successful configuration | Plugin adapter | +| An independent subproject executable requires the library that the same subproject has yet to install | Fresh actual musl cross-build fails at that future archive | Core target-contract export | +| Reconfiguration ignores a newly introduced toolchain while retaining old flags | Actual previously configured CMake directory warns that the toolchain is unused | Plugin build-directory identity | +| Switching an existing cross-build from Ninja to Unix Makefiles reuses its incompatible cache | Actual mcpp build fails with the generator mismatch diagnostic | Plugin build-directory identity | +| Changing `options::system_name` retains the previous target system | Actual mcpp build requests Generic but the configured project still reports Linux | Plugin build-directory identity | +| A retargeted Clang driver is identified as a Linux compiler although it emits Windows COFF | A no-SDK static-library control reports `Clang\|\|GNU\|Linux` and names the archive `libprobe.a` | Plugin compiler-ID configuration | +| A source path containing an apostrophe is split in the dependency graph | Pure CMake 4.4.2/Ninja control reproduces without mcpp, a toolchain or response files | External CMake/Ninja path handling | + +The consuming package's own libraries are not target configuration. Core PR +#889 excludes package link flags and libraries from the foreign-build contract +while retaining them on the consumer's actual link. No CMake primitive belongs +in that correction. + +## Selected representation + +The action supplies a generated CMake toolchain. It reads the engine contract +when configuration runs, after the build program has planned the action. +The generated toolchain is an explicit action input. Its contract path is +propagated through `CMAKE_TRY_COMPILE_PLATFORM_VARIABLES`, so ordinary compiler +ABI checks and source-file compile probes load the same settings. + +JSON arrays are read by index and written to compiler response files. GNU +driver quoting and MSVC command-line quoting are separate encodings selected +by the contract's dialect. MSVC files include a UTF-8 BOM. CMake receives each +response-file reference as one native option and performs generator escaping; +literal argument contents do not pass through Ninja, Make or a shell. +One grouped reference also preserves repeated options and their order. +Literal empty GNU argument words are rejected with a configure diagnostic: +Clang's GNU response parser discards them even when quoted. Empty string +values within nonempty options remain supported. No process wrapper or +second argument grammar is introduced for this invalid-invocation boundary. + +The response-file name hashes the dialect and encoded contents. Changed +arguments therefore change the generated compile command. Identical contents +are retained without rewriting the file. A new representation uses its own +build-directory key, since an old cache does not begin loading a toolchain +merely because a later configure command names one. + +Immutable generator selections (`-G`, `-A`, `-T`, and +`CMAKE_GENERATOR*` cache definitions) and the effective cross system name +participate in the build-directory identity. Joined and separated short +options, and typed and untyped cache definitions, use the same identity. +Ordinary `-D` definitions reconfigure the existing directory. Returning to +a previous generator and system reuses that selection's directory. + +Cross-build contract options follow CMake's language and configuration flags. +`options::config` continues to select the CMake configuration; a conflicting +contract profile option takes precedence in the compiler argument order. +The subproject still owns its language standard and target-specific options. +Compiler-ID probing does not consume directory compile options. The adapter +therefore also supplies `CMAKE__COMPILER_TARGET` from the last actual +`--target` or `-target` compiler option, retaining the original argument words. +It does not substitute `contract.target`, which may be a logical platform +alias. This follows CMake's documented [Clang cross-toolchain configuration](https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html#cross-compiling-using-clang). +CMake 3.19 is the minimum for reading the JSON contract. + +## Actual controls + +The adapter passes all 52 current plugin-logic cases and the existing actual +musl cross-build. Compiler controls use its generated toolchain, rather than +a second implementation of the serializer. GCC and Clang execute C and C++ +programs under Ninja and Unix Makefiles with literal quotes, backslashes, +dollar signs, semicolons, tabs, empty strings and Unicode. They also check +language isolation, repeated-option order, actual compile probes, unchanged +objects after reconfiguration and invalidation when one contract word changes. +Linux controls inspect the actual ELF runpath for literal `$ORIGIN` content. +A separate empty-word control places an empty argument after `-D`. GCC and +Clang reject that native invocation; removing the word consumes a valid +fallback definition and succeeds. The generated helper rejects that empty +word during configuration, preventing LLVM's response parser from silently +changing the invalid invocation into a valid one. + +An actual clang-cl invocation consumes the MSVC response file and produces +COFF; its initialized data retains the expected literal values. This is a +compiler-parser control, not native MSVC or Visual Studio generator coverage. + +An actual GNU-style Clang 22 driver is retargeted to Windows without an SDK. +Ninja, Unix Makefiles and Ninja Multi-Config build static libraries whose +members are AMD64 COFF. The resulting identity is +`Clang|MSVC|GNU|Windows`, pointer width is eight, and the archive is named +`probe.lib`. A deliberately different logical contract target remains unused +for compiler identity. The same control fails with the old reader at its +identity assertion, although that reader can emit COFF. + +The real adapter also passes ten successive selections: Ninja/Linux, +Make/Linux, Make no-op, Ninja return, Ninja no-op, Ninja/Generic, Generic +no-op, Linux return, Ninja Multi-Config/Linux and Multi-Config no-op. +Each build executes the actual musl program. Incompatible selections have +distinct CMake directories; return and no-op cases retain object timestamps. +This matrix passes with the diagnostic corrected core and independently with +the published mcpp 2026.11.5.1 binary. It does not depend on the consumer-link +correction. Before the system-name key correction, the first five selections +pass and the Generic selection still reports Linux. + +The actual sidecar failure remains with the response-file adapter and the old +core. Replacing only two core modules fixes it: the library installs, both the +independent sidecar and the consuming musl program execute, and the consumer's +Ninja graph retains its archive. Extended core E2E 982 fails with the old module +and passes with the corrected contract while retaining the consumer link flag. + +The apostrophe negative remains separate. The positive argument test keeps an +apostrophe in the response-file directory; source paths contain spaces, +Unicode and dollar signs. It does not claim to fix native dependency parsing +of source paths containing apostrophes. + +## Remaining qualification + +The response-file and cache-selection regressions can run against the +currently published mcpp 2026.11.5.1. They do not require waiting for the core +release. Native MSVC, Visual Studio and Xcode generators, custom link drivers +and standalone executables over graph-supplied runtimes are not established +by the Linux compiler controls. The original platform CI remains required. + +Core 2026.11.5.2 must complete its final CI, independent ecosystem review, +publication and ordinary installation. Then the plugin CI pin and real +sidecar fixture can follow the public version. The package's protocol floor +remains distinct from a feature's required bug-fix version. All final plugin +CI and review gates remain required before merge and release. + +The final sequence is local adapter verification and documentation, a +reviewable PR #52 update, public core installation and sidecar integration, +then the full plugin CI and ecosystem review. The sidecar uses ordinary +prebuilt musl; it does not establish graph-runtime executable support. + +Public reports contain fixed assertions and synthetic fixtures. Raw user logs, +usernames, private paths and private project names are not published. diff --git a/.github/scripts/check-cmake-cache-identity.py b/.github/scripts/check-cmake-cache-identity.py new file mode 100644 index 0000000..cc7cd7c --- /dev/null +++ b/.github/scripts/check-cmake-cache-identity.py @@ -0,0 +1,85 @@ +#!/usr/bin/env python3 +"""Exercise CMake cache selection through the real mcpp adapter.""" +import argparse +import json +from pathlib import Path +import shutil +import subprocess +import tempfile + + +def checked(command, project, log): + result = subprocess.run(command, cwd=project, stdout=log, stderr=subprocess.STDOUT) + if result.returncode: + log.flush() + details = Path(log.name).read_text(errors="replace")[-8000:] + details = details.replace(str(project), "").replace(str(Path.home()), "") + raise AssertionError(f"mcpp exited {result.returncode}\n{details}") + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("mcpp") + parser.add_argument("--target", default="x86_64-linux-musl") + args = parser.parse_args() + binary = str(Path(shutil.which(args.mcpp) or args.mcpp).resolve()) + repository = Path(__file__).resolve().parents[2] + fixture = repository / "tests/cmake-cross" + with tempfile.TemporaryDirectory(prefix="mcpp-cmake-cache-") as temporary: + project = Path(temporary) / "project" + shutil.copytree(fixture, project, ignore=shutil.ignore_patterns("target", ".mcpp", ".xlings")) + manifest = project / "mcpp.toml" + text = manifest.read_text().replace('path = "../.."', "path = " + json.dumps(str(repository))) + assert text != manifest.read_text(), "fixture dependency path was not replaced" + manifest.write_text(text) + cmake = project / "answer/CMakeLists.txt" + cmake.write_text(cmake.read_text() + r''' +file(WRITE "${CMAKE_BINARY_DIR}/mcpp-cache-probe.json" + "{\"epoch\":${MCPP_CACHE_PROBE_EPOCH},\"generator\":\"${CMAKE_GENERATOR}\",\"system\":\"${CMAKE_SYSTEM_NAME}\"}") +''') + selections = [ + ("Ninja", "Linux"), ("Unix Makefiles", "Linux"), ("Unix Makefiles", "Linux"), + ("Ninja", "Linux"), ("Ninja", "Linux"), + ("Ninja", "Generic"), ("Ninja", "Generic"), ("Ninja", "Linux"), + ("Ninja Multi-Config", "Linux"), ("Ninja Multi-Config", "Linux")] + seen = {} + for epoch, (generator, system) in enumerate(selections, 1): + objects = {p: p.stat().st_mtime_ns for p in project.glob("target/.build-mcpp/out/*/cmake-cross/deps-cmake/answer/build-*/**/*.o")} + options = ["-G", generator, f"-DMCPP_CACHE_PROBE_EPOCH={epoch}"] + words = ", ".join(json.dumps(value) for value in options) + (project / "build.mcpp").write_text('''import std; +import mcpp; +import mcpp.deps.cmake; +int main() { + mcpp::deps::cmake::options o; + o.source = "answer"; + o.libraries = {"answer"}; +''' + f' o.cache_args = {{{words}}};\n o.system_name = "{system}";\n' + ''' return mcpp::deps::cmake::use(o) ? 0 : 1; +} +''') + with (project / "private.log").open("w") as log: + checked([binary, "build", "--target", args.target, "--profile", "dev"], project, log) + current = [] + for path in project.glob("target/.build-mcpp/out/*/cmake-cross/deps-cmake/answer/build-*/mcpp-cache-probe.json"): + record = json.loads(path.read_text()) + if record["epoch"] == epoch: + current.append((path.parent, record)) + assert len(current) == 1, "the active CMake configure did not write its current epoch" + build, record = current[0] + assert record["generator"] == generator and record["system"] == system, record + selection = (generator, system) + if selection in seen: + assert build == seen[selection], "returning to a selection lost its CMake cache" + assert all(p.stat().st_mtime_ns == stamp for p, stamp in objects.items() if p.is_relative_to(build)), "unchanged compilation objects rebuilt" + else: + assert build not in seen.values(), "incompatible selections share a CMake cache" + seen[selection] = build + applications = list((project / "target" / args.target).glob("*/bin/cmake-cross")) + assert len(applications) == 1 + output = subprocess.check_output([str(applications[0])], text=True).strip() + assert output == "cmake-cross: the subproject was compiled against not glibc", output + print(f"PASS: selection {epoch}: {generator}/{system}; actual target program and cache identity") + + +if __name__ == "__main__": + main() diff --git a/.github/scripts/check-cmake-contract-argv.py b/.github/scripts/check-cmake-contract-argv.py new file mode 100644 index 0000000..6a8345c --- /dev/null +++ b/.github/scripts/check-cmake-contract-argv.py @@ -0,0 +1,217 @@ +#!/usr/bin/env python3 +"""Exercise the generated contract toolchain with actual compiler arguments.""" +import argparse +import json +import sys +from pathlib import Path +import shutil +import subprocess +import tempfile + + +def checked(command, cwd, log): + result = subprocess.run(command, cwd=cwd, stdout=log, stderr=subprocess.STDOUT) + if result.returncode: + log.flush() + details = Path(log.name).read_text(errors="replace")[-8000:] + details = details.replace(str(cwd), "").replace(str(Path.home()), "") + raise AssertionError(f"{Path(str(command[0])).name} exited {result.returncode}\n{details}") + + +def check_clang_target(root, toolchain, manifest, cmake, clang): + compiler = Path(clang).resolve() + ar, rc = (compiler.parent / name for name in ("llvm-ar", "llvm-rc")) + assert ar.is_file() and rc.is_file(), "missing LLVM archive/resource tools" + source = root / "clang-target-source" + source.mkdir() + (source / "probe.c").write_text('''#if !defined(_WIN32) || !defined(_MSC_VER) || !defined(_M_X64) +#error wrong_target +#endif +typedef char pointer_width[(sizeof(void*) == 8) ? 1 : -1]; +int answer(void) { return 42; } +''') + (source / "CMakeLists.txt").write_text('''cmake_minimum_required(VERSION 3.19) +project(target_identity C) +add_library(probe STATIC probe.c) +file(WRITE "${CMAKE_BINARY_DIR}/identity.txt" "${CMAKE_C_COMPILER_ID}|${CMAKE_C_SIMULATE_ID}|${CMAKE_C_COMPILER_FRONTEND_VARIANT}|${CMAKE_C_PLATFORM_ID}|${CMAKE_SIZEOF_VOID_P}|${CMAKE_C_COMPILER_TARGET}") +''') + contract = { + "schema": 1, "dialect": "gnu", "target": "logical-platform-alias", + "compile": {name: {"program": str(compiler), "args": [ + "--target=aarch64-linux-gnu", "-target", "x86_64-pc-windows-msvc"]} + for name in ("c", "cxx", "asm")}, + "link": {"program": str(compiler.parent / "lld-link"), "archiver": str(ar), "args": []}} + manifest.write_text(json.dumps(contract)) + for generator in ("Ninja", "Unix Makefiles", "Ninja Multi-Config"): + build = root / ("clang-target-" + generator.replace(" ", "-")) + with (root / "private.log").open("w") as log: + checked([cmake, "-S", str(source), "-B", str(build), "-G", generator, + "-DCMAKE_SYSTEM_NAME=Windows", "-DCMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY", + "-DCMAKE_TOOLCHAIN_FILE=" + str(toolchain), "-DMCPP_CONTRACT=" + str(manifest), + "-DCMAKE_RC_COMPILER=" + str(rc)], root, log) + checked([cmake, "--build", str(build), "--config", "Release"], root, log) + identity = (build / "identity.txt").read_text() + assert identity == "Clang|MSVC|GNU|Windows|8|x86_64-pc-windows-msvc", f"compiler target identity differs: {identity}" + archive = build / "Release/probe.lib" if generator == "Ninja Multi-Config" else build / "probe.lib" + assert archive.is_file() and not (build / "libprobe.a").exists(), "target archive naming differs" + member = subprocess.check_output([str(ar), "p", str(archive)]) + assert member[:2] == b"\x64\x86", "archive member is not AMD64 COFF" + print(f"PASS: {generator}: retargeted Clang identity, ABI, archive naming and actual COFF") + + +def check_empty_argument(root, source, toolchain, manifest, contract, cmake): + (source / "CMakeLists.txt").write_text("cmake_minimum_required(VERSION 3.19)\nproject(empty_argument NONE)\n") + unit = root / "empty-argument.c" + unit.write_text("int answer(void) { return 42; }\n") + probe = json.loads(json.dumps(contract)) + # Some drivers ignore an empty input filename. Place the empty word where + # -D must consume it: dropping it would instead consume the valid fallback. + probe["compile"]["c"]["args"] = ["-DMCPP_EMPTY_ARGUMENT_CONTROL=1", "-D"] + build = root / "empty-argument-build" + with (root / "private.log").open("w") as log: + configure = [cmake, "-S", str(source), "-B", str(build), "-G", "Ninja", + "-DCMAKE_TOOLCHAIN_FILE=" + str(toolchain), "-DMCPP_CONTRACT=" + str(manifest)] + for empty in (False, True): + if empty: + probe["compile"]["c"]["args"].append("") + manifest.write_text(json.dumps(probe)) + if empty: + # LLVM's GNU response parser discards quoted empty words. + # Reject the lossy representation instead of running it. + result = subprocess.run(configure, cwd=root, capture_output=True, text=True) + assert result.returncode > 0 and "empty GNU argument" in result.stdout + result.stderr + command = [probe["compile"]["c"]["program"], *probe["compile"]["c"]["args"], + "MCPP_EMPTY_WORD_FALLBACK=1", "-x", "c", "-c", str(unit), + "-o", str(root / "empty-argument.o")] + native = subprocess.run(command, cwd=root, capture_output=True, text=True) + assert native.returncode > 0, "the empty native argument control must be invalid" + continue + before = set(toolchain.parent.glob("*.rsp")) + checked(configure, root, log) + added = set(toolchain.parent.glob("*.rsp")) - before + response = [p for p in added if b"MCPP_EMPTY_ARGUMENT_CONTROL" in p.read_bytes()] + assert len(response) == 1, "expected a newly addressed response file" + command = [probe["compile"]["c"]["program"], "@" + str(response[0]), + "MCPP_EMPTY_WORD_FALLBACK=1", "-x", "c", "-c", str(unit), + "-o", str(root / "empty-argument.o")] + checked(command, root, log) + print("PASS: a lossy empty GNU argv word is rejected; the actual native invocation is invalid") + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("toolchain", type=Path) + parser.add_argument("--cmake", default="cmake") + parser.add_argument("--cc", default="cc") + parser.add_argument("--cxx", default="c++") + parser.add_argument("--msvc-preprocessor", help="Also compile a COFF probe with clang-cl or cl") + parser.add_argument("--target-clang", help="Also verify retargeted Clang identity and a Windows static library without an SDK") + args = parser.parse_args() + helper = args.toolchain.resolve().read_bytes() + values = ["hello world", 'literal "quotes"', "back\\slash\\", "$MCPP_ARG_WORD", "a;b", "tab\tvalue", "", "café Ω"] + with tempfile.TemporaryDirectory(prefix="mcpp-cmake-argv-") as temp: + root = Path(temp) / "space Ω $literal" + root.mkdir() + helper_dir = root / "contract 'quote" + helper_dir.mkdir() + toolchain = helper_dir / "contract.cmake" + toolchain.write_bytes(helper) + source = root / "source" + (source / "child").mkdir(parents=True) + (source / "CMakeLists.txt").write_text('''cmake_minimum_required(VERSION 3.16) +project(contract_words C CXX) +include(CheckCSourceCompiles) +check_c_source_compiles("#ifndef ONLY_C\\n#error contract_missing\\n#endif\\nint main(void){return 0;}" CONTRACT_PROBE) +if(NOT CONTRACT_PROBE) + message(FATAL_ERROR "contract arguments did not reach try_compile") +endif() +add_subdirectory(child) +''') + (source / "child/CMakeLists.txt").write_text("add_executable(probe_c probe.c)\nadd_executable(probe_cxx probe.cpp)\n") + code = '''#include +#ifdef __cplusplus +#if !defined(ONLY_CXX) || defined(ONLY_C) +#error wrong_language_options +#endif +#else +#if !defined(ONLY_C) || defined(ONLY_CXX) +#error wrong_language_options +#endif +#endif +#if !defined(PROBE_ORDER) || PROBE_ORDER != 1 +#error option_group_order_lost +#endif +int main(void) { +''' + code += "".join(f"puts(PROBE_{i});\n" for i in range(len(values))) + "return 0;\n}\n" + for name in ("probe.c", "probe.cpp"): + (source / "child" / name).write_text(code) + definitions = [f"-DPROBE_{i}={json.dumps(v, ensure_ascii=False).replace(chr(92) + 't', chr(9))}" for i, v in enumerate(values)] + contract = {"schema": 1, "dialect": "gnu", "compile": { + "c": {"program": shutil.which(args.cc), "args": [*definitions, "-DONLY_C", "-DPROBE_ORDER=1", "-UPROBE_ORDER", "-DPROBE_ORDER=1"]}, + "cxx": {"program": shutil.which(args.cxx), "args": [*definitions, "-DONLY_CXX", "-DPROBE_ORDER=1", "-UPROBE_ORDER", "-DPROBE_ORDER=1"]}, + "asm": {"program": shutil.which(args.cc), "args": []}}, + "link": {"program": shutil.which(args.cxx), "archiver": shutil.which("ar"), "args": []}} + assert all(v["program"] for v in contract["compile"].values()), "missing compiler" + if sys.platform.startswith("linux"): + contract["link"]["args"] = ["-Wl,-rpath,$ORIGIN/probe space"] + manifest = root / "contract.json" + for generator in ("Ninja", "Unix Makefiles"): + build = root / generator.replace(" ", "-") + manifest.write_text(json.dumps(contract)) + configure = [args.cmake, "-S", str(source), "-B", str(build), "-G", generator, + "-DCMAKE_TOOLCHAIN_FILE=" + str(toolchain), "-DMCPP_CONTRACT=" + str(manifest)] + with (root / "private.log").open("w") as log: + checked(configure, root, log) + checked([args.cmake, "--build", str(build), "--verbose"], root, log) + for name in ("probe_c", "probe_cxx"): + output = subprocess.check_output([str(build / "child" / name)], text=True) + assert output.splitlines() == values, (generator, name) + if sys.platform.startswith("linux"): + dynamic = subprocess.check_output(["readelf", "-d", str(build / "child" / name)], text=True) + assert "$ORIGIN/probe space" in dynamic, "literal linker path changed" + objects = {p: p.stat().st_mtime_ns for p in build.rglob("*.o")} + assert objects, "no compilation objects" + checked(configure, root, log) + if generator == "Ninja": + checked([shutil.which("ninja"), "-C", str(build), "-d", "explain", "-n"], root, log) + checked([args.cmake, "--build", str(build), "--verbose"], root, log) + rebuilt = [str(p.relative_to(build)) for p, stamp in objects.items() if p.stat().st_mtime_ns != stamp] + if rebuilt: + log.flush() + details = Path(log.name).read_text(errors="replace")[-8000:].replace(str(root), "").replace(str(Path.home()), "") + raise AssertionError(f"unchanged flags rebuilt objects: {rebuilt}\n{details}") + changed = json.loads(json.dumps(contract)) + changed["compile"]["c"]["args"][0] = '-DPROBE_0="changed value"' + manifest.write_text(json.dumps(changed)) + checked(configure, root, log) + checked([args.cmake, "--build", str(build)], root, log) + assert subprocess.check_output([str(build / "child/probe_c")], text=True).splitlines() == ["changed value", *values[1:]] + assert subprocess.check_output([str(build / "child/probe_cxx")], text=True).splitlines() == values + print(f"PASS: {generator}: literal arguments, language scope, try_compile and incremental identity") + check_empty_argument(root, source, toolchain, manifest, contract, args.cmake) + if args.msvc_preprocessor: + contract["dialect"] = "msvc" + for spec in contract["compile"].values(): + spec["program"] = str(Path(args.msvc_preprocessor).resolve()) + spec["args"] = ["/utf-8", *[f"/DPROBE_{i}={json.dumps(v, ensure_ascii=False).replace(chr(92) + 't', chr(9))}" for i, v in enumerate(values)]] + manifest.write_text(json.dumps(contract)) + (source / "CMakeLists.txt").write_text("cmake_minimum_required(VERSION 3.16)\nproject(response_only NONE)\n") + with (root / "private.log").open("w") as log: + checked([args.cmake, "-S", str(source), "-B", str(root / "msvc"), "-G", "Ninja", + "-DCMAKE_TOOLCHAIN_FILE=" + str(toolchain), "-DMCPP_CONTRACT=" + str(manifest)], root, log) + generated = [p for p in helper_dir.glob("*.rsp") if p.read_bytes().startswith(b"\xef\xbb\xbf") and b"/DPROBE_0=" in p.read_bytes()] + assert len(generated) == 1, "expected one content-addressed MSVC response file" + probe = root / "coff.c" + probe.write_text("".join(f"const char probe_{i}[] = PROBE_{i};\n" for i in range(len(values)))) + output = root / "coff.obj" + checked([args.msvc_preprocessor, "/nologo", "/c", "/TC", "@" + str(generated[0]), str(probe), "/Fo" + str(output)], root, log) + data = output.read_bytes() + assert all(v.encode() + b"\0" in data for v in values), "COFF literal values differ" + print("PASS: MSVC response quoting: actual compiler preserves all literal values in COFF") + if args.target_clang: + check_clang_target(root, toolchain, manifest, args.cmake, args.target_clang) + + +if __name__ == "__main__": + main() diff --git a/.github/scripts/check-deps-and-qt.sh b/.github/scripts/check-deps-and-qt.sh index 9ab9af6..20652ca 100644 --- a/.github/scripts/check-deps-and-qt.sh +++ b/.github/scripts/check-deps-and-qt.sh @@ -210,6 +210,38 @@ cmake_consumer() { echo "ok: a file added to the subproject re-ran its installation" } +# A BUILD FOR ANOTHER TARGET (0.21.0): the subproject is compiled with the +# compilers and words of the engine's contract for the target, read when the +# action runs, and not with the machine's toolset. On a glibc machine, +# `--target x86_64-linux-musl` must give a subproject compiled against musl. +cmake_cross() { + cd "$ROOT/tests/cmake-cross" + rm -rf target + mkdir -p target/ci + "$MCPP" build --target x86_64-linux-musl > target/ci/build.log 2>&1 || { cat target/ci/build.log; fail "the cross build failed"; } + script=$(find target -path '*deps-cmake/answer/answer.cmake' | head -1) + [ -n "$script" ] || fail "no deps-cmake script was written" + grep -q "the engine's contract for x86_64-linux-musl" "$script" || { cat "$script"; fail "the script does not read the contract"; } + cache=$(find target -path '*deps-cmake/answer/build-*/CMakeCache.txt' | head -1) + grep -q '^CMAKE_SYSTEM_NAME:.*=Linux' "$cache" || { grep CMAKE_SYSTEM_NAME "$cache"; fail "CMAKE_SYSTEM_NAME is not Linux"; } + grep -q '^CMAKE_TRY_COMPILE_TARGET_TYPE:.*=STATIC_LIBRARY' "$cache" || fail "the try-compile links" + "$MCPP" run --target x86_64-linux-musl | tee target/ci/run.log + grep -q '^cmake-cross: the subproject was compiled against not glibc$' target/ci/run.log || + fail "the subproject was compiled with the machine's C library" + sidecar=$(find target -path '*deps-cmake/answer/install/bin/answer-sidecar' -type f | head -1) + [ -n "$sidecar" ] || fail "the CMake subproject did not install its own executable" + "$sidecar" | tee target/ci/sidecar-run.log + grep -q '^cmake-sidecar: the subproject was compiled against not glibc$' target/ci/sidecar-run.log || + fail "the independent CMake executable did not use the target runtime" + # The contract excludes consumer libraries, but the consumer still links + # the installation. Inspect its own rule, not a reference elsewhere. + graph=$(find target -path '*/build.ninja' ! -path '*/.build-mcpp/*' | head -1) + [ -n "$graph" ] || fail "the consumer Ninja graph is missing" + grep -q '^ldflags[[:space:]]*=.*libanswer\.a' "$graph" || + fail "the consuming program lost the subproject archive from its link flags" + echo "ok: the target CMake library, independent executable and consumer link all work" +} + # AN INSTALLATION IS BUILT ONCE (0.18.0). deps-cmake keeps what a subproject # installed under a key that holds no path of the machine, and copies it into # the prefix of any later build with the same key. Only a toolset the engine @@ -673,6 +705,7 @@ case "${1:-}" in vcpkg-workspace) vcpkg_workspace ;; cmake-consumer) cmake_consumer ;; cmake-cache) cmake_cache ;; + cmake-cross) cmake_cross ;; qt-consumer) qt_consumer ;; qt-widgets-consumer) qt_widgets_consumer ;; qt-sdk-consumer) qt_sdk_consumer ;; @@ -682,5 +715,5 @@ case "${1:-}" in vcpkg-managed) vcpkg_managed ;; vcpkg-msbuild-refused) vcpkg_msbuild_refused ;; vcpkg-make-port) vcpkg_make_port ;; - *) echo "usage: $0 vcpkg-consumer|vcpkg-libcxx|archive-consumer|vcpkg-workspace|cmake-consumer|cmake-cache|qt-consumer|qt-widgets-consumer|qt-sdk-consumer|qt-import-only|plugin-logic|tool-sources|vcpkg-crt|vcpkg-managed|vcpkg-msbuild-refused|vcpkg-make-port"; exit 2 ;; + *) echo "usage: $0 vcpkg-consumer|vcpkg-libcxx|archive-consumer|vcpkg-workspace|cmake-consumer|cmake-cache|cmake-cross|qt-consumer|qt-widgets-consumer|qt-sdk-consumer|qt-import-only|plugin-logic|tool-sources|vcpkg-crt|vcpkg-managed|vcpkg-msbuild-refused|vcpkg-make-port"; exit 2 ;; esac diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6e2a2b3..59dba96 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -83,18 +83,27 @@ env: # phase -- and `mcpp-package.json` beside every staged tree, which # `mcpp::plugins::package` reads. Until that release exists the jobs build it # from `MCPP_SOURCE_REF` (the repository variable names the branch). - MCPP_VERSION: 2026.10.27.1 + # + # 2026.11.7.1 IS WHAT 0.21.0'S CMAKE CROSS LANE NEEDS (mcpp#885): the engine + # completes a cross subproject's contract (its compilers, archiver and target) + # when the action runs, which a subproject that links an executable or a + # shared library needs. The package floor stays 2026.10.30.1: every other + # member, and a cross subproject that only builds archives, works on the + # older engine, and docs/deps.md states the requirement where cross builds + # are described. So here, as for 0.20.0's SYCL lane, `MCPP_VERSION` is the + # newest engine a lane needs and the floor the oldest the collection loads on. + MCPP_VERSION: 2026.11.7.1 # THE RELEASE EVERY JOB FETCHES, WHICH IS NOT ALWAYS THE ONE UNDER TEST. # - # `MCPP_VERSION` states the engine this collection needs -- the package floor - # (`[package] mcpp`) is the same statement -- and a release of it may not exist + # `MCPP_VERSION` states the engine this collection's lanes need -- the package + # floor (`[package] mcpp`) the oldest it loads on -- and a release of it may not exist # yet: 0.19.0 was written against mcpp#755 and pinned 2026.10.1.3 before that # release was published. Every fetch therefore names this variable instead, and # the dispatch input `mcpp_source_ref` builds the engine under review with it. # # It moves to `MCPP_VERSION` once that release exists, so the ordinary path # runs the engine this collection states. - MCPP_BOOTSTRAP: 2026.10.27.1 + MCPP_BOOTSTRAP: 2026.11.7.1 # AN ENGINE BUILT FROM SOURCE, WHEN A DISPATCH NAMES ONE. # # Empty on every push and pull request, so the steps run the release above, @@ -117,7 +126,7 @@ env: # one-line change here. The mcpp is the bootstrap release, which also builds # the engine under review when `MCPP_SOURCE_REF` names one. MCPPLS_VERSION: 0.0.15 - MCPP_EDITOR_VERSION: 2026.10.27.1 + MCPP_EDITOR_VERSION: 2026.11.7.1 # PINNED, AND WITHOUT IT THE CACHE BELOW CACHED NOTHING. # # A released mcpp is self-contained: with no `MCPP_HOME`, `mcpp self env` @@ -222,6 +231,16 @@ jobs: fi echo "ok: one version, two files agreeing" + # NO HEADER AND NO PREPROCESSOR BRANCH (0.21.0, docs/plugin-development.md + # §4): the operating system is mcpplibs.os's, the C library std's, and a + # platform difference of the building machine is `if constexpr`. + - name: no unit includes a header or branches in the preprocessor + run: | + found=$(grep -nE '^[[:space:]]*#[[:space:]]*(include|if|ifdef|ifndef|elif)\b' \ + src/*.cppm src/*/*.cppm rules/*.cppm dist/*.cppm deps/*.cppm deps/*/*.cppm tools/*.cppm 2>/dev/null || true) + [ -z "$found" ] || { echo "FAIL: a unit includes a header or branches in the preprocessor"; echo "$found"; exit 1; } + echo "ok: every unit reaches the system through modules" + # ONE DERIVATION, AND THE CHECK THAT KEEPS IT ONE. # # `common_base_dir` and `namespace_of` were written in `rules/spirv.cppm` @@ -1470,10 +1489,39 @@ jobs: "$MCPP" build test -f $(sh "$GITHUB_WORKSPACE/tests/out-dir.sh")/saxpy.sycl.o test -f $(sh "$GITHUB_WORKSPACE/tests/out-dir.sh")/sycl_device_link.o + python3 verify-images.py sm_89 + "$MCPP" build --accel 'sycl+{nvptx64-sm_89,nvptx64-sm_90}' + python3 verify-images.py sm_89,sm_90 + "$MCPP" build --accel 'sycl+{spir64,nvptx64-sm_89}' + python3 verify-images.py spir64,sm_89 + "$MCPP" build --no-accel + "$MCPP" run --no-accel | tee run.log + grep -q '^12 24 36 48' run.log + + # A SYCL library of two NVIDIA architectures and a program that links it (mcpp#899): the library's + # device link reads its own objects, so the SDK never unbundles an archive of several architectures, + # and each image is its architecture's. + - name: rules-sycl through a library of two NVIDIA architectures + working-directory: tests/sycl-library-consumer/app + run: | + "$MCPP" build + python3 ../../sycl-consumer/verify-images.py sm_89,sm_90 "$MCPP" build --no-accel "$MCPP" run --no-accel | tee run.log grep -q '^12 24 36 48' run.log + - name: an ordinary worker imports the progress writer without build-program modules + working-directory: tests/progress-worker + run: python3 verify.py "$MCPP" + + - name: CUDA and SYCL coexist with independent code targets + working-directory: tests/multi-accel-consumer + run: | + "$MCPP" build + python3 ../sycl-consumer/verify-images.py sm_89 --cuda-target=sm_90 + # Both host objects execute; this does not submit work to a GPU. + "$MCPP" run + # EVERY RULE COMPILED FOR THIS HOST, which is a different claim from any # step above. Each of those exercises one rule and needs that rule's # payload, so a rule is only ever compiled on the platforms its payload @@ -1756,6 +1804,24 @@ jobs: - name: deps-cmake keeps an installation and takes it for the same key run: bash .github/scripts/check-deps-and-qt.sh cmake-cache + # 0.21.0: a subproject of a build for another target is compiled from the + # engine's contract for it (mcpp 2026.10.30.1+): musl from a glibc machine. + - name: deps-cmake compiles a cross build's subproject from the engine's contract + run: bash .github/scripts/check-deps-and-qt.sh cmake-cross + + - name: CMake contract argv survives both generators and compiler dialects + run: | + helper=$(find tests/cmake-cross/target -name answer-contract.cmake | head -1) + test -n "$helper" + llvm="${MCPP_HOME:-$HOME/.mcpp}/registry/data/xpkgs/xim-x-llvm/22.1.8/bin" + test -x "$llvm/clang-cl" + python3 .github/scripts/check-cmake-contract-argv.py "$helper" \ + --cc /usr/bin/cc --cxx /usr/bin/c++ --msvc-preprocessor "$llvm/clang-cl" \ + --target-clang "$llvm/clang" + python3 .github/scripts/check-cmake-contract-argv.py "$helper" \ + --cc "$llvm/clang" --cxx "$llvm/clang++" + python3 .github/scripts/check-cmake-cache-identity.py "$MCPP" + - name: rules-qt runs moc, rcc, lupdate and lrelease, with the SDK the project declares run: bash .github/scripts/check-deps-and-qt.sh qt-consumer @@ -2528,6 +2594,10 @@ jobs: # The test kit on this host: the same stated contexts, so a decision that # depends on the host the build program runs on is caught here. + - name: standalone progress uses Unicode channels on this host + working-directory: tests/progress-worker + run: python3 verify.py "$MCPP" + - name: the deps members' decisions hold against stated build contexts (mcpp.plugins.testing) run: bash .github/scripts/check-deps-and-qt.sh plugin-logic diff --git a/README.md b/README.md index cfdd8f0..4376274 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ module name the member declares, and configures it there. ```toml [build-dependencies.mcpp] -plugins = { version = "0.20.0", features = ["rules-spirv"], host-module = true } +plugins = { version = "0.21.0", features = ["rules-spirv"], host-module = true } ``` ```cpp @@ -51,12 +51,17 @@ naming it to avoid a download, and the scenarios on both sides -- is in ## Members -From 0.20.0 the package as a whole needs mcpp 2026.10.27.1 (`[package] mcpp`), -whose protocol 17 states a platform's fields and whose `mcpp pack` writes -`mcpp-package.json` beside every staged tree (0.19.0 needed 2026.10.1.3, whose +From 0.21.0 the package as a whole needs mcpp 2026.10.30.1 (`[package] mcpp`), +which writes the configuration's contract that `deps-cmake` compiles a cross +build from (0.20.0 needed 2026.10.27.1, whose protocol 17 states a platform's +fields and whose `mcpp pack` writes `mcpp-package.json` beside every staged +tree; 0.19.0 needed 2026.10.1.3, whose protocol 15 states where each tool and payload comes from); the column states the release each member's behaviour first needed. +Cross CMake subprojects that link executables or shared libraries require +mcpp 2026.11.7.1 or later; cross configuration requires CMake 3.19 or later. + | feature | module | mcpp floor | what it does | doc | |---|---|---|---|---| | `rules-ascendc` | `mcpp.rules.ascendc` | 2026.10.1.3 | Compiles Ascend C (`*.asc`) with BiSheng in mixed mode, so the object joins the ordinary link. | [rules](docs/rules.md#rules-ascendc) | @@ -76,7 +81,7 @@ the release each member's behaviour first needed. | `dist-web` | `mcpp.dist.web` | 2026.9.13.1 | Copies a `wasm32-emscripten` program and its files into a web directory with an `index.html`. | [dist](docs/dist.md#dist-web) | | `dist-apk` | `mcpp.dist.apk` | 2026.10.1.3 | Packs the native closure into a signed APK or App Bundle, with Java, Kotlin and Maven libraries. | [dist-apk](docs/dist-apk.md) | | `deps-vcpkg` | `mcpp.deps.vcpkg` | 2026.10.1.3 | Installs a `vcpkg.json` manifest as an action with the toolset mcpp resolved, and maps the prefix into the build. | [deps](docs/deps.md#deps-vcpkg) | -| `deps-cmake` | `mcpp.deps.cmake` | 2026.10.1.3 | Builds and installs a CMake subproject as an action with Ninja and the toolset mcpp resolved, keeps the installation for any build with the same key, and maps the prefix into the build. | [deps](docs/deps.md#deps-cmake) | +| `deps-cmake` | `mcpp.deps.cmake` | 2026.10.1.3; a cross build 2026.10.30.1 | Builds and installs a CMake subproject as an action with Ninja and the toolset mcpp resolved -- for another target, the compilers and words of the engine's contract -- keeps the installation for any build with the same key, and maps the prefix into the build. | [deps](docs/deps.md#deps-cmake) | | `deps-archive` | `mcpp.deps.archive` | 2026.10.1.3 | Extracts a zip archive the project keeps and places its tree beside the program. | [deps](docs/deps.md#deps-archive) | Some features add a sub-capability to a member: `dist-apk-kotlin` and @@ -103,6 +108,8 @@ The first three need mcpp 2026.10.1.3. `surface`, the name of `plugins-core` bef mechanism runs; the libraries and SDKs a program links are the project's declaration (0.15.0 removed `rules-qt-xim*`; see [rules-qt](docs/rules-qt.md)). +`mcpp.plugins.accel` reads complete implementation-owned target requests; `mcpp.plugins.progress` enables optional action telemetry. Worker programs can select `plugins-progress` for `mcpp.plugins.progress.writer`, the atomic snapshot writer. See [plugin-development](docs/plugin-development.md#optional-action-progress). + ## Layout ``` diff --git a/deps/cmake.cppm b/deps/cmake.cppm index b96290f..c950f9b 100644 --- a/deps/cmake.cppm +++ b/deps/cmake.cppm @@ -52,6 +52,7 @@ export module mcpp.deps.cmake; import std; import mcpp; +import mcpp.plugins.progress; import mcpp.plugins; import mcpp.deps; import mcpp.plugins.fs; @@ -69,6 +70,7 @@ struct layout { }; struct options { + bool progress = false; // optional configure/build/install stage telemetry // The subproject's source directory, relative to the package root. std::string source; // Names the action and the prefix. Empty takes the source directory's name. @@ -118,8 +120,142 @@ struct options { // `~/.cache/mcpp-plugins/deps-cmake`). Read when the action runs; the place // is no part of what is built. See docs/deps.md. std::string cache; + // `CMAKE_SYSTEM_NAME` of a build for another target (0.21.0). Empty + // derives it from the target's system: Linux, Android, Darwin, iOS, + // Windows, Emscripten, Generic for bare metal, and Linux for an ok image, + // whose C library is POSIX at the source level. + std::string system_name; }; +// The `CMAKE_SYSTEM_NAME` of a target system (`mcpp::target_os()`). +inline std::string cmake_system_name(std::string_view os) { + if (os == "linux" || os == "openkal") return "Linux"; + if (os == "android") return "Android"; + if (os == "macos") return "Darwin"; + if (os == "ios") return "iOS"; + if (os == "windows") return "Windows"; + if (os == "emscripten") return "Emscripten"; + return "Generic"; +} + +// The target toolchain, evaluated by CMake and its try_compile projects when +// the action runs. Compiler response files retain the contract's argv words; +// CMake's native option handling escapes each response-file reference for the +// selected generator. A content-derived file name also invalidates commands +// when the arguments change, without rewriting unchanged response files. +inline std::string contract_reader() { + return R"cmake(include_guard(GLOBAL) +if(CMAKE_VERSION VERSION_LESS 3.19) + message(FATAL_ERROR "mcpp.deps.cmake: target contracts require CMake 3.19 or later") +endif() +if(NOT EXISTS "${MCPP_CONTRACT}") + message(FATAL_ERROR "mcpp.deps.cmake: missing target build contract '${MCPP_CONTRACT}'; mcpp 2026.10.30.1 and later write it when the action runs") +endif() +list(APPEND CMAKE_TRY_COMPILE_PLATFORM_VARIABLES MCPP_CONTRACT) +file(READ "${MCPP_CONTRACT}" _mcpp_contract) +string(JSON _mcpp_dialect GET "${_mcpp_contract}" dialect) +if(NOT _mcpp_dialect MATCHES "^(gnu|msvc)$") + message(FATAL_ERROR "mcpp.deps.cmake: unsupported command dialect '${_mcpp_dialect}'") +endif() +function(_mcpp_rsp out) + string(JSON _args GET "${_mcpp_contract}" ${ARGN}) + string(JSON _n LENGTH "${_args}") + set(_content "") + if(_n GREATER 0) + math(EXPR _last "${_n} - 1") + foreach(_i RANGE ${_last}) + string(JSON _word GET "${_args}" ${_i}) + if(_mcpp_dialect STREQUAL "gnu" AND _word STREQUAL "") + message(FATAL_ERROR "mcpp.deps.cmake: an empty GNU argument cannot be represented portably in a compiler response file") + endif() + if(_mcpp_dialect STREQUAL "msvc") + set(_encoded "\"") + set(_slashes "") + string(LENGTH "${_word}" _length) + if(_length GREATER 0) + math(EXPR _end "${_length} - 1") + foreach(_j RANGE ${_end}) + string(SUBSTRING "${_word}" ${_j} 1 _char) + if(_char STREQUAL [[\]]) + string(APPEND _slashes [[\]]) + else() + if(_char STREQUAL [["]]) + string(APPEND _encoded "${_slashes}${_slashes}" [[\"]]) + else() + string(APPEND _encoded "${_slashes}${_char}") + endif() + set(_slashes "") + endif() + endforeach() + endif() + string(APPEND _encoded "${_slashes}${_slashes}\"") + else() + string(REPLACE [[\]] [[\\]] _encoded "${_word}") + string(REPLACE [["]] [[\"]] _encoded "${_encoded}") + set(_encoded "\"${_encoded}\"") + endif() + string(APPEND _content "${_encoded}\n") + endforeach() + endif() + if(_mcpp_dialect STREQUAL "msvc") + string(ASCII 239 187 191 _bom) + string(PREPEND _content "${_bom}") + endif() + string(SHA256 _digest "${_mcpp_dialect}:${_content}") + set(_file "${CMAKE_CURRENT_LIST_DIR}/mcpp-contract-${_digest}.rsp") + set(_old "") + if(EXISTS "${_file}") + file(READ "${_file}" _old) + endif() + if(NOT EXISTS "${_file}" OR NOT _old STREQUAL _content) + file(WRITE "${_file}" "${_content}") + endif() + # SHELL uses CMake's UNIX parser on every host; the generator then owns + # escaping this one argument for its actual native command representation. + string(REPLACE "'" "'\\''" _quoted "${_file}") + set(${out} "SHELL:@'${_quoted}'" PARENT_SCOPE) +endfunction() +function(_mcpp_compiler_target out) + string(JSON _args GET "${_mcpp_contract}" ${ARGN}) + string(JSON _n LENGTH "${_args}") + set(_target "") + set(_next FALSE) + if(_n GREATER 0) + math(EXPR _last "${_n} - 1") + foreach(_i RANGE ${_last}) + string(JSON _word GET "${_args}" ${_i}) + if(_next) + set(_target "${_word}") + set(_next FALSE) + elseif(_word STREQUAL "--") + break() + elseif(_word STREQUAL "--target" OR _word STREQUAL "-target") + set(_next TRUE) + elseif(_word MATCHES "^--?target=(.*)$") + set(_target "${CMAKE_MATCH_1}") + endif() + endforeach() + endif() + set(${out} "${_target}" PARENT_SCOPE) +endfunction() +foreach(_lang c cxx asm) + string(TOUPPER "${_lang}" _upper) + string(JSON CMAKE_${_upper}_COMPILER GET "${_mcpp_contract}" compile ${_lang} program) + # Compiler-ID probing does not consume directory compile options. Use the + # actual compiler argument, not the contract's logical platform alias. + _mcpp_compiler_target(_target compile ${_lang} args) + if(NOT _target STREQUAL "") + set(CMAKE_${_upper}_COMPILER_TARGET "${_target}") + endif() + _mcpp_rsp(_option compile ${_lang} args) + add_compile_options("$<$:${_option}>") +endforeach() +string(JSON CMAKE_AR GET "${_mcpp_contract}" link archiver) +_mcpp_rsp(_option link args) +add_link_options("${_option}") +)cmake"; +} + // The variable a `-D[:]=` argument defines; empty for any // other argument. inline std::string_view defined_variable(std::string_view arg) { @@ -159,6 +295,35 @@ inline bool chooses_generator(std::span args) { return defines(args, "CMAKE_GENERATOR"); } +// Generator, architecture, toolset and instance cannot share a CMake cache +// with another selection. Ordinary -D options can change by reconfiguration. +inline std::string generator_key(std::span args) { + std::string key; + auto add = [&](std::string_view word) { + key += std::to_string(word.size()) + ":" + std::string(word); + }; + for (std::size_t i = 0; i < args.size(); ++i) { + const std::string_view arg = args[i]; + if (arg.starts_with("-G") || arg.starts_with("-A") || arg.starts_with("-T")) { + add(arg.substr(0, 2)); + auto value = arg.substr(2); + if (arg.size() == 2 && i + 1 < args.size()) value = args[++i]; + add(value); + continue; + } + std::string_view definition; + if (arg == "-D" && i + 1 < args.size()) definition = args[i + 1]; + else if (arg.starts_with("-D")) definition = arg.substr(2); + const auto name = definition.substr(0, definition.find_first_of(":=")); + if (name != "CMAKE_GENERATOR" && !name.starts_with("CMAKE_GENERATOR_")) continue; + add(name); + const auto equals = definition.find('='); + add(equals == std::string_view::npos ? std::string_view{} : definition.substr(equals + 1)); + if (arg == "-D") ++i; + } + return key; +} + // The prefix, by name (SPEC-007 R1.3). struct prefix { std::string root, include, lib, bin; @@ -175,6 +340,42 @@ inline mcpp::plugins::tool::found cmake_exe(const options& opt) { namespace detail { +// Stage telemetry is interpreted here, at the adapter that owns CMake. +// The helper invocation isolates optional I/O errors from the actual build. +inline std::string progress_prelude() { + return R"cmake( +if(DEFINED MCPP_PROGRESS_PHASE) + set(_file "$ENV{MCPP_ACTION_PROGRESS_FILE}") + if(NOT _file STREQUAL "") + file(WRITE "${_file}.tmp" "{\"v\":1,\"seq\":${MCPP_PROGRESS_SEQ},\"phase\":\"${MCPP_PROGRESS_PHASE}\"}") + file(RENAME "${_file}.tmp" "${_file}" RESULT _ignored) + endif() + return() +endif() +function(mcpp_progress phase) + if("$ENV{MCPP_ACTION_PROGRESS_FILE}" STREQUAL "") + return() + endif() + string(TIMESTAMP _now "%s" UTC) + get_property(_last GLOBAL PROPERTY _mcpp_progress_time) + # At most one snapshot per second: short stages may finish between reports. + if(_last STREQUAL _now) + return() + endif() + get_property(_seq GLOBAL PROPERTY _mcpp_progress_seq) + if(NOT _seq) + set(_seq 0) + endif() + math(EXPR _seq "${_seq}+1") + set_property(GLOBAL PROPERTY _mcpp_progress_seq "${_seq}") + set_property(GLOBAL PROPERTY _mcpp_progress_time "${_now}") + execute_process(COMMAND "${CMAKE_COMMAND}" "-DMCPP_PROGRESS_PHASE=${phase}" + "-DMCPP_PROGRESS_SEQ=${_seq}" -P "${CMAKE_CURRENT_LIST_FILE}" + OUTPUT_QUIET ERROR_QUIET RESULT_VARIABLE _ignored) +endfunction() +)cmake"; +} + // The script of a keyed installation: `steps` (configure, build, install) runs // only when the cache holds no installation with the key. The key is // computed when the action runs, from `recipe` (the plugin's version, the @@ -376,11 +577,20 @@ inline prefix use(const options& opt) { const bool chosen = chooses_compiler(opt.cache_args); const bool generatorChosen = chooses_generator(opt.cache_args); const bool visualStudio = opt.generator == options::generator_kind::visual_studio; - auto tools = visualStudio ? ts::resolve(opt.toolset) : ts::resolve_named(opt.toolset); - if (!tools) { - std::cerr << std::format("{}: {}\n", who, tools.error()); + // A BUILD FOR ANOTHER TARGET IS COMPILED AS THE ENGINE COMPILES IT (0.21.0): + // the compilers, target words and archiver of the engine's contract, which + // the action reads when it runs. The toolset this program resolves is the + // machine's, and a cross build linked with it would not be the target's. + const std::string targetTriple = mcpp::target(), hostTriple = mcpp::host(); + const bool contract = !chosen && !targetTriple.empty() && !hostTriple.empty() + && targetTriple != hostTriple; + auto resolvedTools = visualStudio ? ts::resolve(opt.toolset) : ts::resolve_named(opt.toolset); + if (!resolvedTools && !contract) { + std::cerr << std::format("{}: {}\n", who, resolvedTools.error()); return {}; } + const ts::resolved_tools noTools{}; + const ts::resolved_tools* tools = resolvedTools ? &*resolvedTools : &noTools; const std::string crt = opt.crt_linkage.empty() ? tools->crt : opt.crt_linkage; if (!crt.empty() && crt != "static" && crt != "dynamic") { std::cerr << std::format("{}: options::crt_linkage is '{}'; it is \"static\" or \"dynamic\".\n", @@ -388,17 +598,25 @@ inline prefix use(const options& opt) { return {}; } const std::string ninja = mcpp::ninja_program(); - const bool named = !chosen && tools->how == ts::mechanism::chain && !tools->cxx.empty(); - const bool useNinja = named && !generatorChosen && !ninja.empty(); - const bool instance = !chosen && !generatorChosen && tools->how == ts::mechanism::instance; + const bool named = !contract && !chosen && tools->how == ts::mechanism::chain && !tools->cxx.empty(); + const bool useNinja = (named || contract) && !generatorChosen && !ninja.empty(); + const bool instance = !contract && !chosen && !generatorChosen && tools->how == ts::mechanism::instance; + const std::string systemName = opt.system_name.empty() + ? cmake_system_name(mcpp::target_os()) : opt.system_name; // One build directory per toolset statement: CMake refuses a cache made // with another generator or instance. What 0.16.0 configured stays in // `build/`. std::string key; - if (useNinja) key = "ninja;" + tools->identity + ";" + tools->cxx; + // A CMake cache configured without a toolchain does not begin loading a + // newly supplied toolchain on reconfigure. Give this representation its + // own build directory instead of retaining legacy flattened CMAKE_*_FLAGS. + if (contract) key = "contract-rsp1;" + targetTriple + ";system;" + systemName; + else if (useNinja) key = "ninja;" + tools->identity + ";" + tools->cxx; else if (named) key = "named;" + tools->identity + ";" + tools->cxx; else if (instance) key = "instance;" + tools->instance_dir + ";" + tools->toolset_version; + if (const auto generator = generator_key(opt.cache_args); !generator.empty()) + key += ";generator;" + generator; const std::string name = opt.name.empty() ? source.filename().string() : opt.name; const fs::path base = fs::path(mcpp::out_dir()) / "deps-cmake" / name; const fs::path build = base / (key.empty() ? std::string("build") : "build-" + mcpp::deps::short_name(key)); @@ -409,8 +627,9 @@ inline prefix use(const options& opt) { p.include = mcpp::deps::generic(root / opt.dirs.include); p.lib = mcpp::deps::generic(root / opt.dirs.lib); p.bin = mcpp::deps::generic(root / opt.dirs.bin); - p.mechanism = std::string(ts::name(named ? ts::mechanism::chain - : instance ? ts::mechanism::instance : ts::mechanism::detected)); + p.mechanism = contract ? std::string("contract") + : std::string(ts::name(named ? ts::mechanism::chain + : instance ? ts::mechanism::instance : ts::mechanism::detected)); const auto tool = cmake_exe(opt); const std::string cmake = tool.program; @@ -430,6 +649,7 @@ inline prefix use(const options& opt) { const std::string id = "deps-cmake:" + name; const std::string desc = "CMAKE " + name; const fs::path script = base / (name + ".cmake"); + const fs::path contractScript = base / (name + "-contract.cmake"); using mcpp::deps::bracket; // Each configure argument, and the same argument as the cache key // states it: with no path of this machine. The subproject's three @@ -486,21 +706,40 @@ inline prefix use(const options& opt) { + (crt == "dynamic" ? "DLL" : "")); } for (auto const& x : opt.cache_args) same(x); + if (contract) { + same("-DCMAKE_SYSTEM_NAME=" + systemName); + // Compiler probes need no executable startup files or link + // runtime; subproject targets retain their actual target kind. + same("-DCMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY"); + arg("-DCMAKE_TOOLCHAIN_FILE=" + mcpp::deps::generic(contractScript), + "-DCMAKE_TOOLCHAIN_FILE="); + mcpp::plugins::fs::write_if_changed(contractScript, contract_reader()); + keyed.push_back("contract " + targetTriple); + } - std::string steps = "execute_process(COMMAND ${CMAKE_COMMAND}"; + std::string steps; + if (opt.progress) steps += "mcpp_progress(configure)\n"; + steps += "execute_process(COMMAND ${CMAKE_COMMAND}"; for (auto const& x : configure) steps += "\n " + bracket(x); + if (contract) steps += "\n \"-DMCPP_CONTRACT:FILEPATH=${MCPP_CONTRACT}\""; steps += "\n RESULT_VARIABLE rc)\n" "if(NOT rc EQUAL 0)\n message(FATAL_ERROR \"configure exited ${rc}\")\nendif()\n" + + (opt.progress ? std::string("mcpp_progress(build-install)\n") : std::string{}) + "execute_process(COMMAND ${CMAKE_COMMAND} --build " + bracket(buildS) + " --config " + bracket(opt.config) + " --target install --parallel\n RESULT_VARIABLE rc)\n" "if(NOT rc EQUAL 0)\n message(FATAL_ERROR \"build and install exited ${rc}\")\nendif()\n"; std::string text = "# Written by mcpp.deps.cmake: configure, build and install " + name + ".\n" - "# toolset: " + ts::describe(*tools) + "\n"; + "# toolset: " + (contract ? std::string("the engine's contract for ") + targetTriple + : ts::describe(*tools)) + "\n"; + if (opt.progress) text = detail::progress_prelude() + text; const auto files = mcpp::deps::files_under(source); // A key states the compilers through the toolset's identity. On the // MSVC ABI that is the MSVC toolset, which a row compiler (clang-cl, // `compiler::row`) is not, so such an installation is not kept. - const bool identified = instance || (named && (!tools->msvc_abi || opt.toolset.cc == ts::compiler::abi_native)); + // A contract build is not kept either: its compilers are named by the + // contract when the action runs, after this key is written. + const bool identified = !contract + && (instance || (named && (!tools->msvc_abi || opt.toolset.cc == ts::compiler::abi_native))); if (!identified || opt.cache == "off") { text += steps; } else { @@ -517,10 +756,13 @@ inline prefix use(const options& opt) { mcpp::plugins::fs::write_if_changed(script, text); const std::string scriptS = mcpp::deps::generic(script); mcpp::action a; + mcpp::plugins::progress::enable(a, opt.progress); a.id = id.c_str(); a.role = mcpp::roles::prepare; a.description = desc.c_str(); - a.arg(cmake.c_str()).arg("-P").arg(scriptS.c_str()); + a.arg(cmake.c_str()); + if (contract) a.arg("-DMCPP_CONTRACT=${mcpp.contract}"); + a.arg("-P").arg(scriptS.c_str()); // Named tools run with the environment the engine runs them with, and // their directories first on PATH. if (named) { @@ -542,6 +784,7 @@ inline prefix use(const options& opt) { } a.input(cmake.c_str()); a.input(scriptS.c_str()); + if (contract) a.input(mcpp::deps::generic(contractScript).c_str()); for (auto const& f : files) a.input(f.c_str()); mcpp::deps::watch_tree(source); a.output(stamp.c_str()); diff --git a/dist/apk.cppm b/dist/apk.cppm index 3650aa2..4188e1c 100644 --- a/dist/apk.cppm +++ b/dist/apk.cppm @@ -96,12 +96,6 @@ // caller's configuration, not this member's (rule 4, 2026-09-12 design // record: "a name is cache-safe, a path is not"). -module; -#include -#if !defined(_WIN32) -#include -#endif - export module mcpp.dist.apk; import std; @@ -790,47 +784,15 @@ inline std::string sha256_file(const std::string& path) { // ─── Running a tool at plan time ───────────────────────────────────────── -// POSIX single quotes: the only character that needs care inside them is the -// quote itself. This member runs on the hosts that build Android rows, which -// are Linux and macOS (xim:android-ndk publishes no Windows table). -inline std::string shell_quote(const std::string& s) { - std::string out = "'"; - for (std::size_t i = 0; i < s.size(); ++i) { - if (s[i] == '\'') out += "'\\''"; - else out += s[i]; - } - return out + "'"; -} - -// Runs `argv` through `/bin/sh`, returning its exit status and its combined -// output -- the same `popen` the rules members use to probe a tool at plan -// time. Only the explicit Maven modes and AAR extraction call it; an ordinary -// build of an unchanged project runs nothing here. -inline int run_capture(const std::vector& argv, std::string& output) { - std::string cmd; - for (std::size_t i = 0; i < argv.size(); ++i) { - if (i) cmd += ' '; - cmd += shell_quote(argv[i]); - } - cmd += " 2>&1"; - output.clear(); -#if defined(_WIN32) - FILE* p = ::_popen(cmd.c_str(), "r"); -#else - FILE* p = ::popen(cmd.c_str(), "r"); -#endif - if (!p) return -1; - char buf[4096]; - std::size_t n; - while ((n = std::fread(buf, 1, sizeof buf, p)) > 0) output.append(buf, n); -#if defined(_WIN32) - // Compiled on every host (`tests/all-rules-compile`); an Android row is - // never packed on Windows, so this branch only has to be well-formed. - return ::_pclose(p); -#else - const int status = ::pclose(p); - return WIFEXITED(status) ? WEXITSTATUS(status) : -1; -#endif +// Runs `argv`, returning its exit status and its combined output, through +// `mcpp::plugins::tool::capture`: no shell between, so no argument is quoted. +// Only the explicit Maven modes and AAR extraction call it; an ordinary build +// of an unchanged project runs nothing here. +int run_capture(const std::vector& argv, std::string& output, + const std::filesystem::path& cwd = {}) { + auto r = mcpp::plugins::tool::capture(argv, /*with_errors=*/true, cwd); + output = std::move(r.out); + return r.status; } // ─── Merging library manifests into the application's ──────────────────── @@ -1196,8 +1158,7 @@ inline bool unpack_aar(const std::string& aar, const fs::path& dest, const std:: fs::remove_all(dest, ec); fs::create_directories(dest, ec); std::string output; - const int rc = run_capture({"/bin/sh", "-c", "cd \"$1\" && \"$2\" xf \"$3\"", "unpack-aar", - dest.string(), jarTool, aar}, output); + const int rc = run_capture({jarTool, "xf", aar}, output, dest); if (rc != 0) { message = std::format("mcpp.dist.apk: unpacking {} with {} failed ({}): {}", aar, jarTool, rc, output); return false; diff --git a/dist/appimage.cppm b/dist/appimage.cppm index 782ce8c..33da40d 100644 --- a/dist/appimage.cppm +++ b/dist/appimage.cppm @@ -36,9 +36,6 @@ // the edge depends only on the program binary, and a metadata-only change would // leave the previous AppImage in place, reported as up to date. -module; -#include - export module mcpp.dist.appimage; import std; diff --git a/dist/apple.cppm b/dist/apple.cppm index d2d468b..d745ed6 100644 --- a/dist/apple.cppm +++ b/dist/apple.cppm @@ -187,9 +187,6 @@ // reaches the device through the runner named `app` this member supplies, // `devicectl-run` (`xim:apple-device-tools`), as `macapp-run` serves macOS. -module; -#include - export module mcpp.dist.apple; import std; diff --git a/dist/web.cppm b/dist/web.cppm index 0c33e6f..fe827a7 100644 --- a/dist/web.cppm +++ b/dist/web.cppm @@ -85,9 +85,6 @@ // the side file as the copy action's input is what makes an edit reach // `/web/index.html`. -module; -#include - export module mcpp.dist.web; import std; diff --git a/dist/wix.cppm b/dist/wix.cppm index bc9f5dc..c1960aa 100644 --- a/dist/wix.cppm +++ b/dist/wix.cppm @@ -93,9 +93,6 @@ // shims into an executable named like an installer). A project with its own // bootstrapper application supplies its own bundle definition. -module; -#include - export module mcpp.dist.wix; import std; diff --git a/docs/deps.md b/docs/deps.md index d9fcd74..f081949 100644 --- a/docs/deps.md +++ b/docs/deps.md @@ -188,6 +188,9 @@ Not supported: vcpkg's classic mode; the debug libraries under `debug/lib`. ## `deps-cmake`: a CMake subproject +Set `options::progress = true` to report the configure and build-install stages when running with mcpp 2026.11.5.1 or later. The default is false. Older engine APIs continue without telemetry. Counts are omitted because this adapter does not know the number of compiler tasks; a restored cached installation need not run either stage. Optional snapshot I/O errors do not fail configure/build/install. The CMake snapshot helper uses `file(RENAME ... RESULT ...)` (CMake 3.21+); older tools can still build while telemetry remains unavailable. + + ```cpp import mcpp.deps.cmake; @@ -254,7 +257,38 @@ resolved toolset. A compiler launcher (`ccache`, `sccache`) is CMake's Each toolset statement configures its own build directory, because CMake refuses a cache made with another generator or instance; the directory 0.16.0 -configured stays `build/`. +configured stays `build/`. Explicit generator, architecture, toolset and instance +selections in `cache_args` use separate directories. Returning to a selection +reuses its directory. Ordinary `-D` settings reconfigure the same directory. + +**A build for another target (0.21.0, mcpp 2026.10.30.1+, CMake 3.19+).** When the build's +target is not the machine (`mcpp build --target x86_64-linux-musl`, an ok image +`--target x86_64-openkal`), the toolset above is the machine's and is not used. +The subproject is compiled as the engine compiles the target: the action reads +the engine's contract for the configuration (`${mcpp.contract}`, see mcpp's +docs/30) when it runs, and configures a toolchain with its compilers, +target, sysroot, target-side headers, platform and profile flags, link words +and archiver. Compiler response files retain argument boundaries and literal +quotes, backslashes and dollar signs. The same toolchain reaches CMake's +compile probes. An empty GNU argument is rejected; an option containing an +empty string value, such as `-DNAME=""`, is retained. Ninja is the default; +a generator in `cache_args` takes precedence. `CMAKE_SYSTEM_NAME` is the target's system (Linux, +Android, Darwin, iOS, Windows, Emscripten, `Generic` for bare metal, and Linux +for an ok image, whose C library is POSIX at the source level), or +`options::system_name`; `CMAKE_TRY_COMPILE_TARGET_TYPE` is `STATIC_LIBRARY`, +so compile probes do not require executable startup files or a link runtime. +Changing `options::system_name` selects a separate CMake build directory. +The libraries the subproject installs are linked into the consuming program +by the engine. A tool the subproject runs at build time for the machine +(LLVM's `llvm-tblgen`) is built in the host graph, as a build dependency's tool, +and named to the subproject with a `cache_args` entry. An installation of a +cross build is not kept in the cache below: its compilers are named by the +contract when the action runs, after the key is written. Compilers or a +toolchain file in `cache_args` decide here too. + +Cross subprojects that link executables or shared libraries require mcpp +2026.11.7.1 or later. Contract profile options follow CMake's configuration +flags; `options::config` still selects the CMake configuration. ### An installation is built once (0.18.0) diff --git a/docs/plugin-development.md b/docs/plugin-development.md index 02a964b..6345594 100644 --- a/docs/plugin-development.md +++ b/docs/plugin-development.md @@ -52,7 +52,7 @@ mcpp-index refuses such a package. ```toml [build-dependencies.mcpp] -plugins = { version = "0.20.0", features = ["plugins-core"], host-module = true, reexport = true } +plugins = { version = "0.21.0", features = ["plugins-core"], host-module = true, reexport = true } ``` `reexport = true` is needed when the plugin's consumers import L2 modules in @@ -202,6 +202,8 @@ Known differences, each one met in this package: - **A type that holds a container of itself does not use `std::pair` or `std::tuple` for the element.** Under clang with libstdc++'s `std` module, `std::vector>` inside `value` instantiates the pair while `value` is incomplete and fails with "incomplete type used in type trait expression"; GCC and clang with libc++ accept it. The element is a plain struct, and a member function that walks the container is defined after both types are complete (`mcpp::plugins::json::member`, 0.19.1). - **No compiler's leniency is relied on.** An extension, a missing `typename`, or a template accepted only because one compiler instantiates it later is a failure in another lane. - **A build program does not use `std::println`.** On macOS 14 the build program links the system libc++, which lacks a symbol it needs; `std::format` into a string and `std::cout` work everywhere. +- **No unit includes a header or branches on the platform in the preprocessor.** The C library comes from `std`, which exports the functions and types of ISO C (`std::isalnum`, `std::getenv`, `std::FILE`) but no macro; a macro of ISO C (`stdout`, `errno`, `SEEK_SET`) comes from `mcpplibs.mwrap.libc` under its own name (`mcpplibs::mwrap::libc::macros::stdout()`). The operating system comes from `mcpplibs.os`: a program a member asks for a fact runs through `mcpp::plugins::tool::capture` (its arguments as a vector, no shell, so nothing is quoted and no null device is named), a library opens through `mcpplibs::os::library::open`, and a child's environment is stated in its spawn options rather than written into this process's. A build program runs on the machine that builds, so a platform difference is a compile-time question of that machine: `if constexpr (mcpplibs::os::is_windows)`. CI refuses an `#include`, `#if` or `#ifdef` in a unit (0.21.0). +- **A function that reaches mcpplibs.os is not `inline`.** A rule's helper that runs a program (`mcpp::plugins::tool::capture`) or opens a library is defined without `inline`, so it is compiled once in its module and an importer sees its declaration only. Under GCC 16 an importer that reads such a body from an inline function fails: "failed to read compiled module cluster: Bad file data", or an unresolved `std::optional>` at the link (`tests/metal-consumer`, 0.21.0). - **A module does not instantiate `std::filesystem::path`'s iterator.** Under MSVC 14.52 every importer that then touches `std::filesystem` fails inside the STL; `mcpp::plugins::names::components` splits a path as a string instead. ## 5. A new plugin @@ -215,3 +217,11 @@ A plugin in this package consists of: - a version bump. A project's own plugin is a module in the project (a path dependency with `host-module = true`) and follows §2 in the same way. + +## Optional action progress + +`mcpp.plugins.progress` is part of `plugins-core`. `enable(action, requested)` sets the optional action flag when the engine API provides it. With older APIs it reports that telemetry is unavailable and leaves the action usable. Engine support starts at mcpp 2026.11.5.1 (protocol 18); this optional path does not raise the package's 2026.10.30.1 floor. + +Worker programs select `plugins-progress` and import `mcpp.plugins.progress.writer`. `writer` reads `MCPP_ACTION_PROGRESS_FILE`; an absent channel is a no-op. `report({.phase="compile", .done=3, .total=9})` atomically replaces the latest schema-1 snapshot, assigns increasing sequence numbers and publishes at most five times per second. Omit counts when the total is unknown. `report()` returns whether it published; this result does not determine whether the worker's actual task succeeded. The optional writer feature uses `mcpplibs.os.fs` for portable atomic replacement; `plugins-core` depends on `mcpplibs.os` as well (§4). + +The writer feature is usable by an ordinary standalone worker with an ordinary dependency; it imports no build-program-only `mcpp` module. Importing `mcpp.plugins.progress.writer` also exposes the `snapshot` type. Channel paths preserve UTF-8 on every host, including Windows with a non-UTF-8 ANSI code page. `tests/progress-worker` checks disabled channels, Unicode paths and optional write failure on Linux, macOS and Windows. diff --git a/docs/rules.md b/docs/rules.md index ef10937..0471dae 100644 --- a/docs/rules.md +++ b/docs/rules.md @@ -14,11 +14,13 @@ Module `mcpp.rules.cuda`; engine floor: 2026.10.1.3 (0.19.0, mcpp#755; 2026.9.6. **Needs and behaviour.** `[build] accel = "cuda…"`, a constrained glob for `*.cu`; the clang route with an LLVM toolchain, the nvcc route with a GCC one +The rule supplies toolkit search directories. The consumer selects the CUDA runtime link policy; for the Linux static-runtime arrangement used by the fixtures, the CUDA configuration states `ldflags = ["-lcudart_static", "-lrt", "-lpthread", "-ldl"]`. + ## `rules-hip` Module `mcpp.rules.hip`; engine floor: 2026.10.1.3 (0.19.0, mcpp#755; 2026.9.6.6 before). -**Needs and behaviour.** `[build] accel = "hip, cuda12.9+{sm_89}"`, a constrained glob for `*.hip`. On the NVIDIA platform HIP is a header layer over the CUDA runtime, so the compiler is the project's own clang and there is no ROCm on the machine +**Needs and behaviour.** `[build] accel = "hip+{sm_89,sm_90}"`, a constrained glob for `*.hip`. On the NVIDIA platform HIP is a header layer over the CUDA runtime, so the compiler is the project's own clang and there is no ROCm on the machine ## `rules-metal` @@ -50,4 +52,16 @@ Module `mcpp.rules.sycl`; engine floor: 2026.10.1.3 (0.19.0, mcpp#755; 2026.9.6. **One C++ runtime, libstdc++ (0.20.0).** `libsycl.so` is compiled against libstdc++ and throws `sycl::exception` on it, and a typed `catch` does not match across two C++ runtimes in one process (mcpp#838). The program's own C++ library is therefore libstdc++: a gcc toolchain, or llvm with `gcc.libstdcxx` in the graph (`[target.'cfg(accelerator = "sycl")'.dependencies] gcc.libstdcxx = "16.1.0.1"`). On Linux the rule refuses any other C++ library before a unit is compiled, naming both remedies, and `compat:sycl-runtime` states the same requirement. The rule names no C++ library on the link line (0.19 added `-l:libstdc++.so.6` beside an llvm build's libc++); `--gcc-install-dir` follows the build's GCC: the graph's supplier (`MCPP_CXX_ABI_ROOT`), else the gcc toolchain's payload, else the `xim:gcc` the rule declares. -**Needs and behaviour.** `[build] accel = "sycl"` or `"sycl, cuda12.9+{sm_89}"`, a constrained glob for `*.sycl`, and `compat:sycl-runtime` so the artifact can reach `libsycl.so.9` at run time. Its own engine need is `.sycl` in the device-source table, 2026.9.6.1 +**Needs and behaviour.** `[build] accel = "sycl"` or `"sycl+{nvptx64-sm_89}"`, a constrained glob for `*.sycl`, and `compat:sycl-runtime` so the artifact can reach `libsycl.so.9` at run time. Its own engine need is `.sycl` in the device-source table, 2026.9.6.1 + + +**SYCL targets belong to SYCL.** `sycl+{nvptx64-sm_89,nvptx64-sm_90}` produces both NVIDIA images. `sycl+{spir64,nvptx64-sm_89}` carries portable SPIR-V beside an NVIDIA image. `sycl` and `sycl+{spir64}` request portable SPIR-V; `sm_89` remains an alias for `nvptx64-sm_89`. An unknown target is refused before compilation. AMD `gfx*` requests are recognized but refused because this rule does not yet supply a ROCm compiler/runtime closure. + +Each selected implementation owns its targets. A repository may retain CUDA and SYCL source groups while selecting only `sycl+{nvptx64-sm_89}`; the CUDA group is excluded by its source constraint and the CUDA rule need not be imported. Selecting `cuda+{sm_90},sycl+{nvptx64-sm_89}` selects both implementations with different targets. The rule passes the same target vector to compilation and device linking. Its CUDA compiler payload is requested only for a SYCL NVIDIA target; `options::cuda` accepts the same explicit choice as other rule tools. + +The older `sycl,cuda+{sm_89}` form still inherits CUDA targets when the SYCL member has no targets, with a migration warning. Both named implementations remain selected. Explicit SYCL targets are never replaced by a CUDA member. Move the device target to the SYCL member to select only SYCL. GPU execution still requires a compatible device and driver; CI verifies the emitted images and the CPU fallback separately. + +**Several NVIDIA architectures in a library.** Each package's device link reads that package's own objects (`-fsycl-link` over its `.sycl.o`), so a library's images are registered inside the library and a consumer's link never asks the SDK to unbundle an archive; a library of `sycl+{nvptx64-sm_89,nvptx64-sm_90}` links into a program with each image its own architecture's (`tests/sycl-library-consumer`). Not supported: handing the SDK an archive of several NVIDIA architectures to device-link (a prebuilt SYCL `.a`, or a kernel that calls a `SYCL_EXTERNAL` function of another package). DPC++ 7.1.0's `clang-offload-bundler` matches the five-part bundle IDs its driver writes (`sycl-nvptx64-nvidia-cuda-sm_89`) to every NVPTX request, so unbundling such an archive for one architecture yields the members of all of them and the device link reports duplicate definitions (mcpp#899, an SDK defect). There, build that archive for one architecture, or keep the device code of each architecture in the package that launches it. The rule keeps every requested architecture; it neither drops one nor reuses one architecture's code for another. + + +The `tests/multi-accel-consumer` fixture selects `cuda+{sm_90},sycl+{nvptx64-sm_89}` and imports both rules. CI inspects each rule's actual image target and runs both host objects without submitting GPU work. This establishes build/link coexistence with one C++ runtime; device execution, interoperation and synchronization require their own runtime checks. diff --git a/docs/tools-embed.md b/docs/tools-embed.md index cb08b3b..9f48ef5 100644 --- a/docs/tools-embed.md +++ b/docs/tools-embed.md @@ -75,3 +75,6 @@ there is always exactly one name to give: `table_options::identifier` (default the array and the struct. `out_dir`, `name_space`, `elem` and `width` mean what they mean in `options`, `element::word32`'s "not a multiple of 4" refusal included -- checked per input, since a table has several. + + +For object surfaces, the execution-time body generator records a payload-content checksum in its `.S` output. A payload edit, including an equal-size edit, changes that source and rebuilds its embedding object. Identical payload bytes preserve the generated assembly timestamp; the separately generated interface remains stable. Planning reads no payload bytes. diff --git a/mcpp.toml b/mcpp.toml index c2beb0f..f6e17f9 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,7 +1,7 @@ [package] name = "plugins" namespace = "mcpp" -version = "0.20.0" +version = "0.21.0" # The first mcpp release that speaks protocol 17 (SPEC-013, mcpp#847): the # platform's directives `mcpp.plugins.platform` states (`link_driver`, # `link_input`, `artifact_name`, `seal`, `run_host`), the `platform` phase, and @@ -9,7 +9,7 @@ version = "0.20.0" # reads. It carries protocol 15 (mcpp#755): `xpkg_source`, `xpkg_program`, # `xpkg_request`, `mcpp::decision`, `mcpp::toolchain`, `mcpp::phase`, and the # `provision = "on-request"` entries below. -mcpp = ">=2026.10.27.1" +mcpp = ">=2026.10.30.1" description = "Official mcpp build plugins: rule packages under mcpp.rules.*, build-time utilities under mcpp.tools.*, each member selected by a feature" license = "Apache-2.0" authors = ["mcpp-community"] @@ -92,9 +92,16 @@ default = [] # build system (`instance`, `chain`, `detected`) # mcpp.plugins.fs deterministic file generation and placement [features.plugins-core] -sources = ["src/declare.cppm", "src/toolset.cppm", "src/fs.cppm", "src/tool.cppm", +sources = ["src/declare.cppm", "src/toolset.cppm", "src/fs.cppm", "src/tool.cppm", "src/accel.cppm", "src/progress.cppm", "src/compat/detected_toolset.cppm"] +# The operating system of every member: the programs a member asks for a fact +# (`mcpp.plugins.tool::capture`), the libraries it opens and the platform it +# builds on, through mcpplibs.os. No unit includes a system header (0.21.0, +# docs/plugin-development.md §4). +[feature-deps.plugins-core] +mcpplibs.os = "0.4.0" + # `mcpp.plugins.toolchain` -- the builders a root build program states its # build toolchain with, in the toolchain phase of a project whose `[toolchain]` # says `configure = "build.mcpp"` (mcpp#755). Its own feature rather than part @@ -308,6 +315,15 @@ implies = ["plugins-core"] sources = ["deps/vcpkg.cppm"] implies = ["deps"] +[features.plugins-progress] +# Standalone workers need no build-program-only mcpp module. +sources = ["src/progress.cppm", "src/progress_writer.cppm"] + +# Since 0.3.1 a name in Windows' long spelling is the name it spells (the progress channel's path may +# be one). One release, the one mcpp's own modules name, so a build is the same at any later time. +[feature-deps.plugins-progress] +mcpplibs.os = "0.4.0" + [features.deps-cmake] sources = ["deps/cmake.cppm"] implies = ["deps"] @@ -408,19 +424,10 @@ implies = ["plugins-core"] "xim:glibc" = "" "xim:linux-headers" = "" -# Only when the SYCL build actually names an NVIDIA target. `accelerator` is a -# SET, so this is a conjunction of two membership tests rather than a -# contradiction -- and without it every SYCL build would download a CUDA -# toolkit it may have no target for. -# -# `linux` is part of the predicate for a reason that is measured rather than -# stylistic: upstream does not build the CUDA plugin for Windows, and the -# published Windows asset carries Level Zero and OpenCL adapters only. The rule -# refuses that combination on that host, and a refusal that arrives after a -# multi-gigabyte download is a worse refusal -- provisioning runs before the -# rule does. -[target.'cfg(all(accelerator = "sycl", accelerator = "cuda", linux))'.feature-xlings.rules-sycl] -"xim:cuda-nvcc" = "12.9.86" +# The toolkit is visible to SYCL without selecting CUDA source compilation. +# Only a SYCL NVIDIA target requests it; other SYCL targets install nothing. +# Windows has no published CUDA adapter, and the rule refuses it there. +"xim:cuda-nvcc" = { version = "12.9.86", provision = "on-request" } # ONE COMPILER PER PLATFORM, AND NOT THE SAME ONE EVERYWHERE. # diff --git a/rules/ascendc.cppm b/rules/ascendc.cppm index 304dd44..f0e6996 100644 --- a/rules/ascendc.cppm +++ b/rules/ascendc.cppm @@ -50,10 +50,6 @@ // own -- and it is stated here so the boundary is not mistaken for an // oversight. -module; - -#include - export module mcpp.rules.ascendc; import std; diff --git a/rules/cuda.cppm b/rules/cuda.cppm index 2c422dc..3521979 100644 --- a/rules/cuda.cppm +++ b/rules/cuda.cppm @@ -33,18 +33,13 @@ // - whether the embedded PTX can be JIT-compiled by this driver, as an // advisory, because the SASS for the named architectures still runs. -module; -#include -#include -#if !defined(_WIN32) -#include -#endif - export module mcpp.rules.cuda; import std; import mcpp; +import mcpp.plugins.accel; import mcpp.plugins.tool; +import mcpplibs.os; // WHY NOTHING HERE USES `std::println`, AND WHY THAT IS NOT A STYLE CHOICE. @@ -76,13 +71,8 @@ export namespace mcpp::rules::cuda { // from those two facts, so both are stated once here rather than at each site // -- a `std::filesystem::exists` against the wrong spelling does not fail, it // answers false, and this rule reads several such answers as decisions. -#if defined(_WIN32) -inline constexpr bool kWindows = true; -inline constexpr const char* kExe = ".exe"; -#else -inline constexpr bool kWindows = false; -inline constexpr const char* kExe = ""; -#endif +inline constexpr bool kWindows = mcpplibs::os::is_windows; +inline constexpr const char* kExe = kWindows ? ".exe" : ""; enum class route { automatic, clang, nvcc }; @@ -127,6 +117,7 @@ struct target { std::vector archs; // {"sm_89"} std::string ptx; // "89" when a portable form is embedded bool present = false; + std::string error; }; // Split on one character. Written out rather than taken from : GCC 16 @@ -150,32 +141,16 @@ inline std::string_view trim(std::string_view s) { return s; } -inline target parse_target(std::string_view accel) { +inline target parse_target(std::string_view text) { target t; - for (std::size_t i = 0; i <= accel.size();) { - auto comma = accel.find(',', i); - auto open = accel.find('{', i), close = accel.find('}', i); - if (open != std::string_view::npos && close != std::string_view::npos - && comma != std::string_view::npos && comma > open && comma < close) - comma = accel.find(',', close); - auto chunk = trim(comma == std::string_view::npos ? accel.substr(i) - : accel.substr(i, comma - i)); - i = comma == std::string_view::npos ? accel.size() + 1 : comma + 1; - if (!chunk.starts_with("cuda")) continue; + const auto parsed = mcpp::plugins::accel::parse(text); + if (!parsed) { t.error = parsed.error(); return t; } + for (const auto& entry : *parsed) { + if (entry.backend != "cuda") continue; t.present = true; - auto plus = chunk.find('+'); - t.version = std::string(trim(chunk.substr(4, plus == std::string_view::npos - ? chunk.size() - 4 : plus - 4))); - if (plus != std::string_view::npos) { - auto o = chunk.find('{', plus), c = chunk.find('}', plus); - if (o != std::string_view::npos && c != std::string_view::npos) - for (auto part : split(chunk.substr(o + 1, c - o - 1), ',')) - if (auto a = trim(part); !a.empty()) t.archs.emplace_back(a); - auto tail = chunk.substr(c == std::string_view::npos ? chunk.size() : c + 1); - for (auto key : {"ptx>=", "floor>="}) - if (auto p = tail.find(key); p != std::string_view::npos) - t.ptx = std::string(trim(tail.substr(p + std::string_view(key).size()))); - } + t.version = entry.version; + t.archs = entry.archs; + t.ptx = entry.portable_floor; } return t; } @@ -343,30 +318,22 @@ inline std::optional find_toolkit(const options& opt, bool& pending) { // The driver's version through the driver's own library, reached through the // sentinel package rather than /usr/lib. "" when there is no driver here, // which is a fact about the machine and not a failure of the build. -inline std::string driver_version(const toolkit& t) { -#if defined(_WIN32) - return {}; -#else +std::string driver_version(const toolkit& t) { + if constexpr (kWindows) return {}; if (t.driver_dir.empty()) return {}; const auto lib = t.driver_dir + "/lib/libcuda.so.1"; if (!std::filesystem::exists(lib)) return {}; // What would change the answer is the library the answer was read from. mcpp::rerun_if_changed(lib.c_str()); - void* h = ::dlopen(lib.c_str(), RTLD_LAZY | RTLD_LOCAL); + auto h = mcpplibs::os::library::open(lib); if (!h) { - const char* why = ::dlerror(); - mcpp::warning(std::format("could not open the driver library {}: {}", lib, - why ? why : "(no reason given)").c_str()); + mcpp::warning(std::format("could not open the driver library {}: {}", lib, h.error().message()).c_str()); return {}; } - using fn = int (*)(int*); - auto get = reinterpret_cast(::dlsym(h, "cuDriverGetVersion")); + auto get = h->function("cuDriverGetVersion"); int v = 0; - std::string out; - if (get && get(&v) == 0 && v > 0) out = std::format("{}.{}", v / 1000, (v % 1000) / 10); - ::dlclose(h); - return out; -#endif + if (get && get(&v) == 0 && v > 0) return std::format("{}.{}", v / 1000, (v % 1000) / 10); + return {}; } inline int major_of(std::string_view v) { @@ -476,40 +443,22 @@ inline bool libc_declares_c23_pi_math(std::string_view sysroot) { // ` -dumpversion` → major. The host compiler is the toolchain mcpp // resolved for this build, so its version is a fact of the build, not a guess. -inline int compiler_major(const std::string& cc) { -#if defined(_WIN32) - (void)cc; return 0; -#else - std::string cmd = cc + " -dumpversion 2>/dev/null"; - if (FILE* p = ::popen(cmd.c_str(), "r")) { - char buf[64] = {}; - std::string s; - if (std::fgets(buf, sizeof buf, p)) s = buf; - ::pclose(p); - return major_of(s); - } - return 0; -#endif +int compiler_major(const std::string& cc) { + if constexpr (kWindows) return 0; + return major_of(mcpp::plugins::tool::first_line({cc, "-dumpversion"})); } // The first back-end stage nvcc names but cannot resolve, from its own plan. // nvcc invokes cicc, cudafe++, ptxas and fatbinary by bare name on a PATH it // states in the plan; a stage that does not resolve there fails the compile // with `sh: 1: cicc: not found`, naming nothing that helps. -inline std::optional unreachable_stage(const toolkit& t, const std::string& ccbin) { -#if defined(_WIN32) - (void)t; (void)ccbin; return std::nullopt; -#else +std::optional unreachable_stage(const toolkit& t, const std::string& ccbin) { + if constexpr (kWindows) return std::nullopt; const auto probe = std::filesystem::temp_directory_path() / "mcpp-rules-cuda-dryrun.cu"; { std::ofstream(probe) << "__global__ void k() {}\n"; } - std::string cmd = std::format("{} --dryrun -ccbin {} -c {} -o /dev/null 2>&1", - t.nvcc(), ccbin, probe.string()); - std::string text; - if (FILE* p = ::popen(cmd.c_str(), "r")) { - char buf[4096]; - while (std::fgets(buf, sizeof buf, p)) text += buf; - ::pclose(p); - } + const std::string text = mcpp::plugins::tool::capture( + {t.nvcc(), "--dryrun", "-ccbin", ccbin, "-c", probe.string(), "-o", "/dev/null"}, + /*with_errors=*/true).out; std::filesystem::remove(probe); std::string path; std::vector stages; @@ -533,7 +482,6 @@ inline std::optional unreachable_stage(const toolkit& t, const std: if (!found) return stage; } return std::nullopt; -#endif } // ─── Planning ────────────────────────────────────────────────────────────── @@ -571,11 +519,8 @@ struct edge { // the same way it does for every ordinary C++ translation unit on this host. inline route decide(route asked) { if (asked != route::automatic) return asked; -#if defined(_WIN32) - return route::clang; -#else + if constexpr (kWindows) return route::clang; return std::string_view(mcpp::compiler()) == "clang" ? route::clang : route::nvcc; -#endif } inline std::vector plan(std::span sources, options opt = {}) { @@ -586,6 +531,7 @@ inline std::vector plan(std::span sources, options opt return out; } const auto tg = parse_target(mcpp::accel()); + if (!tg.error.empty()) { std::cerr << "mcpp.rules.cuda: " << tg.error << '\n'; return {}; } if (!tg.present || tg.archs.empty()) { // C19: a device build that names no device is refused HERE, not at // run time as `no kernel image is available for execution`. @@ -841,7 +787,9 @@ inline bool compile(options opt = {}) { // `accel` itself, which is what the rule already does. A rule whose // backend this build does not name has nothing to do, and that is not a // mistake and must not be reported as one. - if (!parse_target(mcpp::accel()).present) return true; + const auto requested = parse_target(mcpp::accel()); + if (!requested.error.empty()) { std::cerr << "mcpp.rules.cuda: " << requested.error << '\n'; return false; } + if (!requested.present) return true; const auto sources = device_sources(); if (sources.empty()) { mcpp::warning("[build] accel names cuda but no constrained glob matched a `.cu`; " diff --git a/rules/hip.cppm b/rules/hip.cppm index 4342a0e..e282ff6 100644 --- a/rules/hip.cppm +++ b/rules/hip.cppm @@ -29,15 +29,13 @@ // directories, through `mcpp::link_search`; and the HIP version it compiled // against, through `mcpp::fact`, so a build log answers "which HIP" without // anyone reproducing the build. -module; -#include -#include - export module mcpp.rules.hip; import std; import mcpp; +import mcpp.plugins.accel; import mcpp.plugins.tool; +import mcpplibs.os; // WHY NOTHING HERE USES `std::println`, AND WHY THAT IS NOT A STYLE CHOICE. @@ -98,6 +96,7 @@ struct options { // many programming models reach it, so `sm_89` does not acquire a second // spelling because the source file says `hip` instead of `cu`. struct target { + std::string error; bool hip = false; // the `hip` chunk is present std::string cuda_version; // "12.9", from a `cuda12.9` chunk std::vector cuda_archs; // {"sm_89"} @@ -135,25 +134,25 @@ inline std::vector parse_arch_set(std::string_view chunk) { return archs; } -inline target parse_target(std::string_view accel) { +inline target parse_target(std::string_view text) { target t; - for (auto raw : split(accel, ',')) { - auto chunk = trim(raw); - if (chunk.starts_with("hip")) { + const auto parsed = mcpp::plugins::accel::parse(text); + if (!parsed) { t.error = parsed.error(); return t; } + for (const auto& entry : *parsed) { + if (entry.backend == "hip") { t.hip = true; - t.amd_archs = parse_arch_set(chunk); - } else if (chunk.starts_with("cuda")) { - auto rest = chunk.substr(std::string_view("cuda").size()); - auto plus = rest.find('+'); - t.cuda_version = std::string(trim( - plus == std::string_view::npos ? rest : rest.substr(0, plus))); - // A version may be followed by a space-separated clause such as - // `ptx>=89`; the version is the leading numeric run. - if (auto sp = t.cuda_version.find(' '); sp != std::string::npos) - t.cuda_version.resize(sp); - t.cuda_archs = parse_arch_set(chunk); + for (const auto& arch : entry.archs) { + if (arch.starts_with("sm_")) t.cuda_archs.push_back(arch); + else if (arch.starts_with("gfx")) t.amd_archs.push_back(arch); + else { t.error = "unsupported HIP target '" + arch + "'"; return t; } + } } } + for (const auto& entry : *parsed) + if (entry.backend == "cuda") { + t.cuda_version = entry.version; + if (t.cuda_archs.empty() && t.amd_archs.empty()) t.cuda_archs = entry.archs; + } return t; } @@ -276,12 +275,13 @@ inline std::vector plan(std::span sources, options opt } const auto tg = parse_target(mcpp::accel()); + if (!tg.error.empty()) { std::cerr << "mcpp.rules.hip: " << tg.error << '\n'; return out; } auto which = opt.which; if (which == platform::automatic) which = tg.cuda_archs.empty() && !tg.amd_archs.empty() ? platform::amd : platform::nvidia; - if (which == platform::amd) { + if (which == platform::amd || !tg.amd_archs.empty()) { // Stated rather than approximated. The AMD platform needs a ROCm // runtime and device library, and this ecosystem publishes neither // yet; compiling for it would produce an object nothing on this @@ -289,7 +289,7 @@ inline std::vector plan(std::span sources, options opt std::cerr << std::format("mcpp.rules.hip: [build] accel names AMD architectures ({}) and no ROCm\n" " payload is published in this ecosystem yet, so nothing could link or run\n" " the result. The NVIDIA platform is available today:\n" - " accel = \"hip, cuda12.9+{{sm_89}}\"\n" + " accel = \"hip+{{sm_89}}\"\n" " which reaches the device through the CUDA runtime, with HIP as the API.", tg.amd_archs.empty() ? std::string("none") : tg.amd_archs.front()) << '\n'; return out; @@ -301,7 +301,7 @@ inline std::vector plan(std::span sources, options opt std::cerr << std::format("mcpp.rules.hip: [build] accel names no device architecture (accel = \"{}\").\n" " On the NVIDIA platform HIP compiles through the CUDA back end, and the\n" " device is spelled the way every other rule in this ecosystem spells it:\n" - " accel = \"hip, cuda12.9+{{sm_89}}\"\n" + " accel = \"hip+{{sm_89}}\"\n" " The set a build compiles for is a decision; the machine's own hardware is\n" " a poor default for it.", mcpp::accel()) << '\n'; @@ -393,11 +393,7 @@ inline std::vector plan(std::span sources, options opt // because the alternative is a path that is wrong on a host this rule // will one day be asked about, and a wrong path reports itself as a // missing toolchain. -#if defined(_WIN32) - const std::string cc = tcdir + "/bin/clang++.exe"; -#else - const std::string cc = tcdir + "/bin/clang++"; -#endif + const std::string cc = tcdir + (mcpplibs::os::is_windows ? "/bin/clang++.exe" : "/bin/clang++"); if (tcdir.empty() || !std::filesystem::exists(cc)) { std::cerr << std::format("mcpp.rules.hip: the NVIDIA platform compiles through clang, and this " "project's\n toolchain has no clang++ at {}.\n" @@ -523,7 +519,9 @@ inline bool compile(options opt = {}) { // `accel` itself, which is what the rule already does. A rule whose // backend this build does not name has nothing to do, and that is not a // mistake and must not be reported as one. - if (!parse_target(mcpp::accel()).hip) return true; + const auto requested = parse_target(mcpp::accel()); + if (!requested.error.empty()) { std::cerr << "mcpp.rules.hip: " << requested.error << '\n'; return false; } + if (!requested.hip) return true; const auto sources = device_sources(); if (sources.empty()) { mcpp::warning("[build] accel names hip but no constrained glob matched a `.hip`; " @@ -531,7 +529,7 @@ inline bool compile(options opt = {}) { return true; } auto edges = plan(sources, std::move(opt)); - if (edges.empty()) return false; + if (edges.empty()) return mcpp::xpkg_pending(); return submit(edges); } diff --git a/rules/metal.cppm b/rules/metal.cppm index 9c8b219..b8d625b 100644 --- a/rules/metal.cppm +++ b/rules/metal.cppm @@ -35,14 +35,13 @@ // fragment shader once per blend mode passes one `shader` per output, each with // its own definitions and name; the sources need not be in the project's tree. -module; -#include - export module mcpp.rules.metal; import std; import mcpp; import mcpp.plugins; +import mcpp.plugins.tool; +import mcpplibs.os; // Nothing here uses `std::println`: both of its overloads reach into the libc++ // dylib for symbols macOS 14 does not ship, so a build program that printed @@ -113,50 +112,19 @@ inline std::vector device_shaders() { // ─── The toolchain ───────────────────────────────────────────────────────── -// The first line a command prints, or empty. `popen` is POSIX; this module is -// compiled on every host (`tests/all-rules-compile`), so Windows is spelled. -inline std::string first_line_of(const std::string& cmd) { -#if defined(_WIN32) - FILE* p = ::_popen(cmd.c_str(), "r"); -#else - FILE* p = ::popen(cmd.c_str(), "r"); -#endif - if (!p) return {}; - char buf[1024]; - std::string line; - if (std::fgets(buf, sizeof buf, p)) line = buf; -#if defined(_WIN32) - ::_pclose(p); -#else - ::pclose(p); -#endif - while (!line.empty() && (line.back() == '\n' || line.back() == '\r')) line.pop_back(); - return line; -} - // The SDK's path, or empty when `xcrun` cannot locate it. -inline std::string sdk_path(const std::string& sdk) { -#if defined(_WIN32) - (void)sdk; - return {}; -#else - const std::string path = first_line_of( - "/usr/bin/xcrun --sdk " + sdk + " --show-sdk-path 2>/dev/null"); +std::string sdk_path(const std::string& sdk) { + if constexpr (mcpplibs::os::is_windows) return {}; + const std::string path = mcpp::plugins::tool::first_line({"/usr/bin/xcrun", "--sdk", sdk, "--show-sdk-path"}); std::error_code ec; return !path.empty() && std::filesystem::is_directory(path, ec) ? path : std::string(); -#endif } -inline std::string find_tool(const std::string& sdk, const char* tool) { -#if defined(_WIN32) - (void)sdk; (void)tool; - return {}; -#else - const std::string path = first_line_of( - "/usr/bin/xcrun --sdk " + sdk + " --find " + tool + " 2>/dev/null"); +std::string find_tool(const std::string& sdk, const char* tool) { + if constexpr (mcpplibs::os::is_windows) return {}; + const std::string path = mcpp::plugins::tool::first_line({"/usr/bin/xcrun", "--sdk", sdk, "--find", tool}); std::error_code ec; return !path.empty() && std::filesystem::is_regular_file(path, ec) ? path : std::string(); -#endif } inline std::string stem_of(const std::string& path) { diff --git a/rules/qt.cppm b/rules/qt.cppm index 55a8e9d..737a4a8 100644 --- a/rules/qt.cppm +++ b/rules/qt.cppm @@ -47,9 +47,6 @@ // `lrelease` reads that file, the order Qt's Visual Studio integration runs // them in. -module; -#include - export module mcpp.rules.qt; import std; diff --git a/rules/slang.cppm b/rules/slang.cppm index bb31324..dd37420 100644 --- a/rules/slang.cppm +++ b/rules/slang.cppm @@ -38,10 +38,6 @@ // `sizeof` for this style and would not for a style that terminates its output, // so the generated accessor uses what the compiler stated rather than a // coincidence that holds today. -module; -#include -#include - export module mcpp.rules.slang; import std; @@ -52,6 +48,7 @@ import mcpp; import mcpp.plugins; import mcpp.plugins.declare; import mcpp.plugins.tool; +import mcpplibs.os; // `std::println` is avoided here for the reason every file in this package // records: it is not header-only, and the symbols its overloads reach for were @@ -189,13 +186,8 @@ inline bool is_file(const std::string& p) { // Windows spells the separator differently and its paths contain the character // the other platforms separate on, so this is not one constant used twice. -#if defined(_WIN32) -inline constexpr char kPathSep = ';'; -inline constexpr const char* kExeSuffix = ".exe"; -#else -inline constexpr char kPathSep = ':'; -inline constexpr const char* kExeSuffix = ""; -#endif +inline constexpr char kPathSep = mcpplibs::os::is_windows ? ';' : ':'; +inline constexpr const char* kExeSuffix = mcpplibs::os::is_windows ? ".exe" : ""; inline std::string first_on_path(const char* exe) { const char* path = std::getenv("PATH"); @@ -239,27 +231,9 @@ inline mcpp::plugins::tool::found find_compiler(const options& opt) { return f; } -inline std::string run_and_capture(const std::string& cmd) { -#if defined(_WIN32) - FILE* p = ::_popen(cmd.c_str(), "r"); -#else - FILE* p = ::popen(cmd.c_str(), "r"); -#endif - if (!p) return {}; - std::string text; - char buf[512]; - while (std::fgets(buf, sizeof buf, p)) text += buf; -#if defined(_WIN32) - ::_pclose(p); -#else - ::pclose(p); -#endif - return text; -} - // `slangc -v` prints the release on its own, e.g. `2026.14.1`. -inline std::string compiler_version(const std::string& exe) { - auto text = run_and_capture("\"" + exe + "\" -v 2>&1"); +std::string compiler_version(const std::string& exe) { + auto text = mcpp::plugins::tool::capture({exe, "-v"}, /*with_errors=*/true).out; for (auto& c : text) if (c == '\r') c = '\n'; auto nl = text.find('\n'); return std::string(trim(nl == std::string::npos ? text : text.substr(0, nl))); diff --git a/rules/spirv.cppm b/rules/spirv.cppm index 7ed3fe5..ef849e9 100644 --- a/rules/spirv.cppm +++ b/rules/spirv.cppm @@ -51,19 +51,17 @@ // two compilers that produce an equivalent header from one shader are still // two different answers to "what compiled this". -module; -#include -#include - export module mcpp.rules.spirv; import std; import mcpp; +import mcpp.plugins.accel; // The lib root, which carries `mcpp::plugins::surface` -- the declarations a // consumer names, written once for every member that embeds a payload. import mcpp.plugins; import mcpp.plugins.declare; import mcpp.plugins.tool; +import mcpplibs.os; // WHY NOTHING HERE USES `std::println`, AND WHY THAT IS NOT A STYLE CHOICE. @@ -161,6 +159,7 @@ inline std::string include_dir(const options& opt) { struct target { std::string version; // "1.2", from `vulkan1.2` bool present = false; + std::string error; }; inline std::string_view trim(std::string_view s) { @@ -184,21 +183,14 @@ inline std::vector split(std::string_view s, char sep) { return out; } -inline target parse_target(std::string_view accel) { +inline target parse_target(std::string_view text) { target t; - for (std::size_t i = 0; i <= accel.size();) { - auto comma = accel.find(',', i); - auto chunk = trim(comma == std::string_view::npos ? accel.substr(i) - : accel.substr(i, comma - i)); - i = comma == std::string_view::npos ? accel.size() + 1 : comma + 1; - if (!chunk.starts_with("vulkan")) continue; + const auto parsed = mcpp::plugins::accel::parse(text); + if (!parsed) { t.error = parsed.error(); return t; } + for (const auto& entry : *parsed) { + if (entry.backend != "vulkan") continue; t.present = true; - auto rest = chunk.substr(std::string_view("vulkan").size()); - // A bare `vulkan` is legitimate and means "whatever the loader offers"; - // the target environment then falls back to the default below. - auto plus = rest.find('+'); - t.version = std::string(trim(plus == std::string_view::npos ? rest - : rest.substr(0, plus))); + t.version = entry.version; } return t; } @@ -249,13 +241,8 @@ inline flavour classify(const std::string& path) { // properties of the HOST and not of the target being compiled for -- a cross // build from Linux to Windows still looks for `glslc`, because that is the // binary about to be executed. -#if defined(_WIN32) -inline constexpr std::string_view kExeSuffix = ".exe"; -inline constexpr char kPathSep = ';'; -#else -inline constexpr std::string_view kExeSuffix = ""; -inline constexpr char kPathSep = ':'; -#endif +inline constexpr std::string_view kExeSuffix = mcpplibs::os::is_windows ? ".exe" : ""; +inline constexpr char kPathSep = mcpplibs::os::is_windows ? ';' : ':'; // The first of `/` and `/.exe` that exists. Both are // tried on every host rather than only the one whose suffix matches: a payload @@ -366,40 +353,8 @@ inline compiler find_compiler(const options& opt) { // magic, the second is the release. The release is what a floor compares, and // stating it as a fact is what makes a build log answer "which compiler // produced this SPIR-V" without anyone having to reproduce the build. -// `popen` is POSIX and Windows spells it `_popen`; the null device differs -// too. Both are named here so the call sites below read the same on every -// host -- the alternative is a `#if` around each one, and the one that gets -// forgotten is the one nobody compiles. -inline FILE* open_pipe(const std::string& cmd) { -#if defined(_WIN32) - return ::_popen(cmd.c_str(), "r"); -#else - return ::popen(cmd.c_str(), "r"); -#endif -} -inline void close_pipe(FILE* p) { -#if defined(_WIN32) - ::_pclose(p); -#else - ::pclose(p); -#endif -} -inline constexpr const char* kNullDevice = -#if defined(_WIN32) - "NUL"; -#else - "/dev/null"; -#endif - -inline std::string run_and_capture(const std::string& cmd) { - FILE* p = open_pipe(cmd); - if (!p) return {}; - std::string text; - char buf[512]; - while (std::fgets(buf, sizeof buf, p)) text += buf; - close_pipe(p); - return text; -} +// The compiler is asked through `mcpp::plugins::tool::capture`: its arguments +// as a vector, so the call reads the same on every host. // THE OPTIMISER IS OPTIONAL AND ITS ABSENCE IS NOT A BUILD ERROR. // @@ -413,14 +368,13 @@ inline std::string run_and_capture(const std::string& cmd) { // says so once — refusing would make an optional pass a requirement, and // passing it silently would let the manifest claim an optimisation that did // not happen. -inline bool has_optimizer(const std::string& exe) { - const auto out = run_and_capture("\"" + exe + "\" -Os --version 2>&1"); +bool has_optimizer(const std::string& exe) { + const auto out = mcpp::plugins::tool::capture({exe, "-Os", "--version"}, /*with_errors=*/true).out; return out.find("optimizer not linked") == std::string::npos; } -inline std::string compiler_version(const compiler& cc) { - const std::string text = run_and_capture("\"" + cc.path + "\" --version 2>" - + std::string(kNullDevice)); +std::string compiler_version(const compiler& cc) { + const std::string text = mcpp::plugins::tool::capture({cc.path, "--version"}).out; // glslc: `shaderc v2026.3 2fbab05...` on the first line. glslang: // `Glslang Version: 11:15.1.0`, whose first field is the SPIR-V generator // magic and whose second is the release. @@ -600,6 +554,8 @@ inline bool write_header(const std::string& header, const std::string& inc, inline bool compile(std::span shaders, options opt = {}) { if (shaders.empty()) return true; + const auto requested = parse_target(mcpp::accel()); + if (!requested.error.empty()) { std::cerr << "mcpp.rules.spirv: " << requested.error << '\n'; return false; } const auto cc = find_compiler(opt); // Asked for: the engine installs the payload and runs this program again. @@ -921,7 +877,9 @@ inline bool compile(options opt = {}) { // `accel` itself, which is what the rule already does. A rule whose // backend this build does not name has nothing to do, and that is not a // mistake and must not be reported as one. - if (!parse_target(mcpp::accel()).present) return true; + const auto requested = parse_target(mcpp::accel()); + if (!requested.error.empty()) { std::cerr << "mcpp.rules.spirv: " << requested.error << '\n'; return false; } + if (!requested.present) return true; const auto shaders = device_shaders(); if (shaders.empty()) { mcpp::warning("[build] accel names vulkan but no constrained glob matched a " diff --git a/rules/swift.cppm b/rules/swift.cppm index 15eee2e..e0c3006 100644 --- a/rules/swift.cppm +++ b/rules/swift.cppm @@ -60,14 +60,13 @@ // - Swift Package Manager dependencies. A SwiftPM package is not a package in // the mcpp graph. -module; -#include - export module mcpp.rules.swift; import std; import mcpp; import mcpp.plugins; +import mcpp.plugins.tool; +import mcpplibs.os; // Nothing here uses `std::println`, for the reason `rules/metal.cppm` gives. @@ -168,49 +167,18 @@ inline std::vector device_swift_sources() { // ─── The toolchain ───────────────────────────────────────────────────────── -// The first line a command prints, or empty. `popen` is POSIX; this module is -// compiled on every host (`tests/all-rules-compile`), so Windows is spelled. -inline std::string first_line_of(const std::string& cmd) { -#if defined(_WIN32) - FILE* p = ::_popen(cmd.c_str(), "r"); -#else - FILE* p = ::popen(cmd.c_str(), "r"); -#endif - if (!p) return {}; - char buf[1024]; - std::string line; - if (std::fgets(buf, sizeof buf, p)) line = buf; -#if defined(_WIN32) - ::_pclose(p); -#else - ::pclose(p); -#endif - while (!line.empty() && (line.back() == '\n' || line.back() == '\r')) line.pop_back(); - return line; -} - -inline std::string sdk_path(const std::string& sdk) { -#if defined(_WIN32) - (void)sdk; - return {}; -#else - const std::string path = first_line_of( - "/usr/bin/xcrun --sdk " + sdk + " --show-sdk-path 2>/dev/null"); +std::string sdk_path(const std::string& sdk) { + if constexpr (mcpplibs::os::is_windows) return {}; + const std::string path = mcpp::plugins::tool::first_line({"/usr/bin/xcrun", "--sdk", sdk, "--show-sdk-path"}); std::error_code ec; return !path.empty() && std::filesystem::is_directory(path, ec) ? path : std::string(); -#endif } -inline std::string find_swiftc(const std::string& sdk) { -#if defined(_WIN32) - (void)sdk; - return {}; -#else - const std::string path = first_line_of( - "/usr/bin/xcrun --sdk " + sdk + " --find swiftc 2>/dev/null"); +std::string find_swiftc(const std::string& sdk) { + if constexpr (mcpplibs::os::is_windows) return {}; + const std::string path = mcpp::plugins::tool::first_line({"/usr/bin/xcrun", "--sdk", sdk, "--find", "swiftc"}); std::error_code ec; return !path.empty() && std::filesystem::is_regular_file(path, ec) ? path : std::string(); -#endif } // ─── The rule ────────────────────────────────────────────────────────────── diff --git a/rules/sycl.cppm b/rules/sycl.cppm index ff28e9a..e5f6ca2 100644 --- a/rules/sycl.cppm +++ b/rules/sycl.cppm @@ -59,15 +59,13 @@ // points the device compiler at the same GCC the program uses: the graph's // supplier (`MCPP_CXX_ABI_ROOT`), else the gcc toolchain's own payload, // else the `xim:gcc` this rule declares. -module; -#include -#include - export module mcpp.rules.sycl; import std; import mcpp; +import mcpp.plugins.accel; import mcpp.plugins.tool; +import mcpplibs.os; // WHY NOTHING HERE USES `std::println`, AND WHY THAT IS NOT A STYLE CHOICE. @@ -110,21 +108,22 @@ struct options { // C++ libraries (`xim:gcc`, `xim:glibc`, `xim:linux-headers`) are payloads // too, and an override in `[xlings.overrides]` names another of each. mcpp::plugins::tool::choice compiler; + // NVIDIA assembler and libdevice; requested only by NVIDIA targets. + mcpp::plugins::tool::choice cuda; std::string out_dir = std::string(mcpp::out_dir()); }; // ─── What the engine said ────────────────────────────────────────────────── -// THE PROGRAMMING MODEL AND THE DEVICE ARE TWO CHUNKS, NOT ONE. +// EACH PROGRAMMING MODEL OWNS ITS DEVICE-CODE TARGETS. // // accel = "sycl" -- SPIR-V, compiled by the runtime for // whatever device it finds: a CPU, a // Level Zero GPU, an OpenCL device. -// accel = "sycl, cuda12.9+{sm_89}" -- ahead of time for NVIDIA. +// accel = "sycl+{nvptx64-sm_89}" -- ahead of time for NVIDIA. // -// The device is spelled the way every other rule in this ecosystem spells it, -// so `sm_89` does not acquire a second spelling because the source file says -// `.sycl` instead of `.cu`. +// `nvptx64-sm_89` names NVIDIA code generated by SYCL; `sm_89` remains +// an alias. A separate CUDA member selects CUDA source compilation. // THE TWO PUBLISHED SYCL TOOLCHAINS DIFFER IN MORE THAN A FILE SUFFIX. // // Upstream publishes `sycl_linux.tar.gz` and `sycl_windows.tar.gz` from one @@ -141,17 +140,15 @@ struct options { // plugins are not built for Windows, and the asset agrees -- it carries // Level Zero and OpenCL adapters and no others. An ahead-of-time NVIDIA // build is therefore refused there rather than attempted. -#if defined(_WIN32) -inline constexpr bool kWindows = true; -inline constexpr const char* kExe = ".exe"; -#else -inline constexpr bool kWindows = false; -inline constexpr const char* kExe = ""; -#endif +inline constexpr bool kWindows = mcpplibs::os::is_windows; +inline constexpr const char* kExe = kWindows ? ".exe" : ""; struct target { bool sycl = false; - std::vector cuda_archs; // {"sm_89"} + bool portable = true; + bool legacy_targets = false; + std::string error; + std::vector cuda_archs; // {"sm_89", "sm_90"} std::vector amd_archs; // {"gfx1100"} }; @@ -186,13 +183,41 @@ inline std::vector parse_arch_set(std::string_view chunk) { return archs; } -inline target parse_target(std::string_view accel) { +inline target parse_target(std::string_view text) { target t; - for (auto raw : split(accel, ',')) { - auto chunk = trim(raw); - if (chunk.starts_with("sycl")) t.sycl = true; - else if (chunk.starts_with("cuda")) t.cuda_archs = parse_arch_set(chunk); - else if (chunk.starts_with("hip")) t.amd_archs = parse_arch_set(chunk); + const auto parsed = mcpp::plugins::accel::parse(text); + if (!parsed) { t.error = parsed.error(); return t; } + const mcpp::plugins::accel::request* own = nullptr; + for (const auto& entry : *parsed) + if (entry.backend == "sycl") own = &entry; + if (!own) return t; + t.sycl = true; + t.portable = own->archs.empty(); + for (const auto& arch : own->archs) { + if (arch == "spir64") t.portable = true; + else if (arch.starts_with("nvptx64-sm_")) t.cuda_archs.push_back(arch.substr(8)); + else if (arch.starts_with("sm_")) t.cuda_archs.push_back(arch); + else if (arch.starts_with("gfx")) t.amd_archs.push_back(arch); + else { + t.error = "unsupported SYCL target '" + arch + "'; name spir64 or nvptx64-sm_"; + return t; + } + } + // Preserve the old spelling's target inheritance, while keeping BOTH + // explicitly named implementations selected. New requests put targets on + // sycl itself; they never activate a CUDA implementation implicitly. + if (own->archs.empty()) { + for (const auto& entry : *parsed) { + if (entry.backend == "cuda" && !entry.archs.empty()) { + t.cuda_archs = entry.archs; + t.portable = false; + t.legacy_targets = true; + } + if (entry.backend == "hip" && !entry.archs.empty()) { + t.amd_archs = entry.archs; + t.legacy_targets = true; + } + } } return t; } @@ -245,29 +270,8 @@ inline std::string cxx_gcc_root() { // `dpcpp --version` states the release and the intel/llvm revision it was // built from. Stated as a fact so a build log answers "which SYCL compiler" // without anyone reproducing the build. -inline FILE* open_pipe(const std::string& cmd) { -#if defined(_WIN32) - return ::_popen(cmd.c_str(), "r"); -#else - return ::popen(cmd.c_str(), "r"); -#endif -} -inline void close_pipe(FILE* p) { -#if defined(_WIN32) - ::_pclose(p); -#else - ::pclose(p); -#endif -} - -inline std::string compiler_version(const std::string& exe) { - FILE* p = open_pipe("\"" + exe + "\" --version 2>" - + (kWindows ? std::string("NUL") : std::string("/dev/null"))); - if (!p) return {}; - std::string text; - char buf[512]; - while (std::fgets(buf, sizeof buf, p)) text += buf; - close_pipe(p); +std::string compiler_version(const std::string& exe) { + const std::string text = mcpp::plugins::tool::capture({exe, "--version"}).out; for (auto line : split(text, '\n')) { auto at = line.find("DPC++ compiler "); if (at == std::string_view::npos) continue; @@ -341,6 +345,10 @@ inline std::vector plan(std::span sources, options opt } const auto tg = parse_target(mcpp::accel()); + if (!tg.error.empty()) { std::cerr << "mcpp.rules.sycl: " << tg.error << '\n'; return out; } + if (tg.legacy_targets) + mcpp::warning("mcpp.rules.sycl: legacy targets inherited from another explicitly selected implementation; " + "put SYCL targets on sycl+{nvptx64-sm_89}. Both named implementations remain selected."); if constexpr (kWindows) { if (!tg.cuda_archs.empty()) { std::cerr << std::format("mcpp.rules.sycl: [build] accel names an NVIDIA target and this host's SYCL\n" @@ -352,11 +360,11 @@ inline std::vector plan(std::span sources, options opt return out; } } - if (!tg.amd_archs.empty() && tg.cuda_archs.empty()) { + if (!tg.amd_archs.empty()) { std::cerr << std::format("mcpp.rules.sycl: [build] accel names AMD architectures and this ecosystem\n" " publishes no ROCm payload yet, so nothing could link or run the result.\n" " Available today: accel = \"sycl\" (SPIR-V, any device the runtime finds)\n" - " or accel = \"sycl, cuda12.9+{{sm_89}}\" (ahead of time for NVIDIA).") << '\n'; + " or accel = \"sycl+{{nvptx64-sm_89}}\" (ahead of time for NVIDIA).") << '\n'; return out; } @@ -389,7 +397,19 @@ inline std::vector plan(std::span sources, options opt const std::string dpcpp = dpcppTool.program; const std::string dpcppRoot = dpcppTool.root; const std::string gcc = cxx_gcc_root(); - const std::string cuda = tg.cuda_archs.empty() ? std::string{} : payload("cuda-nvcc"); + std::string cuda; + if (!tg.cuda_archs.empty()) { + const mcpp::plugins::tool::spec cudaSpec{ + .who = "mcpp.rules.sycl", .package = "cuda-nvcc", .programs = {"ptxas"}, + .option = "options::cuda" }; + const auto cudaTool = mcpp::plugins::tool::resolve(cudaSpec, opt.cuda); + if (cudaTool.pending()) return out; + if (!cudaTool) { + std::cerr << mcpp::plugins::tool::describe_missing(cudaSpec, cudaTool) << '\n'; + return out; + } + cuda = cudaTool.root; + } // THE C LIBRARY IS THE SAME QUESTION AS THE C++ ONE, ONE LAYER DOWN. // // `--gcc-install-dir` below names the C++ standard library because, left @@ -478,9 +498,10 @@ inline std::vector plan(std::span sources, options opt // did not produce and says so as a warning rather than an error. std::vector targeting; if (!tg.cuda_archs.empty()) { - targeting = { "-fsycl-targets=nvptx64-nvidia-cuda", "--cuda-path=" + cuda }; + targeting = { tg.portable ? "-fsycl-targets=spir64,nvptx64-nvidia-cuda" + : "-fsycl-targets=nvptx64-nvidia-cuda", "--cuda-path=" + cuda }; for (auto const& a : tg.cuda_archs) { - targeting.push_back("-Xsycl-target-backend"); + targeting.push_back("-Xsycl-target-backend=nvptx64-nvidia-cuda"); targeting.push_back("--cuda-gpu-arch=" + a); } } @@ -526,13 +547,13 @@ inline std::vector plan(std::span sources, options opt "kernels are compiled to SPIR-V and the runtime chooses. A back end " "that cannot consume SPIR-V -- CUDA is one -- fails inside the SYCL " "scheduler, where the program cannot catch it. Name the device to " - "compile ahead of time: accel = \"sycl, cuda12.9+{sm_89}\"."); + "compile ahead of time: accel = \"sycl+{nvptx64-sm_89}\"."); - std::cout << std::format("mcpp.rules.sycl: {} -- {} for {}", - tg.cuda_archs.empty() ? "SPIR-V, compiled by the runtime" - : "ahead of time, NVIDIA back end", - std::filesystem::path(exe).filename().string(), - tg.cuda_archs.empty() ? std::string("any device") : tg.cuda_archs.front()) << '\n'; + std::string targets = tg.portable ? "spir64" : ""; + for (const auto& arch : tg.cuda_archs) + targets += (targets.empty() ? "" : ",") + std::string("nvptx64-") + arch; + std::cout << std::format("mcpp.rules.sycl: {} -- {}", targets, + std::filesystem::path(exe).filename().string()) << '\n'; std::vector objects; for (auto const& src : sources) { @@ -617,7 +638,9 @@ inline bool compile(options opt = {}) { // `accel` itself, which is what the rule already does. A rule whose // backend this build does not name has nothing to do, and that is not a // mistake and must not be reported as one. - if (!parse_target(mcpp::accel()).sycl) return true; + const auto requested = parse_target(mcpp::accel()); + if (!requested.error.empty()) { std::cerr << "mcpp.rules.sycl: " << requested.error << '\n'; return false; } + if (!requested.sycl) return true; const auto sources = device_sources(); if (sources.empty()) { mcpp::warning("[build] accel names sycl but no constrained glob matched a `.sycl`; " @@ -625,7 +648,7 @@ inline bool compile(options opt = {}) { return true; } auto edges = plan(sources, std::move(opt)); - if (edges.empty()) return false; + if (edges.empty()) return mcpp::xpkg_pending(); return submit(edges); } diff --git a/src/accel.cppm b/src/accel.cppm new file mode 100644 index 0000000..4e3e655 --- /dev/null +++ b/src/accel.cppm @@ -0,0 +1,124 @@ +// Shared reading of the engine's accelerator requests. Targets are opaque; +// each rule interprets only the member whose backend name it owns. +export module mcpp.plugins.accel; +import std; + +export namespace mcpp::plugins::accel { +struct request { + std::string backend; + std::string version; + std::vector archs; + std::string portable_floor; +}; +std::expected, std::string> parse(std::string_view text); +} // namespace mcpp::plugins::accel + +namespace mcpp::plugins::accel { +namespace { + +std::string_view trim_sv(std::string_view s) { + while (!s.empty() && (s.front() == ' ' || s.front() == '\t')) s.remove_prefix(1); + while (!s.empty() && (s.back() == ' ' || s.back() == '\t')) s.remove_suffix(1); + return s; +} + +std::expected, std::string> +read_request_targets(std::string_view s, std::size_t& pos) { + std::vector targets; + // Empty is the existing unconstrained set, also emitted for a bare name. + if (pos < s.size() && s[pos] == '}') return targets; + while (true) { + const auto begin = pos; + while (pos < s.size() && s[pos] != ',' && s[pos] != '}') { + if (s[pos] == '{' || s[pos] == '+') return std::unexpected(std::format( + "invalid accelerator request at byte {}: invalid target-set delimiter", pos + 1)); + ++pos; + } + const auto value = trim_sv(s.substr(begin, pos - begin)); + if (value.empty() || std::ranges::any_of(value, [](unsigned char c) { return std::isspace(c); })) + return std::unexpected(std::format( + "invalid accelerator request at byte {}: expected one nonempty target token", begin + 1)); + targets.emplace_back(value); + if (pos == s.size() || s[pos] == '}') return targets; + ++pos; + } +} + +} // namespace + +std::expected, std::string> parse(std::string_view s) { + std::size_t pos = 0; + auto spaces = [&] { + while (pos < s.size() && (s[pos] == ' ' || s[pos] == '\t')) ++pos; + }; + auto error = [&](std::string_view why) { + return std::unexpected(std::format("invalid accelerator request at byte {}: {}", + pos + 1, why)); + }; + spaces(); + if (pos == s.size() || trim_sv(s.substr(pos)) == "(none)") return std::vector{}; + + std::vector out; + std::set names; + while (pos < s.size()) { + spaces(); + const auto begin = pos; + while (pos < s.size() && std::isalpha(static_cast(s[pos]))) ++pos; + if (pos == begin) return error("expected an implementation name"); + request a; + a.backend = std::string(s.substr(begin, pos - begin)); + if (!names.insert(a.backend).second) + return error(std::format("implementation '{}' appears more than once; combine its targets in one set", + a.backend)); + const auto version = pos; + if (pos < s.size() && std::isdigit(static_cast(s[pos]))) { + while (pos < s.size()) { + const auto c = static_cast(s[pos]); + if (!std::isalnum(c) && c != '.' && c != '-' && c != '_') break; + ++pos; + } + } + a.version = std::string(s.substr(version, pos - version)); + spaces(); + if (pos < s.size() && s[pos] == '+') { + ++pos; + spaces(); + if (pos == s.size() || s[pos] != '{') return error("expected '{' after '+'"); + ++pos; + spaces(); + auto targets = read_request_targets(s, pos); + if (!targets) return std::unexpected(targets.error()); + a.archs = std::move(*targets); + if (pos == s.size() || s[pos] != '}') return error("expected '}' to close the target set"); + ++pos; + spaces(); + } + const auto tail = s.substr(pos); + const std::size_t floorPrefix = tail.starts_with("floor>=") ? 7 : tail.starts_with("ptx>=") ? 5 : 0; + if (floorPrefix) { + pos += floorPrefix; + spaces(); + const auto floor = pos; + while (pos < s.size() && s[pos] != ',' && s[pos] != ' ' && s[pos] != '\t') { + if (s[pos] == '{' || s[pos] == '}' || s[pos] == '+') return error("invalid portable-form floor"); + ++pos; + } + if (pos == floor) return error("expected a portable-form floor"); + a.portable_floor = std::string(s.substr(floor, pos - floor)); + spaces(); + } + if (pos < s.size() && s[pos] != ',') return error("unexpected text after the implementation request"); + std::ranges::sort(a.archs); + a.archs.erase(std::unique(a.archs.begin(), a.archs.end()), a.archs.end()); + out.push_back(std::move(a)); + if (pos == s.size()) break; + ++pos; + spaces(); + if (pos == s.size()) return error("expected an implementation after ','"); + } + std::ranges::sort(out, {}, &request::backend); + return out; +} + + +} // namespace mcpp::plugins::accel diff --git a/src/plugins.cppm b/src/plugins.cppm index 8e30a34..49e71ba 100644 --- a/src/plugins.cppm +++ b/src/plugins.cppm @@ -17,9 +17,6 @@ // because only the lib root is built first. So shared code lives here rather // than in a file of its own, and a member that needs it writes // `import mcpp.plugins;`. -module; -#include - export module mcpp.plugins; import std; @@ -49,7 +46,7 @@ export namespace mcpp::plugins { // // One package, one version: the number lives in mcpp.toml, and the CI step // `the collection states its own version` compares the two. -inline constexpr std::string_view version = "0.20.0"; +inline constexpr std::string_view version = "0.21.0"; } // namespace mcpp::plugins @@ -1307,14 +1304,31 @@ inline emitted outputs(const options& opt) { } +// The assembly's meaning includes the bytes named by .incbin. Put a content +// checksum in its source so restat cannot prune the assembler after a payload +// changes while the .incbin path stays the same. This runs in the body action, +// after its declared payload inputs exist; planning still reads no payload. +inline std::optional payload_checksum(const std::string& file) { + std::ifstream input(file, std::ios::binary); + if (!input) return std::nullopt; + std::uint64_t sum = 14695981039346656037ULL; + std::array bytes; + while (input) { + input.read(bytes.data(), static_cast(bytes.size())); + for (std::streamsize i = 0; i < input.gcount(); ++i) { + sum ^= static_cast(bytes[static_cast(i)]); + sum *= 1099511628211ULL; + } + } + return input.bad() ? std::nullopt : std::optional(sum); +} + // Writes ONE half of the surface. The names come from `outputs`, which the // caller has already asked; this produces the content. // -// Nothing here reads a payload's BYTES. Under object storage the assembly names -// each payload in `.incbin` and the assembler reads it later, which is why this -// can be an action whose command runs before, after or independently of the -// payload's own producer -- what matters is that the payload is a declared -// INPUT of that action, and the graph then holds the edge. +// The interface reads no payload. The object body runs after its declared +// payload inputs, records their content checksum and names them in .incbin; +// the assembler reads the bytes after that body action has completed. inline bool write(std::span items, const options& opt, half which) { if (items.empty()) return true; if (opt.module_name.empty()) { @@ -1520,8 +1534,16 @@ blob& load(const char* path, blob& b) {{ it.identifier); return false; } - if (!write_if_different(out.assembly_file, - assembly_for(items, opt, opt.target_os))) { + auto assembly = assembly_for(items, opt, opt.target_os); + for (const auto& it : items) { + const auto checksum = payload_checksum(it.payload_path); + if (!checksum) { + std::cerr << std::format("mcpp.plugins.surface: cannot read payload {}\n", it.payload_path); + return false; + } + assembly += std::format("// payload content: {:016x}\n", *checksum); + } + if (!write_if_different(out.assembly_file, assembly)) { std::cerr << std::format("mcpp.plugins.surface: cannot write {}\n", out.assembly_file); return false; diff --git a/src/progress.cppm b/src/progress.cppm new file mode 100644 index 0000000..bad549a --- /dev/null +++ b/src/progress.cppm @@ -0,0 +1,35 @@ +// Optional action telemetry; useful in worker programs as well as adapters. +export module mcpp.plugins.progress; +import std; + +export namespace mcpp::plugins::progress { +struct snapshot { + std::string phase; + std::optional done, total; + std::string detail; +}; + +// Compile against older mcpp build-program APIs without requiring the field. +// The worker naturally becomes a no-op when the engine supplies no channel. +template bool enable(Action& action, bool requested = true) { + if constexpr (requires(Action& a) { a.progress = true; }) { + action.progress = requested; + return requested; + } else { + if (requested) std::cerr << "mcpp.plugins.progress: this engine does not support action progress; continuing without telemetry\n"; + return false; + } +} + +inline bool valid_text(std::string_view value, std::size_t limit) { + return value.size() <= limit && std::ranges::none_of(value, [](unsigned char c) { + return c < 32 || c == 127; + }); +} +inline std::string quote(std::string_view value) { + std::string out = "\""; + for (char c : value) { if (c == '"' || c == '\\') out += '\\'; out += c; } + return out + '"'; +} + +} diff --git a/src/progress_writer.cppm b/src/progress_writer.cppm new file mode 100644 index 0000000..b82ebc2 --- /dev/null +++ b/src/progress_writer.cppm @@ -0,0 +1,39 @@ +// Atomic latest-state writer for action worker programs. +export module mcpp.plugins.progress.writer; +import std; +export import mcpp.plugins.progress; +import mcpplibs.os.fs; +import mcpplibs.os.env; +export namespace mcpp::plugins::progress { +class writer { +public: + writer() { + if (auto file = mcpplibs::os::env::get("MCPP_ACTION_PROGRESS_FILE"); file && !file->empty()) + file_ = mcpplibs::os::fs::from_utf8(*file); + } + bool enabled() const { return !file_.empty(); } + // At most five atomic latest-state replacements per second. Reporting + // failure, missing telemetry, and rate limiting do not affect the work. + bool report(const snapshot& state) { + if (!enabled() || !valid_text(state.phase, 128) || state.phase.empty() + || !valid_text(state.detail, 1024) + || (state.done && state.total && *state.done > *state.total)) return false; + const auto now = std::chrono::steady_clock::now(); + if (last_ && now - *last_ < std::chrono::milliseconds(200)) return false; + std::string text = "{\"v\":1,\"seq\":" + std::to_string(seq_) + + ",\"phase\":" + quote(state.phase); + if (state.done) text += ",\"done\":" + std::to_string(*state.done); + if (state.total) text += ",\"total\":" + std::to_string(*state.total); + if (!state.detail.empty()) text += ",\"detail\":" + quote(state.detail); + text += '}'; + // fs's replacement has the same semantics on Linux, macOS and Windows. + if (!mcpplibs::os::fs::write_atomic(file_, text)) return false; + ++seq_; last_ = now; + return true; + } +private: + std::filesystem::path file_; + std::uint64_t seq_ = 0; + std::optional last_; +}; +} diff --git a/src/testing.cppm b/src/testing.cppm index 2810f40..080face 100644 --- a/src/testing.cppm +++ b/src/testing.cppm @@ -28,8 +28,9 @@ // } // // HOW. The build program is run by mcpp as usual; `run` starts the same -// executable once per case (`--mcpp-plugins-testing-case=`) with the case's -// context in its environment and its standard output in a file, and runs the +// executable once per case (`--mcpp-plugins-testing-case=`) through +// mcpplibs.os, with the case's context stated as the child's environment (this +// process's own is never written) and its standard output kept in a file, and runs the // check on what it wrote. A separate process per case is what makes the // context complete: every accessor of `mcpp.core` reads the environment, and // a function's `static` state starts fresh. A failed check fails the build @@ -37,35 +38,14 @@ // written to `/plugins-testing/results.txt`, one `PASS` or `FAIL` // line per case, for a CI step to count. -module; -#include -// The environment as an array, so the kit can clear the real build's -// MCPP_XPKG_*. Darwin does not export `environ` to anything but a main program, -// and offers `_NSGetEnviron()` instead; elsewhere the symbol is declared here -// rather than taken from , whose declaration a module purview does not -// see. -#if defined(__APPLE__) -#include -#elif !defined(_WIN32) -extern "C" char** environ; -#endif - export module mcpp.plugins.testing; import std; import mcpp; +import mcpplibs.os; namespace mcpp::plugins::testing::detail { -inline void set_env(const std::string& name, const std::string& value) { -#if defined(_WIN32) - ::_putenv_s(name.c_str(), value.c_str()); -#else - if (value.empty()) ::unsetenv(name.c_str()); - else ::setenv(name.c_str(), value.c_str(), 1); -#endif -} - // Every key an accessor of `mcpp.core` reads that a plugin's logic depends on. // A case sets each of them -- empty unless its context states a value -- so // nothing of the real build reaches the function under test. @@ -99,16 +79,9 @@ inline constexpr std::string_view kKeys[] = { // Enumerated rather than listed: the keys are derived from package names, so no // fixed list can cover the next payload a member reads. inline std::vector inherited_payload_keys() { -#if defined(_WIN32) - char** env = _environ; -#elif defined(__APPLE__) - char** env = *_NSGetEnviron(); -#else - char** env = environ; -#endif std::vector out; - for (char** e = env; e && *e; ++e) { - std::string_view entry(*e); + for (auto const& variable : mcpplibs::os::env::variables()) { + std::string_view entry(variable); const auto eq = entry.find('='); if (eq == std::string_view::npos) continue; const auto name = entry.substr(0, eq); @@ -316,7 +289,6 @@ inline std::string expand(std::string text, const std::string& root) { text.replace(at, 6, root); return text; } -inline std::string quoted(const std::string& s) { return "\"" + s + "\""; } } // namespace detail2 // Runs `cases` (see the header). Returns the build program's exit code: 0 @@ -359,18 +331,22 @@ inline int run(int argc, char** argv, std::span cases) { fs::create_directories(f.parent_path(), ec); std::ofstream(f, std::ios::binary) << detail2::expand(content, rootS); } + // The case's context, every other key absent: stated for the child alone. + mcpplibs::os::process::spawn_options o; + o.program = self; + o.args = {std::string(flag) + std::to_string(n)}; for (auto const& k : keys) { - std::string v; - for (auto const& kv : c.ctx.values) if (kv.first == k) v = detail2::expand(kv.second, rootS); - detail::set_env(k, v); + std::optional v; + for (auto const& kv : c.ctx.values) + if (kv.first == k) v = detail2::expand(kv.second, rootS); + if (v && v->empty()) v.reset(); + o.env.emplace_back(k, std::move(v)); } + auto ran = mcpplibs::os::process::run(std::move(o)); + const int status = ran ? ran->status.shell_code() : -1; const fs::path so = root / "stdout.txt", se = root / "stderr.txt"; - std::string cmd = detail2::quoted(self.string()) + " " + std::string(flag) + std::to_string(n) - + " > " + detail2::quoted(so.string()) + " 2> " + detail2::quoted(se.string()); -#if defined(_WIN32) - cmd = "\"" + cmd + "\""; // cmd.exe strips one outer pair of quotes -#endif - const int status = std::system(cmd.c_str()); + std::ofstream(so, std::ios::binary) << (ran ? ran->out : std::string()); + std::ofstream(se, std::ios::binary) << (ran ? ran->err : ran.error().message()); result r; r.exit_code = status; @@ -396,7 +372,6 @@ inline int run(int argc, char** argv, std::span cases) { } } std::ofstream(base / "results.txt", std::ios::binary) << verdicts; - for (auto const& k : keys) detail::set_env(k, ""); std::cerr << report << std::format("mcpp.plugins.testing: {} of {} cases passed\n", cases.size() - failed, cases.size()); return failed == 0 ? 0 : 1; diff --git a/src/tool.cppm b/src/tool.cppm index 8cfa831..8bf2783 100644 --- a/src/tool.cppm +++ b/src/tool.cppm @@ -28,13 +28,11 @@ // reports `Using cmake (mcpp.deps.cmake) ← /usr/bin/cmake [program · // build.mcpp:9]` and `mcpp why tool cmake` answers from the same record. -module; -#include - export module mcpp.plugins.tool; import std; import mcpp; +import mcpplibs.os; namespace mcpp::plugins::tool::detail { @@ -47,13 +45,9 @@ inline std::string generic(const std::filesystem::path& p) { return p.lexically_normal().generic_string(); } -#if defined(_WIN32) -inline constexpr char kPathSep = ';'; -inline constexpr std::string_view kExe = ".exe"; -#else -inline constexpr char kPathSep = ':'; -inline constexpr std::string_view kExe = ""; -#endif +// A build program runs on the machine that builds: its own system's conventions. +inline constexpr char kPathSep = mcpplibs::os::is_windows ? ';' : ':'; +inline constexpr std::string_view kExe = mcpplibs::os::is_windows ? ".exe" : ""; // `/`, with and without `.exe`. // @@ -352,4 +346,49 @@ inline std::string describe_missing(const spec& s, const found& f) { return out; } +// WHAT A PROGRAM PRINTS (0.21.0). A member asks a tool for a fact -- `xcrun +// --show-sdk-path`, a compiler's version, a dry run's plan -- through +// mcpplibs.os: the program and its arguments as a vector, with no shell between, +// so nothing is quoted and no null device is named on any host. The standard +// error is dropped, or appended to the output when `with_errors`. A program that +// does not end within a minute is ended; `status` is then the shell's code for +// it, and -1 when the program could not be started at all. +// +// NOT INLINE: the two are compiled once, in this unit, and an importer sees +// only their declarations. GCC 16 fails an importer that instantiates +// mcpplibs.os's types from another module's inline body: it cannot read the +// module back ("failed to read compiled module cluster: Bad file data") or +// leaves `std::optional>` unresolved at the link +// (`tests/metal-consumer`; docs/plugin-development.md §4). +struct printed { + int status = -1; + std::string out; +}; + +printed capture(const std::vector& argv, bool with_errors = false, + const std::filesystem::path& cwd = {}) { + namespace process = mcpplibs::os::process; + if (argv.empty()) return {}; + process::spawn_options o; + o.program = argv.front(); + o.args.assign(argv.begin() + 1, argv.end()); + o.cwd = cwd; + o.in = process::stdio::null; + o.err = with_errors ? process::stdio::pipe : process::stdio::null; + auto r = process::run(std::move(o), process::bounds{}); + if (!r) return {}; + // The standard error after the output, as a reader of either finds it. + if (with_errors) r->out += r->err; + return {r->status.shell_code(), std::move(r->out)}; +} + +// The first line a program prints, without its line end; empty when it prints +// nothing or cannot be started. +std::string first_line(const std::vector& argv) { + auto text = capture(argv).out; + if (auto nl = text.find('\n'); nl != std::string::npos) text.resize(nl); + while (!text.empty() && (text.back() == '\r' || text.back() == ' ')) text.pop_back(); + return text; +} + } // namespace mcpp::plugins::tool diff --git a/tests/all-rules-compile/mcpp.toml b/tests/all-rules-compile/mcpp.toml index 2c2a6a4..08290a2 100644 --- a/tests/all-rules-compile/mcpp.toml +++ b/tests/all-rules-compile/mcpp.toml @@ -10,14 +10,13 @@ # of a rule that are written for a host are exactly the halves that host never # compiles. # -# A rule's host-dependent code is `#if defined(_WIN32)`, a `.exe` suffix, a -# directory named `lib/x64`, a listing command. None of it is compiled by a -# Linux build, so a Windows branch can be syntactically wrong for as long as -# nobody builds on Windows -- and the first person to do so gets a compile -# error out of a package they only wanted to use. A dist member's host- -# dependent code is the same shape -- `dist-wix`'s tool discovery carries its -# own `#if defined(_WIN32)` -- so it belongs in this fixture for the same -# reason a rule does. +# A rule's host-dependent code is an `if constexpr (mcpplibs::os::is_windows)`, +# a `.exe` suffix, a directory named `lib/x64`, a listing command. Since 0.21.0 +# no unit branches in the preprocessor, so every host compiles every branch; +# what only one host can show is whether that host's half links and behaves, +# and a library or a tool named for one host is still met only there. A dist +# member's host-dependent code is the same shape, so it belongs in this fixture +# for the same reason a rule does. # # This fixture names no accelerator and packages nothing (`mcpp build`, never # `mcpp pack`). Every rule's `compile()` and every dist member's `plan_for()` diff --git a/tests/cmake-consumer/build.mcpp b/tests/cmake-consumer/build.mcpp index 55cab6f..7164df8 100644 --- a/tests/cmake-consumer/build.mcpp +++ b/tests/cmake-consumer/build.mcpp @@ -5,6 +5,7 @@ import mcpp.deps.cmake; int main() { mcpp::deps::cmake::options o; o.source = "greet"; + o.progress = true; o.libraries = { "greet" }; o.shared = true; return mcpp::deps::cmake::use(o) ? 0 : 1; diff --git a/tests/cmake-cross/answer/CMakeLists.txt b/tests/cmake-cross/answer/CMakeLists.txt new file mode 100644 index 0000000..af4e356 --- /dev/null +++ b/tests/cmake-cross/answer/CMakeLists.txt @@ -0,0 +1,15 @@ +# A static library whose one function says which C library compiled it. +cmake_minimum_required(VERSION 3.16) +project(answer LANGUAGES C) + +add_library(answer STATIC answer.c) +target_include_directories(answer PUBLIC $) + +install(TARGETS answer ARCHIVE DESTINATION lib) +install(FILES include/answer.h DESTINATION include) + +# This executable belongs to the subproject. Its link must use this build's +# archive, without depending on the consumer's future installation prefix. +add_executable(answer-sidecar sidecar.c) +target_link_libraries(answer-sidecar PRIVATE answer) +install(TARGETS answer-sidecar RUNTIME DESTINATION bin) diff --git a/tests/cmake-cross/answer/answer.c b/tests/cmake-cross/answer/answer.c new file mode 100644 index 0000000..35ff5e3 --- /dev/null +++ b/tests/cmake-cross/answer/answer.c @@ -0,0 +1,14 @@ +#include "answer.h" +#include + +/* glibc defines __GLIBC__ in , which does not include; + does, and so does every glibc header that a musl one is not. */ +#include + +const char* answer_libc(void) { +#if defined(__GLIBC__) + return "glibc"; +#else + return "not glibc"; +#endif +} diff --git a/tests/cmake-cross/answer/include/answer.h b/tests/cmake-cross/answer/include/answer.h new file mode 100644 index 0000000..c5d972c --- /dev/null +++ b/tests/cmake-cross/answer/include/answer.h @@ -0,0 +1,8 @@ +#pragma once +#ifdef __cplusplus +extern "C" { +#endif +const char* answer_libc(void); +#ifdef __cplusplus +} +#endif diff --git a/tests/cmake-cross/answer/sidecar.c b/tests/cmake-cross/answer/sidecar.c new file mode 100644 index 0000000..202c479 --- /dev/null +++ b/tests/cmake-cross/answer/sidecar.c @@ -0,0 +1,7 @@ +#include +#include + +int main(void) { + printf("cmake-sidecar: the subproject was compiled against %s\n", answer_libc()); + return 0; +} diff --git a/tests/cmake-cross/build.mcpp b/tests/cmake-cross/build.mcpp new file mode 100644 index 0000000..dadfce0 --- /dev/null +++ b/tests/cmake-cross/build.mcpp @@ -0,0 +1,10 @@ +import std; +import mcpp; +import mcpp.deps.cmake; + +int main() { + mcpp::deps::cmake::options o; + o.source = "answer"; + o.libraries = { "answer" }; + return mcpp::deps::cmake::use(o) ? 0 : 1; +} diff --git a/tests/cmake-cross/mcpp.toml b/tests/cmake-cross/mcpp.toml new file mode 100644 index 0000000..6a24aab --- /dev/null +++ b/tests/cmake-cross/mcpp.toml @@ -0,0 +1,22 @@ +# Fixture: a CMake subproject built for ANOTHER TARGET (0.21.0). Built with +# `--target x86_64-linux-musl` on a glibc machine, the subproject is compiled +# with the compilers and words of the engine's contract for that target, not +# with the machine's toolset, and linked into the static program. +[package] +name = "cmake-cross" +version = "0.1.0" +description = "Fixture: a CMake subproject compiled for the build's target from the engine's contract" +license = "Apache-2.0" +authors = ["mcpp-community"] + +[language] +standard = "c++23" +modules = true +import_std = true + +[build-dependencies.mcpp] +plugins = { path = "../..", features = ["deps-cmake"], host-module = true } + +[targets.cmake-cross] +kind = "bin" +main = "src/main.cpp" diff --git a/tests/cmake-cross/src/main.cpp b/tests/cmake-cross/src/main.cpp new file mode 100644 index 0000000..863bec1 --- /dev/null +++ b/tests/cmake-cross/src/main.cpp @@ -0,0 +1,8 @@ +#include + +#include + +int main() { + std::printf("cmake-cross: the subproject was compiled against %s\n", answer_libc()); + return 0; +} diff --git a/tests/multi-accel-consumer/build.mcpp b/tests/multi-accel-consumer/build.mcpp new file mode 100644 index 0000000..4796f2b --- /dev/null +++ b/tests/multi-accel-consumer/build.mcpp @@ -0,0 +1,5 @@ +import mcpp.rules.cuda; +import mcpp.rules.sycl; +int main() { + return mcpp::rules::cuda::compile() && mcpp::rules::sycl::compile() ? 0 : 1; +} diff --git a/tests/multi-accel-consumer/mcpp.toml b/tests/multi-accel-consumer/mcpp.toml new file mode 100644 index 0000000..ff0cbbb --- /dev/null +++ b/tests/multi-accel-consumer/mcpp.toml @@ -0,0 +1,20 @@ +[package] +name = "multi-accel-consumer" +version = "0.1.0" +accelerators = ["cuda", "sycl"] +[toolchain] +default = "llvm@22.1.8" +[build-dependencies.mcpp] +plugins = { path = "../..", features = ["rules-cuda", "rules-sycl"], host-module = true } +[dependencies.compat] +cuda-runtime = "2026.09.05" +sycl-runtime = "2026.09.07" +[dependencies.gcc] +libstdcxx = "16.1.0.1" +[build] +accel = "cuda+{sm_90},sycl+{nvptx64-sm_89}" +ldflags = ["-lcudart_static", "-lrt", "-lpthread", "-ldl"] +sources = ["src/main.cpp", { glob = "src/*.cu", accel = "cuda" }, { glob = "src/*.sycl", accel = "sycl" }] +[targets.multi-accel-consumer] +kind = "bin" +main = "src/main.cpp" diff --git a/tests/multi-accel-consumer/src/kernel.cu b/tests/multi-accel-consumer/src/kernel.cu new file mode 100644 index 0000000..347e6f3 --- /dev/null +++ b/tests/multi-accel-consumer/src/kernel.cu @@ -0,0 +1,2 @@ +__global__ void cuda_kernel(int* value){*value=1;} +extern "C" int cuda_anchor(){return 1;} diff --git a/tests/multi-accel-consumer/src/kernel.sycl b/tests/multi-accel-consumer/src/kernel.sycl new file mode 100644 index 0000000..0a552a0 --- /dev/null +++ b/tests/multi-accel-consumer/src/kernel.sycl @@ -0,0 +1,6 @@ +#include +extern "C" void sycl_kernel(int* value){ + sycl::queue queue; + queue.single_task([=]{*value=2;}).wait_and_throw(); +} +extern "C" int sycl_anchor(){return 2;} diff --git a/tests/multi-accel-consumer/src/main.cpp b/tests/multi-accel-consumer/src/main.cpp new file mode 100644 index 0000000..f3e3dad --- /dev/null +++ b/tests/multi-accel-consumer/src/main.cpp @@ -0,0 +1,4 @@ +extern "C" int cuda_anchor(); +extern "C" int sycl_anchor(); +// Exercise both host objects without requiring a physical GPU or a queue. +int main(){return cuda_anchor()+sycl_anchor()==3?0:1;} diff --git a/tests/plugin-logic/build.mcpp b/tests/plugin-logic/build.mcpp index 7ac9292..a7c1a03 100644 --- a/tests/plugin-logic/build.mcpp +++ b/tests/plugin-logic/build.mcpp @@ -14,6 +14,13 @@ import mcpp.rules.qt; import mcpp.plugins.tool; import mcpp.plugins.platform; import mcpp.plugins; +import mcpp.plugins.accel; +import mcpp.plugins.progress; +import mcpp.plugins.progress.writer; +import mcpp.rules.cuda; +import mcpp.rules.sycl; +import mcpp.rules.hip; +import mcpp.rules.spirv; namespace t = mcpp::plugins::testing; namespace ts = mcpp::plugins::toolset; @@ -139,6 +146,148 @@ mcpp::plugins::tool::found tool_resolve(mcpp::plugins::tool::choice c = {}) { int main(int argc, char** argv) { return t::run(argc, argv, { + { "surface: equal-size payload changes reach assembly and identical bytes keep its timestamp", + t::row::linux_gcc().file("pkg/payload.bin", "abcd"), + [] { + namespace sf = mcpp::plugins::surface; + const auto root=std::filesystem::path(mcpp::manifest_dir()); + sf::options options; + options.module_name="fixture.embed"; + options.out_dir=mcpp::out_dir(); + options.store=sf::storage::object; + options.target_os="linux"; + options.has_gas_assembler=true; + const std::vector items{{.identifier="data",.payload_path=(root/"payload.bin").string()}}; + if (!sf::write(items,options,sf::half::body)) return 1; + const auto file=sf::outputs(options).assembly_file; + auto read=[&]{std::ifstream in(file);return std::string(std::istreambuf_iterator(in),{});}; + const auto before=read(); + std::ofstream(root/"payload.bin",std::ios::binary|std::ios::trunc)<<"abce"; + if (!sf::write(items,options,sf::half::body) || read()==before) return 2; + const auto sentinel=std::filesystem::file_time_type::clock::now()-std::chrono::hours(1); + std::filesystem::last_write_time(file,sentinel); + if (!sf::write(items,options,sf::half::body)) return 3; + return std::filesystem::last_write_time(file)==sentinel ? 0 : 4; + }, + [](const t::result& r,t::checker& c){c.expect(r.exit_code==0,"the body tracks payload content without rewriting unchanged source");} }, + + { "progress: no channel leaves work independent of telemetry", + t::row::linux_gcc(), + [] { + mcpp::plugins::progress::writer writer; + return !writer.enabled() && !writer.report({.phase="configure"}) ? 0 : 1; + }, + [](const t::result& r, t::checker& c) { c.expect(r.exit_code == 0, "absence is a no-op"); } }, + { "progress: snapshots replace atomically and publishing is rate limited", + t::row::linux_gcc().set("MCPP_ACTION_PROGRESS_FILE", "{root}/telemetry/state.json") + .file("telemetry/keep"), + [] { + mcpp::plugins::progress::writer writer; + if (!writer.enabled() || !writer.report({.phase="configure"})) return 1; + if (writer.report({.phase="too-soon"})) return 2; + std::this_thread::sleep_for(std::chrono::milliseconds(210)); + return writer.report({.phase="build",.done=3,.total=9}) ? 0 : 3; + }, + [](const t::result& r, t::checker& c) { + c.expect(r.exit_code == 0, "two bounded reports succeeded"); + const auto text=r.file("telemetry/state.json"); + c.expect(text.find("\"seq\":1") != std::string::npos + && text.find("\"phase\":\"build\"") != std::string::npos + && text.find("configure") == std::string::npos, "only the latest snapshot remains"); + } }, + { "progress: invalid counts and terminal controls are refused", + t::row::linux_gcc().set("MCPP_ACTION_PROGRESS_FILE", "{root}/telemetry/state.json") + .file("telemetry/keep"), + [] { + mcpp::plugins::progress::writer writer; + return !writer.report({.phase="bad\nphase"}) + && !writer.report({.phase="build",.done=4,.total=3}) ? 0 : 1; + }, + [](const t::result& r, t::checker& c) { + c.expect(r.exit_code == 0, "invalid optional data was refused"); + c.expect(r.file("telemetry/state.json").empty(), "invalid telemetry wrote no document"); + } }, + + { "accel: targets stay on their implementation across internal commas", + t::row::linux_gcc(), + [] { + const auto parsed = mcpp::plugins::accel::parse( + "sycl+{spir64,nvptx64-sm_90,nvptx64-sm_89},cuda12.9+{sm_90,sm_89}"); + return parsed && parsed->size() == 2 && (*parsed)[0].backend == "cuda" + && (*parsed)[0].archs.size() == 2 && (*parsed)[1].backend == "sycl" + && (*parsed)[1].archs.size() == 3 ? 0 : 1; + }, + [](const t::result& r, t::checker& c) { c.expect(r.exit_code == 0, "member ownership survives set commas"); } }, + + { "accel: implementation membership is exact rather than a prefix", + t::row::linux_gcc(), + [] { + const char* text = "cudaextra+{x},syclish+{y},hipster+{z},vulkanother1.0"; + return !mcpp::rules::cuda::parse_target(text).present + && !mcpp::rules::sycl::parse_target(text).sycl + && !mcpp::rules::hip::parse_target(text).hip + && !mcpp::rules::spirv::parse_target(text).present ? 0 : 1; + }, + [](const t::result& r, t::checker& c) { c.expect(r.exit_code == 0, "no rule takes another implementation's prefix"); } }, + + { "sycl: multiple NVIDIA targets do not select a CUDA implementation", + t::row::linux_gcc(), + [] { + const auto a = mcpp::rules::sycl::parse_target("sycl+{nvptx64-sm_90,nvptx64-sm_89}"); + return a.error.empty() && a.sycl && !a.portable && !a.legacy_targets + && a.cuda_archs == std::vector{"sm_89", "sm_90"} + && !mcpp::rules::cuda::parse_target("sycl+{nvptx64-sm_90,nvptx64-sm_89}").present ? 0 : 1; + }, + [](const t::result& r, t::checker& c) { c.expect(r.exit_code == 0, "the NVIDIA target belongs to SYCL"); } }, + + { "sycl: generic and NVIDIA forms coexist in one request", + t::row::linux_gcc(), + [] { + const auto a = mcpp::rules::sycl::parse_target("sycl+{spir64,nvptx64-sm_89}"); + return a.error.empty() && a.portable && a.cuda_archs.size() == 1 ? 0 : 1; + }, + [](const t::result& r, t::checker& c) { c.expect(r.exit_code == 0, "both forms are preserved"); } }, + + { "sycl: explicit targets are not replaced by a coexisting CUDA request", + t::row::linux_gcc(), + [] { + const auto a = mcpp::rules::sycl::parse_target("cuda12.9+{sm_90},sycl+{nvptx64-sm_89}"); + return a.error.empty() && !a.legacy_targets + && a.cuda_archs == std::vector{"sm_89"} ? 0 : 1; + }, + [](const t::result& r, t::checker& c) { c.expect(r.exit_code == 0, "each implementation keeps its targets"); } }, + + { "sycl: the old explicit two-member spelling retains its migration boundary", + t::row::linux_gcc(), + [] { + const auto a = mcpp::rules::sycl::parse_target("sycl,cuda12.9+{sm_89,sm_90}"); + return a.error.empty() && a.legacy_targets && a.cuda_archs.size() == 2 + && mcpp::rules::cuda::parse_target("sycl,cuda12.9+{sm_89,sm_90}").present ? 0 : 1; + }, + [](const t::result& r, t::checker& c) { c.expect(r.exit_code == 0, "CUDA remains selected in the legacy spelling"); } }, + + { "sycl: an unknown target is refused instead of defaulting to SPIR-V", + t::row::linux_gcc(), + [] { return mcpp::rules::sycl::parse_target("sycl+{unknown-device}").error.empty() ? 1 : 0; }, + [](const t::result& r, t::checker& c) { c.expect(r.exit_code == 0, "unknown target has a diagnostic"); } }, + + { "hip: AMD target sets retain all targets", + t::row::linux_gcc(), + [] { + const auto a = mcpp::rules::hip::parse_target("hip6.4+{gfx942,gfx1100}"); + return a.error.empty() && a.hip && a.amd_archs.size() == 2 ? 0 : 1; + }, + [](const t::result& r, t::checker& c) { c.expect(r.exit_code == 0, "HIP does not split the set into members"); } }, + + { "accel: malformed and duplicate requests are diagnosed before a rule can no-op", + t::row::linux_gcc(), + [] { + for (auto text : {"sycl+{sm_89", "sycl+{sm_89}junk", "sycl+{a,,b}", "sycl,sycl"}) + if (mcpp::plugins::accel::parse(text)) return 1; + return 0; + }, + [](const t::result& r, t::checker& c) { c.expect(r.exit_code == 0, "all malformed requests have an error"); } }, + // ── deps-vcpkg ────────────────────────────────────────────────────── { "vcpkg: a Visual Studio toolset is selected through its instance", with_vcpkg(t::row::windows_visual_studio()), @@ -290,6 +439,45 @@ int main(int argc, char** argv) { } }, // ── deps-cmake ────────────────────────────────────────────────────── + { "cmake: immutable generator selections have distinct cache identities", + t::row::linux_gcc(), + [] { + using mcpp::deps::cmake::generator_key; + const auto key = [](std::initializer_list args) { + return generator_key(std::span(args.begin(), args.size())); + }; + std::printf("generator-diff=%d\n", key({"-G", "Ninja"}) != key({"-G", "Unix Makefiles"})); + std::printf("joined-same=%d\n", key({"-GNinja", "-Ax64"}) == key({"-G", "Ninja", "-A", "x64"})); + std::printf("typed-same=%d\n", key({"-DCMAKE_GENERATOR:STRING=Ninja"}) == key({"-D", "CMAKE_GENERATOR=Ninja"})); + std::printf("instance-diff=%d\n", key({"-DCMAKE_GENERATOR_INSTANCE=one"}) != key({"-DCMAKE_GENERATOR_INSTANCE=two"})); + std::printf("ordinary-same=%d\n", key({"-DBUILD_TESTING=ON"}) == key({"-DBUILD_TESTING=OFF"})); + return 0; + }, + [](const t::result& r, t::checker& c) { + for (const auto* name : {"generator-diff=", "joined-same=", "typed-same=", "instance-diff=", "ordinary-same="}) + c.expect(r.has_line(name, {"1"}), name); + } }, + + { "cmake: another target uses a contract toolchain without flattening argv", + with_subproject(t::row::linux_gcc().set("MCPP_TARGET", "aarch64-linux-gnu") + .set("MCPP_TARGET_ARCH", "aarch64")), + [] { return cmake_use(); }, + [](const t::result& r, t::checker& c) { + c.expect(r.exit_code == 0, "the member succeeds"); + const auto script = r.file("out/deps-cmake/greet/greet.cmake"); + const auto toolchain = r.file("out/deps-cmake/greet/greet-contract.cmake"); + c.expect(script.find("-DCMAKE_TOOLCHAIN_FILE=") != std::string::npos, + "CMake and its probes load the same contract toolchain"); + c.expect(script.find("-DCMAKE_C_FLAGS=") == std::string::npos, + "contract arguments are not flattened into a native flags string"); + c.expect(toolchain.find("add_compile_options") != std::string::npos + && toolchain.find("add_link_options") != std::string::npos, + "native CMake options carry response-file references"); + c.expect(r.line("mcpp:action=", {"deps-cmake:greet"}).find("greet-contract.cmake") + != std::string::npos, + "the generated toolchain is an action input"); + } }, + { "cmake: a Visual Studio toolset is named, with Ninja, by default", with_subproject(t::row::windows_visual_studio("14.43.34808", "14.44.35207")), [] { return cmake_use(); }, diff --git a/tests/plugin-logic/mcpp.toml b/tests/plugin-logic/mcpp.toml index b5b4c70..9474215 100644 --- a/tests/plugin-logic/mcpp.toml +++ b/tests/plugin-logic/mcpp.toml @@ -18,7 +18,7 @@ modules = true import_std = true [build-dependencies.mcpp] -plugins = { path = "../..", features = ["deps-vcpkg", "deps-cmake", "rules-qt", "plugins-platform", "plugins-testing"], host-module = true } +plugins = { path = "../..", features = ["plugins-progress", "deps-vcpkg", "deps-cmake", "rules-qt", "plugins-platform", "plugins-testing", "rules-cuda", "rules-sycl", "rules-hip", "rules-spirv"], host-module = true } [targets.plugin-logic] kind = "bin" diff --git a/tests/progress-worker/mcpp.toml b/tests/progress-worker/mcpp.toml new file mode 100644 index 0000000..52599c2 --- /dev/null +++ b/tests/progress-worker/mcpp.toml @@ -0,0 +1,8 @@ +[package] +name = "progress-worker" +version = "0.1.0" +[dependencies.mcpp] +plugins = { path = "../..", features = ["plugins-progress"] } +[targets.progress-worker] +kind = "bin" +main = "src/main.cpp" diff --git a/tests/progress-worker/src/main.cpp b/tests/progress-worker/src/main.cpp new file mode 100644 index 0000000..ff4afaf --- /dev/null +++ b/tests/progress-worker/src/main.cpp @@ -0,0 +1,12 @@ +import std; +import mcpp.plugins.progress.writer; +int main(int argc, char**) { + const bool probe = argc > 1; + mcpp::plugins::progress::writer writer; + if (!writer.enabled()) return probe ? 3 : 0; + const mcpp::plugins::progress::snapshot start{.phase="configure"}; + const bool began = writer.report(start); + std::this_thread::sleep_for(std::chrono::milliseconds(210)); + const bool finished = writer.report({.phase="compile",.done=1,.total=1}); + return probe && (!began || !finished) ? 2 : 0; +} diff --git a/tests/progress-worker/verify.py b/tests/progress-worker/verify.py new file mode 100644 index 0000000..b1d641c --- /dev/null +++ b/tests/progress-worker/verify.py @@ -0,0 +1,44 @@ +"""Exercise the standalone writer's environment and file contract on this host.""" + +import ctypes +import json +import os +from pathlib import Path +import subprocess +import sys +import tempfile + + +compiler = str(Path(sys.argv[1]).resolve()) +project = Path(__file__).resolve().parent +environment = os.environ.copy() +environment.pop("MCPP_ACTION_PROGRESS_FILE", None) +environment.pop("MCPP_ACTION_PROGRESS_ROOT", None) + +if os.name == "nt": + code_page = ctypes.windll.kernel32.GetACP() + print(f"Windows ACP: {code_page}", flush=True) + if code_page == 65001: + print("UTF-8 ACP: this run validates paths without reproducing ANSI loss.", flush=True) + + +def run(env, probe=False): + command = [compiler, "run"] + if probe: + command += ["--", "probe"] + subprocess.run(command, cwd=project, env=env, check=True, timeout=600) + + +run(environment) +with tempfile.TemporaryDirectory(prefix="mcpp-progress-") as directory: + root = Path(directory) / "channel-通道 Ω" + root.mkdir() + channel = root / "snapshot.json" + run(dict(environment, MCPP_ACTION_PROGRESS_FILE=str(channel)), probe=True) + snapshot = json.loads(channel.read_text(encoding="utf-8")) + assert snapshot == dict(v=1, seq=1, phase="compile", done=1, total=1), snapshot + # A directory cannot be replaced with the snapshot file. Telemetry failure + # remains optional for a worker whose actual task succeeds. + run(dict(environment, MCPP_ACTION_PROGRESS_FILE=str(root))) + +print("PASS: disabled, Unicode channel and optional write failure", flush=True) diff --git a/tests/sycl-consumer/mcpp.toml b/tests/sycl-consumer/mcpp.toml index 580234a..7deb03c 100644 --- a/tests/sycl-consumer/mcpp.toml +++ b/tests/sycl-consumer/mcpp.toml @@ -3,9 +3,8 @@ name = "sycl-consumer" namespace = "example" version = "0.1.0" description = "A SYCL kernel behind a seam module, reaching an NVIDIA device, with a CPU fallback" -# BOTH, for the reason the HIP consumer records: the glob below names the -# NVIDIA target the SYCL unit is compiled for, so `cuda` is a backend this -# package's sources are written for. +# The repository keeps a CUDA source group, but this build selects SYCL. +# The NVIDIA code target belongs to the SYCL request. accelerators = ["sycl", "cuda"] [language] @@ -55,18 +54,17 @@ libstdcxx = "16.1.0.1" # and C libraries the SYCL unit compiles against -- dpcpp's clang is not the # clang mcpp resolved and is configured with neither, so left alone its search # list reaches /usr/include with no ecosystem glibc on it); and cuda-nvcc, -# whose libdevice the NVIDIA back end needs -- declared only when the accel -# actually names cuda, so a SYCL build for another target does not pay for it. +# whose libdevice the NVIDIA back end needs -- requested only by a SYCL NVIDIA +# target, so a different SYCL target does not pay for it. [build] -# Two chunks: the programming model, and the device. Written `sycl` alone, the -# unit is compiled to SPIR-V and the runtime picks a device at run time; with -# the second chunk it is compiled ahead of time for that architecture. -accel = "sycl, cuda12.9+{sm_89}" +# One selected implementation, with its own NVIDIA device-code target. +accel = "sycl+{nvptx64-sm_89}" sources = [ "src/*.cppm", "src/*.cpp", - { glob = "src/kernels/**/*.sycl", accel = "sycl, cuda12.9+{sm_89}" }, + { glob = "src/kernels/**/*.sycl", accel = "sycl" }, + { glob = "src/cuda/**/*.cu", accel = "cuda" }, ] include_dirs = ["include"] diff --git a/tests/sycl-consumer/src/cuda/unused.cu b/tests/sycl-consumer/src/cuda/unused.cu new file mode 100644 index 0000000..48c1ee1 --- /dev/null +++ b/tests/sycl-consumer/src/cuda/unused.cu @@ -0,0 +1,3 @@ +// A CUDA implementation retained beside SYCL. A pure SYCL build must +// leave it out without requiring a CUDA-source rule to consume it. +extern "C" __global__ void unused_cuda_kernel() {} diff --git a/tests/sycl-consumer/verify-images.py b/tests/sycl-consumer/verify-images.py new file mode 100644 index 0000000..a36e19e --- /dev/null +++ b/tests/sycl-consumer/verify-images.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python3 +"""Check actual rule outputs; no device or installed GPU driver is required.""" +from pathlib import Path +import shlex +import subprocess +import sys +import tempfile + +expected = sys.argv[1].split(',') +cuda_target = next((arg.split('=', 1)[1] for arg in sys.argv[2:] + if arg.startswith('--cuda-target=')), None) +graph = max(Path('target').glob('*/*/build.ninja'), key=lambda p: p.stat().st_mtime) +text = graph.read_text() +assert 'src/cuda/unused.cu' not in text, 'a pure SYCL request compiled the retained CUDA group' +commands = [line.strip().removeprefix('command = ') for line in text.splitlines() + if line.strip().startswith('command = ') and '-fsycl-link ' in line] +assert len(commands) == 1, 'expected one rule device-link command' +command = shlex.split(commands[0]) +with tempfile.TemporaryDirectory(prefix='sycl-images-') as temporary: + work = Path(temporary) + # Keep the rule's exact compiler, flags and inputs; save intermediates from + # the device link so the test verifies images rather than just flag spelling. + command.insert(1, '-save-temps=obj') + command[command.index('-o') + 1] = str(work / 'device-link.o') + subprocess.run(command, cwd=work, check=True, timeout=180) + images = [] + for table in work.glob('*.table'): + lines = table.read_text().splitlines() + if not lines or lines[0] != '[Code|Properties|Symbols]': + continue + for line in lines[1:]: + path = Path(line.split('|')[0]) + assert path.is_file() and path.stat().st_size > 100, str(path) + images.append((table.name, path.read_bytes()[:4])) + for target in expected: + if target == 'spir64': + assert any(magic == bytes.fromhex('03022307') for _, magic in images), images + else: + assert any(target in name and magic == bytes.fromhex('50ed55ba') + for name, magic in images), (target, images) + if cuda_target: + cuda_commands = [line.strip().removeprefix('command = ') for line in text.splitlines() + if line.strip().startswith('command = ') and ' -x cuda ' in line] + assert len(cuda_commands) == 1, 'expected a separate CUDA compilation action' + cuda = shlex.split(cuda_commands[0]) + cuda.insert(1, '-save-temps=obj') + cuda[cuda.index('-o') + 1] = str(work / 'cuda.o') + subprocess.run(cuda, cwd=work, check=True, timeout=180) + ptx = [file.read_text() for file in work.glob('*.s')] + assert any('.target ' + cuda_target in file for file in ptx), cuda_target + print('PASS: the independent CUDA action emits', cuda_target) + print('PASS: actual device images:', ', '.join(expected)) diff --git a/tests/sycl-library-consumer/app/mcpp.toml b/tests/sycl-library-consumer/app/mcpp.toml new file mode 100644 index 0000000..65e248c --- /dev/null +++ b/tests/sycl-library-consumer/app/mcpp.toml @@ -0,0 +1,30 @@ +[package] +name = "app" +version = "0.1.0" +description = "A program that links the SYCL library built for two NVIDIA architectures" +accelerators = ["sycl"] + +[language] +standard = "c++23" +modules = true +import_std = true + +[toolchain] +default = "llvm@22.1.8" + +[dependencies.example] +saxpy = { path = "../saxpy" } + +[target.'cfg(accelerator = "sycl")'.dependencies.compat] +sycl-runtime = "2026.09.07" + +[target.'cfg(accelerator = "sycl")'.dependencies.gcc] +libstdcxx = "16.1.0.1" + +[build] +accel = "sycl+{nvptx64-sm_89,nvptx64-sm_90}" +sources = ["src/*.cpp"] + +[targets.app] +kind = "bin" +main = "src/main.cpp" diff --git a/tests/sycl-library-consumer/app/src/main.cpp b/tests/sycl-library-consumer/app/src/main.cpp new file mode 100644 index 0000000..d158891 --- /dev/null +++ b/tests/sycl-library-consumer/app/src/main.cpp @@ -0,0 +1,12 @@ +import std; +import saxpy; + +int main() { + const std::vector x{1, 2, 3, 4}, y{10, 20, 30, 40}; + auto out = app::saxpy(2.0f, x, y); + if (!out) { std::println("device unavailable"); return 1; } + for (auto v : *out) std::print("{} ", v); + std::println(""); + const std::vector want{12, 24, 36, 48}; + return *out == want ? 0 : 1; +} diff --git a/tests/sycl-library-consumer/saxpy/build.mcpp b/tests/sycl-library-consumer/saxpy/build.mcpp new file mode 100644 index 0000000..475d740 --- /dev/null +++ b/tests/sycl-library-consumer/saxpy/build.mcpp @@ -0,0 +1,9 @@ +import std; +import mcpp; +import mcpp.rules.sycl; + +int main() { + mcpp::rules::sycl::options opt; + opt.includes = { "include" }; + return mcpp::rules::sycl::compile(opt) ? 0 : 1; +} diff --git a/tests/sycl-library-consumer/saxpy/include/saxpy/saxpy.h b/tests/sycl-library-consumer/saxpy/include/saxpy/saxpy.h new file mode 100644 index 0000000..8d82d97 --- /dev/null +++ b/tests/sycl-library-consumer/saxpy/include/saxpy/saxpy.h @@ -0,0 +1,21 @@ +// The device island's interface. +// +// `extern "C"` and free of standard-library types, on purpose. The island is +// compiled by nvcc driving a host compiler that mcpp did not choose, so the two +// sides do not share a C++ ABI and must not exchange anything that depends on +// one. Keeping the boundary this narrow is also what lets the island publish a +// C-surface compatibility tag. +#ifndef MCPP_EXAMPLE_SAXPY_H +#define MCPP_EXAMPLE_SAXPY_H + +#ifdef __cplusplus +extern "C" { +#endif + +// out[i] = a * x[i] + y[i], computed on the device. Returns 0 on success. +int saxpy_device(float a, const float* x, const float* y, float* out, unsigned n); + +#ifdef __cplusplus +} +#endif +#endif diff --git a/tests/sycl-library-consumer/saxpy/mcpp.toml b/tests/sycl-library-consumer/saxpy/mcpp.toml new file mode 100644 index 0000000..dfbc590 --- /dev/null +++ b/tests/sycl-library-consumer/saxpy/mcpp.toml @@ -0,0 +1,39 @@ +[package] +name = "saxpy" +namespace = "example" +version = "0.1.0" +description = "A SYCL library of several NVIDIA architectures, behind a seam module" +accelerators = ["sycl"] + +[language] +standard = "c++23" +modules = true +import_std = true + +[toolchain] +default = "llvm@22.1.8" + +[build-dependencies.mcpp] +plugins = { path = "../../..", features = ["rules-sycl"], host-module = true } + +[target.'cfg(accelerator = "sycl")'.dependencies.compat] +sycl-runtime = "2026.09.07" + +[target.'cfg(accelerator = "sycl")'.dependencies.gcc] +libstdcxx = "16.1.0.1" + +# The library's own device link reads its own objects (rules/sycl.cppm): its +# images are registered inside the library, so a consumer's link never asks the +# SDK to unbundle an archive of several architectures (mcpp#899). +[build] +sources = [ + "src/*.cppm", + { glob = "src/kernels/**/*.sycl", accel = "sycl" }, +] +include_dirs = ["include"] + +[target.'cfg(not(accelerator = "sycl"))'.build] +sources = ["src/cpu/*.cpp"] + +[targets.saxpy] +kind = "lib" diff --git a/tests/sycl-library-consumer/saxpy/src/cpu/saxpy.cpp b/tests/sycl-library-consumer/saxpy/src/cpu/saxpy.cpp new file mode 100644 index 0000000..1881cbf --- /dev/null +++ b/tests/sycl-library-consumer/saxpy/src/cpu/saxpy.cpp @@ -0,0 +1,11 @@ +// The CPU implementation behind the same seam. Compiled only when the build +// asks for no accelerator (`mcpp build --no-accel`), through the +// `cfg(not(accelerator = "sycl"))` section of the manifest; the device island +// and this file define the same symbol and are never in one link. +#include "saxpy/saxpy.h" + +extern "C" int saxpy_device(float a, const float* x, const float* y, + float* out, unsigned n) { + for (unsigned i = 0; i < n; ++i) out[i] = a * x[i] + y[i]; + return 0; +} diff --git a/tests/sycl-library-consumer/saxpy/src/kernels/saxpy.sycl b/tests/sycl-library-consumer/saxpy/src/kernels/saxpy.sycl new file mode 100644 index 0000000..d50ccf7 --- /dev/null +++ b/tests/sycl-library-consumer/saxpy/src/kernels/saxpy.sycl @@ -0,0 +1,100 @@ +// The island, written in SYCL. Nothing here is visible to the module graph: a +// device translation unit is never scanned and never produces a BMI. +// +// THIS ISLAND USES THE C++ STANDARD LIBRARY, AND ITS SIBLINGS DELIBERATELY DO +// NOT. +// +// tests/cuda-consumer and tests/hip-consumer say in their own comments that +// their islands touch no standard library. A SYCL island cannot make that +// promise: SYCL IS a C++ library, `sycl::queue` and `sycl::buffer` are class +// templates over standard types, and `libsycl.so` is itself compiled against +// libstdc++ and throws `sycl::exception` on it. +// +// So the whole program uses libstdc++ (0.20.0): the manifest supplies it to the +// llvm build through `gcc.libstdcxx`, and `mcpp.rules.sycl` points the device +// compiler at the same GCC. One C++ runtime in the process is what lets a +// `catch (sycl::exception const&)` match what the runtime threw; until 0.19 +// the program linked libc++ beside it, and only `catch (...)` matched +// (mcpp-community/mcpp#838). The seam stays `extern "C"` for the reason the +// siblings give: it is the one place a backend can be exchanged. +// +// WHAT THIS ISLAND CAN AND CANNOT TURN INTO A RETURN CODE. +// +// It catches the synchronous half (no usable device, a rejected submit) and +// installs a handler for the asynchronous half, which SYCL otherwise delivers +// to a default handler that calls `std::terminate`. What it cannot catch is a +// missing device image: a build compiled to SPIR-V, run against a back end +// that does not consume SPIR-V, throws from inside the scheduler +// (`ProgramManager::getDeviceImage`) in neither of those two paths. That is +// why the manifest names the device -- and why `mcpp.rules.sycl` warns at +// build time when it does not. +#include "saxpy/saxpy.h" +#include +#include + +extern "C" int saxpy_device(float a, const float* x, const float* y, + float* out, unsigned n) { + // THE QUEUE TAKES AN ASYNC HANDLER, AND WITHOUT ONE THIS FUNCTION CANNOT + // KEEP ITS PROMISE. + // + // SYCL splits errors in two. A synchronous one -- no usable device, a bad + // submit -- is thrown where it happens and an ordinary catch sees it. An + // ASYNCHRONOUS one, which is most of what a device reports, is delivered + // to the queue's exception handler; a queue constructed WITHOUT one gets + // the default handler, and the default handler calls `std::terminate`. + // + // It is not something a catch can intercept, because it does not travel as + // an exception through this frame. Measured, on a build whose SPIR-V no + // device on this machine could consume, with every `try` below already in + // place: + // + // terminate called after throwing an instance of 'sycl::_V1::exception' + // terminate called recursively + // + // A device island that promises the rest of the program a return code has + // to install a handler, or the promise is only kept on machines where + // nothing goes wrong. + int rc = 0; + auto on_async = [&rc](sycl::exception_list errors) { + for (std::exception_ptr const& e : errors) { + try { std::rethrow_exception(e); } + catch (const sycl::exception& ex) { + std::fprintf(stderr, "sycl (async): %s\n", ex.what()); + } + } + rc = -1; + }; + + try { + sycl::queue q{on_async}; + { + sycl::buffer bx(x, sycl::range<1>(n)); + sycl::buffer by(y, sycl::range<1>(n)); + sycl::buffer bo(out, sycl::range<1>(n)); + // The catch is INSIDE the buffer scope on purpose: a buffer + // destructor blocks until the work that reads it has finished, and + // letting an exception propagate through three of those means + // unwinding through destructors that are themselves finishing + // failed work. Caught here, the scope ends normally. + try { + q.submit([&](sycl::handler& h) { + sycl::accessor ax(bx, h, sycl::read_only); + sycl::accessor ay(by, h, sycl::read_only); + sycl::accessor ao(bo, h, sycl::write_only, sycl::no_init); + h.parallel_for(sycl::range<1>(n), + [=](sycl::id<1> i) { ao[i] = a * ax[i] + ay[i]; }); + }); + q.wait_and_throw(); + } catch (const sycl::exception& e) { + std::fprintf(stderr, "sycl: %s\n", e.what()); + rc = -1; + } + } + } catch (const sycl::exception& e) { + // The queue itself: a machine with no device the runtime can use fails + // here, before any buffer exists. + std::fprintf(stderr, "sycl: no usable device: %s\n", e.what()); + rc = -1; + } + return rc; +} diff --git a/tests/sycl-library-consumer/saxpy/src/saxpy.cppm b/tests/sycl-library-consumer/saxpy/src/saxpy.cppm new file mode 100644 index 0000000..33195dd --- /dev/null +++ b/tests/sycl-library-consumer/saxpy/src/saxpy.cppm @@ -0,0 +1,27 @@ +// The seam. +// +// Its reason for existing is not that a device compiler rejects modules. It is +// that this is the one place a backend can be exchanged: the island underneath +// can become CUDA, SYCL or a CPU fallback without a single consumer of this +// module changing, and a `cfg(accelerator = ...)` section has somewhere to +// apply. Remove the seam and every importer becomes backend-specific. +module; +#include "saxpy/saxpy.h" +export module saxpy; +import std; + +export namespace app { + +// The device interface is raw pointers and a count because it has to be. The +// seam is where that becomes a C++ interface again. +std::optional> +saxpy(float a, std::span x, std::span y) { + if (x.size() != y.size()) return std::nullopt; + std::vector out(x.size()); + if (saxpy_device(a, x.data(), y.data(), out.data(), + static_cast(x.size())) != 0) + return std::nullopt; + return out; +} + +} // namespace app diff --git a/tools/embed.cppm b/tools/embed.cppm index 21863af..afa16c1 100644 --- a/tools/embed.cppm +++ b/tools/embed.cppm @@ -23,10 +23,6 @@ // large file across several arrays, and does not emit a `std::span` accessor. // Each is reachable from what it emits, and a tool in this collection earns a // feature by being needed by more than one consumer. -module; -#include -#include - export module mcpp.tools.embed; import std; diff --git a/tools/embed_main.cpp b/tools/embed_main.cpp index 65eb2a4..c5932fe 100644 --- a/tools/embed_main.cpp +++ b/tools/embed_main.cpp @@ -27,8 +27,6 @@ // own: `mcpp.plugins.surface` decides every name, and this reads what the rule // wrote down and calls it. The two forms cannot drift, because they are the // same code -- which is why that module imports only `std`. -#include -#include import std; import mcpp.plugins; diff --git a/tools/island.cppm b/tools/island.cppm index 7549df4..4e9a39a 100644 --- a/tools/island.cppm +++ b/tools/island.cppm @@ -8,9 +8,6 @@ // project does, directly from its own `build.mcpp`, exactly as it uses // `mcpp.tools.embed`. Leaving it in the lib root gave it to every consumer // whether or not they asked, while `tools-embed` next to it required a feature. -module; -#include - export module mcpp.tools.island; import std;