ci: cram: add diff-based component selection
Automatic component jobs need one reviewed source for path-to-board and path-to-component decisions. Keeping that policy beside the existing test buckets lets shared code classify changed tests consistently. Add the six-board selection matrix and all board-specific, cross-cutting, and explicit no-op paths. Validate the matrix while loading the manifest. Add a resolver for variable and label precedence, Git diffs, all-match component unions, full collapse, empty reasons, and report output. Exercise union routing, test classification, collapse, precedence, stand-down reasons, defensive label errors, and six-board reporting in an offline synthetic repository. Component coverage stays green. Assisted-by: codex:gpt-5.6-sol References: CI-9 Signed-off-by: Petr Štetiar <petr.stetiar@prplfoundation.org>
This commit is contained in:
Petr Štetiar
parent
d001612fee
commit
15c1b5bd17
@ -76,6 +76,74 @@ uses those paths to distinguish full, prplMesh, and prplOS coverage.
|
||||
`cram_component_bucketing.py` uses underscores because it is an importable
|
||||
module. Executable scripts follow the repository's hyphenated CLI convention.
|
||||
|
||||
## Diff-based selection
|
||||
|
||||
Merge request pipelines map changed paths to component lists for six hardware
|
||||
boards. Board-specific paths select their owning board. Cross-cutting paths
|
||||
select only the two priority boards, Freedom and OSPv2; the other four boards
|
||||
remain available for manual runs and weekly full coverage.
|
||||
|
||||
| Changed path | Board scope | Selection |
|
||||
|---|---|---|
|
||||
| `profiles/mvebu.yml`, `ipq807x.yml`, `mtk_filogic.yml`, or `airoha_an7581.yml` | Omnia, Haze, Mozart, or Valyrian respectively | `full` |
|
||||
| `profiles/qca_ipq95xx.yml` | Freedom | `full` |
|
||||
| `profiles/mxl_x86_osp_tb341_v2.yml` or `mxl_wlan_hostap_ng_wav700.yml` | OSPv2 | `full` |
|
||||
| `profiles/secure_boot_emmc.yml` | Freedom + OSPv2 | `full` |
|
||||
| `profiles/feed_amx.yml` | Freedom + OSPv2 | `full` |
|
||||
| `profiles/feed_prplmesh.yml` | Freedom + OSPv2 | `prplmesh` |
|
||||
| `profiles/feed_prplos.yml` | Freedom + OSPv2 | `prplos` |
|
||||
| `profiles/lcm.yml` | Freedom + OSPv2 | `lcm` |
|
||||
| `profiles/prpl_core.yml`, `prpl.yml`, or `security.yml` | Freedom + OSPv2 | `full` |
|
||||
| `profiles/mgmt.yml`, `cellular.yml`, or `thread.yml` | Freedom + OSPv2 | `prplos` |
|
||||
| `.gitlab/tests/cram/<board>/**` | owning board | touched test's bucket |
|
||||
| `.gitlab/tests/cram/generic/**` | Freedom + OSPv2 | touched test's bucket |
|
||||
| `.gitlab/tests/cram/post/**` or a sanity-listed test | Freedom + OSPv2 | `prplos` |
|
||||
| `.gitlab/testbed/<board>.yml` | owning board | `prplos` |
|
||||
| `.gitlab/testbed.yml` | Freedom + OSPv2 | `prplos` |
|
||||
| `package/**`, `target/**`, `toolchain/**`, `include/**`, `config/**`, `tools/**`, `feeds.conf.default`, `Makefile`, or `rules.mk` | Freedom + OSPv2 | `full` |
|
||||
|
||||
A `by-test` row asks the manifest which bucket owns each touched test. Multiple
|
||||
matches form an ordered component union; a `full` match or all three split
|
||||
buckets collapse to `full`. Unmatched paths, including documentation and other
|
||||
CI files, do not arm a board.
|
||||
|
||||
Profiles without a cram build consumer are explicit no-ops:
|
||||
`debug.yml`, `extender_full.yml`, `extender_minimal.yml`, `ipq40xx.yml`,
|
||||
`mxl_x86.yml`, `mxl_x86_sec.yml`, `mxl_x86_osp_tb341.yml`, `mxl_wlan.yml`,
|
||||
`mxl_wlan_hostap_ng.yml`, and `mxl_wlan_hostap_ng_gw.yml`. Changes to
|
||||
`components.yml` and its scripts are also no-ops because lint validates them
|
||||
and the author deliberately chooses any DUT run.
|
||||
|
||||
### Labels and precedence
|
||||
|
||||
The scoped labels `cram::full`, `cram::prplmesh`, `cram::lcm`, and
|
||||
`cram::prplos` override the computed selection. `cram::skip` suppresses
|
||||
automatic selection for the merge request. The labels share the `cram::`
|
||||
scope, so GitLab keeps at most one on an MR.
|
||||
|
||||
Selection precedence is:
|
||||
|
||||
1. A set `CRAM_COMPONENTS` pipeline variable stands the `[auto]` jobs down and
|
||||
controls the existing component jobs. `CRAM_COMPONENTS=none` matches no
|
||||
component.
|
||||
2. `cram::skip` stands `[auto]` jobs down; another `cram::*` label forces that
|
||||
component.
|
||||
3. Otherwise, the merge request diff is resolved through the matrix above.
|
||||
4. Any `full` result, or the union of all split buckets, collapses to `full`.
|
||||
|
||||
Labels are checked both when GitLab creates the pipeline and again when the
|
||||
job starts. A label added after pipeline creation but before the build is
|
||||
played therefore still controls the tests that run.
|
||||
|
||||
### QEMU end-state
|
||||
|
||||
CI-5 will add QEMU as a selection board without changing the resolver or job
|
||||
template. `test:qemu:smoke` remains an unfiltered MR pre-gate,
|
||||
`test:qemu:full` remains an always-full non-blocking equivalence run, and a
|
||||
future `cram qemu [auto]` receives cross-cutting selections. At that point
|
||||
`profiles/x86_64.yml`, currently an explicit no-op reserved for CI-5, maps to
|
||||
QEMU `full`.
|
||||
|
||||
## Selecting pipeline jobs
|
||||
|
||||
Non-scheduled pipelines create every component job and the non-DUT
|
||||
|
||||
@ -49,3 +49,69 @@ hw_only:
|
||||
- "generic/acceleration-plan-components/040-usb-filter.t"
|
||||
|
||||
# post/** is invoked separately for every component job and is not bucketed.
|
||||
|
||||
selection:
|
||||
schema: 1
|
||||
boards:
|
||||
wnc-freedom: {alias: Freedom, priority: true}
|
||||
mxl25641-hdk-6: {alias: OSPv2, priority: true}
|
||||
turris-omnia: {alias: Omnia, priority: false}
|
||||
prpl-haze: {alias: Haze, priority: false}
|
||||
arcadyan-mozart: {alias: Mozart, priority: false}
|
||||
nokia-valyrian: {alias: Valyrian, priority: false}
|
||||
paths:
|
||||
- {glob: profiles/mvebu.yml, boards: [turris-omnia], component: full}
|
||||
- {glob: profiles/ipq807x.yml, boards: [prpl-haze], component: full}
|
||||
- {glob: profiles/mtk_filogic.yml, boards: [arcadyan-mozart], component: full}
|
||||
- {glob: profiles/airoha_an7581.yml, boards: [nokia-valyrian], component: full}
|
||||
- {glob: profiles/qca_ipq95xx.yml, boards: [wnc-freedom], component: full}
|
||||
- {glob: profiles/mxl_x86_osp_tb341_v2.yml, boards: [mxl25641-hdk-6], component: full}
|
||||
- {glob: profiles/mxl_wlan_hostap_ng_wav700.yml, boards: [mxl25641-hdk-6], component: full}
|
||||
- {glob: profiles/secure_boot_emmc.yml, boards: priority, component: full}
|
||||
- {glob: profiles/feed_amx.yml, boards: priority, component: full}
|
||||
- {glob: profiles/feed_prplmesh.yml, boards: priority, component: prplmesh}
|
||||
- {glob: profiles/feed_prplos.yml, boards: priority, component: prplos}
|
||||
- {glob: profiles/lcm.yml, boards: priority, component: lcm}
|
||||
- {glob: profiles/prpl_core.yml, boards: priority, component: full}
|
||||
- {glob: profiles/prpl.yml, boards: priority, component: full}
|
||||
- {glob: profiles/mgmt.yml, boards: priority, component: prplos}
|
||||
- {glob: profiles/security.yml, boards: priority, component: full}
|
||||
- {glob: profiles/cellular.yml, boards: priority, component: prplos}
|
||||
- {glob: profiles/thread.yml, boards: priority, component: prplos}
|
||||
- {glob: .gitlab/tests/cram/generic/**, boards: priority, component: by-test}
|
||||
- {glob: .gitlab/tests/cram/post/**, boards: priority, component: by-test}
|
||||
- {glob: .gitlab/testbed.yml, boards: priority, component: prplos}
|
||||
- {glob: package/**, boards: priority, component: full}
|
||||
- {glob: target/**, boards: priority, component: full}
|
||||
- {glob: toolchain/**, boards: priority, component: full}
|
||||
- {glob: include/**, boards: priority, component: full}
|
||||
- {glob: config/**, boards: priority, component: full}
|
||||
- {glob: tools/**, boards: priority, component: full}
|
||||
- {glob: feeds.conf.default, boards: priority, component: full}
|
||||
- {glob: Makefile, boards: priority, component: full}
|
||||
- {glob: rules.mk, boards: priority, component: full}
|
||||
- {glob: .gitlab/tests/cram/turris-omnia/**, boards: [turris-omnia], component: by-test}
|
||||
- {glob: .gitlab/tests/cram/prpl-haze/**, boards: [prpl-haze], component: by-test}
|
||||
- {glob: .gitlab/tests/cram/arcadyan-mozart/**, boards: [arcadyan-mozart], component: by-test}
|
||||
- {glob: .gitlab/tests/cram/nokia-valyrian/**, boards: [nokia-valyrian], component: by-test}
|
||||
- {glob: .gitlab/tests/cram/wnc-freedom/**, boards: [wnc-freedom], component: by-test}
|
||||
- {glob: .gitlab/tests/cram/mxl25641-hdk-6/**, boards: [mxl25641-hdk-6], component: by-test}
|
||||
- {glob: .gitlab/testbed/turris-omnia.yml, boards: [turris-omnia], component: prplos}
|
||||
- {glob: .gitlab/testbed/prpl-haze.yml, boards: [prpl-haze], component: prplos}
|
||||
- {glob: .gitlab/testbed/arcadyan-mozart.yml, boards: [arcadyan-mozart], component: prplos}
|
||||
- {glob: .gitlab/testbed/nokia-valyrian.yml, boards: [nokia-valyrian], component: prplos}
|
||||
- {glob: .gitlab/testbed/wnc-freedom.yml, boards: [wnc-freedom], component: prplos}
|
||||
- {glob: .gitlab/testbed/mxl25641-hdk-6.yml, boards: [mxl25641-hdk-6], component: prplos}
|
||||
- {glob: profiles/debug.yml, component: noop}
|
||||
- {glob: profiles/extender_full.yml, component: noop}
|
||||
- {glob: profiles/extender_minimal.yml, component: noop}
|
||||
- {glob: profiles/x86_64.yml, component: noop}
|
||||
- {glob: profiles/ipq40xx.yml, component: noop}
|
||||
- {glob: profiles/mxl_x86.yml, component: noop}
|
||||
- {glob: profiles/mxl_x86_sec.yml, component: noop}
|
||||
- {glob: profiles/mxl_x86_osp_tb341.yml, component: noop}
|
||||
- {glob: profiles/mxl_wlan.yml, component: noop}
|
||||
- {glob: profiles/mxl_wlan_hostap_ng.yml, component: noop}
|
||||
- {glob: profiles/mxl_wlan_hostap_ng_gw.yml, component: noop}
|
||||
- {glob: .gitlab/tests/cram/components.yml, component: noop}
|
||||
- {glob: .gitlab/tests/cram/scripts/**, component: noop}
|
||||
|
||||
@ -5,6 +5,7 @@ from __future__ import annotations
|
||||
|
||||
import glob
|
||||
import os
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Dict, Iterable, List, Mapping, Sequence, Set, Tuple
|
||||
@ -15,12 +16,39 @@ import yaml
|
||||
SCHEMA_VERSION = 1
|
||||
COMPONENT_NAMES = ("lcm", "prplmesh")
|
||||
RESERVED_NAMES = {"full", "prplos", "sanity", "smoke", "hw_only"}
|
||||
SELECTION_COMPONENT_ORDER = ("prplmesh", "lcm", "prplos")
|
||||
|
||||
|
||||
class BucketingError(Exception):
|
||||
"""An input or bucketing invariant is invalid."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SelectionBoard:
|
||||
"""A DUT board available to diff-based selection."""
|
||||
|
||||
name: str
|
||||
alias: str
|
||||
priority: bool
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SelectionPath:
|
||||
"""One repository path mapping in the selection matrix."""
|
||||
|
||||
glob: str
|
||||
boards: Tuple[str, ...]
|
||||
component: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Selection:
|
||||
"""Validated diff-based selection data."""
|
||||
|
||||
boards: Mapping[str, SelectionBoard]
|
||||
paths: Tuple[SelectionPath, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Manifest:
|
||||
"""Validated component manifest and its expanded file matches."""
|
||||
@ -30,6 +58,7 @@ class Manifest:
|
||||
component_patterns: Mapping[str, Tuple[str, ...]]
|
||||
smoke_patterns: Tuple[str, ...]
|
||||
hw_only_patterns: Tuple[str, ...]
|
||||
selection: Selection
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@ -167,6 +196,136 @@ def _validate_include_section(
|
||||
return tuple(patterns)
|
||||
|
||||
|
||||
def _glob_regex(pattern: str) -> re.Pattern[str]:
|
||||
"""Compile a repository glob where ``*`` never crosses a slash."""
|
||||
|
||||
expression = ""
|
||||
index = 0
|
||||
while index < len(pattern):
|
||||
character = pattern[index]
|
||||
if character != "*":
|
||||
expression += re.escape(character)
|
||||
index += 1
|
||||
continue
|
||||
if pattern[index : index + 2] == "**":
|
||||
index += 2
|
||||
if index < len(pattern) and pattern[index] == "/":
|
||||
expression += "(?:.*/)?"
|
||||
index += 1
|
||||
else:
|
||||
expression += ".*"
|
||||
else:
|
||||
expression += "[^/]*"
|
||||
index += 1
|
||||
return re.compile(f"^{expression}$")
|
||||
|
||||
|
||||
def path_matches_glob(path: str, pattern: str) -> bool:
|
||||
"""Return whether a POSIX repository path matches a validated glob."""
|
||||
|
||||
return bool(_glob_regex(pattern).fullmatch(path))
|
||||
|
||||
|
||||
def _validate_selection(value: object) -> Selection:
|
||||
selection = _require_mapping(value, "selection")
|
||||
unknown = set(selection) - {"schema", "boards", "paths"}
|
||||
missing = {"schema", "boards", "paths"} - set(selection)
|
||||
if missing:
|
||||
raise BucketingError(
|
||||
f"selection is missing fields: {', '.join(sorted(missing))}"
|
||||
)
|
||||
if unknown:
|
||||
raise BucketingError(
|
||||
f"selection has unknown fields: {', '.join(sorted(unknown))}"
|
||||
)
|
||||
if selection["schema"] != SCHEMA_VERSION:
|
||||
raise BucketingError(
|
||||
f"selection.schema must be {SCHEMA_VERSION}, got "
|
||||
f"{selection['schema']!r}"
|
||||
)
|
||||
|
||||
raw_boards = _require_mapping(selection["boards"], "selection.boards")
|
||||
if not raw_boards:
|
||||
raise BucketingError("selection.boards must not be empty")
|
||||
boards: Dict[str, SelectionBoard] = {}
|
||||
aliases: Set[str] = set()
|
||||
for name, raw_board in raw_boards.items():
|
||||
board = _require_mapping(raw_board, f"selection.boards.{name}")
|
||||
if set(board) != {"alias", "priority"}:
|
||||
raise BucketingError(
|
||||
f"selection.boards.{name} must contain alias and priority"
|
||||
)
|
||||
alias = board["alias"]
|
||||
priority = board["priority"]
|
||||
if not isinstance(alias, str) or not alias:
|
||||
raise BucketingError(f"selection.boards.{name}.alias must be a string")
|
||||
if alias in aliases:
|
||||
raise BucketingError(f"duplicate selection board alias: {alias}")
|
||||
if not isinstance(priority, bool):
|
||||
raise BucketingError(f"selection.boards.{name}.priority must be a boolean")
|
||||
aliases.add(alias)
|
||||
boards[name] = SelectionBoard(name, alias, priority)
|
||||
|
||||
raw_paths = selection["paths"]
|
||||
if not isinstance(raw_paths, list) or not raw_paths:
|
||||
raise BucketingError("selection.paths must be a non-empty list")
|
||||
paths: List[SelectionPath] = []
|
||||
seen_globs: Set[str] = set()
|
||||
allowed_components = {"full", "prplmesh", "lcm", "prplos", "by-test", "noop"}
|
||||
for index, raw_path in enumerate(raw_paths):
|
||||
context = f"selection.paths[{index}]"
|
||||
path = _require_mapping(raw_path, context)
|
||||
if set(path) - {"glob", "boards", "component"}:
|
||||
raise BucketingError(f"{context} has unknown fields")
|
||||
if {"glob", "component"} - set(path):
|
||||
raise BucketingError(f"{context} requires glob and component")
|
||||
pattern = path["glob"]
|
||||
component = path["component"]
|
||||
if not isinstance(pattern, str) or not pattern:
|
||||
raise BucketingError(f"{context}.glob must be a non-empty string")
|
||||
posix_pattern = PurePosixPath(pattern)
|
||||
if (
|
||||
posix_pattern.is_absolute()
|
||||
or ".." in posix_pattern.parts
|
||||
or "\\" in pattern
|
||||
or any(character in pattern for character in "?[]")
|
||||
):
|
||||
raise BucketingError(f"{context}.glob is invalid: {pattern!r}")
|
||||
_glob_regex(pattern)
|
||||
if pattern in seen_globs:
|
||||
raise BucketingError(f"duplicate selection glob: {pattern}")
|
||||
seen_globs.add(pattern)
|
||||
if component not in allowed_components:
|
||||
raise BucketingError(f"{context}.component is invalid: {component!r}")
|
||||
|
||||
raw_targets = path.get("boards")
|
||||
if component == "noop":
|
||||
if raw_targets is not None:
|
||||
raise BucketingError(f"{context} noop must not name boards")
|
||||
targets: Tuple[str, ...] = ()
|
||||
elif raw_targets == "priority":
|
||||
targets = tuple(name for name, board in boards.items() if board.priority)
|
||||
if not targets:
|
||||
raise BucketingError(f"{context} priority selects no boards")
|
||||
elif isinstance(raw_targets, list) and raw_targets:
|
||||
if any(not isinstance(target, str) for target in raw_targets):
|
||||
raise BucketingError(f"{context}.boards must contain strings")
|
||||
targets = tuple(raw_targets)
|
||||
if len(targets) != len(set(targets)):
|
||||
raise BucketingError(f"{context}.boards contains duplicates")
|
||||
missing_targets = set(targets) - set(boards)
|
||||
if missing_targets:
|
||||
raise BucketingError(
|
||||
f"{context}.boards names unknown boards: "
|
||||
f"{', '.join(sorted(missing_targets))}"
|
||||
)
|
||||
else:
|
||||
raise BucketingError(f"{context}.boards must be priority or a list")
|
||||
paths.append(SelectionPath(pattern, targets, component))
|
||||
|
||||
return Selection(boards=boards, paths=tuple(paths))
|
||||
|
||||
|
||||
def load_manifest(path: Path) -> Manifest:
|
||||
"""Load and validate manifest structure without expanding its globs."""
|
||||
|
||||
@ -179,7 +338,14 @@ def load_manifest(path: Path) -> Manifest:
|
||||
raise BucketingError(f"cannot parse manifest {path}: {error}") from error
|
||||
|
||||
document = _require_mapping(raw, "manifest")
|
||||
expected_top_level = {"version", "sanity", "components", "smoke", "hw_only"}
|
||||
expected_top_level = {
|
||||
"version",
|
||||
"sanity",
|
||||
"components",
|
||||
"smoke",
|
||||
"hw_only",
|
||||
"selection",
|
||||
}
|
||||
missing = expected_top_level - set(document)
|
||||
unknown = set(document) - expected_top_level
|
||||
if missing:
|
||||
@ -221,12 +387,11 @@ def load_manifest(path: Path) -> Manifest:
|
||||
sanity_patterns = _validate_include_section(
|
||||
document["sanity"], "sanity", {"include"}
|
||||
)
|
||||
smoke_patterns = _validate_include_section(
|
||||
document["smoke"], "smoke", {"include"}
|
||||
)
|
||||
smoke_patterns = _validate_include_section(document["smoke"], "smoke", {"include"})
|
||||
hw_only_patterns = _validate_include_section(
|
||||
document["hw_only"], "hw_only", {"include"}
|
||||
)
|
||||
selection = _validate_selection(document["selection"])
|
||||
|
||||
return Manifest(
|
||||
path=path,
|
||||
@ -234,6 +399,7 @@ def load_manifest(path: Path) -> Manifest:
|
||||
component_patterns=component_patterns,
|
||||
smoke_patterns=smoke_patterns,
|
||||
hw_only_patterns=hw_only_patterns,
|
||||
selection=selection,
|
||||
)
|
||||
|
||||
|
||||
@ -241,7 +407,9 @@ def _relative_test_path(test_root: Path, path: Path) -> str:
|
||||
try:
|
||||
return path.relative_to(test_root).as_posix()
|
||||
except ValueError as error:
|
||||
raise BucketingError(f"test is outside test root {test_root}: {path}") from error
|
||||
raise BucketingError(
|
||||
f"test is outside test root {test_root}: {path}"
|
||||
) from error
|
||||
|
||||
|
||||
def discover_tests(test_root: Path, include_post: bool = False) -> List[Path]:
|
||||
@ -355,9 +523,7 @@ def partition_tests(
|
||||
rendered = "\n".join(
|
||||
f" - {path}: {', '.join(names)}" for path, names in overlaps
|
||||
)
|
||||
raise BucketingError(
|
||||
"tests match more than one partition bucket:\n" + rendered
|
||||
)
|
||||
raise BucketingError("tests match more than one partition bucket:\n" + rendered)
|
||||
return buckets
|
||||
|
||||
|
||||
@ -391,7 +557,67 @@ def select_component_tests(
|
||||
|
||||
sanity = list(buckets["sanity"])
|
||||
sanity_set = set(sanity)
|
||||
return sanity + [path for path in full if path in selected and path not in sanity_set]
|
||||
return sanity + [
|
||||
path for path in full if path in selected and path not in sanity_set
|
||||
]
|
||||
|
||||
|
||||
def classify_changed_test(manifest: Manifest, repository_path: str) -> str | None:
|
||||
"""Classify one changed cram test path for automatic selection."""
|
||||
|
||||
prefix = ".gitlab/tests/cram/"
|
||||
if not repository_path.startswith(prefix):
|
||||
raise BucketingError(f"not a cram test path: {repository_path}")
|
||||
relative = repository_path[len(prefix) :]
|
||||
if not relative.endswith(".t"):
|
||||
return None
|
||||
if relative.startswith("post/") or any(
|
||||
path_matches_glob(relative, pattern) for pattern in manifest.sanity_patterns
|
||||
):
|
||||
return "prplos"
|
||||
explicit = [
|
||||
component
|
||||
for component, patterns in manifest.component_patterns.items()
|
||||
if any(path_matches_glob(relative, pattern) for pattern in patterns)
|
||||
]
|
||||
if len(explicit) > 1:
|
||||
raise BucketingError(
|
||||
f"changed test matches multiple components: {repository_path}: "
|
||||
f"{', '.join(explicit)}"
|
||||
)
|
||||
return explicit[0] if explicit else "prplos"
|
||||
|
||||
|
||||
def select_changed_components(
|
||||
manifest: Manifest, board: str, changed_paths: Sequence[str]
|
||||
) -> tuple[str, bool]:
|
||||
"""Resolve changed paths to one board's ordered component selection."""
|
||||
|
||||
if board not in manifest.selection.boards:
|
||||
raise BucketingError(f"unknown selection board: {board}")
|
||||
selected: Set[str] = set()
|
||||
matched_for_board = False
|
||||
for changed_path in changed_paths:
|
||||
for mapping in manifest.selection.paths:
|
||||
if not path_matches_glob(changed_path, mapping.glob):
|
||||
continue
|
||||
if board not in mapping.boards:
|
||||
continue
|
||||
matched_for_board = True
|
||||
component = mapping.component
|
||||
if component == "by-test":
|
||||
component = classify_changed_test(manifest, changed_path)
|
||||
if component is None:
|
||||
continue
|
||||
if component == "full":
|
||||
return "full", False
|
||||
if component != "noop":
|
||||
selected.add(component)
|
||||
|
||||
if set(SELECTION_COMPONENT_ORDER).issubset(selected):
|
||||
return "full", False
|
||||
ordered = [name for name in SELECTION_COMPONENT_ORDER if name in selected]
|
||||
return ",".join(ordered), matched_for_board and not ordered
|
||||
|
||||
|
||||
def render_test_paths(paths: Sequence[Path]) -> str:
|
||||
|
||||
151
.gitlab/tests/cram/scripts/resolve-changed-components.py
Executable file
151
.gitlab/tests/cram/scripts/resolve-changed-components.py
Executable file
@ -0,0 +1,151 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Resolve changed repository paths to per-board cram components."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Sequence
|
||||
|
||||
from cram_component_bucketing import (
|
||||
BucketingError,
|
||||
Manifest,
|
||||
load_manifest,
|
||||
resolve_manifest_path,
|
||||
select_changed_components,
|
||||
)
|
||||
|
||||
|
||||
def parse_args(argv: Sequence[str]) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--test-root", required=True, type=Path)
|
||||
parser.add_argument("--manifest", required=True, type=Path)
|
||||
parser.add_argument("--base-ref", required=True)
|
||||
parser.add_argument("--head-ref", default="HEAD")
|
||||
target = parser.add_mutually_exclusive_group(required=True)
|
||||
target.add_argument("--board")
|
||||
target.add_argument("--report", action="store_true")
|
||||
parser.add_argument("--cram-components", default=None)
|
||||
parser.add_argument("--labels", default="")
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def _run_git(test_root: Path, arguments: Sequence[str]) -> str:
|
||||
command = ["git", "-C", os.fspath(test_root), *arguments]
|
||||
try:
|
||||
result = subprocess.run(
|
||||
command,
|
||||
check=True,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
)
|
||||
except FileNotFoundError as error:
|
||||
raise BucketingError("git is required for selection") from error
|
||||
except subprocess.CalledProcessError as error:
|
||||
detail = error.stderr.strip() or error.stdout.strip()
|
||||
raise BucketingError(
|
||||
f"command failed ({' '.join(command)}): {detail}"
|
||||
) from error
|
||||
return result.stdout
|
||||
|
||||
|
||||
def _changed_paths(test_root: Path, base_ref: str, head_ref: str) -> list[str]:
|
||||
output = _run_git(
|
||||
test_root,
|
||||
["diff", "--name-only", "--no-renames", f"{base_ref}..{head_ref}"],
|
||||
)
|
||||
return [path for path in output.splitlines() if path]
|
||||
|
||||
|
||||
def _label_override(labels: str) -> tuple[str | None, str | None]:
|
||||
values = [label.strip() for label in labels.split(",") if label.strip()]
|
||||
cram_labels = [label for label in values if label.startswith("cram::")]
|
||||
if len(cram_labels) > 1:
|
||||
raise BucketingError(
|
||||
"multiple cram selection labels: " + ", ".join(cram_labels)
|
||||
)
|
||||
if not cram_labels:
|
||||
return None, None
|
||||
label = cram_labels[0]
|
||||
value = label.removeprefix("cram::")
|
||||
if value == "skip":
|
||||
return "", "stand-down:label"
|
||||
if value not in {"full", "prplmesh", "lcm", "prplos"}:
|
||||
raise BucketingError(f"unknown cram selection label: {label}")
|
||||
return value, f"label:{value}"
|
||||
|
||||
|
||||
def _resolve_board(
|
||||
manifest: Manifest,
|
||||
board: str,
|
||||
changed_paths: Sequence[str] | None,
|
||||
cram_components: str | None,
|
||||
labels: str,
|
||||
) -> tuple[str, str]:
|
||||
if board not in manifest.selection.boards:
|
||||
raise BucketingError(f"unknown selection board: {board}")
|
||||
if cram_components is not None:
|
||||
return "", "stand-down:variable"
|
||||
label_selection, label_reason = _label_override(labels)
|
||||
if label_reason is not None:
|
||||
return label_selection or "", label_reason
|
||||
assert changed_paths is not None
|
||||
selection, drift = select_changed_components(manifest, board, changed_paths)
|
||||
if selection:
|
||||
return selection, "computed"
|
||||
return "", "DRIFT:coarse-match-without-component" if drift else "nothing-mapped"
|
||||
|
||||
|
||||
def _empty_reason(alias: str, reason: str) -> str:
|
||||
return f"nothing selected for {alias}: {reason}"
|
||||
|
||||
|
||||
def main(argv: Sequence[str] | None = None) -> int:
|
||||
args = parse_args(sys.argv[1:] if argv is None else argv)
|
||||
try:
|
||||
manifest_path = resolve_manifest_path(args.test_root, args.manifest)
|
||||
manifest = load_manifest(manifest_path)
|
||||
_, label_reason = _label_override(args.labels)
|
||||
needs_diff = args.cram_components is None and label_reason is None
|
||||
changed = (
|
||||
_changed_paths(args.test_root, args.base_ref, args.head_ref)
|
||||
if needs_diff
|
||||
else None
|
||||
)
|
||||
boards = tuple(manifest.selection.boards) if args.report else (args.board,)
|
||||
results = []
|
||||
for board in boards:
|
||||
selection, reason = _resolve_board(
|
||||
manifest,
|
||||
board,
|
||||
changed,
|
||||
args.cram_components,
|
||||
args.labels,
|
||||
)
|
||||
results.append((board, selection, reason))
|
||||
except BucketingError as error:
|
||||
print(f"error: {error}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
if args.report:
|
||||
for board, selection, reason in results:
|
||||
alias = manifest.selection.boards[board].alias
|
||||
rendered = selection or "(empty)"
|
||||
print(f"{alias}: {rendered} [{reason}]")
|
||||
if not selection:
|
||||
print(_empty_reason(alias, reason), file=sys.stderr)
|
||||
else:
|
||||
board, selection, reason = results[0]
|
||||
print(selection)
|
||||
if not selection:
|
||||
alias = manifest.selection.boards[board].alias
|
||||
print(_empty_reason(alias, reason), file=sys.stderr)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@ -1,4 +1,5 @@
|
||||
---
|
||||
# reserved: QEMU board (CI-5)
|
||||
profiles:
|
||||
- generic
|
||||
target: x86
|
||||
|
||||
Loading…
Reference in New Issue
Block a user