From e9e16622bfc00c5690fc5b87196f2218cb857965 Mon Sep 17 00:00:00 2001 From: "markus.schu" Date: Mon, 22 Dec 2025 15:12:35 +0100 Subject: [PATCH] inital setup for mle and sup.11 in aspice --- process/standards/aspice_40/aspice.rst | 3 +- process/standards/aspice_40/iic/iic-01.rst | 26 +++- process/standards/aspice_40/iic/iic-03.rst | 21 +++- process/standards/aspice_40/iic/iic-04.rst | 17 ++- process/standards/aspice_40/iic/iic-08.rst | 31 ++++- process/standards/aspice_40/iic/iic-11.rst | 10 +- process/standards/aspice_40/iic/iic-13.rst | 15 ++- process/standards/aspice_40/iic/iic-16.rst | 12 +- process/standards/aspice_40/iic/iic-19.rst | 12 +- process/standards/aspice_40/mle/index.rst | 22 ++++ process/standards/aspice_40/mle/mle.1.rst | 116 ++++++++++++++++++ process/standards/aspice_40/mle/mle.2.rst | 107 +++++++++++++++++ process/standards/aspice_40/mle/mle.3.rst | 101 ++++++++++++++++ process/standards/aspice_40/mle/mle.4.rst | 131 +++++++++++++++++++++ process/standards/aspice_40/sup/index.rst | 1 + process/standards/aspice_40/sup/sup.11.rst | 114 ++++++++++++++++++ 16 files changed, 721 insertions(+), 18 deletions(-) create mode 100644 process/standards/aspice_40/mle/index.rst create mode 100644 process/standards/aspice_40/mle/mle.1.rst create mode 100644 process/standards/aspice_40/mle/mle.2.rst create mode 100644 process/standards/aspice_40/mle/mle.3.rst create mode 100644 process/standards/aspice_40/mle/mle.4.rst create mode 100644 process/standards/aspice_40/sup/sup.11.rst diff --git a/process/standards/aspice_40/aspice.rst b/process/standards/aspice_40/aspice.rst index f65513da85..8caa666d6b 100644 --- a/process/standards/aspice_40/aspice.rst +++ b/process/standards/aspice_40/aspice.rst @@ -68,6 +68,7 @@ Tailoring man/index swe/index + mle/index sup/index spl/index reu/index @@ -418,7 +419,7 @@ Generic practices For guidance on how to perform process improvements see the Process Improvement process (PIM.3). -.. needextend:: "c.this_doc()" +.. needextend:: "c.this_doc()" :+tags: aspice40_gp3 Appendix diff --git a/process/standards/aspice_40/iic/iic-01.rst b/process/standards/aspice_40/iic/iic-01.rst index bc5e10b8a2..581cefbfa3 100644 --- a/process/standards/aspice_40/iic/iic-01.rst +++ b/process/standards/aspice_40/iic/iic-01.rst @@ -1,4 +1,4 @@ -.. +.. # ******************************************************************************* # Copyright (c) 2025 Contributors to the Eclipse Foundation # @@ -49,5 +49,25 @@ - The name of work products and an associated reference (to file, to tool artifact) - Configuration item attributes and properties -.. needextend:: "c.this_doc()" - :+tags: aspice40_iic01 +.. std_req:: 01-53 Trained ML model + :id: std_req__aspice_40__iic-01-53 + :status: valid + + - The trained ML model is the output of the training process. It consists of the software representing the ML architecture, the set of weights which were optimized during the training, and the final set of hyperparameters. + +.. std_req:: 01-54 Hyperparameter + :id: std_req__aspice_40__iic-01-54 + :status: valid + + + - Hyperparameters are used to control the ML model which has to be trained, e.g.: + - Learn rate of training + - Scaling of network (number of layers or neurons per layer) + - Loss function + - Minimum characteristics: + - Description + - Initial value + - Final value upon communicating the results of the ML training + +.. needextend:: "c.this_doc()" + :+tags: aspice40_iic01 diff --git a/process/standards/aspice_40/iic/iic-03.rst b/process/standards/aspice_40/iic/iic-03.rst index 446ee038af..b5af9d3071 100644 --- a/process/standards/aspice_40/iic/iic-03.rst +++ b/process/standards/aspice_40/iic/iic-03.rst @@ -59,5 +59,22 @@ - reviews such as optical inspections à findings record - analyses: values -.. needextend:: "c.this_doc()" - :+tags: aspice40_iic03 +.. std_req:: 03-51 ML data set + :id: std_req__aspice_40__iic-03-51 + :status: valid + + - Selection of ML Data for e.g., ML model training (ML Training and Validation Data Set) or test of the trained and deployed ML model (ML Test Data Set). + +.. std_req:: 03-53 ML data + :id: std_req__aspice_40__iic-03-53 + :status: valid + + - Datum to be used for Machine Learning. The datum has to be attributed by metadata, e.g., unique ID and data characteristics. Examples: + - Visual data like a photo or videos (but a video could also be considered as sequence of photos depending on the intended use) + - Audio recording + - Sensor data + - Data created by an algorithm + - Data might be processed to create additional data. E.g., processing could add noise, change colors or merge pictures. + +.. needextend:: "c.this_doc()" + :+tags: aspice40_iic03 diff --git a/process/standards/aspice_40/iic/iic-04.rst b/process/standards/aspice_40/iic/iic-04.rst index 60458a4194..6abc315387 100644 --- a/process/standards/aspice_40/iic/iic-04.rst +++ b/process/standards/aspice_40/iic/iic-04.rst @@ -76,6 +76,19 @@ - semi-formal languages (e.g, UML, SysML) - formal languages (e.g, model-based approach) +.. std_req:: 04-51 ML architecture + :id: std_req__aspice_40__iic-04-51 + :status: valid + + - An ML architecture is basically a special part of a software architecture (see 04-04). Additionally + + - ML architecture describes the overall structure of the ML-based software element + - ML architecture specifies ML architectural elements including an ML model and other ML architectural elements, provided to train, deploy, and test the ML model. + - describes interfaces within the ML-based software element and to other software elements + - ML architecture describes details of the ML model like used layers, activation functions, loss function, and backpropagation + - ML architecture contains defined hyperparameter ranges and initial values for training start + - resource consumption objectives are defined + - ML architecture contains allocated ML requirements -.. needextend:: "c.this_doc()" - :+tags: aspice40_iic04 +.. needextend:: "c.this_doc()" + :+tags: aspice40_iic04 diff --git a/process/standards/aspice_40/iic/iic-08.rst b/process/standards/aspice_40/iic/iic-08.rst index 21dac1eb24..2d0a3ebe5b 100644 --- a/process/standards/aspice_40/iic/iic-08.rst +++ b/process/standards/aspice_40/iic/iic-08.rst @@ -146,6 +146,33 @@ - Measures including criteria for monitoring effectiveness, suitability, and adequacy of the standard process - Method for collecting and analyzing the monitoring measures +.. std_req:: 08-64 ML test approach + :id: std_req__aspice_40__iic-08-64 + :status: valid + + - The ML test approach describes + - ML test scenarios with distribution of data characteristics (e.g., gender, weather conditions, street conditions within the ODD) defined by ML requirements + - quantity of each ML test scenario inside the test data set + - expected test result per test datum + - pass/fail criteria for the ML testing + - entry and exit criteria for the ML testing + - the required ML testing infrastructure and environment configuration + +.. std_req:: 08-65 ML training and validation approach + :id: std_req__aspice_40__iic-08-65 + :status: valid + + Process Monitoring Method may have the following characteristics: + + - ML Training and Validation approach describes at least: + - entry and exit criteria of the ML training + - approaches for hyperparameter tuning / optimization to be used in the training + - approach for data set creation and modification + - training environment, including required training hardware (e.g., GPU, or supercomputer to be used) + - interface adapter for provision of input data and storage of output data + - if required, actions to organize the data set and training environment + - The ML training and validation approach may additionally include robustification methods like random dropout + .. std_req:: 08-66 Measures against deviations in quantitative process analysis :id: std_req__aspice_40__iic-08-66 :status: valid @@ -157,5 +184,5 @@ - Effective implementation of these counter measures -.. needextend:: "c.this_doc()" - :+tags: aspice40_iic08 +.. needextend:: "c.this_doc()" + :+tags: aspice40_iic08 diff --git a/process/standards/aspice_40/iic/iic-11.rst b/process/standards/aspice_40/iic/iic-11.rst index 856f058418..40db7be477 100644 --- a/process/standards/aspice_40/iic/iic-11.rst +++ b/process/standards/aspice_40/iic/iic-11.rst @@ -71,6 +71,12 @@ commented source code, auto-code, an object file, a library, an executable, or an executable model as input to verification +.. std_req:: 11-50 Deployed ML model + :id: std_req__aspice_40__iic-11-50 + :status: valid + + - It is derived from the trained ML model (see 01-53) and is to be integrated into the target system. + - It may differ from the trained ML model which often requires powerful hardware and uses interpretative languages. -.. needextend:: "c.this_doc()" - :+tags: aspice40_iic11 +.. needextend:: "c.this_doc()" + :+tags: aspice40_iic11 diff --git a/process/standards/aspice_40/iic/iic-13.rst b/process/standards/aspice_40/iic/iic-13.rst index ea86d5932f..ee1071fe4a 100644 --- a/process/standards/aspice_40/iic/iic-13.rst +++ b/process/standards/aspice_40/iic/iic-13.rst @@ -166,6 +166,17 @@ - Information about the verification execution (date, “object-under-verification”, etc.) - Abstraction or summary of verification results +.. std_req:: 13-50 ML test results + :id: std_req__aspice_40__iic-13-50 + :status: valid + + - Test data and logs + - Test data with correct results + - Test data with incorrect results + - Test data not executed, and a rationale + - Information about the test execution (date, participants, model version etc.) + - Abstraction or summary of ML test results + .. std_req:: 13-51 Consistency Evidence :id: std_req__aspice_40__iic-13-51 :status: valid @@ -234,5 +245,5 @@ - for non-standard and critical resources and infrastructure. -.. needextend:: "c.this_doc()" - :+tags: aspice40_iic13 +.. needextend:: "c.this_doc()" + :+tags: aspice40_iic13 diff --git a/process/standards/aspice_40/iic/iic-16.rst b/process/standards/aspice_40/iic/iic-16.rst index b417849be0..b85ebaf3cc 100644 --- a/process/standards/aspice_40/iic/iic-16.rst +++ b/process/standards/aspice_40/iic/iic-16.rst @@ -57,6 +57,14 @@ - Disciplinary reporting line - Organizational units and sub-units, if applicable +.. std_req:: 16-52 ML data management system + :id: std_req__aspice_40__iic-16-52 + :status: valid + + - The ML data management system is part of the configuration management system (see 16-03) and + - Supports data management activities like data collection, description, ingestion, exploration, profiling, labeling/annotation, selection, structuring and cleansing + - Provides the data for different purposes, e.g., training, testing + - Supports the relevant sources of ML data -.. needextend:: "c.this_doc()" - :+tags: aspice40_iic16 +.. needextend:: "c.this_doc()" + :+tags: aspice40_iic16 diff --git a/process/standards/aspice_40/iic/iic-19.rst b/process/standards/aspice_40/iic/iic-19.rst index 5c5c3c2753..7f6662eedc 100644 --- a/process/standards/aspice_40/iic/iic-19.rst +++ b/process/standards/aspice_40/iic/iic-19.rst @@ -30,6 +30,14 @@ - disciplines (e.g., different configuration management approaches for software and hardware, or combined approaches) - options due to socio-cultural differences +.. std_req:: 19-50 ML data quality approach + :id: std_req__aspice_40__iic-19-50 + :status: valid + + - The ML data quality approach + - Defines Quality criteria (see 18-07) e.g., the relevant data sources, reliability and consistency of labelling, completeness against ML data requirements + - Describes analysis activities of the data + - Describes activities to ensure the quality of the data to avoid issues e.g., data bias, bad labeling -.. needextend:: "c.this_doc()" - :+tags: aspice40_iic19 +.. needextend:: "c.this_doc()" + :+tags: aspice40_iic19 diff --git a/process/standards/aspice_40/mle/index.rst b/process/standards/aspice_40/mle/index.rst new file mode 100644 index 0000000000..2f392d1aa4 --- /dev/null +++ b/process/standards/aspice_40/mle/index.rst @@ -0,0 +1,22 @@ +.. + # ******************************************************************************* + # Copyright (c) 2025 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + mle.1 + mle.2 + mle.3 + mle.4 diff --git a/process/standards/aspice_40/mle/mle.1.rst b/process/standards/aspice_40/mle/mle.1.rst new file mode 100644 index 0000000000..ae5ad8b29b --- /dev/null +++ b/process/standards/aspice_40/mle/mle.1.rst @@ -0,0 +1,116 @@ +.. + # ******************************************************************************* + # Copyright (c) 2025 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +MLE.1 Machine Learning Requirements Analysis +--------------------------------------------- + +The purpose is to refine the machine learning-related software requirements into a set of ML +requirements. + +Process outcomes +~~~~~~~~~~~~~~~~ + +1. The ML requirements including ML data requirements are identified and specified based on + the software requirements and the components of the software architecture. +2. ML requirements are structured and prioritized. +3. ML requirements are analyzed for correctness and verifiability. +4. The impact of ML requirements on the ML operating environment is analyzed. +5. Consistency and bidirectional traceability are established between ML requirements and + software requirements, and between ML requirements and software architecture. +6. The ML requirements are agreed and communicated to all affected parties. + +Base practices +~~~~~~~~~~~~~~ + +.. std_req:: MLE.1.BP1: Specify ML requirements + :id: std_req__aspice_40__MLE-1-BP1 + :status: valid + :links: std_req__aspice_40__iic-17-00 + + Use the software requirements and the software architecture to identify and specify functional and non-functional ML requirements, as well as ML data requirements specifying data characteristics (e.g., gender, weather conditions, street conditions within the ODD) and their expected distributions. + + .. note:: + + Non-functional requirements may include relevant characteristics of the ODD and KPIs as robustness, performance, and level of trustworthiness. + + .. note:: + + The ML data requirements are input for SUP.11 Machine Learning Data Management but also for other MLE processes. + + .. note:: + + In case of ML development only, stakeholder requirements represent the software requirements. + +.. std_req:: MLE.1.BP2: Structure ML requirements + :id: std_req__aspice_40__MLE-1-BP2 + :status: valid + :links: std_req__aspice_40__iic-17-00, std_req__aspice_40__iic-17-54 + + Structure and prioritize the ML requirements. + + .. note:: + + Examples for structuring criteria can be grouping (e.g., by functionality) or variants identification. + + .. note:: + + Prioritization can be done according to project or stakeholder needs via e.g., definition of release scopes. Refer to SPL.2.BP1. + +.. std_req:: MLE.1.BP3: Analyze ML requirements + :id: std_req__aspice_40__MLE-1-BP3 + :status: valid + :links: std_req__aspice_40__iic-17-54,std_req__aspice_40__iic-15-51 + + Analyze the specified ML requirements including their interdependencies to ensure correctness, technical feasibility, and ability for machine learning model testing, and to support project management regarding project estimates. + + .. note:: + + See MAN.3.BP3 for project feasibility and MAN.3.BP5 for project estimates. + +.. std_req:: MLE.1.BP4: Analyze the impact on the ML operating environment + :id: std_req__aspice_40__MLE-1-BP4 + :status: valid + :links: std_req__aspice_40__iic-15-51 + + Analyze the impact that the ML requirements will have on interfaces of software components and the ML operating environment. + + .. note:: + + The ML operating environment is defined as the infrastructure and information which both the trained ML model and the deployed ML model need for execution. + +.. std_req:: MLE.1.BP5: Ensure consistency and establish bidirectional traceability + :id: std_req__aspice_40__MLE-1-BP5 + :status: valid + :links: std_req__aspice_40__iic-13-51 + + Ensure consistency and establish bidirectional traceability between ML requirements and software requirements and between ML requirements and the software architecture. + + .. note:: + + Bidirectional traceability supports consistency, facilitates impact analyses of change requests, and verification coverage demonstration. Traceability alone, e.g., the existence of links, does not necessarily mean that the information is consistent with each other. + + .. note:: + + Redundant traceability is not intended, but at least one out of the given traceability paths. + +.. std_req:: MLE.1.BP6: Communicate agreed ML requirements and impact on the operating environment + :id: std_req__aspice_40__MLE-1-BP6 + :status: valid + :links: std_req__aspice_40__iic-13-52 + + Communicate the agreed ML requirements, and the results of the impact analysis on the ML operating environment to all affected parties. + + +.. needextend:: "c.this_doc()" + :+tags: aspice40_mle1 diff --git a/process/standards/aspice_40/mle/mle.2.rst b/process/standards/aspice_40/mle/mle.2.rst new file mode 100644 index 0000000000..5eeee26ff1 --- /dev/null +++ b/process/standards/aspice_40/mle/mle.2.rst @@ -0,0 +1,107 @@ +.. + # ******************************************************************************* + # Copyright (c) 2025 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +MLE.2 Machine Learning Architecture +------------------------------------ + +The purpose is to establish an ML architecture supporting training and deployment, consistent +with the ML requirements, and to evaluate the ML architecture against defined criteria. + +Process outcomes +~~~~~~~~~~~~~~~~ + +1. A ML architecture is developed. +2. Hyperparameter ranges and initial values are determined as a basis for the training. +3. Evaluation of ML architectural elements is conducted. +4. Interfaces of the ML architectural elements are defined. +5. Resource consumption objectives for the ML architectural elements are defined. +6. Consistency and bidirectional traceability are established between the ML architectural + elements and the ML requirements. +7. The ML architecture is agreed and communicated to all affected parties. + +Base practices +~~~~~~~~~~~~~~ + +.. std_req:: MLE.2.BP1: Develop ML architecture + :id: std_req__aspice_40__MLE-2-BP1 + :status: valid + :links: std_req__aspice_40__iic-04-51, std_req__aspice_40__iic-01-54, std_req__aspice_40__iic-15-51 + + Develop and document the ML architecture that specifies ML architectural elements including details of the ML model, pre- and postprocessing, and hyperparameters which are required to create, train, test, and deploy the ML model. + + .. note:: + + Necessary details of the ML model may include layers, activation functions, and backpropagation. The level of detail of the ML model may not need to cover aspects like single neurons. + + .. note:: + + The details of the ML model may differ between the ML model used during training and the deployed ML model. + +.. std_req:: MLE.2.BP2: Determine hyperparameter ranges and initial values + :id: std_req__aspice_40__MLE-2-BP2 + :status: valid + :links: std_req__aspice_40__iic-04-51,std_req__aspice_40__iic-01-54 + + Determine and document the hyperparameter ranges and the initial values as a basis for the training. + +.. std_req:: MLE.2.BP3: Analyze ML architectural elements + :id: std_req__aspice_40__MLE-2-BP3 + :status: valid + :links: std_req__aspice_40__iic-04-51,std_req__aspice_40__iic-15-51 + + Define criteria for analysis of the ML architectural elements. Analyze ML architectural elements according to the defined criteria. + + .. note:: + + Trustworthiness and explainability might be criteria for the analysis of the ML architectural elements. + +.. std_req:: MLE.2.BP4: Define interfaces of the ML architectural elements + :id: std_req__aspice_40__MLE-2-BP4 + :status: valid + :links: std_req__aspice_40__iic-04-51 + + Determine and document the internal and external interfaces of each ML architectural element including its interfaces to related software components. + +.. std_req:: MLE.2.BP5: Define resource consumption objectives for the ML architectural elements + :id: std_req__aspice_40__MLE-2-BP5 + :status: valid + :links: std_req__aspice_40__iic-04-51 + + Determine and document the resource consumption objectives for all relevant ML architectural elements during training and deployment. + +.. std_req:: MLE.2.BP6: Ensure consistency and establish bidirectional traceability + :id: std_req__aspice_40__MLE-2-BP6 + :status: valid + :links: std_req__aspice_40__iic-13-51 + + Ensure consistency and establish bidirectional traceability between the ML architectural elements and the ML requirements. + + .. note:: + + Bidirectional traceability supports consistency, and facilitates impact analyses of change requests, and verification coverage demonstration. Traceability alone, e.g., the existence of links, does not necessarily mean that the information is consistent with each other. + + .. note:: + + The bidirectional traceability should be established on a reasonable level of abstraction to the ML architectural elements. + +.. std_req:: MLE.2.BP7: Communicate agreed ML architecture + :id: std_req__aspice_40__MLE-2-BP7 + :status: valid + :links: std_req__aspice_40__iic-13-52 + + Inform all affected parties about the agreed ML architecture including the details of the ML model and the initial hyperparameter values. + + +.. needextend:: "c.this_doc()" + :+tags: aspice40_mle2 diff --git a/process/standards/aspice_40/mle/mle.3.rst b/process/standards/aspice_40/mle/mle.3.rst new file mode 100644 index 0000000000..268fe7c673 --- /dev/null +++ b/process/standards/aspice_40/mle/mle.3.rst @@ -0,0 +1,101 @@ +.. + # ******************************************************************************* + # Copyright (c) 2025 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +MLE.3 Machine Learning Training +-------------------------------- + +The purpose is to optimize the ML model to meet the defined ML requirements. + +Process outcomes +~~~~~~~~~~~~~~~~ + +1. A ML training and validation approach is specified. +2. The data set for ML training and ML validation is created. +3. The ML model, including hyperparameter values, is optimized to meet the defined ML + requirements. +4. Consistency and bidirectional traceability are established between the ML training and + validation data set and the ML data requirements. +5. Results of optimization are summarized, and the trained ML model is agreed and + communicated to all affected parties. + +Base practices +~~~~~~~~~~~~~~ + +.. std_req:: MLE.3.BP1: Specify ML training and validation approach + :id: std_req__aspice_40__MLE-3-BP1 + :status: valid + :links: std_req__aspice_40__iic-08-65 + + Specify an approach which supports the training and validation of the ML model to meet the defined ML requirements. The ML training and validation approach includes + + * entry and exit criteria of the training and validation, + * approaches for hyperparameter tuning / optimization, + * approach for data set creation and modification, and + * training and validation environment + + .. note:: + + The ML training and validation approach may include random dropout and other robustification methods. + + .. note:: + + ML validation is the optimization of the hyperparameters during Machine Learning Training (MLE.3). The term "validation" has a different meaning than VAL.1. + + .. note:: + + The training environment should reflect the environment of the deployed model. + +.. std_req:: MLE.3.BP2: Create ML training and validation data set + :id: std_req__aspice_40__MLE-3-BP2 + :status: valid + :links: std_req__aspice_40__iic-03-51 + + Select data from the ML data collection provided by SUP.11 and assign them to the data set for training and validation of the ML model according to the specified ML training and validation approach. + + .. note:: + + The ML training and validation data set may include corner cases, unexpected cases, and normal cases depending on the ML requirements. + + .. note:: + + A separated data set for training and validation might not be required in some cases (e.g., k-fold cross validation, no optimization of hyperparameters). + +.. std_req:: MLE.3.BP3: Create and optimize ML model + :id: std_req__aspice_40__MLE-3-BP3 + :status: valid + :links: std_req__aspice_40__iic-01-53,std_req__aspice_40__iic-01-54 + + Create the ML model according to the ML architecture and train it, using the identified ML training and validation data set according to the ML training and validation approach to meet the defined ML requirements, and training and validation exit criteria. + +.. std_req:: MLE.3.BP4: Ensure consistency and establish bidirectional traceability + :id: std_req__aspice_40__MLE-3-BP4 + :status: valid + :links: std_req__aspice_40__iic-13-51 + + Ensure consistency and establish bidirectional traceability between the ML training and validation data set and the ML data requirements. + + .. note:: + + Bidirectional traceability supports consistency and facilitates impact analyses of change requests. Traceability alone, e.g., the existence of links, does not necessarily mean that the information is consistent with each other. + +.. std_req:: MLE.3.BP5: Summarize and communicate agreed trained ML model + :id: std_req__aspice_40__MLE-3-BP5 + :status: valid + :links: std_req__aspice_40__iic-13-52 + + Summarize the results of the optimization and inform all affected parties about the agreed trained ML model. + + +.. needextend:: "c.this_doc()" + :+tags: aspice40_mle3 diff --git a/process/standards/aspice_40/mle/mle.4.rst b/process/standards/aspice_40/mle/mle.4.rst new file mode 100644 index 0000000000..56052a9e0b --- /dev/null +++ b/process/standards/aspice_40/mle/mle.4.rst @@ -0,0 +1,131 @@ +.. + # ******************************************************************************* + # Copyright (c) 2025 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +MLE.4 Machine Learning Model Testing +------------------------------------- + +The purpose is to ensure compliance of the trained ML model and the deployed ML model with +the ML requirements. + +Process outcomes +~~~~~~~~~~~~~~~~ + +1. A ML test approach is defined. +2. A ML test data set is created. +3. The trained ML model is tested. +4. The deployed ML model is derived from the trained ML model and tested. +5. Consistency and bidirectional traceability are established between the ML test approach and + the ML requirements, and the ML test data set and the ML data requirements; and + bidirectional traceability is established between the ML test approach and ML test results. +6. Results of the ML model testing are summarized and communicated with the deployed ML + model to all affected parties. + +Base practices +~~~~~~~~~~~~~~ + +.. std_req:: MLE.4.BP1: Specify an ML test approach + :id: std_req__aspice_40__MLE-4-BP1 + :status: valid + :links: std_req__aspice_40__iic-08-64 + + Specify an ML test approach suitable to provide evidence for compliance of the trained ML model and the deployed ML model with the ML requirements. The ML test approach includes + + * ML test scenarios with distribution of data characteristics (e.g., gender, weather conditions, street conditions within the ODD) defined by ML requirements, + * distribution and frequency of each ML test scenario inside the ML test data set, + * expected test result per test datum, + * entry and exit criteria of the testing, + * approach for data set creation and modification, and + * the required testing infrastructure and environment setup. + + .. note:: + + Expected test result per test datum might require labeling of test data to support comparison of output of the ML model with the expected output. + + .. note:: + + Test datum is the smallest amount of data which is processed by the ML model into only one output. E.g., one image in photo processing or an audio sequence in voice recognition. + + .. note:: + + Data characteristic is one property of the data that may have different expressions in the ODD. E.g., weather condition may contain expressions like sunny, foggy or rainy. + + .. note:: + + An ML test scenario is a combination of expressions of all defined data characteristics e.g., weather conditions = sunny, street conditions = gravel road. + +.. std_req:: MLE.4.BP2: Create ML test data set + :id: std_req__aspice_40__MLE-4-BP2 + :status: valid + :links: std_req__aspice_40__iic-03-51 + + Create the ML test data set needed for testing of the trained ML model and testing of the deployed ML model from the ML data collection provided by SUP.11 considering the ML test approach. The ML test data set shall not be used for training. + + .. note:: + + The ML test data set for the trained ML model might differ from the test data set of the deployed ML model. + + .. note:: + + Additional data sets might be used for special purposes like assurance of safety, fairness, robustness. + +.. std_req:: MLE.4.BP3: Test trained ML model + :id: std_req__aspice_40__MLE-4-BP3 + :status: valid + :links: std_req__aspice_40__iic-13-50 + + Test the trained ML model according to the ML test approach using the created ML test data set. Record and evaluate the ML test results. + + .. note:: + + Evaluation of test logs might include pattern analysis of failed test data to support e.g., trustworthiness. + +.. std_req:: MLE.4.BP4: Derive deployed ML model + :id: std_req__aspice_40__MLE-4-BP4 + :status: valid + :links: std_req__aspice_40__iic-11-50, std_req__aspice_40__iic-13-50 + + Derive the deployed ML model from the trained ML model according to the ML architecture. The deployed ML model shall be used for testing and delivery to software integration. + + .. note:: + + The deployed ML model will be integrated into the target system and may differ from the trained ML model which often requires powerful hardware and uses interpretative languages. + +.. std_req:: MLE.4.BP5: Test deployed ML model + :id: std_req__aspice_40__MLE-4-BP5 + :status: valid + :links: std_req__aspice_40__iic-11-50, std_req__aspice_40__iic-13-50 + + Test the deployed ML model according to the ML test approach using the created ML test data set. Record and evaluate the ML test results. + +.. std_req:: MLE.4.BP6: Ensure consistency and establish bidirectional traceability + :id: std_req__aspice_40__MLE-4-BP6 + :status: valid + :links: std_req__aspice_40__iic-13-51 + + Ensure consistency and establish bidirectional traceability between the ML test approach and the ML requirements, and the ML test data set and the ML data requirements; and bidirectional traceability is established between the ML test approach and ML test results. + + .. note:: + + Bidirectional traceability supports consistency, and facilitates impact analyses of change requests, and verification coverage demonstration. Traceability alone, e.g., the existence of links, does not necessarily mean that the information is consistent with each other. + +.. std_req:: MLE.4.BP7: Summarize and communicate results + :id: std_req__aspice_40__MLE-4-BP7 + :status: valid + :links: std_req__aspice_40__iic-13-52 + + Summarize the ML test results of the ML model. Inform all affected parties about the agreed results and the deployed ML model. + + +.. needextend:: "c.this_doc()" + :+tags: aspice40_mle4 diff --git a/process/standards/aspice_40/sup/index.rst b/process/standards/aspice_40/sup/index.rst index 24667f5950..77818c3a88 100644 --- a/process/standards/aspice_40/sup/index.rst +++ b/process/standards/aspice_40/sup/index.rst @@ -20,3 +20,4 @@ sup.8 sup.9 sup.10 + sup.11 diff --git a/process/standards/aspice_40/sup/sup.11.rst b/process/standards/aspice_40/sup/sup.11.rst new file mode 100644 index 0000000000..121ad92587 --- /dev/null +++ b/process/standards/aspice_40/sup/sup.11.rst @@ -0,0 +1,114 @@ +.. + # ******************************************************************************* + # Copyright (c) 2025 Contributors to the Eclipse Foundation + # + # See the NOTICE file(s) distributed with this work for additional + # information regarding copyright ownership. + # + # This program and the accompanying materials are made available under the + # terms of the Apache License Version 2.0 which is available at + # https://www.apache.org/licenses/LICENSE-2.0 + # + # SPDX-License-Identifier: Apache-2.0 + # ******************************************************************************* + +SUP.11 Machine Learning Data Management +--------------------------------------- + +The purpose is to define and align ML data with ML data requirements, maintain the integrity +and quality of the ML data, and make them available to affected parties. + +Process outcomes +~~~~~~~~~~~~~~~~ + +1. A ML data management system including an ML data lifecycle is established. +2. A ML data quality approach is developed including ML data quality criteria. +3. Collected ML data are processed for consistency with ML data requirements. +4. ML data are verified against defined ML data quality criteria and updated as needed. +5. ML data are agreed and communicated to all affected parties. + +Base practices +~~~~~~~~~~~~~~ + +.. std_req:: SUP.11.BP1: Establish an ML data management system + :id: std_req__aspice_40__SUP-11-BP1 + :status: valid + :links: std_req__aspice_40__iic-16-52 + + Establish an ML data management system which supports + + * ML data management activities, + * relevant sources of ML data, + * ML data life cycle including a status model, and + * interfaces to affected parties. + + .. note:: + + Supported ML data management activities may include data collection, labeling/annotation, and structuring. + +.. std_req:: SUP.11.BP2: Develop an ML data quality approach + :id: std_req__aspice_40__SUP-11-BP2 + :status: valid + :links: std_req__aspice_40__iic-19-50 + + Develop an approach to ensure that the quality of ML data is analyzed based on defined ML data quality criteria and activities are performed to support avoidance of biases of data. + + .. note:: + + Examples of ML data quality criteria are relevant data sources, reliability and consistency of labelling, completeness against ML data requirements. + + .. note:: + + The ML data management system should support the quality criteria and activities of the ML data quality approach. + + .. note:: + + Biases to avoid may include sampling bias (e.g., gender, age) and feedback loop bias. + + .. note:: + + For creation of ML data sets see MLE.3.BP2 and MLE.4.BP2. + +.. std_req:: SUP.11.BP3: Collect ML data + :id: std_req__aspice_40__SUP-11-BP3 + :status: valid + :links: std_req__aspice_40__iic-03-53 + + Relevant sources for raw data are identified and continuously monitored for changes. The raw data is collected according to the ML data requirements. + + .. note:: + + The identification and collection of ML data might be an organizational responsibility. + + .. note:: + + Continuous monitoring should include the ODD and may lead to changes of the ML requirements. + +.. std_req:: SUP.11.BP4: Process ML data + :id: std_req__aspice_40__SUP-11-BP4 + :status: valid + :links: std_req__aspice_40__iic-03-53 + + The raw data are processed (annotated, analyzed, and structured) according to the ML data requirements. + +.. std_req:: SUP.11.BP5: Assure quality of ML data + :id: std_req__aspice_40__SUP-11-BP5 + :status: valid + :links: std_req__aspice_40__iic-03-53 + + Perform the activities according to the ML data quality approach to ensure that the ML data meets the defined ML data quality criteria. + + .. note:: + + These activities may include sample-based reviews or statistical methods. + +.. std_req:: SUP.11.BP6: Communicate agreed processed ML data + :id: std_req__aspice_40__SUP-11-BP6 + :status: valid + :links: std_req__aspice_40__iic-13-52 + + Inform all affected parties about the agreed processed ML data and provide them to the affected parties. + + +.. needextend:: "c.this_doc()" + :+tags: aspice40_sup11