Added Benchmark Evaluation Framework with CORE Benchmark Suite #402
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Integrated NanoChat’s benchmark components into Plato to enable direct evaluation of Plato models using the NanoChat benchmark. Added files: - common.py: shared utilities and configurations for the benchmark - core_eval.py: implements the CORE benchmark evaluation logic - evaluate_model.py: main entry point to run model evaluation from Plato - report.py: handles result aggregation and reporting - tokenizer.py: provides tokenization utilities for language model evaluation
… models.
Features:
- Automatically sets up the NanoChat datasets under `.cache/nanochat`.
- Downloads and unpacks the CORE evaluation bundle if not already available.
- Invokes `evaluate_model.py` with the specified HuggingFace model path.
- Adds argument parsing for `<model_path>` and optional `[max_per_task]`.
- Defaults `max_per_task` to 16 when not provided.
Usage:
bash evaluate_model.sh <model_path> [optional: max_per_task]
…chmarks/language_model.
- Introduced eval_model() in testing.py to define a placeholder interface for benchmark-based evaluation. - The default strategy now raises NotImplementedError to prompt use of specialized testing strategies.
- Added static methods save_benchmark_result() and load_benchmark_result() in base.py for saving and loading benchmark evaluation results.
- Implemented benchmark evaluation pipeline in plato/trainers/composable.py. - Added eval_model(), eval(), and eval_process() methods.
- Enabled benchmark evaluation in split learning to test and validate benchmark implementations.
✅ Deploy Preview for platodocs canceled.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces a benchmark evaluation framework to Plato, enabling systematic evaluation of trained federated learning models on user-specified benchmarks. This PR also adds the CORE benchmark (borrowed from NanoChat) for language model evaluations.
Description
Benchmark registry and CORE benchmark implementation:
plato/benchmarks/base.py): Abstract interface with thread-safe download guards and evaluation contract.plato/benchmarks/registry.py): Runtime benchmark selection via configuration (registered_benchmarks dict).plato/benchmarks/core.py): Language model evaluation suite with multiprocessing support, task loading, and metric aggregation.plato/benchmarks/core_helpers): Task evaluation logic and necessary tokenizer wrapper for compatibility.Configuration support (
plato/config.py):[benchmark]section withtypefield to define the type/name of the benchmark; fortype="core", meaning using CORE benchmark, addedrandom_seed, andmax_per_taskfields in particularTrainer integration (
plato/trainers/composable.py):eval(),eval_model(), andeval_process()to run the model evaluation in the testing strategysave_benchmark_result()andload_benchmark_result()utilities for trainer max concurrency modeServer orchestration (
plato/servers/fedavg.py):benchmarks_registry.get()benchmark.get_formatted_result()Testing strategy interface (
plato/trainers/testing_strategies/default.py):eval_model()to testing strategy to run model evaluationSplit learning example validation (
examples/split_learning/llm_split_learning):eval_model()insplit_learning_trainer.pyHow has this been tested?
Command:
Output showing successful CORE benchmark evaluation on 22 tasks after 10 rounds of FL training session:
Types of changes
Checklist:
ruff format) and checked using the Ruff linter (ruff check --fix).