[feat][evaluation] add skip evaluation feature for evaluators#512
Open
[feat][evaluation] add skip evaluation feature for evaluators#512
Conversation
Implement ShouldSkipEvaluator method to allow evaluators to skip execution based on input data. This includes adding the interface method, default implementations for prompt and code evaluators, and integration into the evaluation workflow. The feature helps optimize evaluation by skipping unnecessary runs.
HearyShen
changed the title
HearyShen
changed the title
… and create record when skipped Modify ShouldSkip methods to return EvaluatorOutputData instead of EvaluatorRecord Update ShouldSkipEvaluator to create a record with output data when skipped Adjust tests and mocks to reflect the new behavior
Add mock expectation for ShouldSkipEvaluator in all test cases to ensure proper test coverage of evaluator skipping logic
Codecov Report❌ Patch coverage is
@@ Coverage Diff @@
## main #512 +/- ##
==========================================
+ Coverage 77.13% 77.15% +0.01%
==========================================
Files 650 650
Lines 72599 72657 +58
==========================================
+ Hits 56001 56059 +58
- Misses 13257 13258 +1
+ Partials 3341 3340 -1
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 4 files with indirect coverage changes Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
Add error handling for ShouldSkipEvaluator call and log warning when skip check fails
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement ShouldSkipEvaluator method to allow evaluators to skip execution based on input data. This includes adding the interface method, default implementations for prompt and code evaluators, and integration into the evaluation workflow. The feature helps optimize evaluation by skipping unnecessary runs.
What type of PR is this?
Check the PR title
(Optional) Translate the PR title into Chinese
(Optional) More detailed description for this PR(en: English/zh: Chinese)
en:
zh(optional):
(Optional) Which issue(s) this PR fixes