scoring: remove IDF from BM25 scoring #912
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We remove IDF because we want to use BM25 scoring for keyword search and our query-time calculation of IDF won't work anymore if terms are AND-ed (keyword search) instead of OR-ed (codycontext). Our evaluations show a slight improvement which supports the decision to treat IDF as constant. This is also in line with how we calculate line-based BM25. Test plan: updated unit tests
stefanhengl
commented
Feb 12, 2025
| @@ -79,8 +79,8 @@ func TestBM25(t *testing.T) { | |||
| query: &query.Substring{Pattern: "example"}, | |||
| content: exampleJava, | |||
| language: "Java", | |||
| // bm25-score: 0.58 <- sum-termFrequencyScore: 14.00, length-ratio: 1.00 | |||
There was a problem hiding this comment.
I remove the score from the comments because it is redundant and requires an update for every update of the scoring function. sum-termFrequencyScore and length-ratio are more robust.
|
Couple questions
|
|
|
Although I had the same reaction as @mmanela, that we should really avoid deviating from the classic BM25 formula, we did an analysis here that made me feel okay about this: https://linear.app/sourcegraph/issue/SPLF-838/use-bm25-for-multi-term-keyword-searches. TL;DR: eval results on every dataset improved, and IDF may not be as critical for the code search use case. |
jtibshirani
approved these changes
Feb 12, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We remove IDF from our BM25 scoring, effectively treating it as constant.
This is supported by our evaluations which showed that for keyword style queries, IDF can down-weight the score of important keywords too much, leading to a worse ranking. The intuition is that for code search, each keyword is important independently of how frequent it appears in the corpus.
Removing IDF allows us to apply BM25 scoring to a wider range of query types. Previously, BM25 was limited to queries with individual terms combined using OR, as IDF was calculated on the fly at query time.
Test plan:
updated tests