New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MLVU aggregate function is incorrect #533

Open

SCZwangxiao opened this issue Feb 11, 2025 · 0 comments

SCZwangxiao commented Feb 11, 2025

According to results in their leaderboard, the overall score is calculated by the average of each subtype accuracy, not the sample accuracy.

lmms-eval/lmms_eval/tasks/mlvu/utils.py

Lines 117 to 122 in d294646

    
           total_correct = 0 
        
           total_answered = 0 
        
           for k, v in category2score.items(): 
        
               total_correct += v["correct"] 
        
               total_answered += v["answered"] 
        
           eval_logger.info(f"Overall Performance: {100 * total_correct / total_answered if total_answered > 0 else 0 : .1f}%")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment