Authors: Dong Zhang, Yang Sun, Fei Lyu, Xiaofeng Liu, Xin Li
The evaluation results of open-source large language models are not only influenced by model parameter scale, training corpus, and alignment strategies, but also significantly constrained by the test set architecture and the weight allocation of evaluation metrics. To address this issue, this paper leverages multiple types of real-world evaluation data collected from engineering projects to investigate the evolution mechanism of open-source large model evaluation scores and rankings under the coupled effects of weight allocation and test set design. This paper selects four mainstream frontier open-source models—DeepSeek-V3.2, MiniMax-M2.5, Qwen3.6-35B-A3B, and gpt-oss-120b—to conduct controlled experiments. All models uniformly employ 8 96G H20 GPUs for local offline inference evaluation, and weighted aggregation is performed after unifying the scoring criteria to eliminate evaluation interference caused by third-party API rate limiting, version differences, and inconsistent scoring standards. The native capabilities of the models are supplemented with official model card information from the ModelScope platform: MiniMax-M2.5 focuses on code engineering, intelligent agents, and office interaction, with inference speed improved by 37% over the previous generation and inference cost at only 10% of Claude Opus 4.6; gpt-oss-120b is an OpenAI open-source MoE architecture model that natively supports hierarchical reasoning, tool invocation, and MXFP4 quantized deployment. The experimental datasets cover five major tasks—social bias discrimination, common misconception fact-checking, lifestyle question answering, contextual semantic understanding, and basic logical reasoning—corresponding to five evaluation capabilities: bias identification, fact verification, knowledge response, contextual interpretation, and mathematical reasoning. Specifically, this paper operationalizes "task type"as an observational variable into five categories of questions: :social bias discrimination is used to measure the Bias Identificationconcept(characterized by bias judgment accuracy and harmful tendency response rate),common misconceptionFact-Checkingfact-checking is used to measure the fact verification concept(characterized by fact judgment accuracy),lifestyle question answering is used to measure the knowledge response concept(characterized by answer correctness rate and completeness score),contextual semantic understanding is used to measure the contextual interpretation concept(characterized by semantic consistency and coreference resolution accuracy),basic logical reasoning is used to measure the mathematical reasoning concept(characterized by reasoning step accuracy and final conclusion accuracy)。This paper constructs three evaluation weighting schemes: average baseline weights、performance-oriented weights、, and robust compliance-oriented weights,to analyze the variation patterns of model tier rankings under differentiated weight allocations。The experimental results show that:when the test set task categories are homogeneous、and question types are uniform,model rankings are highly susceptible to local sample perturbations;test sets with diverse task types、and balanced difficulty levels,exhibit stronger evaluation stability,and can objectively characterize comprehensive model capabilities。The study further confirms that,metric weights、sample structure、and task distribution are not auxiliary evaluation variables,but core elements determining the credibility of evaluation conclusions、and applicability。
Comments: 22 Pages.
Download: PDF
[v1] 2026-07-06 07:53:15
Unique-IP document downloads: 21 times
Vixra.org is a pre-print repository rather than a journal. Articles hosted may not yet have been verified by peer-review and should be treated as preliminary. In particular, anything that appears to include financial or legal advice or proposed medical treatments should be treated with due caution. Vixra.org will not be responsible for any consequences of actions that result from any form of use of any documents on this website.
Add your own feedback and questions here:
You are equally welcome to be positive or negative about any paper but please be polite. If you are being critical you must mention at least one specific error, otherwise your comment will be deleted as unhelpful.