Upstage (CEO Kim Seong-hoon) and the Korea Intelligence and Information Society Agency (NIA, Director Hwang Jong-seong) announced on the eleventh that they will probably be upgrading the jointly operated ‘Open Ko-LLM Leaderboard’ by adding evaluation indicators.
There are two additional indicators: ‘Inference ability (Winogrande, ability to infer pronouns in text)’ and ‘Arithmetic reasoning ability (GSM8K, reasoning ability in the sector of arithmetic operations)’.
Because it is an indicator also utilized in the Hugging Face Open LLM leaderboard, it should be a possibility to boost the domestic model performance evaluation system to a level reminiscent of international evaluation standards if applied to Korean.
Flitto (CEO Jeongsoo Lee), a specialist in artificial intelligence (AI) language data, participated in the event of additional indicators and supported translation and development tailored to the Korean language.
Park Chan-jun, a senior researcher at Upstage, said that through this upgrade, “We’ll have the opportunity to filter out models which can be overfitted based on the present indicators and more objective evaluation will probably be possible,” adding, “We expect to have the opportunity to pick good models.”
He added, “It seems that it should take a while because we want to reevaluate greater than 1,500 models currently submitted, and the ultimate reflection is targeted for sometime in July at the most recent.”
Meanwhile, the open Ko-LLM leaderboard opened in September last yr, and as of the top of May, the variety of participating large language models (LLMs) exceeded 1,500.
NIA Director Hwang Jong-seong said, “NIA and Upstage plan to constantly improve the leaderboard evaluation indicators and system for the event of Korean LLM. We expect that it will result in more energetic exchange and cooperation between domestic and foreign AI researchers.” He said.
Meanwhile, the main points of the leaderboard jointly hosted by Upstage and the Korea Intelligence and Information Society Agency (NIA) are as follows: NIA homepageme Hugging Face HomepageYou may check it here.
Reporter Jang Se-min semim99@aitimes.com
