Stra (CEO Kim Yong-ho) announced that ‘multilingual AI dubbing’, which expands the applying of artificial intelligence (AI) technology to K-pop idol content, is receiving a very good response.
Stra (CEO Kim Yong-ho), established in 2021, attracted attention for its ‘AI transcription’ technology. This technology analyzes MP3 files, extracts codes and melodies, and creates MIDI files separated by instrument corresponding to guitar, piano, and drums.
Last May, sound source-voice separation and generation AI technology was applied to the ‘K-Pop Pro’ app, gaining popularity with the feature that enables users to finish an ‘AI cover song’ with their very own voice in only one minute. Currently, the service is running stably, with over 20,000 songs created.
Yongho Kim, CEO of Stra, said, “Based on these references and technologies, we at the moment are participating in global dubbing of K-pop content.” ‘AI-based multilingual dubbing’ technology is applied to content videos uploaded to the idol group’s official YouTube channel.
Recently, in partnership with H Music Entertainment, we’re providing AI dubbed versions of official content from idol groups WOOAH and DXMON. It is on the market in Chinese, Japanese, English, Spanish, and Korean.
He explained that, unlike general translation, it could even preserve singers’ intonations and nuances. “It is because it is predicated not only on TTS (text-to-speech), which only conveys meaning by entering text, but in addition on ‘voice cloning,’” he said.
Along with the prevailing voice data needed for voice learning, this technology requires additional guided recording by native speakers. And the tactic is to exchange the recording of the guide’s native speaker with the voice of an idol member. ‘Sound separation technology’, which cleanly erases and extracts all existing images and voices, can also be essential, and this was possible with Stra’s technology. It has develop into possible to supply high-quality dubbed videos without the necessity for extra recording studios or equipment.
This process was also applied to the currently uploaded video. It took 3 to 4 days to supply a 30-minute video. Consequently, high-quality dubbing was accomplished and positive feedback was received from overseas fans. Comments on the YouTube channel included reactions corresponding to “The AI voice is cute,” “I didn’t know you spoke Japanese so well,” and “Please make a German version, too.”
CEO Kim said, “It was effective to implement a tone much like the artist’s original voice and natural intonation,” and added, “Also, providing a subtitled version through the entertainment company’s official account and specifying that it was ‘AI dubbing’ drew a very good response.” “I believe it helped,” he said.
In the longer term, we plan to further simplify the production process. Ultimately, we’re working on upgrading the mix of TTS and voice cloning in order that we are able to mechanically create dubbed videos in greater than 20 countries by uploading only the unique video.
CEO Kim said, “We’ve got been holding meetings with various entertainment corporations since we now have been introducing various businesses and products by fusing K-pop and AI technology. At first, we thought we were simply creating ‘multilingual dubbed songs,’ however the artist’s songs and “The sound source was a novel creation, so it was easy to arouse rejection from fans.”
So what I assumed of instead was ‘Zarkon’. Jakon is a term amongst K-pop fandoms that abbreviates ‘own content’. In accordance with Stra’s own research, popular K-pop artists have, on average, about 1,300 pieces of content per YouTube channel.
It is crucial for K-pop artists to capture the hearts of overseas fandom, but aside from large entertainment corporations, it’s difficult so as to add subtitles because of an absence of editing staff. Among the many contents of small and medium-sized agencies, only about 30% of videos had subtitles. Because the content is predicated on on a regular basis conversations corresponding to vlogs, live shows, and reactions, relatively than standardized dramas or movies, dubbing or translation is harder.
CEO Kim said, “As a fan service for fans in countries that don’t prefer subtitles or young fans who’ve difficulty reading subtitles, I attempted it with the thought that if I could provide AI dubbing with the voice of an idol, I might have the opportunity to speak on a better level. “In the longer term, we wish to expand our service area for more overseas fans of K-pop artists,” he said.
Currently, the service is being provided in the shape of collaborating with the agency’s production team, receiving the unique video and sending the edited video. In the longer term, it’s being developed in order that it could be used inside the company as a ‘web service’. It’s planned to be launched as early as January next yr.
He added that he confirmed the demand for AI dubbing technology in areas apart from entertainment through participation in domestic exhibitions. A representative example is the short-form video of an influencer who exports products corresponding to K-beauty and K-food overseas. AI dubbing technology is used to dub in multiple languages and increase global sales. It was revealed that technology verification (PoC) with actual demand sources can also be in progress.
Yongho Kim, CEO of Stra, said, “K-Pop Pro has currently exceeded 400,000 cumulative downloads, and at one point even entered the highest 100 within the US App Store music rankings.”
“At first, it was a service to assist overseas fans overcome the barrier to song lyrics, but in the longer term, we are going to strive to beat the language barrier of all content,” he said.
Reporter Jang Se-min semim99@aitimes.com
