Multilingual India's race to build own Large Language Model easier said than done

Key Concepts:

• Large Language Model (LLM)
• Generative AI
• Natural Language Processing (NLP)
• Data Digitization
• Language Diversity
• Bias in AI
• GPUs (Graphics Processing Units)
• Data Centers
• Bharat Jen Consortium

Bharat Jen and India's AI Ambitions:

India is developing its own large language model (LLM), aiming to rival OpenAI's ChatGPT. This initiative is driven by the need to cater to India's diverse linguistic landscape. The goal is to create generative AI solutions that address everyday problems across the country.

Hindi Model Example:

An example of this is a Hindi model that can mimic a speaker's voice and tone. This AI model converts text to speech, enabling human-like communication. It's currently used in the Krishath app to assist farmers by answering questions about crop health. This translation capability is crucial for reaching remote regions.

Bharat Jen Consortium:

Professor Ramak Krishnan is part of Bharat Jen, a government-funded consortium. Since its inception last year, it has rolled out 19 language models. Bharat Jen gathers data from magazines, data sources, foundations, and NGOs in local languages to digitize and incorporate it into the foundational model.

Language Diversity Challenge:

India has 22 officially recognized languages and over 10,000 local languages. Coding an AI model that understands most, if not all, of these languages is a complex task. Some languages, like Marathi, share roots with Hindi and Gujarati, while others, like Kannada, Telugu, Tamil, and Malayalam, are entirely different.

Data Scarcity in Indian Languages:

The lack of online content in Indian languages is a significant challenge. Bharat Jen estimates that roughly half of all internet data is in English, while Indian languages make up only about 1%. Many literary works in Indian languages have not been digitized, and traditional knowledge is often passed down verbally.

Data Collection Strategy:

Experts suggest that collecting data from local sources could be half the battle. The diversity of languages and data could help create AI models with fewer biases.

"India is all about scale and complexity. If it is solved in India and if it works in India chances are it'll work in the world that's the opportunity."

AI in Street Food Business:

Guran Yadav's experience with ChatGPT for his street food business highlights the current limitations. The AI understood his question in the local dialect of Bhojpuri but replied in Hindi. He attributes this to the foreign origin of the chatbot.

Infrastructure Investment:

Experts emphasize that to create made-in-India generative AI at scale, firms like Bharat Jen need to invest billions of dollars in graphics processing units (GPUs) and data centers.

Conclusion:

The development of LLMs in India faces challenges related to language diversity and data scarcity. Overcoming these hurdles requires significant investment in infrastructure and data collection efforts. Success in this endeavor could transform India from a major tech service provider to a major tech disruptor in the trillion-dollar AI market.