Bridging the linguistic divide in Gen AI

Over the last 30-40 years, technology has marked a quantum leap in the progress of humanity. Generative Artificial Intelligence (Gen AI)—the latest frontier of innovation—is making significant strides in critical applications across healthcare, security, finance, and practically, every industry vertical.

However, the narrative of this technological marvel remains incomplete.

The crux of the problem lies in the linguistic divide—a perennial problem that has echoed through every phase of technological development—from desktops and smartphones to the internet and now, Gen AI.

Despite over 7,000 languages spoken globally, the internet predominantly caters to English and a handful of other languages. This linguistic bias poses a substantial limitation for GenAI tools.

The lingual divide

Gen AI represents the pinnacle of technological innovation, yet it grapples with a persistent challenge—the linguistic gap.

Ethnologue—a widely referenced resource on world languages—said as of 2022, there are about 7,139 living languages in the world, yet the development of Gen AI tends to favour data-rich languages like English, Spanish, and German, neglecting the diverse linguistic tapestry present worldwide.

The unfortunate reality is that large language models (LLMs) source their training content from the internet, which lacks sufficient diversity of content in different languages.

India—speaking over 120 major languages and 22 officially recognised languages—exemplifies this disparity. Gen AI's trajectory mirrors previous technological phases, offering limited support for languages like Hindi while potentially sidelining others such as Bengali, Tamil and Punjabi.

This bias not only hampers digital inclusion efforts but also endangers the cultural identity entrenched in linguistic diversity. Access to cutting-edge Gen AI tools remains unevenly distributed, leaving non-English speakers, particularly in India, at a disadvantage. The emotional disconnect is tangible, hindering accessibility and growth for these communities.

The resolution of the linguistic disparity in Gen AI tools is an intricate undertaking that necessitates collaborative efforts and strategic interventions across multiple fronts.

Creating diverse and inclusive training corpora

It is not only essential to include content from various languages, but one also needs to ensure the data is representative of different demographics, cultures, and perspectives.

LLMs trained on biased or incomplete data may struggle to generate accurate and culturally sensitive responses, even in languages fairly well-represented in the training data.

It would require combined and concerted efforts from developers, language specialists, linguists, ethicists, and domain experts to identify and address biases in the data. Educational institutions can also add immense contributions in this area.

Building transparency and ethical standards

It is crucial to establish and adopt clear ethical guidelines and standards for AI development and deployment. These should be drafted in conjunction with industry bodies, public bodies, and private organisations.

To build transparency, continuous monitoring and evaluation mechanisms should also be implemented for AI systems post-deployment.

Government regulations and policies

Government backing through policies promoting linguistic diversity in technology is paramount. Incentivising or mandating the inclusion of regional languages in tech development frameworks ensures a more equitable technological landscape.

This support can involve funding, incentives for research, and the implementation of language inclusion guidelines. The government can also play a pivotal role in establishing regulations and policies that guide the responsible development and deployment of AI.

The promotion of audits and assessments of AI systems can also help ensure compliance with ethical standards and regulations.

Individual action

Individuals and organisations—passionate towards the cause of bridging the linguistic gap—can also play their part towards this cause.

They can engage with communities to educate and spread awareness about the need for language diversity in technology, promote language-inclusive technologies, provide feedback to developers, support multilingual initiatives like contributing to open-source projects, volunteer for language data collection, or participate in language preservation efforts.

The journey ahead

The journey towards an inclusive digital realm starts with acknowledging and acting upon the need for linguistic inclusivity. It's a concerted effort, a collective movement toward a future where technology is a unifying force rather than a divisive factor. The journey doesn't stop at technology; it's about embracing diversity as a strength.

As the world charts its course into a future steered by technological advancements, bridging the linguistic gap in Gen AI tools becomes paramount. By embracing and nurturing linguistic diversity, we can forge a digital landscape that harmonises with the myriad voices constituting the fabric of humanity.

The pursuit of linguistic inclusivity in Gen AI is not merely a technological endeavour; it symbolises a commitment to preserving and celebrating the cultural richness inherent in our diverse languages.

Vidushi Kapoor is the CEO of Process9.