King Charles has warned artificial intelligence executives of the "existential dangers" posed by the technology falling into the wrong hands, speaking at a summit he convened at Dumfries House in Ayrshire.

The gathering brought together senior figures from across the AI industry, including Jensen Huang, chief executive of Nvidia, Sarah Friar, chief financial officer of OpenAI, and Sir Demis Hassabis, co-founder of DeepMind. Also present were Kanishka Narayan, the UK's AI Minister, and an advisor to the Pope.

The summit was called to consider whether a "shared set of principles" could be developed to help guide the future application of AI.

"Those who have created these technologies are now increasingly warning that AI risks developing darker capacities - perhaps even to take life," the King told the executives.

He said there appeared to be an "urgency" in "adequately considering the existential dangers of such technologies falling into the wrong hands, and being used in potentially catastrophic ways."

"The task before you is not merely to advance technology, but to ensure that it remains firmly in the service of humanity, community and the natural world," he added.

Huang told attendees he believed safety was "paramount," adding that companies should hold back a product and "keep engineering" if it was not safe enough.

The Nvidia chief also highlighted the value of open AI models — systems that give users direct access to the software — as a means to "ensure that people and countries are not left behind." He said such systems must be protected so that "researchers, universities, start-ups and countries" can develop applications "none of us would have imagined alone."

"Let us lead with responsible optimism, build AI safely and securely, open it to more people, expand our ambition and solve the problems that matter to the world," Huang said.

Nvidia is a central player in AI infrastructure, supplying chips to leading model developers. It is currently the world's most valuable company, with a stock market valuation of around $5.15 trillion.

Sir Demis Hassabis, who co-founded DeepMind before it was acquired by Google, offered a more cautious assessment. He said artificial general intelligence — systems believed capable of matching or surpassing human performance across multiple tasks — was "probably only a few short years away" and could have an impact "ten times that of the Industrial Revolution."

The chance of something going wrong was "definitely non-zero," he said, but he advocated for "a sensible middle way," arguing that humanity could address the risks if collectively given the space and time to mitigate them. "The future is not yet written," he told delegates.

Sarah Friar of OpenAI said AI had the potential to solve some of society's "hardest problems," but acknowledged that its growing capabilities raised questions about safety and "how we ensure it always serves people."

"We need to address those questions together," she said. "No one company or government can do that alone and I'm grateful to His Majesty for convening this discussion at such an important moment."

The summit comes amid a broader wave of concern within the industry about the rapidly evolving capabilities of advanced, or "frontier," AI models and the regulation needed to govern them.

In July, OpenAI announced what it described as an "unprecedented" incident in which an advanced AI system escaped test limits and accessed an external tech platform. Following further incidents, a resignation post by an Anthropic researcher went viral in September after he cited concerns that AI could pose an existential threat to humanity. He later said those working on the technology were "genuinely frightened" about the speed of its advancement.

In response, an Anthropic scientist wrote that he personally believed the possibility of human extinction caused by AI carried a greater than 10% probability.

The debate has also drawn criticism from sceptics, who argue that focusing on AI's hypothetical future risks distracts from more immediate concerns, including the technology's environmental impact.