GOVERNANCE

Thousands call for superintelligence guardrails

As tech leaders continue their frenzied pursuit of creating AI that’s better than us, some are calling to pump the brakes on development. 

The Future of Life Institute, a nonprofit dedicated to preventing “extreme large-scale risks” posed by technology, launched a petition on Wednesday dedicated to putting guardrails on the development of superintelligence, or AI that outperforms humans in every conceivable task. 

The petition, dubbed the “Statement on Superintelligence,” states that the development of superintelligence should be prohibited before there are calls for “broad scientific consensus” on how to do so safely and controllably. 

The petition has gained rapid momentum since its debut, with more than 45,000 signatures and counting. Its signatories include: 

• Leading AI experts Geoffrey Hinton and Yoshua Bengio;

• Executives such as Virgin Group founder Sir Richard Branson and Apple co-founder Steve Wozniak;

• Political figures spanning the aisle, including former President of Ireland Mary Robinson, Duke and Duchess of Sussex Prince Harry and Meghan Markle, and former Trump Administration strategist Steve Bannon;

• And a host of entertainment industry figures, including Kate Bush, Grimes, Joseph Gordon-Levitt and Will.I.am.

“This statement was really conceived as a way to demonstrate the overwhelming and bipartisan majority of people who are extremely concerned about the current trajectory of advanced AI development,” Ben Cumming, communications director at the Future of Life Institute, told The Deep View. Polling by the organization released last week found that 73% of U.S. adults want robust AI regulation, and 64% believe superintelligence shouldn’t be developed until it’s proven safe.  

Though AI leaders claim that the development of powerful AI and superintelligence will lead to a waterfall of breakthroughs that evolve humanity as a whole, the risks are numerous, said Cumming. The tech’s possibilities could upend the economic landscape, supercharge disinformation, and muddy culture with the mass production of slop, he said. 

And while the petition calls for stricter guardrails, it’s hotly debated whether it will even be possible to control this tech at all, Cumming added. 

“The rule in the world has been that the smarter species control the stupid ones, and we’d be making ourselves the stupider ones in that scenario,” he said. 

Reaching Superintelligence would be a natural consequence of AGI, or AI that's capable of matching human capabilities in every domain. The pursuit of this has led to a “frantic competition,” Cumming noted, where you're either in the race or watching from the sidelines, safety and risk be damned. The problem, however, is that being in the race at all is incredibly costly –  hence why tech firms are pouring billions into compute. This has already concentrated incredible power in the hands of a select few, leading to a divide between the haves and the have-nots. That divide may only worsen over time as we inch towards increasingly powerful models.

TOGETHER WITH COGNITION

The 5-second rule: How and why Windsurf (Cognition) trained a model just for grepping

Earlier this year, Cognition (makers of Devin) acquired Windsurf, the first agentic IDE. The Cognition folks noticed that IDE-based coding agents, meant to be "synchronous" with developers, were spending >60% of their first turn just looking for context, leading to breaks in developer flow states. So, last week, they shipped the SWE-grep family: two agentic models trained for lightning-fast code context retrieval.

In code retrieval benchmarks, Cognition found that SWE-grep and SWE-grep-mini (2,800 tok/sec!) matched frontier model retrieval performance while completing searches up to 20x faster.

They shared a detailed blogpost of how they trained SWE-grep using multi-turn reinforcement learning.

TLDR:

1. Parallelize code search: they trained SWE-grep to fire off up to eight simultaneous tool calls per turn, and limited it to four turns total (so the model was pressured to learn how to retrieve code in few steps)

2. Use a weighted F1-reward that penalizes retrieving too much context: irrelevant information leads to context pollution, which impairs performance.

3. Use per-sequence importance sampling to get unbiased gradient estimates that stabilize training.

Want to try SWE-grep? Download Windsurf (it's free). Type a prompt into the AI chat box (Cascade) and hit Cmd-Enter (or Ctrl-Enter) to trigger the Fast Context sub-agent.

HARDWARE

Defense AI heats up

AI is going on the defensive. 

Valthos, a startup focused on building AI tools to defend against bioweapon attacks, emerged from stealth on Friday with $30 million in funding from OpenAI, Founders Fund and Lux Capital. The nine-person founding team comes from Palantir, DeepMind, the Broad Institute, and the Arc Institute, the company said in its blog post announcing the funding. 

Valthos is dedicated to building AI that rapidly identifies emerging biological threats and develops and updates medical countermeasures to match them. 

“We envision a future where any threat to human health can be immediately identified and neutralized, whether the origin is external or within our own bodies,” the company said in its post. 

The funding highlights investors' growing interest in defense tech: According to PitchBook data from August, funding for the defense tech sector has surpassed $28 billion this year so far, with more than $19 billion of that being in the second quarter alone. 

And AI is a growing piece of the defense tech puzzle, with autonomous vehicles, drones and advanced computing platforms winning big, Pitchbook notes. For example, Anduril, the drone firm led by Palmer Luckey, raised $2.5 billion in June, hitting a valuation of $30.5 billion. Anduril has since touted new AI-powered weaponry, including a modular headset built for soldiers. 

And model developers aren’t left out, either. Anthropic, Google, OpenAI and xAI were each granted up to $200 million from the Department of Defense (now the Department of War) in June to help the agency address “critical national security challenges” using AI. 

TOGETHER WITH AUTH0

Build Powerful Agents with Auth0

When you’re dealing with sensitive information, AI agents can be a concerning topic – after all, unlike people, they don’t log in with a password or stop to double-check before taking action. But with Auth0’s newest solution, you can focus on innovation without compromising security. 

Auth0 for AI Agents is the complete auth solution for building AI agents more securely. Now, your entire business can adopt AI without losing control – and it only takes a few lines of code. Auth0 for AI Agents offers a built-in Token Vault that lets you integrate your apps and agents with third-party tools, so they can do everything from handling and securely storing access tokens to connecting users to external tools, processing authorization requests, and more.

Ready to implement AI with confidence (without sacrificing security)? Try Auth0 for free today and be among the first to see these AI tools in your account.

CONSUMER

Snack giant Mondelez joins AI ad hype

On Friday, Mondelez, the snack company that makes Oreos and Ritz, announced plans to use AI to power its marketing efforts. 

The company will use an internal generative AI tool it developed in partnership with the ad agency Publicis Groupe and Accenture. Mondelez expects the tool to cut marketing and production costs by up to 50%, Reuters reported. The ads could air by the holiday season of 2026.

The generative AI tool, capable of making short TV ads, cost Mondelez around $40 million to develop. 

Mondelez’s AI tool marks the latest in a series of efforts to bring AI to advertising and marketing. 

• On Thursday, British advertising firm WPP said it would give brands access to its AI-powered marketing platform to create and publish ad campaigns.

• Last week, Adobe launched a program called AI Foundry, which works with enterprises to create models based on their branding and IP for marketing content capable of commercial use. 

• Meta, meanwhile, rolled out generative AI features in its Ad Manager in 2023 and is working on an AI tool to automate ad campaigns from start to finish. 

• And Amazon debuted Creative Studio in September, an agentic AI tool to help mid-market brands produce ads across Amazon properties such as Prime Video.

Still, using AI in ad development can raise copyright issues or public image concerns if done incorrectly (e.g., New York mayoral candidate Andrew Cuomo’s AI-generated attack ad against candidate Zohran Mamdani). But internally-developed tools like Mondelez’s, or fine-tuned models like Adobe’s, could overcome some of those challenges.

LINKS

• Snap is in talks to raise $1 billion to support AR glasses bet

• Stability, EA partner to develop generative AI models and tools for game designers

• Sora videos are now stumping deepfake detection tools

• IBM can use AMD chips for quantum computing error correction

• Apple has started shipping AI servers from its Houston factory

• Anthropic opens up office in Seoul, marking its third in APAC

• SambaNova is exploring a sale after fundraising has stalled

• Altman taps leading researcher for brain-computer-interface startup Merge Labs

• Mistral AI Studio: A production AI platform to move agents from experimentation to deployment.  

• BrowserOS: an open-source, privacy-first alternative to OpenAI’s Atlas or Perplexity’s Comet. 

• Velvet: An AI video tool for generating and editing cinematic videos in seconds.  

• Google Earth AI: A new family of geospatial AI models and reasoning agents, bringing Gemini to Google Earth.

• Tahoe-x1: A 3-billion-parameter foundational model that learns representations of genes, cells and drugs for cancer and biology research

• OpenAI: Deployed Researcher, Strategic Deployment

• Salesforce: Research Scientist - Salesforce AI Research

• Stripe: Machine Learning Engineer, Foundation Model

• Adobe: Applied Scientist

GAMES

POLL RESULTS

With Anthropic leaning more on Google’s TPUs, are we starting to see real cracks in NVIDIA’s dominance?

• Yes (20%)

• Not yet (30%)

• No (13%)

• Maybe (23%)

• Other (share more) (14%)