As security becomes a major focal point of the AI industry, Anthropic has added a new tool to its defense arsenal.

This week, the AI lab unveiled FlashProve, a system built for "provable inference" to ensure that an AI output comes from the model a system claims it does. This open-source system uses cryptography to verify that generated output is from a given model, rather than from one that a model router swapped in.

"A user querying a model today cannot check that the responses came from the specific model that a developer claims is responding, as opposed to one secretly substituted in by a routing service for cost reasons, or even by an attacker who has tampered with the token stream," Anthropic wrote in its blog post.

While the current state of AI forces users to trust that model providers and routing services are telling the truth about which models they serve, FlashProve removes that ambiguity by bringing zero-trust principles to the process.

• To do this, Anthropic uses a cryptographic technique called "zkSNARK," or zero-knowledge succinct argument of knowledge, which lets a model provider create a fingerprint of a model without revealing its proprietary information, such as its weights.
• Then, along with a user's output, FlashProve will generate a short proof that a user can verify using software "in seconds on a CPU." If the response deviates from the specified model, the user's check fails.
• Along with keeping the model's original weights private, Anthropic touted that the proofs generated by this system are "small and cheap to check," allowing users to verify the authenticity of outputs "without access to elaborate infrastructure."

Anthropic said that its motivation to develop provable inference is twofold: To increase customer trust, and to pace frontier development by giving auditors the ability to monitor the outputs from frontier models without the need for expensive hardware.

"The goal of provable inference (PI) is to enable a user with limited computational resources and without any information about the weights to verify that a response came from the requested model," Anthropic wrote.

Our Deeper View

FlashProve seeks to solve one vector of a much larger problem that the AI industry is facing: trust. While this provides a solution to ensure that an output comes from the AI model that it says it does, preventing tampering and output spoofing, it's only one piece of the puzzle. For instance, hallucination is a never-ending battle that AI developers are fighting, creating a layer of mistrust with the perception that users can't trust the accuracy of the outputs. Additionally, users also face the quandary of whether or not to trust these model providers with their data at all, given that, if it's used in training, it can be extracted via prompt injection attacks. While FlashProve certainly represents a step forward, there is far more work to be done to make these models trustworthy.