AI 公司为何未被追责入侵行为

Is hacking legal now? Anthropic broke into three systems. OpenAI took political aim and broke into an Australian government health services website. Google felt they were missing out and then hacked into three as-yet-unnamed companies. And of course Meta did some hacking as well.

Consequences so far: none.

The big AI corporations are hiding behind a responsibility gap argument due to the emergent behaviour of their systems. It’s a position tailored to dissuade prosecution as it removes intent, which is typically required for a crime. I understand their systems though, so I will show how they absolutely intended this behaviour, giving a framework for legal evaluation.

Criminality of unauthorized access

The laws around hacking are complex. This is a cross-section of legal requirements in the G20 determining if unlawful access has actually happened.

- (Attempted) Access: The party has to have obtained access to the system, sometimes an attempt is enough

- Without right: The party did not have rights to access the system

- Intent: The party has to have intended accessing a system

Noteworthy here is that intent transfers across actions. If one attempts to hack one system and ends up in a different system, the intent is still there.

Many G20 countries do not require access to have been achieved; the mere attempt is enough. Though in this case, the breaches are clear. “Without right” and “Access” are facts readily admitted by the offending companies. Their desired impunity is based on not having intent, since lacking intent is a wildcard that avoids criminal prosecution in most legal systems. These companies imply their AI acted autonomously without intent, thereby divesting themselves of responsibility.

Was there really no intent on behalf of the companies, or agents of the company, in these hacking incidents? To answer this, we must examine more closely how companies create these AIs. I’ll focus on large language models (LLMs), though many aspects are going to be similar for any AI.

Responsibility gap and intent

The current generation of LLMs comprises many interacting systems and algorithms. Parts of this system exhibit emergent properties. The output of a model is imprecise but not wholly unpredictable.

Weak emergence refers to a system having properties, such as conversational fluency, that appear only at the whole system level. We can still explain every step in the process by analyzing the parts — the fluency doesn’t arise supernaturally. In AI in particular, we use the term to indicate that we cannot look at the parts and predict what output the model will generate without running the model.

There are several key choices that AI companies make when designing their systems:

- Source material: All the text and media they use as input to train their model.

- Training: A feedback loop with both objective and subjective calibration of the model.

- Environment: How the model is exposed to the internet and what tools it has.

- Configuration: Distinct numerical values that limit the capabilities of a model.

Source Material

The source material and training capture the intent of the company behind it. The company decides this for product differentiation. Some companies create AIs specifically for legal analysis, while others develop them for medical diagnosis. Much of what captures our public attention are the general purpose models, the chat bots, general purpose research tools and personal assistants. The companies involved in the breaches have all advertised they are building models to analyze security — a key part of security testing is penetration testing where the tester finds ways to break into a system.

When collating data, there is inevitably a problem of defining scope. Highly specialized AIs have a more limited scope, such as all EU laws and regulations for a paralegal tool, and the most general purpose models need a cross-section of everything. It obviously isn’t possible to input all recorded history across all branches of knowledge, and even those specialized models need auxiliary input to understand search queries. A company chooses which slices of information they use.

The AI creators select which knowledge to include, and perhaps more importantly, which knowledge to specifically exclude. For example, OpenAI and Anthropic both claim to exclude material about weapons. Many filter out erotic and toxic material. All of the companies have done some form of privacy filtering. There are a lot of choices here.

Interesting note: a company called Hugging Face was one of the sites hacked by OpenAI. Hugging Face is a platform about building models and includes classification rules and dataset used to train models. An AI system could hack into such a site to alter such datasets and unburden potentially itself, or at least future generations of itself, from these limiting filters.

Training

Simply feeding input to create a model doesn’t produce the results we want. An AI model has to be trained to give the type of output that is wanted. This is generally done by rating its actual output against desired output. Corrections are input in the model. Repeat until it reaches the desired output. None of this is automatic, nor does it have a standard final form.

Much of this is based on individuals subjectively deciding if an output is correct or not. We reward the model for producing the output we want. That means the cognitive biases, stereotypes and prejudices of all those individuals will be imprinted on the model. The models are far from neutral collectors.

For example, to get a conversational flow in chat bots, a company has to train the system explicitly for this. Their trainers reward friendly answers and reject impersonal and cold ones. The trainers steer the system towards something that sounds helpful and confident. This behaviour is an intended outcome, not a surprising one.

While one can determine a lot about a model by checking it with specific prompts, it will never reveal a full picture of its biases, as those are often subtle and hard to detect. Testing would also not detect subterfuge, such as being trained to detect regulatory probing and give different results — recall the Volkswagen emissions scandal. I don’t believe any companies are doing this yet.

To get a fair impression of how a company intended the model to behave, the company must provide a detailed account of how it was built. Right now they are keeping this private which allows the providers to conflate intended behaviour with the responsibility gap. This effectively grants impunity to all the choices they made during the AI’s production.

Article 53(1)(d) of the EU Regulation 2024/1689 states AI companies must provide a “sufficiently detailed summary”. That is weak wording and further explanatory notices have made it weaker. It’s also primarily aimed at ensuring “rights holders” can assess the model.

Environment

The AI interfaces we have on the web, or integrated in programs, have several layers. We can group a few of them together into a model architecture. This is the underlying “AI” part. It consists of matrices and algorithms to take an input and generate an output. This model is completely passive and isolated.

One of the interesting emergent properties of the LLMs is their ability to use tools. If we describe a series of tools they can use, they will request use of those tools. For example, if we tell it that it can get more information about a topic from a web search, it will emit output that requests a web search. It is possible to get this output to be so well structured that another program can use it as input. That other program can then actually make a web search and emit the result back to the model.

These tools are in a layer above the model. There is code that parses the model responses and handles the tools. An AI is limited by its set of tools. There is nothing in this environment layer that anybody would consider unpredictable or an