What if the business got its legal advice from a robot?

This year has seen generative AI tools reach the mainstream. It started with DALL-E, Stable Diffusion and Midjourney generating stunning visual images from simple word prompts.

The end of the year saw the launch of ChatGPT – a chatbot capable of providing detailed and convincing responses across a broad range of knowledge domains. This includes the law.

We wanted to see just how good ChatGPT is so put it head-to-head with three of our top lawyers using 50 questions on data protection and contract law. Those questions are here.

ChatGPT will try to provide legal advice

Asking ChatGPT a legal question is easy. Type in “when must I appoint a data protection officer under the GDPR?” and it comes back with an authoritative answer that is pretty accurate (see Q15).

You can go a stage further and ask it to apply the law to the facts. For example, ask it: “I run a hospital and process medical records. Must I appoint a data protection officer under the GDPR?” and it comes back with a fully reasoned answer that is also broadly right (see Q16).

How did ChatGPT compare with our top lawyers?

The outcome depends on your perspective. For a generalised AI chatbot, the answers it produces are amazing. Regardless of the fact not all of them were actually right, most were close and this demonstrates this technology has advanced substantially.

As a source of legal advice, our lawyers gave it 2.5 out of 5; in other words, generally “poor”. ChatGPT provides very confident and authoritative sounding responses. However, it doesn’t always pick up on the nuance and sometimes the right-sounding answers are so completely wrong relying on them is dangerous.

Just because you can doesn't mean you should 

These tools will inevitably become more sophisticated and more powerful, and might well be specifically trained to provide legal advice. This is definitely an area to keep track of.

However, in the meantime, the business should be kept well away from ChatGPT as a trusted legal adviser.

The questions are here and a longer article on our research is here.

Image: ““a lawyer amazed at the quality of legal advice provided by a robot in a photorealistic sepia style in a professional office setting” With help from Dall*E 2.”