UK: ChatGPT fails solicitor’s conveyancing test

AI chatbot ChatGPT has a “long way to go” before it replaces conveyancers, according to the head of property at an England regional law firm who asked it a range of questions relating to first-time buyers. 

Daniel Chard, a partner at Bird & Co in East Midlands - England, found that the chatbot slipped up on stamp duty and first-time buyer ISAs, though he was “impressed with elements of its capabilities” in answering popular queries.

Mr Chard said it was “clear to see how ChatGPT could be disruptive for many sectors, including the legal sector” but it could not replace the advice of a legal expert, especially in complex situations such as buying a house.

“This comes down to the nuances of the law, which changes frequently, as well as the human element of interactions with lawyers, which simply cannot be replaced by a chatbot.”

Mr Chard said the most “striking issue” with ChatGPT “appearing to give” legal advice was that it was “not able to tailor any advice given to the person asking”, which “could easily lead first-time buyers down a path” where they wasted time and money.

“It is clear AI tools like this could be useful in providing basic understanding to legal queries. It could even be useful as a starting point for many first-time buyers who are curious about the processes involved in buying a home, but responses should be taken as surface-level knowledge.

“However, after carrying out this study, it’s clear ChatGPT has a long way to go before it replaces any form of legal expert in giving actionable advice involving conveyancing.”

He said ChatGPT produced “a great, in-depth answer” when asked ‘what is a first-time buyer?’, including incentives usually associated with first-time buyers.

When asked ‘how to get a mortgage as a first-time buyer’ the chatbot correctly mentioned credit checks and getting pre-approval for a mortgage before putting in an offer.

However, it provided a “very basic and simplified version of events” when it came to actually buying a home, making no mention of putting in an offer and waiting for this to be accepted.

Asked whether first-time buyers were “guaranteed a mortgage”, Mr Chard said ChatGPT delivered another “great answer”, which went into detail about how mortgages were approved, although a solicitor could “offer a more in-depth explanation surrounding available schemes and requirements”.

The chatbot was then asked about stamp duty for first-time buyers in the UK. Mr Chard said its answer was incorrect and out-of-date.

ChatGPT said that “as of September 2021”, first-time buyers paid no stamp duty on property worth up to £300,000 and 5% after that.

In fact, temporary measures applying from September last year mean that first-time buyers are exempt from stamp duty on properties worth up to £425,000, and pay 5% on the next £200,000.

The chatbot slipped up again when asked about first-time buyer ISAs in the UK. It stated that the Help to Buy ISA was replaced by the Lifetime ISA in 2017. Mr Chard said this was incorrect and it was possible to set one up until November 2019.

When asked if a first-time buyer could use the Help to Buy ISA if buying with someone who was not a first-time buyer, the chatbot remembered that the ISA scheme was “closed to new applications in November 2019”, contradicting the previous answer.

Mr Chard said a further contradiction occurred when the response stated that people could “still open and contribute to a Help to Buy ISA”, before mentioning that they could not.