What are the current limitations of generative AI tools?

Large language models are only as good as the data they are trained on. The open web is usually used for the training data. This excludes closed data. This is data which sits behind a paywall or which requires authentication to access.

As much data is not available for training because it's not on the open web, or because of its sensitive, there will be gaps in its learning. Some of the data may be incorrect or may be disputed. For this reason, generative AI tools may hallucinate, or get things wrong.

Algorithmic bias

Data on which the LLM was trained may not be impartial. It may reflect human prejudice, misunderstanding or incomplete data. Flawed data might be non-representative, lack information or be biased towards a particular point of view. As the LLM learns from this data, it produces algorithms which amplify these biases.

In addition, programmers may add unfair weighting into the decision-making process. Weighting is a technique to avoid bias, but it requires conscious or unconscious assumptions which can introduce further bias and inaccuracy.

Black box predictions

We see the output, but not the inner workings. Explanations for output are not known and cannot be traced. This can be caused by the size of the data set (often using millions of data points) and a lack of clear reasoning that occurs in more rule-based systems.

Confirmation bias

If you believe something to be true, you will more likely notice supporting evidence, interpret neutral information as supportive and dismiss or avoid opposing viewpoints. AI can be used to reinforce a user's own biases by providing an output the user expects. It can be introduced by selecting and using data that supports the hypothesis. To counter seek different perspectives, encourage critical thinking and ask “what would disprove my belief”. Generative AI is designed to create plausible responses rather than accurate responses.

Eating its own tail / model collapse

The phrase “AI eating its own tail” is a metaphor that describes a situation where AI systems start learning from or generating content based on other AI-generated content, rather than original human-created data. This is a problem because there is a loss of originality, information quality is lost due to amplification of mistakes or biases (forms it’s own ‘echo chamber’) and the training data sets and output become less representative of the real world. Models need to be continually fine-tuned with real data. Generated content has been published to the open web for many years and is difficult to distinguish from human created content.

False balance

Where genAI output is more balanced between different viewpoints than the evidence supports. It may give the impression that some science is contentious, when the evidence shows it is not. It distorts understanding and can amplify fringe viewpoints. It is important to check against the evidence base and clarify the weight of expert consensus.

Groupthink bias

This occurs when AI models are trained on only one perspective or cultural viewpoint. With lack of diversity there is a re-enforcement of existing biases without questioning. It is possible to become overconfident in models if people agree they are working well even if there are subtle errors. The conformity reduces innovations and can create ethical risks. Combat with diverse teams, open discussion and use of external audits

Over-fitting

When the LLM learns the training data but performs poorly on new unseen data. You can identify if there is high training accuracy, but real-world testing is lower, making the model unviable. It means the model often requires more diverse and representative data or potentially simplified with fewer parameters. The model should be tested on different subsets of data to cross validate and stop training if the validation data starts to drop.

Representation bias

When training data does not accurately represent different groups. This happens if data used is limited i.e. mostly Western countries, one gender, one age group. There may be a historical bias around inequalities and stereotypes which gets passed on to the AI. Group samples may be underrepresented or missing entirely. Audits should be conducted across different groups; more diverse datasets should be used, and fairness metrics should be included in evaluation.

Survivorship bias

Occurs when the model is trained on data which only represents a successful subgroup rather than all subgroups. For example, only on successful outcomes for a drug trial rather than the unsuccessful ones. To avoid ensure complete datasets are used, ask “what is missing from this data?” and use ‘failure cases’ to improve the model robustness.

Temporal bias

These are changes that have happened over time, which were not accounted for in the training e.g. information becomes outdated or unrepresentative of current conditions. It is important to monitor performance, update the training data and re-train if needed. It is also possible to use time-aware models which consider trends and seasonality or include timestamped data to help model and understand temporal patterns.

How can these biases be mitigated?

Ensure the large language model is trained on diverse data. Developers need to test for bias.
Adversarial training. One neural network generates content while another checks for bias. The model then learns to avoid these biases.
Data augmentation. Deliberately include a variety of perspectives into the training data.
Resampling. Resample training data to ensure under-represented groups get more attention.
Tell the generative AI tool when it gets something wrong or is being biased. Tools learn from prompts too.
Be aware of these biases and account for them when assessing the output.
Reflect on the design on your prompts to check for your own biases.

Large language models are becoming better at identifying and mitigating for biases, and dealing with current limitations. Even so, you need to evaluate the outputs to make sure it feels right. Always check the sources used and what the actual evidence is telling you.

Artificial intelligence, The Knowledge Network