Data Preprocessing in Machine Learning: Techniques to Improve Model Accuracy

Introduction

Artificial intelligence (AI) has evolved rapidly in the last few years, particularly in the domain of Natural Language Processing (NLP). A key change agent in the field is the Transformer architecture, which was described by Vaswani et al. in the article “Attention is All You Need” in 2017. Transformers represent a hallmark of AI development, mainly because they are no longer reliant on previous models, which used recurrent or convolutional architectures. Transformers, by their self-attention mechanisms, can represent long-range (in terms of temporal or spatial relationships) dependencies in the data without being dominated by sequential order in processing such input. When you think about your favorite chatbots, document summarizers, and real-time translation applications, they are certainly driven by transformers now, and we discuss key transformer models, BERT, GPT, and glimpses of things done differently with information that extends their parents (BERT, GPT).

Want to learn how machine learning models are trained? Read our Supervised Learning Guide.

What is a Transformer in AI?

A transformer is an architecture of deep learning models that have been developed especially for processing sequential data, i.e., text, without the use of recurrent networks. Its main building block is the self-attention mechanism that enables the model to consider the relative importance of various words within a sentence all at once.

Transformer, Open AI, & BERT

Key Components

The following are the key components of a Transformer:

• Self-Attention
• Multi-Head Attention
• Positional Encoding
• Feedforward Layers
• Layer Normalization

Transformers allow parallel processing (as opposed to RNNs) and scale much more favorably with data, making them the new norm for NLP work.

To understand how we get the data ready for these models, check out our post on Data Preprocessing Techniques for Machine Learning

BERT Ascendance: Bidirectional Encoder Representations from Transformers

BERT is a great evolution in language understanding launched by Google in 2018. Before BERT, language models only read text in order from left to right.

BERT reads text in both directions, left to right and from right to left, which captures more context.

BERT consists of three main components:

• Masked Language Modelling (MLM): Predict the missing word in a sample sentence.
• Next Sentence Prediction (NSP): Determine whether one sentence follows another.
• Bidirectional: Provides better performance overall in Q&A and classification tasks.

BERT can be applied to the following applications:

• Google Search
• Chatbots and other virtual assistants
• Sentiment analysis
• Named Entity Recognition (NER)

GPT: Generative Pre-trained Transformer

GPT (designed by OpenAI) is the autoregressive aspect of transformers. While BERT is good at understanding text, GPT is good at generating coherent text. This is crucial as it is the underpinning of generative AI such as ChatGPT.

AI Innovation

Features

The following are the major features of GPT:

• Unidirectional Training: Reads left-to-right.
• Retraining and then fine-tuning
• Transformer Decoder Architecture

Milestones

Different models of GPT are found; each model has some key advantages:

• GPT-1: Introduced generative pretraining.
• GPT-2: Mega text generation
• GPT-3: 175 billion parameters
• GPT-4 & GPT-4o: Multi-modal models (text, images, code)

Key Difference Between BERT and GPT

Difference between BERT & GPT

Understanding their differences helps you choose the right model based on your application, either classification or generation.

Why does Transformer Direction matter?

Transformer directionality is a vital component of the capabilities of models like BERT and GPT. For example, BERT can model contextual meaning in a bi-directional manner. Using both left and right context, it can more thoroughly understand the meaning of a word. GPT is designed in a left-to-right (or unidirectional) manner so that it can generate coherent, contextual, and relevant sentences. The model architecture we choose directly impacts how we will perform: BERT is better suited to classification tasks, sentiment assessment, and question answering, where understanding the context is more important. On the other hand, GPT is stronger at creative applications such as writing assistants, story creation, or writing code completions. However, it should be noted that unidirectionality does limit the ability to capture full contextual cues, in some cases. This comparison provides an essential architectural trade-off between understanding and generating. T5 and XLNet are examples of models that hope to capitalize on both approaches. Understanding these differences allows developers and researchers to choose from a complete list of best-suited model architectures for their NLP tasks.

Transformer-Based Model Beyond BERT and GPT

Several other powerful models based on transformers have been released recently, all designed to target different aspects of NLP.

• T5 (Text-to-Text Transfer Transformer)

It is used to transform all tasks into a text-to-text framework. This type of model performs well in translation, summarization, and Q&A.

• XLNet

XLNet extends BERT; it looks at all permutations of a sentence. And removes the limitation of masked language modeling.

• Roberta

It stands for Robustly optimized BERT is trained with additional data, and takes much longer training times.

• mBERT and XLM-R

This model is multilingual. It has over 100 languages, and this type of transformer model is great for global AI use cases.

How Transformers Revolutionized AI

Transformers changed the trajectory away from rule-based systems and RNNs, toward large pre-trained models that can:

• Be fine-tuned with less data
• Accomplish multi-NLP tasks
• Generalize across domains and languages

Transformers also enabled transfer learning in NLP, meaning that it's now possible to adapt a model, trained on a large dataset, with little tuning toward downstream task-specific objectives.

Practical Examples of Transformer Models

The following are some of the practical examples of transformer models:

Search engines (Google BERT)

• ChatGPT and Alexa are like Chatbots and virtual assistants
• Legal and medical document processing
• Language translation (Google Translate)
• Automatic summarizers
• GitHub Copilot-like tools that help in code generation and debugging

Potential Issues and Limitations of Transformers

Even though they are impressive, transformer models still have a few limitations:

• An expensive and memory-intensive process
• Bias and fairness issues are implicit in their training data
• Explainability issues
• Energy, consumption, and sustainability issues

Researchers are currently addressing how to make transformers more efficient and responsible.

The Future of Transformer Models in AI

We are already seeing multi-modal transformers that understand text, image, audio, and video together. Here are some examples:

• GPT-4o (OpenAI’s Omni model)
• CLIP (image + text)
• Flamingo and Gemini (multi-modal models)

These models close the gap between how humans think and perceive things in the world and how machines learn about the world, getting us one step closer to reaching Artificial General Intelligence (AGI).

Conclusion

Transformer models are rapidly changing the world of artificial intelligence, particularly language models in NLP. From understanding context (BERT) to generating text that resembles human language (GPT), transformer models are permeating every area of life. As we continue to develop models that are more powerful and multi-modal, transformers will remain the foundation for intelligent systems across applications and industries.

Whether you are building a language translator, chatbot, or document classifier, there is a transformer model that fits your needs, and some other people are likely building them every day.