Building a Transformer for Khmer Language Modeling

Understanding Character-Level Language Models with Real Khmer Sentences

---

Introduction

Language modeling is one of the most fascinating applications of deep learning, and when applied to languages with unique scripts like Khmer, it becomes even more interesting. In this comprehensive tutorial, we’ll build a decoder-only Transformer for autoregressive language modeling at the character level for the Khmer language.

What You’ll Learn

• Transformer Architecture: Understanding self-attention and multi-head attention
• Autoregressive Language Modeling: How to predict the next character in a sequence
• Khmer Language Processing: Working with the Khmer abugida script
• Character-Level Modeling: Why and how to model text at the character level
• Practical Implementation: Complete PyTorch implementation with real Khmer sentences

Why Khmer?

Khmer, the official language of Cambodia, uses an abugida script (Unicode range U+1780–U+17FF) with:

• 33 consonants (ក, ខ, គ, etc.)
• 23 vowels (ា, ិ, ី, etc.)
• Various diacritics and special characters

The non-segmented nature of Khmer script makes word tokenization challenging, making character-level modeling ideal for capturing the language’s unique patterns.

---

1. Understanding the Transformer Architecture

What is a Transformer?

Introduced in the groundbreaking paper “Attention is All You Need” (Vaswani et al., 2017), Transformers revolutionized natural language processing by replacing sequential processing with parallel computation. Unlike Recurrent Neural Networks (RNNs) that process sequences one element at a time, Transformers can process entire sequences simultaneously.

Key Components

1.1 Self-Attention Mechanism

Self-attention is the heart of the Transformer. It allows each position in a sequence to attend to all other positions, learning relationships between different parts of the input.

Mathematical Formulation:

For an input sequence $X \in \mathbb{R}^{n \times d}$ (where $n$ is sequence length and $d$ is embedding dimension):

1. Queries, Keys, and Values:

   • $Q = XW_Q$ (Query matrix)
   • $K = XW_K$ (Key matrix)
   • $V = XW_V$ (Value matrix)

   Where $W_Q, W_K, W_V \in \mathbb{R}^{d \times d}$ are learnable weight matrices.

2. Attention Scores: $\text{Scores} = \frac{QK^T}{\sqrt{d_k}}$

   The scaling factor $\sqrt{d_k}$ prevents the dot products from growing too large.

3. Attention Output: $\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$

Intuition: Think of self-attention as a way for each character to “look at” all other characters and decide which ones are most relevant. For example, in Khmer, a vowel like ‘ា’ might pay more attention to the preceding consonant ‘ក’ to form the syllable ‘កា’.

1.2 Multi-Head Attention

Instead of using a single attention mechanism, Transformers use multiple attention “heads” in parallel:

$\text{head}_i = \text{Attention}(QW_{Q_i}, KW_{K_i}, VW_{V_i})$

$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W_O$

This allows the model to capture different types of relationships simultaneously - some heads might focus on phonetic patterns, others on syntactic structures.

1.3 Positional Encoding

Since Transformers process sequences in parallel, they need a way to understand the order of elements. This is achieved through positional encoding:

$PE(\text{pos}, 2i) = \sin\left(\frac{\text{pos}}{10000^{2i/d}}\right)$

$PE(\text{pos}, 2i+1) = \cos\left(\frac{\text{pos}}{10000^{2i/d}}\right)$

These sinusoidal functions provide unique position information that’s added to the input embeddings: $x + PE$.

1.4 Feed-Forward Networks

Each position is processed independently through a feed-forward network:

$\text{FFN}(x) = \text{ReLU}(xW_1 + b_1)W_2 + b_2$

1.5 Layer Normalization and Residual Connections

These components stabilize training:

$\text{LayerNorm}(x + \text{Sublayer}(x))$

Where LayerNorm normalizes each feature dimension: $\text{LayerNorm}(x)_i = \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}} \cdot \gamma + \beta$

---

2. Implementing the Transformer Components

Let’s implement these concepts in PyTorch:

import torch
import torch.nn as nn
import math

# Positional Encoding
class PositionalEncoding(nn.Module):
    def __init__(self, d_model, max_len=500):
        super(PositionalEncoding, self).__init__()
        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0)
        self.register_buffer('pe', pe)

    def forward(self, x):
        return x + self.pe[:, :x.size(1), :]

# Multi-Head Self-Attention
class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, num_heads):
        super(MultiHeadAttention, self).__init__()
        assert d_model % num_heads == 0
        self.d_k = d_model // num_heads
        self.num_heads = num_heads
        self.q_linear = nn.Linear(d_model, d_model)
        self.k_linear = nn.Linear(d_model, d_model)
        self.v_linear = nn.Linear(d_model, d_model)
        self.out_linear = nn.Linear(d_model, d_model)
        self.dropout = nn.Dropout(0.1)

    def forward(self, x, mask=None):
        batch_size = x.size(0)
        # Linear transformations and reshape for multi-head
        q = self.q_linear(x).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
        k = self.k_linear(x).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
        v = self.v_linear(x).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
        
        # Compute attention scores
        scores = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.d_k)
        
        # Apply causal mask for autoregressive modeling
        if mask is not None:
            scores = scores.masked_fill(mask == 0, -1e9)
        
        # Apply softmax and dropout
        attn = torch.softmax(scores, dim=-1)
        attn = self.dropout(attn)
        
        # Apply attention to values
        context = torch.matmul(attn, v)
        context = context.transpose(1, 2).contiguous().view(batch_size, -1, self.num_heads * self.d_k)
        output = self.out_linear(context)
        return output

# Feed-Forward Network
class FeedForward(nn.Module):
    def __init__(self, d_model, d_ff):
        super(FeedForward, self).__init__()
        self.linear1 = nn.Linear(d_model, d_ff)
        self.linear2 = nn.Linear(d_ff, d_model)
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(0.1)

    def forward(self, x):
        x = self.relu(self.linear1(x))
        x = self.dropout(x)
        x = self.linear2(x)
        return x

# Transformer Decoder Block
class DecoderBlock(nn.Module):
    def __init__(self, d_model, num_heads, d_ff):
        super(DecoderBlock, self).__init__()
        self.self_attn = MultiHeadAttention(d_model, num_heads)
        self.norm1 = nn.LayerNorm(d_model)
        self.ff = FeedForward(d_model, d_ff)
        self.norm2 = nn.LayerNorm(d_model)
        self.dropout = nn.Dropout(0.1)

    def forward(self, x, mask):
        # Self-attention with residual connection
        attn_output = self.self_attn(x, mask)
        x = self.norm1(x + self.dropout(attn_output))
        
        # Feed-forward with residual connection
        ff_output = self.ff(x)
        x = self.norm2(x + self.dropout(ff_output))
        return x

---

3. Autoregressive Language Modeling

What is Autoregressive Modeling?

Autoregressive modeling predicts the next element in a sequence given all previous elements. For language modeling, this means predicting the next character given all previous characters.

Mathematical Formulation:

The model learns to predict $P(x_t | x_1, \dots, x_{t-1})$ for each position $t$ in the sequence.

Training Objective: Minimize the cross-entropy loss: $\mathcal{L} = -\sum_{t=1}^n \log P(x_t | x_1, \dots, x_{t-1})$

Causal Masking

A crucial component for autoregressive modeling is the causal mask, which ensures that when predicting position $t$, the model can only attend to positions $1$ through $t-1$.

# Create Causal Mask
def create_causal_mask(seq_len):
    mask = torch.tril(torch.ones(seq_len, seq_len)).unsqueeze(0).unsqueeze(0)
    return mask

This creates a lower triangular matrix where:

• 1 indicates positions that can be attended to
• 0 indicates positions that should be masked (set to $-∞$)

Complete Transformer Model

class KhmerTransformer(nn.Module):
    def __init__(self, vocab_size, d_model=128, num_heads=4, num_layers=2, d_ff=512):
        super(KhmerTransformer, self).__init__()
        self.embedding = nn.Embedding(vocab_size, d_model)
        self.pos_encoding = PositionalEncoding(d_model)
        self.decoder_layers = nn.ModuleList([
            DecoderBlock(d_model, num_heads, d_ff) for _ in range(num_layers)
        ])
        self.linear = nn.Linear(d_model, vocab_size)
        self.dropout = nn.Dropout(0.1)

    def forward(self, x, mask):
        # Embedding and positional encoding
        x = self.embedding(x)
        x = self.pos_encoding(x)
        x = self.dropout(x)
        
        # Pass through decoder layers
        for layer in self.decoder_layers:
            x = layer(x, mask)
        
        # Final linear layer for vocabulary prediction
        output = self.linear(x)
        return output

---

4. Khmer Language Processing

Why Character-Level Modeling for Khmer?

1. Complex Script: Khmer’s abugida script has complex character combinations
2. No Word Boundaries: Words aren’t separated by spaces
3. Syllable Structure: Characters combine to form syllables (e.g., ‘ក’ + ‘ា’ = ‘កា’)
4. Vocabulary Management: Avoids the need for large vocabularies

Real Khmer Sentences

We’ll use authentic Khmer sentences to train our model:

def prepare_khmer_data(num_samples=1000, max_len=50):
    # Real Khmer sentences with English translations
    khmer_sentences = [
        "ខ្ញុំចូលចិត្តញ៉ាំបាយ។",  # I like to eat rice.
        "តើអ្នកសុខសប្បាយជាទេ?",  # How are you?
        "ថ្ងៃនេះអាកាសធាតុល្អណាស់។",  # The weather is very good today.
        "ខ្ញុំស្រឡាញ់សៀវភៅ។",  # I love books.
        "ភ្នំពេញជារាជធានីនៃប្រទេសកម្ពុជា។",  # Phnom Penh is the capital of Cambodia.
        # ... more sentences
    ]
    
    # Build vocabulary from all characters
    all_chars = set(''.join(khmer_sentences))
    vocab = ['<PAD>', '<SOS>', '<EOS>'] + sorted(list(all_chars))
    char_to_idx = {char: idx for idx, char in enumerate(vocab)}
    idx_to_char = {idx: char for idx, char in enumerate(vocab)}
    
    # Prepare training data
    data = []
    for _ in range(num_samples):
        sentence = random.choice(khmer_sentences)
        if len(sentence) > max_len - 2:
            sentence = sentence[:max_len - 2]
        sequence = ['<SOS>'] + list(sentence) + ['<EOS>']
        data.append([char_to_idx[char] for char in sequence])
    
    return data, char_to_idx, idx_to_char, vocab

Special Tokens

• <PAD>: Padding token for batch processing
• <SOS>: Start of sequence token
• <EOS>: End of sequence token

---

5. Training Process

Teacher Forcing

During training, we use teacher forcing - feeding the ground truth sequence as input and predicting the next character. This helps the model learn faster.

Training Loop

def train_model(model, data, char_to_idx, idx_to_char, epochs=10, batch_size=32, device='cpu'):
    model = model.to(device)
    optimizer = optim.Adam(model.parameters(), lr=0.001)
    criterion = nn.CrossEntropyLoss(ignore_index=char_to_idx['<PAD>'])

    for epoch in range(epochs):
        model.train()
        random.shuffle(data)
        total_loss = 0
        
        for i in range(0, len(data), batch_size):
            batch = data[i:i + batch_size]
            
            # Pad sequences to same length
            max_len = max(len(seq) for seq in batch)
            padded = [seq + [char_to_idx['<PAD>']] * (max_len - len(seq)) for seq in batch]
            input_seq = torch.tensor(padded, dtype=torch.long, device=device)
            
            # Create target sequence (shifted by 1)
            target_seq = input_seq[:, 1:].contiguous()
            input_seq = input_seq[:, :-1]
            
            # Create causal mask
            mask = create_causal_mask(input_seq.size(1)).to(device)
            
            # Forward pass
            optimizer.zero_grad()
            output = model(input_seq, mask)
            loss = criterion(output.view(-1, output.size(-1)), target_seq.view(-1))
            
            # Backward pass
            loss.backward()
            optimizer.step()
            total_loss += loss.item()
        
        print(f"Epoch {epoch + 1}, Loss: {total_loss / (len(data) // batch_size)}")

Key Training Concepts

1. Sequence Shifting: Input is [<SOS>, char1, char2, ...], target is [char1, char2, ..., <EOS>]
2. Causal Masking: Ensures autoregressive property
3. Padding: Handles variable-length sequences in batches
4. Loss Calculation: Cross-entropy loss ignoring padding tokens

---

6. Text Generation

Sampling Strategy

Once trained, we can generate new Khmer text by sampling from the model’s predictions:

def generate_text(model, char_to_idx, idx_to_char, max_len=50, device='cpu'):
    model.eval()
    sequence = [char_to_idx['<SOS>']]
    
    with torch.no_grad():
        for _ in range(max_len):
            input_seq = torch.tensor([sequence], dtype=torch.long, device=device)
            mask = create_causal_mask(input_seq.size(1)).to(device)
            output = model(input_seq, mask)
            
            # Get probabilities for next character
            probs = torch.softmax(output[:, -1, :], dim=-1)
            next_char_idx = torch.multinomial(probs, 1).item()
            
            sequence.append(next_char_idx)
            
            # Stop if end token is generated
            if next_char_idx == char_to_idx['<EOS>']:
                break
    
    # Convert back to characters, excluding special tokens
    return ''.join(idx_to_char[idx] for idx in sequence[1:-1])

Generation Process

1. Start with <SOS> token
2. For each step, predict probability distribution over vocabulary
3. Sample next character from distribution
4. Continue until <EOS> token or max length
5. Return generated sequence

---

7. Complete Implementation

Here’s the complete working code:

import torch
import torch.nn as nn
import torch.optim as optim
import math
import random

# [All the classes and functions from above]

# Main execution
if __name__ == "__main__":
    # Prepare data
    data, char_to_idx, idx_to_char, vocab = prepare_khmer_data()
    print(f"Vocabulary size: {len(vocab)}")
    print(f"Sample sequence: {''.join(idx_to_char[idx] for idx in data[0])}")
    
    # Setup device and model
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model = KhmerTransformer(vocab_size=len(vocab)).to(device)
    
    # Train model
    train_model(model, data, char_to_idx, idx_to_char, device=device)
    
    # Generate text
    generated = generate_text(model, char_to_idx, idx_to_char, device=device)
    print(f"Generated text: {generated}")

---

8. Understanding the Output

What the Model Learns

The model learns to:

• Character Patterns: Common Khmer character combinations
• Syllable Structure: How consonants and vowels combine
• Sentence Patterns: Typical sentence endings and structures
• Contextual Relationships: Which characters follow others

Sample Output

A trained model might generate sequences like:

• ខ្ញុំចូលចិត្តញ៉ាំបាយ។ (I like to eat rice.)
• តើអ្នកសុខសប្បាយជាទេ? (How are you?)

Quality Assessment

The quality depends on:

• Training Data: More diverse sentences = better generalization
• Model Size: Larger models can capture more complex patterns
• Training Time: Longer training = better convergence
• Hyperparameters: Learning rate, batch size, etc.

---

9. Advanced Concepts

Temperature Sampling

For more controlled generation, we can add temperature to the sampling:

def generate_text_with_temperature(model, char_to_idx, idx_to_char, temperature=1.0, max_len=50, device='cpu'):
    model.eval()
    sequence = [char_to_idx['<SOS>']]
    
    with torch.no_grad():
        for _ in range(max_len):
            input_seq = torch.tensor([sequence], dtype=torch.long, device=device)
            mask = create_causal_mask(input_seq.size(1)).to(device)
            output = model(input_seq, mask)
            
            # Apply temperature
            logits = output[:, -1, :] / temperature
            probs = torch.softmax(logits, dim=-1)
            next_char_idx = torch.multinomial(probs, 1).item()
            
            sequence.append(next_char_idx)
            if next_char_idx == char_to_idx['<EOS>']:
                break
    
    return ''.join(idx_to_char[idx] for idx in sequence[1:-1])

• Temperature below 1.0: More focused, deterministic output
• Temperature above 1.0: More diverse, creative output

Perplexity

A common metric for language models is perplexity:

$\text{Perplexity} = 2^{\mathcal{L}/n}$

Where $\mathcal{L}$ is the cross-entropy loss and $n$ is the sequence length. Lower perplexity indicates better model performance.

---

10. Practical Considerations

Data Requirements

For production use, you’d need:

• Large Corpus: Millions of Khmer sentences
• Diverse Sources: News, literature, conversations
• Quality Control: Clean, well-formatted text

Model Scaling

Real applications require:

• More Layers: 12-24 transformer layers
• Larger Embeddings: 512-1024 dimensions
• More Heads: 8-16 attention heads
• Longer Sequences: 512-2048 tokens

Computational Resources

• Training: GPU/TPU clusters
• Inference: Optimized deployment
• Memory: Efficient attention implementations

---

11. Extensions and Improvements

1. Subword Tokenization

Instead of character-level, consider subword tokenization:

• Byte Pair Encoding (BPE)
• WordPiece
• SentencePiece

2. Pre-training and Fine-tuning

• Pre-train on large Khmer corpus
• Fine-tune on specific tasks (translation, summarization)

3. Multi-lingual Models

Train on multiple languages including Khmer for better performance.

4. Evaluation Metrics

• BLEU Score: For translation tasks
• ROUGE: For summarization
• Human Evaluation: For quality assessment

---

Conclusion

This tutorial has walked you through building a complete Transformer-based language model for Khmer text. We’ve covered:

1. Transformer Architecture: Self-attention, multi-head attention, positional encoding
2. Autoregressive Modeling: Causal masking, teacher forcing
3. Khmer Language Processing: Character-level modeling, real sentences
4. Practical Implementation: Complete PyTorch code
5. Training and Generation: Loss functions, sampling strategies

Key Takeaways

• Character-level modeling is ideal for Khmer due to its script complexity
• Causal masking ensures autoregressive properties
• Real data provides authentic language patterns
• Transformer architecture enables parallel processing and long-range dependencies

Next Steps

1. Expand the dataset with more Khmer text
2. Scale up the model with more layers and parameters
3. Experiment with different architectures (GPT-style, BERT-style)
4. Apply to downstream tasks (translation, summarization, question answering)

The foundation you’ve built here can be extended to create powerful Khmer language models for various applications. Happy coding!

---

This tutorial demonstrates the power of modern deep learning techniques applied to less-resourced languages like Khmer, showing how even small models can learn meaningful patterns from authentic data.