RareCode: An Unsupervised Deep Learning Framework for Anomaly Detection in Colorectal Cancer Histopathological Images

Summary

This study presents RareCode, an unsupervised deep learning framework that detects anomalies in colorectal cancer histopathological images by analyzing Codebook Activation Rarity (CAR), with the potential to assist pre-screening without labeled training data.

Abstract

Unsupervised anomaly detection in histopathological images has been widely explored using reconstruction-based approaches that measure reconstruction errors, yet these methods often fail to capture subtle, semantic-level pathological variations due to the inherent heterogeneity of tissue textures. To address this limitation, this study presents RareCode, a vector quantization-based framework that extends the detection paradigm beyond pixel-level reconstruction to incorporate semantic-level codebook activation analysis. The core innovation is Codebook Activation Rarity (CAR) scoring, which profiles the activation frequency of each codebook entry during training on exclusively normal samples and flags infrequent activations as anomaly indicators at inference time, thereby complementing reconstruction errors with semantic-level discrimination. Building upon this single-scale CAR mechanism, a Multi-scale Hierarchical Codebook (MHC) module is further introduced, employing codebooks of varying sizes with learnable fusion weights to capture pathological patterns spanning from coarse tissue-level structures to fine cellular-level details. Leveraging the multi-scale design, hierarchical anomaly heatmaps are generated at each codebook granularity, providing pathologists with interpretable, multi-dimensional visual localization cues indicating both where and at what structural level anomalies occur. Five-fold cross-validation on a clinically annotated colorectal cancer histopathological image dataset obtained from Shenzhen People's Hospital demonstrates that RareCode achieves an area under the curve (AUC) of 96.82%, outperforming baseline methods. Per-class analysis showed stronger performance for cancer detection (AUC = 99.44%, specificity = 94.21%) than for inflammation detection (AUC = 94.30%, specificity = 60.44%). These findings suggest that CAR analysis may offer a promising unsupervised approach for histopathological anomaly detection, potentially aiding clinical pre-screening in colorectal cancer diagnosis.

Introduction

Reliable histopathological screening for colorectal cancer remains constrained by expert annotation and manual review, particularly when inflammatory lesions and malignancies show overlapping morphology^1,2. These constraints motivate unsupervised computational approaches that learn from normal tissue without requiring exhaustive abnormal annotations^3,4.

The clinical urgency of automated histopathological screening is underscored by the growing imbalance between diagnostic demand and available pathology expertise, with the diagnostic workload per US pathologist rising by over 40% in a single decade as the pathologist workforce continued to shrink⁵. In colorectal cancer (CRC), organized screening programs have been associated with mortality reductions of 29-68%⁶, yet the capacity for manual pathological review remains limited by workforce availability. An automated pre-screening system that reliably identifies suspicious cases for priority review could help alleviate this burden and improve diagnostic turnaround times. However, deploying such systems requires both high sensitivity to avoid missing true positives and adequate specificity to prevent excessive false alarms⁷.

Unsupervised anomaly detection (UAD) has become increasingly relevant in medical image analysis⁸ because it can model normal tissue distributions without requiring abnormal training labels^3,9,10. Reconstruction-based models, including autoencoders, variational autoencoders, generative adversarial networks, and memory-augmented variants, identify anomalies through reconstruction errors, but high-capacity decoders may still reconstruct abnormal regions with high fidelity^{11,12,13,14,15,16,17}. Feature-based methods such as PatchCore and PaDiM use pretrained representations, although features learned from natural images may not fully capture microscopic atypia in histopathology^18,19,20,21. Pathology-specific foundation models and diffusion-based anomaly detectors provide powerful alternatives, but their data requirements or computational costs may limit their direct use in high-throughput screening^22,23. Recent automated and evolutionary methods, such as EvoAAE²⁴ and MoARNN-AM²⁵, further illustrate the value of adaptive model optimization for anomaly detection, although their application contexts differ from histopathological image analysis. In contrast, vector quantization (VQ)-based methods impose discrete codebook constraints that can reduce identity mapping; however, existing VQ-based approaches still rely primarily on spatial reconstruction errors and underuse the semantic information contained in codebook activation patterns^26,27.

Although the aforementioned methods demonstrate promising performance in general domains, their application to complex histopathological scenarios faces specific challenges. Histopathological images are characterized by complex tissue heterogeneity across multiple structural levels²⁸. In practice, image analysis typically operates on local image patches extracted from tissue sections, and pathological diagnosis is inherently cross-scale: pathologists assess glandular and tissue morphology at low magnification while examining nuclear pleomorphism and mitotic figures at high magnification²⁹. We identify three key limitations of current approaches: First, pathological anomalies and normal tissues exhibit high similarity in low-level visual features such as staining styles and local textures, causing reconstruction-based methods to fail in detecting subtle anomalies, as both normal and anomalous samples may produce similar reconstruction quality at the semantic level. Second, most existing methods employ single-scale feature extraction, making it challenging to simultaneously capture anomalous features at both tissue and cellular levels^30,31. Third, although mainstream methods can generate pixel-level heatmaps, they often fail to intuitively capture the hierarchical attributes of anomalies, limiting the usefulness of such models as clinical diagnostic aids.

To address the above challenges, we propose RareCode, a UAD framework that explores codebook activation patterns and hierarchical feature representation at multiple levels of abstraction. The framework introduces three main components: (1) Codebook Activation Rarity (CAR) Scoring Mechanism, which computes rarity scores based on entry activation frequencies trained exclusively on normal samples, distinguishing normal from anomalous samples at the semantic level and providing complementary discriminative signals to traditional spatial reconstruction errors. (2) Multi-scale Hierarchical Codebook (MHC) Fusion Architecture, which captures pathological features at multiple levels of granularity—from coarse structural patterns to fine-grained cellular details—by employing codebooks with varying capacities, with adaptive fusion achieved through learnable weights. (3) Hierarchical Interpretable Localization Module, which leverages the multi-scale architecture to generate anomaly heatmaps at different granularities, providing pathologists with multi-scale semantically interpretable diagnostic references.

The primary hypothesis of this study was that codebook activation-frequency profiles derived from codebooks trained exclusively on normal colorectal histopathological images would encode anomaly-relevant semantic information beyond pixel-level reconstruction error, and that multi-scale integration of these complementary signals would improve the discrimination of images containing cancerous or inflammatory tissue from normal images relative to representative UAD methods, as assessed primarily by AUC. To test this hypothesis, we evaluated RareCode using 5-fold cross-validation on a clinically annotated CRC dataset and performed ablation and localization analyses to examine the contributions and interpretability of its core components.

Protocol

This study was approved by the Clinical Research Ethics Committee of Shenzhen People's Hospital (Approval No. LL-KY-2025300-01). The requirement for informed consent was waived by the ethics committee because this retrospective study used existing histopathological images and clinical materials without direct patient contact or intervention. All clinical data and histopathological images were de-identified before analysis, and no personally identifiable information was used in this study. All data used in this research were handled in accordance with the ethical standards of the institutional and national research committees.

Framework overview
The overall architecture of the proposed UAD framework, RareCode, is illustrated in Figure 1. The framework employs a parallel dual-branch architecture. For each 512 x 512 input image, non-overlapping 32 x 32 and 64 x 64 patches are extracted as fine- and coarse-scale inputs, respectively. The 64 x 64 patches are resized to 32 x 32 before being passed to the network so that both branches share the same encoder-decoder input size while preserving different receptive fields. Encoders in each branch map the extracted features into a latent space, where the MHC module imposes discretization constraints. Subsequently, decoders reconstruct the image from quantized features to compute spatial-domain reconstruction errors. The CAR mechanism then measures the degree of semantic-level anomaly by profiling codebook activation frequencies. Finally, the model fuses reconstruction scores and CAR scores through a weighted combination to obtain image-level anomaly scores. The RareCode framework is trained end-to-end using only normal samples, requiring no anomaly annotations. The dual-branch design was used to capture both local cellular features and broader glandular structures. The MHC module was used to model these features with codebooks of different capacities.

Encoder-decoder architecture
RareCode constructs structurally independent encoders and decoders for the fine- and coarse-scale branches. Each branch uses the same encoder-decoder design but does not share parameters. The encoder consists of four convolutional blocks, each including a 3 x 3 convolution, batch normalization, ReLU activation, and dropout. The channel width increases from 64 to 128, then to 256, and the final feature map is flattened and projected into a 64-dimensional latent embedding. The decoder mirrors the encoder with a fully connected projection layer followed by four transposed convolution layers, reconstructing each patch to the original input size of the network. Reconstruction error is calculated as the mean squared error between the input and reconstructed patches. By combining this encoder-decoder structure with discrete codebook constraints, RareCode learns compact representations of normal tissue and measures deviations during inference.

Multi-scale hierarchical codebook
To capture pathological features at multiple levels of abstraction—from broad tissue patterns to localized cellular variations—the MHC module employs K discrete codebooks with varying capacities n₁, n₂, ..., n_K. In the final FourScales configuration, each branch contains four codebooks with 64, 128, 256, and 512 entries, respectively, and each codebook entry has a 64-dimensional embedding. For each encoded patch feature, the nearest codebook entry is selected according to Euclidean distance. The quantized outputs from different codebooks are then fused using learnable weights normalized across codebooks. Smaller codebooks, due to stronger compression constraints, are expected to encode coarse-grained prototypical patterns that abstract away local variations, whereas larger codebooks preserve finer-grained features that capture more specific structural characteristics. For each codebook , where e_k⁽ⁱ⁾  R^d denotes the i-th embedding vector in the k-th codebook, continuous features are mapped to the discrete codebook space via nearest neighbor lookup (Eq. 1 and 2):

To achieve adaptive fusion of multi-scale features, we introduce learnable weights . After softmax normalization, weighted summation is performed over the quantized outputs from each codebook (Eq. 3):

where σ(·) denotes the softmax function, ensuring that weights satisfy Σ_k σ(w_k) = 1. This design enables the model to automatically adjust the contribution ratios of codebooks at different granularities based on the semantic content of the input features.

To optimize codebook learning, we adopt the standard VQ loss function (Eq. 4):

where sg[·] denotes the stop-gradient operation, and β = 0.25 is the commitment loss weight coefficient. The first term encourages codebook vectors to move toward encoder outputs, while the second term encourages encoder outputs to remain consistent with their corresponding codebook vectors. The complete mechanism of multi-scale VQ is illustrated in Figure 2.

Codebook activation rarity scoring
CAR measures semantic-level anomalies based on codebook activation statistics estimated from normal training samples. After model training, all normal training images were passed through the encoder and MHC module without data augmentation. For each branch and each codebook, the assignment index of every patch feature was recorded, and the number of assignments to each codebook entry was counted. The counts were then normalized to obtain the activation-frequency distribution for that codebook (Eq. 5):

During inference, each test image was processed through the same patch extraction, encoding, and nearest-neighbor codebook assignment procedure. For each assigned codebook entry, the rarity score is calculated as the negative logarithm of its activation frequency in the normal training set (Eq. 6):

where ε is a small constant used for numerical stability. A low-frequency codebook entry, therefore, receives a higher rarity score, suggesting that the corresponding patch feature is less consistent with the learned normal-tissue distribution. Patch-level rarity scores were averaged within each image and across the fine- and coarse-scale branches to obtain an image-level rarity response.

To complement activation-frequency rarity, we also compute a percentile-based quantization distance score (Eq. 7):

This score is derived from the Euclidean distance between each encoded feature vector and its nearest codebook entry. The distance distribution is estimated from normal training samples, and test-sample distances are converted into percentile scores. Larger percentile scores indicate that the encoded feature is more difficult to represent using the learned normal codebook prototypes.

The final CAR score combines activation rarity and quantization distance across all codebooks using the learnable codebook weights (Eq. 8):

The same normalized codebook weights used for multi-scale feature fusion are applied to score-level fusion, maintaining consistency between representation learning and anomaly scoring. The resulting CAR score is calculated at the image level and is then combined with the reconstruction score to obtain the final anomaly score. The overall CAR scoring mechanism is illustrated in Figure 3.

Final anomaly score computation
We fuse spatial-domain reconstruction errors with semantic-level CAR scores. Before fusion, both scores are separately normalized using percentile-based min-max normalization to reduce the influence of extreme outliers. The final image-level anomaly score is then defined as (Eq. 9):

where S_recon denotes the normalized mean squared reconstruction error, reflecting sample reconstruction quality in pixel space; S_CAR is the CAR score described above, capturing degrees of semantic-level anomaly. The hyperparameter α [0,1] controls the relative weighting between the two components, with optimal values determined on validation sets.

Training objective
The RareCode framework is trained end-to-end using only normal samples. The total training loss comprises the following components (Eq. 10):

Each term is defined as follows: Reconstruction Loss: , measuring the mean squared error between input patches p and reconstructed patches D(z_q) after quantization. This loss is applied separately to the 32 × 32 patch branch () and 64 × 64 patch branch () to ensure effective learning of features at both spatial scales. 

Dataset description
To evaluate RareCode, experiments were conducted on a clinically annotated CRC histopathological image dataset comprising images of colorectal tissue sections stained with hematoxylin and eosin (H&E), obtained from Shenzhen People's Hospital. All images in this dataset originate from real clinical cases. Images were digitized at 40x magnification with an original resolution of 1024 × 1024 pixels. To ensure annotation reliability and quality, all sample category labels were jointly reviewed and confirmed by two or more senior professional pathologists following a double-blind protocol.

Based on histopathological characteristics, the dataset was categorized into three classes: normal tissue (1,226 images), cancer (1,215 images), and inflammation (1,214 images). In the binary UAD setting, cancer and inflammation samples were treated as anomalous, while normal samples served as the reference distribution. This formulation reflects the intended triage task of separating cases requiring further pathological review from those that are morphologically normal.

All H&E-stained images were resized to 512 × 512 pixels before being used as network input. Stratified 5-fold cross-validation was used for evaluation. In each fold, the normal samples were first divided into training, validation, and held-out test subsets. RareCode was trained only on the training normal samples. The validation subset, which also contained only normal samples, was used for model selection, hyperparameter tuning, and selection of the fusion weight α. The held-out test set, consisting of unseen normal samples and anomalous samples including cancer and inflammation, was used only for final performance evaluation and was not used for model training, hyperparameter selection, or α selection. The representative categories, dataset structure, and validation protocol are shown in Figure 4.

Implementation details
The proposed RareCode framework and comparison baselines were implemented using PyTorch. All experiments were conducted on a workstation equipped with a single NVIDIA GeForce RTX 4060 Ti GPU (16 GB). For network architecture configuration, to capture features at different spatial extents, the dual-branch architecture processes non-overlapping patches at two scales: 32 × 32 pixels (smaller receptive field, capturing local texture patterns) and 64 × 64 pixels (larger receptive field, capturing broader spatial context), respectively. The continuous feature embedding dimension produced by both branch encoders is uniformly set to 64. In the MHC module, we configure four codebooks with different capacities for each branch, with discrete prototype quantities of 64, 128, 256, and 512, respectively.

During optimization, the network was trained end-to-end for 50 epochs with a batch size of 8. We employ the AdamW optimizer for weight updates, with an initial learning rate of 5 × and weight decay of to prevent overfitting. Additionally, to ensure smooth training convergence, a cosine annealing learning rate schedule was used, enabling finer optimization in later training stages. Patch sizes and codebook sizes were selected to balance multi-scale representation and computational cost. The final FourScales configuration was supported by ablation analysis, and α was selected on the validation set before test-set evaluation.

For the decoder-complexity ablation, a complex-decoder variant was implemented based on the FourScales configuration. This variant used the same data split, codebook sizes, training epochs, optimizer, learning rate, validation-based α selection, and evaluation metrics as the FourScales model. The basic decoder was replaced with an EMCAD-style decoder containing multi-scale depthwise convolution blocks and Convolutional Block Attention Modules (CBAM). After each of the first three transposed-convolution upsampling stages, parallel 3 × 3, 5 × 5, and 7 × 7 depthwise convolutions were applied, followed by 1 × 1 pointwise fusion, batch normalization, ReLU activation, a residual connection, and CBAM attention. CBAM included channel attention based on average- and max-pooling descriptors and spatial attention using a 7 × 7 convolution. The final stage used transposed convolution and sigmoid activation to reconstruct 32 × 32 patches.

Results

Comparison with baseline methods
To evaluate the RareCode framework, multiple representative UAD methods were selected as baselines, spanning reconstruction-based methods (CAE³², VAE¹², SAE³³, MAE³⁴, PatchSAE³⁵), memory-augmented reconstruction (MemAE³⁶), knowledge distillation (STFPM³⁷), synthetic anomaly generation (DRAEM

Discussion

The results suggest that codebook activation patterns may provide complementary information to spatial reconstruction errors. The improvement from NoCAR to the full FourScales model supports the potential contribution of CAR scoring, while the selected fusion weights indicate that semantic rarity may play an important role in the final anomaly score. One possible explanation is that CRC-related abnormalities can span different morphological scales, from nuclear atypia to glandular architectural disruption, and therefore ...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
GeForce RTX 4060 Ti GPU	NVIDIA	N/A	16 GB VRAM; single GPU used for all experiments (RRID:SCR_022858)
Matplotlib	Matplotlib Development Team	Version 3.8.4	Data visualization and figure generation (RRID:SCR_008624)
NumPy	NumPy Developers	Version 1.26.4	Numerical computing and array operations (RRID:SCR_008633)
pandas	pandas Development Team	Version 2.2.3	Data organization and tabular result processing (RRID:SCR_018214)
Pillow	Python Imaging Library / Pillow Contributors	Version 10.3.0	Image loading and preprocessing
Python	Python Software Foundation	Version 3.12.3	Programming language used for model development and data analysis (RRID:SCR_008394)
PyTorch	Meta Platforms, Inc.	Version 2.7.0+cu118	Deep learning framework used for model implementation and training (RRID:SCR_018536)
scikit-learn	scikit-learn Developers	Version 1.4.2	Cross-validation, train-test splitting, and evaluation metrics (RRID:SCR_002577)
SciPy	SciPy Developers	Version 1.13.1	Statistical analysis (RRID:SCR_008058)
torchvision	PyTorch Project	Version 0.22.0+cu118	Image preprocessing and data transformation
tqdm	tqdm Developers	Version 4.66.4	Training and evaluation progress monitoring