PillCam SB3 · Deep Learning · AP-HP Saint-Antoine

AI Anomaly Detection in Capsule Endoscopy.

A patient swallows a capsule. It travels the entire GI tract capturing up to 50,000 frames. This pipeline extracts every frame, detects anomalies, draws contours, classifies pathologies across 17 categories, and auto-sorts results into labelled folders.

Frame Extraction Anomaly Detection Contour Segmentation 17 Pathology Classes AP-HP Validated
50k
Max frames per session
17
Pathology classes
4
Colab notebooks
AP-HP
Clinically validated
The Capsule's Journey

From mouth
to rectum.

The PillCam capsule travels naturally through the digestive system over 8–12 hours, capturing two frames per second. Every centimetre of the GI tract is recorded.

Oesophagus
~5 min
Stomach
~1–2 h
Small Intestine
~5–7 h
Colon
~1–2 h
Total frames captured
~50,000
End-to-End Workflow

Five steps from
capsule to diagnosis.

01
💊
Capsule Input
Patient swallows PillCam SB3. Capsule records the full GI tract continuously at 2 fps over 8–12 hours.
02
🎞️
Frame Extractor
OpenCV pipeline extracts every frame from the MPG recording. Checkpoint/resume for 8–12h sessions.
03
🔍
AI Detection
Deep learning scans each frame, detects anomalies, draws precise contour overlays matching clinical annotation protocol.
04
📊
Classification
Ensemble of EfficientNet-B3, ConvNeXt, SwinV2 classifies across 17 pathology categories with TTA.
05
📁
Auto-Sorted Output
Annotated frames sorted into pathology-named folders. Clinician receives a structured, review-ready report.
Real Detection Results

Drag to reveal AI annotation.

Each slider shows the exact transformation produced by the pipeline — raw PillCam frame on the left, AI-annotated output on the right.

Case 01 · Chylous Cyst

Chylous Cyst — Overview

A chylous cyst (lymphangiectasia) of the small bowel mucosa — a pale, well-defined ovoid lesion visible in the lower field. Multiple scattered angioectasias also visible in the background. The AI isolates the primary cyst with a yellow boundary, critical for differential diagnosis with lipoma or polyp.

Kyste ChyleuxYellow ContourMulticlass SegmentationEfficientNet-B3
Timestamp 00:35:16 — Primary detection: chylous cyst (lower field). Background: scattered angioectasias. Overlay generated automatically by the segmentation pipeline.
AI annotated
Raw
Before — Raw
After — AI
00:35:16 · PillCam SB3
Drag to compare
Case 02 · Chylous Cyst

Chylous Cyst — Close View

A larger chylous cyst at close proximity. The AI detects and contours the lesion robustly despite partial occlusion by an intestinal fold and challenging proximity to the mucosa. Single-class yellow boundary output with high-confidence segmentation.

Close-view DetectionOcclusion RobustnessMedSAM AnnotationTTA Ensemble
Timestamp 05:02:13 — Single dominant lesion. High-confidence detection despite partial mucosal fold occlusion. Boundary precisely fits lesion contour.
AI annotated
Raw
Before — Raw
After — AI
05:02:13 · PillCam SB3
Drag to compare
Case 03 · Aphthoid Erosions

Aphthoid Erosions x2

Two aphthoid erosions detected simultaneously. The AI applies a dual-class contour system: an orange outer boundary delineates the lesion zone, a blue inner mask segments the erosion core — replicating the annotation protocol used by the clinical team at Saint-Antoine.

Dual-Class ContourOrange + Blue MaskUNet + EfficientNet-B3Focal + Dice Loss
Timestamp 04:29:13 — Two erosions detected. Orange outer boundary (lesion zone) + blue inner mask (erosion core). Post-TTA confidence applied.
AI annotated — Erosions x2
AI Active
04:29:13 · PillCam SB3
Case 04 · Angioectasia

Angioectasia Lesion

A vascular malformation of the small intestinal mucosa. The AI classifies the frame then applies binary segmentation with a blue contour overlay, isolating the lesion from surrounding healthy mucosa with sub-pixel precision.

Binary SegmentationBlue ContourConvNeXt-SmallSwinV2-Small
Timestamp 01:24:22 — Before (left): raw endoscopic frame. After (right): AI contour overlay precisely delineating the angioectasia.
AI annotated
Raw
Before — Raw
After — AI
01:24:22 · PillCam SB3
Drag to compare
Technical Architecture

How the pipeline works.

🎞️
Frame Extraction
OpenCV-based script extracts frames from MPG PillCam recordings with checkpoint/resume. Handles 8–12h sessions without memory issues.
OpenCVPythonMPG
🏷️
MedSAM Annotation
Custom HTML canvas widget in Colab powered by MedSAM for fast interactive tricolor mask creation — replacing slow manual ImageJ workflows.
MedSAMHTML CanvasColab
🔍
Classification
Ensemble of EfficientNet-B3, ConvNeXt-Small, SwinV2-Small across 17 pathology classes. MixUp augmentation + TTA + stratified splits.
EfficientNet-B3SwinV2MixUp
✂️
Segmentation
UNet + EfficientNet-B3 encoder for multiclass semantic segmentation. Combined Focal + Dice loss. Tricolor and binary mask encoding with TTA at inference.
UNetFocal+DiceTTA
🎨
Overlay Generation
Post-processing converts raw label masks to colorized RGB overlays. Dual-class contour rendering (outer boundary + inner mask) on original frames.
RGB MasksContoursOpenCV
Clinical Validation
Outputs reviewed and validated by Professors of the Endoscopy Service at AP-HP Saint-Antoine. Integrated into real PillCam SB3 reading sessions.
AP-HPPr DraySaint-Antoine
Detection Scope

17 pathology classes detected.

Category
Category
Category
Category
Erosion aphtoïde
Angioectasie
Kyste chyleux
Ulcération
Polype
Saignement actif
Lymphangiectasie
Sténose
Diverticule
Nodule
Xanthome
Muqueuse normale
Résidu alimentaire
Bulle d'air
Papille de Vater
Lésion vasculaire

Interested in collaboration?

This tool is part of ongoing research at AP-HP Saint-Antoine. Open to academic and industrial partnerships.

Get in Touch Back to Portfolio