{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Overview\n", "\n", "Each task involves predicting the expression levels of the 50 most variable genes from 112×112 μm H&E-stained image patches centered on each spatial transcriptomics spot. The tasks are formulated as multivariate regression problems.\n", "\n", "| **Task ID** | **Oncotree** | **Number of Samples** | **Technology** | **Sample ID** |\n", "|-------------|--------------|-----------------------|----------------|---------------|\n", "| Task 1 | IDC | 4 | Xenium | TENX95, TENX99, NCBI783, NCBI785 |\n", "| Task 2 | PRAD | 23 | Visium | MEND139~MEND162 |\n", "| Task 3 | PAAD | 3 | Xenium | TENX116, TENX126, TENX140 |\n", "| Task 4 | SKCM | 2 | Xenium | TENX115, TENX117 |\n", "| Task 5 | COAD | 4 | Xenium | TENX111, TENX147, TENX148, TENX149 |\n", "| Task 6 | READ | 4 | Visium | ZEN36, ZEN40, ZEN48, ZEN49 |\n", "| Task 7 | ccRCC | 24 | Visium | INT1~INT24 |\n", "| Task 8 | LUAD | 2 | Xenium | TENX118, TENX141 |\n", "| Task 9 | IDC-LymphNode | 4 | Visium | NCBI681, NCBI682, NCBI683, NCBI684 |\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Reproducing HEST-Benchmark results \n", "\n", "- Ensure that HEST has been properly installed (see README, Installation)\n", "- Install benchmark dependencies via `pip install -e \".[benchmark]\"`\n", "- Benchmark data (patches, h5ad, splits, genes) are downloaded automatically\n", "- Patch encoders are loaded through TRIDENT\n", "\n", "**Important (model access):**\n", "Many foundation models used by TRIDENT are hosted on Hugging Face and may be gated.\n", "You may need to request access for each model you want to benchmark and authenticate locally.\n", "\n", "Typical setup:\n", "1. Request access on each model page (for example: UNI, CONCH/CONCHv1.5, GigaPath, Virchow, H-Optimus, etc.).\n", "2. Login with your Hugging Face token in the benchmark environment:\n", "\n", "```\n", "huggingface-cli login\n", "```\n", "\n", "Once authenticated, TRIDENT handles model loading directly from the encoder name in the benchmark config.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Launching HEST-bench via CLI" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%bash\n", "export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libffi.so.7\n", "python ../src/hest/bench/benchmark.py --config ../bench_config/bench_config.yaml" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Benchmarking your own model with HEST-Benchmark\n", "\n", "Below is a complete, minimal example using a Hugging Face Vision Transformer as a custom encoder.\n", "\n", "We wrap the model with:\n", "- a `forward()` method returning a patch embedding (CLS token),\n", "- an `eval_transforms` callable,\n", "- a `precision` attribute.\n", "\n", "This aligns with TRIDENT-style encoder behavior and can be passed directly to `benchmark(...)`.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from hest.bench import benchmark\n", "import torch\n", "from torchvision import transforms\n", "from transformers import AutoImageProcessor, AutoModel\n", "\n", "\n", "class HFViTCustomEncoder(torch.nn.Module):\n", " \"\"\"Minimal TRIDENT-style wrapper around a Hugging Face vision model.\"\"\"\n", "\n", " def __init__(self, hf_id=\"google/vit-base-patch16-224-in21k\", precision=torch.float32):\n", " super().__init__()\n", " self.hf_id = hf_id\n", " self.processor = AutoImageProcessor.from_pretrained(hf_id)\n", " self.model = AutoModel.from_pretrained(hf_id)\n", " self.precision = precision\n", "\n", " # Reuse model-recommended preprocessing parameters when available.\n", " mean = self.processor.image_mean if self.processor.image_mean is not None else (0.5, 0.5, 0.5)\n", " std = self.processor.image_std if self.processor.image_std is not None else (0.5, 0.5, 0.5)\n", " size = self.processor.size\n", " target = size[\"shortest_edge\"] if isinstance(size, dict) and \"shortest_edge\" in size else 224\n", "\n", " self.eval_transforms = transforms.Compose([\n", " transforms.Resize(target),\n", " transforms.CenterCrop(target),\n", " transforms.ToTensor(),\n", " transforms.Normalize(mean=mean, std=std),\n", " ])\n", "\n", " def forward(self, x):\n", " outputs = self.model(pixel_values=x)\n", " # Return CLS token as patch embedding.\n", " return outputs.last_hidden_state[:, 0, :]\n", "\n", "\n", "PATH_TO_CONFIG = \"../bench_config/bench_config.yaml\"\n", "custom_encoder = HFViTCustomEncoder(hf_id=\"google/vit-base-patch16-224-in21k\", precision=torch.float32)\n", "\n", "# This wrapper exposes `eval_transforms` and `precision`, so it can be passed directly.\n", "benchmark(\n", " custom_encoder,\n", " None,\n", " None,\n", " config=PATH_TO_CONFIG,\n", " exp_code=\"custom_hf_vit\",\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Reproducing: finding genes of interest for HEST benchmark" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import scanpy as sc\n", "from hest import get_k_genes\n", "\n", "# TODO add full paths to samples of interest here\n", "sample_paths = ['TENX118.h5ad', 'TENX141.h5ad']\n", "\n", "ad_list = [ sc.read_h5ad(sample_path) for sample_path in sample_paths]\n", "genes = get_k_genes(ad_list, k=50, criteria=\"var\", min_cells_pct=0.1)\n", "print(len(genes))" ] } ], "metadata": { "kernelspec": { "display_name": "hest", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.14" } }, "nbformat": 4, "nbformat_minor": 4 }