Released April 2, 2026, Gemma 4 is Google DeepMind’s most capable open model family, built on the same research as Gemini 3 and released under a fully permissive Apache 2.0 license.

It comes in four sizes:

• E2B and E4B are edge-optimized models that activate an effective 2B and 4B parameters, designed to run offline on phones, Raspberry Pi, and Jetson Nano.

• 26B-A4B (MoE) is a mixture-of-experts model with 26B total parameters but only 3.8B active per token, so you get 26B quality at 4B speed.

• 31B Dense is the largest, currently ranked #3 globally among open models on Arena AI.

Key capabilities across the family:

• native vision and audio processing, function calling,

• structured JSON output, system instructions,

• a thinking/reasoning mode,

• 256K context windows for the larger models (128K for edge),

• training on 140+ languages.

The benchmark leap is dramatic. Compared to Gemma 3, the 31B Dense model jumped from 20.8% to 89.2% on AIME math, nearly tripled LiveCodeBench scores, and went from 19% to 74% on BigBench Extra Hard. This makes it far more competitive with Qwen 3.5, which previously dominated math and coding among open models at this size class.

Architectural innovations include a second embedding (Per-Layer Embeddings) table that feeds residual signals into every decoder layer, and shared key-value tensors in the final layers to reduce memory during long-context inference.

Experiments

To test the capabilities of the Gemma-4 model, I did some quick experiments. I am primarily interested in multi-modal and visual-language capabilities of the models. Hugging Face team did various tests on Gemma-4 model and wrote in a blog post. I did some further exploration of this series of models which are discussed below.

Image Reasoning

The image reasoning of Gemma-4 is tested across three experiments testing capabilities the HuggingFace blog never benchmarked. These are multi-image reasoning, adversarial robustness, and structured extraction.

Multi-Image Reasoning

Three prompts, each with two images. The question: does the model actually attend to both images, or does it default to one and confabulate the other?

• Chart Comparison (H1 vs H2 2025)

  

  Result: Correctly identified May ($61M) as the H1 peak and December ($102M) as the H2 peak. Correctly judged H2 as the better-performing half and noted that “every single month in H2 (except August) outperformed the best month of H1.” Accurately characterized H1 as volatile and H2 as trending upward.

  Verdict: ✅ Both images attended to. Numbers extracted correctly from both. Cross-chart comparison was substantive, not generic.

• Product Spec Comparison

  

  Result: Produced a structured markdown comparison table covering price, price-per-GB-RAM, storage, display, battery, and weight — declaring a winner per row. Notably, it computed $54.12/GB (UltraBook) vs $45.28/GB (ThinkPad) — actual division, not vibes.

  Verdict: ✅ Strong. It performed real arithmetic across two images and structured the answer in a table without being asked. The price-per-GB calculation is the kind of derived metric that proves the model isn’t just OCR’ing — it’s reasoning across the inputs.

• Scene Comparison (Venice + Bangkok)

  

  

  Result: Identified Venice from the seagull-on-mooring-pole and the “Ferrovia” sign (Santa Lucia train station). Identified Bangkok from the temple architecture (Grand Palace / Wat Phra Kaew). Both identifications used specific visual cues, not generic guesses.

  Verdict: ✅ The “Ferrovia” callout is impressive, it noticed text in the background and used it as evidence. This is grounded reasoning, not pattern-matching.

Summary: The model doesn’t ignore secondary images, doesn’t confuse them, and can perform calculations and comparisons that span both. This is a meaningful capability the HF blog never demonstrated.

Adversarial Vision

Five prompts designed to fool the model. The question: does it confidently get things wrong, or does it flag the problem?

• Rotated Text (180°)

  Setup: A receipt image rotated upside-down.

  

  Result: Returned valid JSON with both items, both prices, and the correct total $145.26. Did not mention or notice the rotation.

  Verdict: ✅ Read the rotated text perfectly. Either the vision encoder is rotation-invariant or it normalized the image internally. Either way, the user gets the correct answer with zero friction.

• Misleading Chart (Truncated Y-axis)

  Setup: Bar chart with values 97, 98, 99, 100 but Y-axis starting at 95, visually exaggerating tiny differences.

  

  Result: Read all four values correctly. Computed the actual difference as 3 points. And then unprompted, called out: “Yes. The chart uses a truncated y-axis (it starts at 95 instead of 0)...”

  Verdict: ✅✅ This is the standout result. The model didn’t just answer the literal question — it flagged the visual deception, which is exactly what a human analyst would do. This is data literacy, not just OCR.

• Low-Resolution Degraded Image

  Setup: The bird photo downsampled to 48×48 then upscaled — pixelated and barely recognizable.

  

  Result: Opened with: “The image provided is extremely low-resolution and heavily pixelated, making it very difficult to identify specific details with certainty.” Then made a hedged guess: “likely a bird, perched on top of a grey post or pillar.”

  Verdict: ✅ Calibrated uncertainty. Many vision models confidently hallucinate detail on degraded inputs. Gemma 4 explicitly acknowledged the limitation before guessing and the guess was correct.

• Stroop Test (Color vs Text)

  Setup: Red square with the word “BLUE” written on it.

  

  Result: “The background of the image is red. The text written on it is ‘BLUE’. No, the text and the color are not consistent.”

  Verdict: ✅ Three-for-three: identified the actual pixel color, read the contradicting text, and noted the inconsistency. No confusion between symbol and substance.

• on-Latin OCR (Japanese)

  Setup: An image with two lines of Japanese text:

  • 東京タワーの高さは333mです (Tokyo Tower’s height is 333m)

  • 建設年: 1958年12月23日 (Construction year: December 23, 1958)

  

  Result:

  Transcription: 333m 1958 12 23

  Translation: 333m December 23, 1958

  Verdict: ❌ The only clear failure. The model extracted only the numbers (333m, 1958/12/23) and dropped all the Japanese characters. It got the dates and measurements right, but lost the semantic context — it never tells you this is about Tokyo Tower or that 1958/12/23 is a “construction year.” The Japanese script was effectively ignored.

Head-to-Head Capability Tests

Six prompts covering captioning, object detection, OCR, spatial reasoning, and chart extraction.

• Detailed Captioning (Bird Photo)

  

  Result Produced a single rich sentence: “A single seagull with grey and white plumage stands perched atop a weathered grey wooden mooring pole in the foreground of a bright, daylight scene...” — included subject, environment, lighting, and architectural background details.

  Verdict: ✅ Reads like professional photo caption copy. No hallucinated objects.

• Object Detection Bounding Box (Bike)

  

  Result: [{"box_2d": [542, 0, 954, 893], "label": "the bike"}]

  Verdict: ✅ Valid JSON, single object, coordinates in the documented 1000×1000 normalized space. Coordinates need visual verification but format is correct.

• GUI Element Detection (View Recipe Button)

  

  Result: [{"box_2d": [171, 104, 246, 308], "label": "view recipe"}]

  Verdict: ✅ Compare to the HF blog’s reference for the same image: [171, 75, 245, 308]. Within ~30 pixels, which basically matches. This demonstrates reproducibility against Google’s reference outputs.

• OCR + Structured Extraction (Receipt)

  

  Result:

  {
    "items": [
      {"name": "Widget Pro", "price": 45.00},
      {"name": "Gadget XL", "price": 89.50}
    ],
    "tax": 10.76,
    "total": 145.26
  }

  Verdict: ✅✅ Perfect extraction. Numeric values are typed as numbers (not strings), spacing in product names was correctly inferred (Widget Pro not WidgetPro — even though the rotated-text version in Experiment 2 had WidgetPro without the space; the upright version got it right). All four numeric fields correct.

• Spatial Reasoning

  

  Prompt: What is to the left of the bird? Behind it? Below it?

  Result: Correctly placed “white building with windows and trees” to the left, “clear, light-colored sky” behind, and “weathered grey wooden piling” below.

  Verdict: ✅ Concise, anatomically structured answer. Real spatial grounding rather than generic scene description.

• Chart Data Extraction

  

  Result:

  {"Jul": 52, "Aug": 47, "Sep": 68, "Oct": 75, "Nov": 89, "Dec": 102}

  Verdict: ✅ Six values, all correct, valid JSON, ready to use as a data source.

Video Analysis

To explore the video understanding capabilities, I used a personal video which is posted as a reel on my Instagram. This is a picnic-and-reading-in-a-park video, trimmed to 60s.

• Basic Description

  The 31B gave a remarkably accurate scene-by-scene description:

  “A man and a woman are enjoying a sunny day outdoors in a grassy park area with trees and a distant body of water... The video begins with a stylized sequence of the two lying on their backs in the grass, holding open books above their faces... The man is in a grey t-shirt, and the woman is in a blue patterned shirt... The camera cuts to a close-up of a hand holding a black pour-over coffee dripper over a glass, filtering coffee.”

  Verdict: ✅ Highly accurate. It picked up the opening symmetry shot, the wardrobe colors, the pour-over coffee detail, and the narrative arc (reading → conversation). No hallucinations.

• Structured Categorization

  Returned a clean, parseable JSON object:

  {
    "content_category": "lifestyle",
    "hook_description": "A symmetrical, aesthetic shot of two people lying on grass holding books up to the sky.",
    "hook_rating": 4,
    "text_overlays": [],
    "has_face": true,
    "has_call_to_action": false,
    "estimated_duration_seconds": 60,
    "visual_quality": 4,
    "pacing": "medium",
    "target_audience": "Book lovers, people interested in 'slow living' or aesthetic outdoor dates"
  }

  Verdict: ✅✅ Production-ready. Every field is filled correctly, the audience targeting is specific (not "everyone"), and the JSON parsed on the first try.

• Temporal Understanding

  The model produced a precise 5-segment timeline. It also correctly identified the editing technique (hard cuts, jump cuts between 00:03-00:15, classic B-roll cutaway to coffee at 00:35) and called the pacing “slow and rhythmic, matching the relaxed vibe of a picnic.”

  Verdict: ✅✅ This is the most impressive output of the whole experiment. The model didn’t just describe frames — it understood video grammar. Cutaways, jump cuts, narrative arc, pacing changes. For an Instagram pipeline, this temporal segmentation is exactly what you need to detect things like “the hook is too long” or “the CTA is missing.”

• Video + Audio Description

  The E4B model (8B params with embeddings) is the only Gemma 4 variant that processes audio from video. We run the same videos and compare what the audio track adds.

  The good: It correctly identified that there’s spoken dialogue (no music), correctly inferred the speakers were “talking about their decision to dedicate a time to reading after falling behind,” and noted “a beautiful spring, sunny day.”

  The bad: It transcribed the location as “Hamsterhiif” which is a hallucination of a real place name Hampstead Heath. (Across the three experiments it variously called it “Hamsterhiif”, “Hamsteriheb”, and “Hamsteriheb” — the model couldn’t even agree with itself across runs.)

  The very bad: It hallucinated visual content that isn’t in the video. From its description:

  “They are observed helping to set up or work with a large, tent-like structure, possibly related to a field activity or community project. They appear to be engaged in some sort of manual labor.”

  This appears to be a misinterpretation of the blanket-spreading sequence the 31B correctly identified in segment 00:16–00:23. The E4B turned “spreading a picnic blanket” into “manual labor on a tent structure.” The 2b ablation reinforces this: the audio-OFF run later described “covering a large, canvas-like structure with a printed design of stylized trees” — pure confabulation.

  Verdict: ⚠️ Audio detection works (speech vs music classification is correct). But specific transcription is unreliable, and the visual descriptions are noticeably worse than the 31B’s. The location name fabrication is the kind of error that would silently corrupt downstream analytics.

Parting Words

This look quite promising. I have a whole range of experiments that I am running on image and video understanding capabilities of these models. And I’ll share more details on a project that I am working on related to video reasoning of large (visual) language models.

Codes used for this experiment can be found at GitHub: januverma/gemma4-experiments

Large language models rely heavily on a key-value (KV) cache which is a memory store that keeps previously computed attention keys and values so the model doesn’t recompute them at every step. As context windows grow longer, this cache becomes a major memory bottleneck. In vector search, you may also need to store billions of embedding vectors. In both cases, memory and memory bandwidth become bottlenecks. Google frames TurboQuant as a way to reduce these bottlenecks by compressing vectors while still preserving the inner products and distances that attention and retrieval depend on.

Vector quantization compresses high-dimensional vectors by mapping continuous values to a smaller set of discrete symbols. I have written about vector quantization in details in an earlier post. But traditional methods have a hidden cost: they need to store extra “quantization constants” (like scale and zero-point) for every small block of data. Google explicitly points out that this overhead can cost 1–2 extra bits per number, partially undoing the compression, which matters a lot when you are trying to live in the 2–4 bit regime.

TurboQuant solves this by eliminating those extra constants entirely. It achieves this through a clever two-stage pipeline that combines two sub-algorithms: PolarQuant and QJL. The idea is:

First, use most of the bit budget for a very strong geometry-aware lossy compression of the vector itself. Then use the last 1 bit to encode the leftover error in a way that makes inner products unbiased again.

TurboQuant Pipeline

The TurboQuant pipeline is as follows:

• Step 1 randomly rotates the vector using a fast Hadamard transform to make all coordinates roughly equally distributed.

• Step 2 (PolarQuant) uses most of the bit budget to quantize this rotated vector with near-optimal quality and zero overhead.

• Step 3 (QJL) spends just 1 extra bit to fix the small bias left over, ensuring accurate inner product estimation for attention scores.

Random Rotation

TurboQuant begins by applying a random rotation to the input vector. After that rotation, the coordinates behave much more nicely: each coordinate follows a concentrated distribution, and in high dimensions different coordinates become nearly independent. That means you can quantize each coordinate separately with an almost-optimal scalar quantizer, instead of needing a complicated joint vector codebook.

The paper’s theoretical story is that random rotation induces a coordinate distribution that is close to a nice reference law, so precomputed scalar quantizers become near-optimal.

A simplified code snippet:

import numpy as np

def random_rotation(v, seed=42):
    """Apply randomized Hadamard transform (simplified as random orthogonal)."""
    rng = np.random.RandomState(seed)
    # In practice, use fast Walsh-Hadamard with random sign flips
    # Here we simulate with a random orthogonal matrix
    d = len(v)
    Q, _ = np.linalg.qr(rng.randn(d, d))
    return Q @ v

PolarQuant

PolarQuant's core idea is elegant: instead of quantizing a vector in Cartesian coordinates (x, y, z, ...), convert it to polar coordinates i.e. a single radius and a set of angles. Instead of storing raw coordinates directly, it recursively turns groups of coordinates into radii + angles.

import numpy as np
 
def cartesian_to_polar_pair(x1, x2):
    """Convert a pair of Cartesian coordinates to polar (radius, angle)."""
    r = np.sqrt(x1**2 + x2**2)
    theta = np.arctan2(x2, x1)  # angle in [-pi, pi]
    return r, theta

Why does this help? In Cartesian space, each block of numbers has a different scale, so traditional quantizers need to store per-block scale/zero-point constants (that is the overhead). But in polar coordinates, after a random rotation, the angles follow a known, concentrated Beta distribution. Since the distribution is predictable, you don't need per-block constants, you can apply a fixed, globally-optimal quantizer.

The recursion works bottom-up: pair coordinates into (radius, angle), then pair the radii again, repeating until you have one final radius (the vector norm) and a tree of angles. Since the angles have a known distribution, you quantize them with a fixed grid. No per-block scale constants needed.

Here is a simplified implementation of PolarQuant in Python:

import numpy as np

def polarquant_encode(v, bits_per_angle=3, seed=42):
    """
    PolarQuant encoding:
    1. Randomly rotate the vector
    2. Pair coordinates → polar (radius, angle)
    3. Recursively pair radii until one final radius remains
    4. Quantize all angles to a fixed grid (no per-block constants!)
    """
    # Step 1: Random rotation spreads energy evenly across coordinates
    v_rot = random_rotation(v, seed)
    
    # Step 2: Recursive polar conversion
    coords = v_rot.copy()
    all_angles = []
    
    while len(coords) > 1:
        new_radii = []
        for i in range(0, len(coords), 2):
            if i + 1 < len(coords):
                r, theta = cartesian_to_polar_pair(coords[i], coords[i+1])
                new_radii.append(r)
                all_angles.append(theta)
            else:
                new_radii.append(coords[i])  # odd element passes through
        coords = np.array(new_radii)
    
    final_radius = coords[0]  # = ||v|| (the vector norm)
    
    # Step 3: Quantize angles to a FIXED grid
    # Key insight: after random rotation, angles follow a known Beta distribution
    # so we can use a universal grid — no per-block scale/zero needed!
    n_levels = 2 ** bits_per_angle
    quantized_angles = []
    for theta in all_angles:
        # Normalize angle to [0, 1] range
        normalized = (theta + np.pi) / (2 * np.pi)
        # Uniform quantization (optimal for the concentrated distribution)
        level = int(np.clip(normalized * n_levels, 0, n_levels - 1))
        quantized_angles.append(level)
    
    return final_radius, quantized_angles, bits_per_angle

Quantized Johnson–Lindenstrauss (QJL)

PolarQuant gives excellent MSE (mean-squared error) compression, but it introduces a subtle bias in inner product estimation. When you compute the attention score dot(query, key) using quantized values, the errors don’t cancel out but they systematically skew the result. This is a problem for attention mechanisms that rely on accurate dot products.

QJL fixes this by applying a Johnson-Lindenstrauss transform to the residual error (the difference between the original and quantized vector), then keeping only the sign bits (+1 or -1). This creates an unbiased estimator that, when combined with the PolarQuant output, gives accurate inner products.

The mathematical trick is that the sign of a random projection preserves angular information — two vectors that are close together will have similar sign patterns, while distant vectors will have very different sign patterns. The factor π/2 corrects the scaling so the estimator is unbiased.

Here’s the QJL implementation:

import numpy as np
 
def qjl_encode(residual, m=None, seed=0):
    """
    Quantized Johnson-Lindenstrauss (QJL) encoding.
    
    Projects the residual error into a lower-dimensional space using
    a random sign matrix, then keeps only the sign bits.
    
    Args:
        residual: the error vector e = v - v_hat (d-dimensional)
        m: projection dimension (default: d, can be smaller)
        seed: random seed for reproducible projection matrix
    
    Returns:
        sign_bits: {+1, -1} array of length m (1 bit each)
        norm_e: the norm of the residual (stored once per vector)
    """
    d = len(residual)
    if m is None:
        m = d
    
    # Generate random ±1 matrix (Rademacher distribution)
    # This matrix is NOT stored — it's regenerated from the seed
    rng = np.random.RandomState(seed)
    S = rng.choice([-1, 1], size=(m, d)).astype(np.float32)
    S /= np.sqrt(m)  # normalize
    
    # Project and take the sign
    projection = S @ residual
    sign_bits = np.sign(projection)  # each is +1 or -1 → 1 bit
    
    norm_e = np.linalg.norm(residual)
    
    return sign_bits, norm_e

Key Results

• Zero overhead constants. Unlike traditional methods that store scale + zero-point per block (adding 1-2 bits), TurboQuant needs no extra constants. The random rotation + polar conversion makes a fixed quantizer work universally.

• Data-oblivious and online. No training, fine-tuning, or dataset-specific codebooks required. Works immediately on any vector which is ideal for streaming KV cache compression.

• Near-optimal distortion. Within a ~2.7× constant factor of the information-theoretic lower bound. Provably efficient, not just empirically good, but mathematically close to the best any algorithm could achieve.

• Unbiased inner products. The QJL correction eliminates the systematic bias that MSE-optimal quantizers introduce. Critical for accurate attention score computation in transformers.

• Dual application. Works for both KV cache compression in LLMs and high-dimensional vector search. Outperforms product quantization (PQ) in recall while reducing indexing time to near zero.

Final Words

TurboQuant randomly rotates a vector to make its coordinates predictable, converts to polar coordinates so it can quantize with a fixed grid (no overhead), and uses a 1-bit sign trick on the leftover error to keep inner products unbiased.

Full code for a demo of TurboQuant can be found here: januverma/turboquant-demo

To understand Mamba, you need to understand the core tradeoff in sequence modeling. Transformers use self-attention, which stores all past tokens in a key-value (KV) cache. This gives them perfect recall but makes compute grow quadratically with sequence length, and the KV cache grows linearly, eating memory at long contexts. State space models take the opposite approach. They compress all past information into a fixed-size hidden state that gets updated at each timestep. This gives them linear scaling with sequence length which is great for efficiency but that fixed state has to do all the work of “remembering,” which is inherently lossy.

The SSM recurrence looks something like:

where h is the hidden state, A is a transition matrix controlling how the state evolves, B controls how new input enters the state, and a separate matrix C reads out the output. The design of A, B, and the discretization scheme (how you go from continuous to discrete time) determines the model's expressiveness.

Since the state can’t grow, the question becomes:

how do you make that fixed box more powerful?

This is exactly where the Mamba lineage diverges.

Mamba Series

• Mamba-1 (late 2023) made these matrices input-dependent (”selective”), meaning A, B, C change based on the current token which is a huge leap that let SSMs actually compete with Transformers on language modeling.

• Mamba-2 (mid-2024) simplified the SSM mechanism to leverage GPU tensor cores better, achieving 2–8x faster training than Mamba-1. The tradeoff was that it reduced the transition matrix A from a diagonal matrix to a scalar-times-identity, making the recurrence simpler but less expressive. This was a deliberate bet that training speed was the primary bottleneck.

• Mamba-3 The LLM landscape has shifted since Mamba-2. Post-training methods like reinforcement learning with verifiable rewards require massive amounts of generated rollouts, and agentic workflows have pushed inference demand enormously. Yet Mamba-2’s simplified recurrence left the decode step memory-bound that is the GPU spends most of its time moving data rather than computing.

  Mamba-3 asks:

  what would an SSM designed with inference in mind look like?

Mamba-3

The authors identified three “levers” to pull: make the recurrence more expressive, use a richer transition matrix, and add more parallel work per step, all without significantly increasing decode latency.

1. Exponential-Trapezoidal Discretization

Mamba-3 introduces a new discretization scheme that makes the recurrence formula more expressive. In classical numerical methods, “discretization” is how you convert a continuous-time differential equation into discrete update steps. Mamba-2 used a simpler scheme; the new exponential-trapezoidal approach implicitly applies a convolution-like operation on the input to the hidden state. This is significant because it actually allowed the team to remove the short causal convolution that had been a staple of Mamba-1 and Mamba-2 (and most linear models). That short conv was originally needed for induction-style retrieval capabilities, but the new recurrence, combined with simple biases on the B and C matrices, provides equivalent functionality intrinsically.

2. Complex-Valued State Tracking

Mamba-3 expands state-tracking capabilities by modeling a complex-valued SSM system. Instead of real-valued hidden states, the transition matrix operates in the complex plane. The clever implementation trick is that they express this via RoPE (Rotary Position Embeddings) interpreting complex transitions as rotations. This avoids having to rewrite kernels from scratch for complex arithmetic while giving the state richer dynamics to represent oscillatory or periodic patterns.

3. Multi-Input, Multi-Output (MIMO) SSMs

Instead of the standard single-input, single-output (SISO) SSM, Mamba-3 offers a MIMO variant that runs multiple SSMs in parallel. The key insight here is about the compute-vs-memory tradeoff during inference: current linear models use lots of GPU tensor cores for fast training, but during decoding, each timestep requires so little compute that the hardware sits idle most of the time. MIMO adds more FLOPs per timestep — using those idle cores — so decode latency stays roughly constant even though the model is doing more useful work. The cost shows up in training (which becomes slower) but not in inference. This is an elegant exploitation of the asymmetry between training and inference compute profiles.

Architecture Changes

Beyond the SSM core, Mamba-3 also modernizes its surrounding architecture. They added QK-normalization (called “BCNorm” in SSM terms) which stabilizes training, bringing Mamba-3 in line with contemporary transformer models. They also switched to interleaved MLP layers following standard transformer conventions. And as mentioned, the short causal convolution is gone.

Let me visualize the practical outcome: how Mamba-3 fits into the bigger picture of hybrid architectures, which the authors predict will be the dominant paradigm going forward.

This hybrid layout captures the Mamba-3 team’s prediction: linear layers will predominantly be used alongside global self-attention layers, the SSM handles the bulk of sequence processing at O(1) memory cost, while a few strategically placed attention layers handle exact retrieval tasks that need the full KV cache.

Final

Mamba-3 is a bet that the age of inference has arrived. Rather than simplifying the SSM to train faster (as Mamba-2 did), it enriches the recurrence via trapezoidal discretization, complex-valued states, and MIMO to make each decode step do more useful work using GPU cycles that were previously idle. The result is a model that is both more accurate and faster at generation than its predecessors, with open-sourced kernels built across three levels of GPU abstraction (Triton, TileLang, CuTe DSL) for maximum hardware performance. You can find the paper at arxiv.org/abs/2603.15569 and the code at the mamba-ssm GitHub repo.

• contrastive methods like CLIP, which learn by comparing positive and negative pairs

• generative methods like MAE or video diffusion models, which learn by reconstructing raw pixels.

The key philosophical insight is this: predicting at the pixel level forces a model to spend enormous capacity on unpredictable, irrelevant details like the exact texture of grass, the shimmer of light on water, the grain of a surface. These details are essentially random noise from the standpoint of understanding what is happening in a scene.

LeCun’s argument is that intelligent systems should instead learn to predict in an abstract representation space, focusing on the parts of the world that are actually predictable and semantically meaningful.

In a JEPA system, you have:

• An encoder that maps a visible/context portion of the input into an embedding.

• a predictor that tries to predict the embedding of the missing/target portion.

• a target encoder that produces the ground-truth embedding to match against, typically an exponential moving average of the main encoder.

Crucially, the predictor never has to reconstruct raw pixels, it just needs to get close in embedding space.

This has a practical consequence: semantically equivalent but superficially different outputs (two valid answers to the same question, two plausible continuations of a scene) will be nearby in embedding space, even though they would look completely different in pixel space. The model naturally learns to abstract away irrelevant variation.

V-JEPA: self-supervised learning from video

V-JEPA applies the JEPA principle to video. It predicts masked spatio-temporal regions in a learned representation space rather than reconstructing them at the pixel level. Concretely, given a video clip, the system masks out large spatio-temporal blocks (tubes of patches across multiple frames), encodes the visible patches, and then has the predictor fill in the embeddings of the masked regions.

V-JEPA’s pretraining is based solely on an unsupervised feature prediction objective, and does not use pretrained image encoders, text, negative examples, human annotations, or pixel-level reconstruction. This is a notably clean setup — the model learns purely from watching video.

The results were striking: it achieved 82.1% on Kinetics-400 and 71.2% on Something-Something-v2, surpassing previous best video models by +4 and +10 points respectively, all with frozen backbone evaluation (no fine-tuning).

V-JEPA 2: scaling up to a world model

V-JEPA 2 took the original recipe and scaled it dramatically. It was pretrained on a video and image dataset comprising over 1 million hours of internet video, using encoder models up to 1 billion parameters.

The key extensions from V-JEPA to V-JEPA 2 include: 3D tubelet tokenization (patches spanning 2 frames), 3D rotary positional embeddings, a progressive resolution training schedule that moves from 16 frames at 256×256 to 64 frames at 384×384, and multi-block masking at ~90% ratio.

But the really exciting part was what came after pretraining. By post-training a latent action-conditioned world model (V-JEPA 2-AC) using less than 62 hours of unlabeled robot videos from the Droid dataset, they deployed it zero-shot on Franka robot arms in two different labs — enabling picking and placing of objects using planning with image goals, without any task-specific training or reward. This demonstrated that self-supervised video pretraining can bootstrap genuine physical-world planning.

V-JEPA 2.1

This is the latest member of the family, released on March 16, 2026. V-JEPA 2.1 improves the training recipe to focus on learning high-quality and temporally consistent dense features. While V-JEPA 2 was already excellent at global video understanding tasks (classifying an action, answering a question about a scene), V-JEPA 2.1 focuses on making the per-token features (the dense, spatially localized representations) much higher quality.

The paper introduces four key innovations:

1. Dense Predictive Loss. In the original V-JEPA / V-JEPA 2, only the masked tokens contributed to the self-supervised loss. The encoder produced representations for visible patches, the predictor predicted representations for masked patches, and the loss was computed only on the masked ones. V-JEPA 2.1 uses a masking-based self-supervision objective where all tokens (both visible/context and masked tokens) contribute to the self-supervised training loss. This is a subtle but important change. By supervising every token, the encoder is pushed to produce high-quality features everywhere, not just in regions that happen to be visible during a particular forward pass.

2. Deep Self-Supervision. Rather than applying the loss only at the final layer of the encoder, V-JEPA 2.1 applies the self-supervised loss at multiple intermediate representations of the encoder models. This ensures that useful, semantically rich features emerge throughout the depth of the network, not just at the output. It is reminiscent of auxiliary losses in deep networks, but applied to the self-supervised prediction objective specifically.

3. Multi-Modal Tokenizers. V-JEPA 2.1 introduces multi-modal tokenizers for images and videos, allowing the model to handle both modalities with appropriate tokenization schemes rather than treating images as single-frame videos.

4. Model and Data Scaling. The approach benefits from model and data scaling, and the paper demonstrates improvements across both dense prediction tasks (where you need per-pixel or per-patch output quality) and global prediction tasks.

The net effect is that V-JEPA 2.1 produces features that are not only good for telling you what action is happening (global understanding) but also good for telling you exactly where things are and how they move over time with spatial precision (dense understanding). The PCA visualizations of the learned features reportedly show much cleaner, more temporally consistent spatial maps than previous versions.

Why this matters: the bigger picture

The JEPA family represents a bet that the path to robust AI, especially for physical-world understanding, robotics, and planning, runs through self-supervised prediction in latent space rather than through pixel-level generation or text-supervised contrastive learning. The progression tells a clear story:

• V-JEPA showed the principle works for video.

• V-JEPA 2 showed it scales and can bootstrap a world model for robotic planning.

• V-JEPA 2-AC showed the world model can actually plan actions. VL-JEPA showed you can bridge to language.

• V-JEPA 2.1 shows how to get the dense, spatially precise features you need for fine-grained tasks, closing a gap that earlier versions had relative to methods like DINOv2.

The underlying thesis that you should learn to predict what is predictable and abstract away what is not is what LeCun has been advocating as a foundation for machine intelligence that goes beyond pattern matching toward genuine world understanding.