SAM3 for fixed prompt lists · ONNX + TensorRT 10.3 · measured on RTX 4070 · Oct 2026

SAM3 model file map

How SAM3 went from 4.0 GB of VRAM to 1.47 GB while it runs, with the model itself down from 3.7 GB to 1.2 GB: which model files exist, which ones stay loaded, and which run only once.

GPU memory held by SAM3 · 4K frames3.7 GB → 1.2 GBModel memory 3,766 → 1,262 MiB. Whole process in nvidia-smi, CUDA context included: 4,028 → 1,474 MiB.

Parameters on the GPU481Mwas 835M with the stage-2 split

Detections that differ from FP321 of 72stage-2 split: 8 of 72

The journey

Three stages, one workload: a fixed list of prompts, one image at a time.

STAGE 1

SAM3 base

facebook/sam3

Meta's PyTorch model in FP16, 840.4M parameters. Every prompt runs the whole network, and every mask is resized to the full frame on the GPU.

4,028 MiBwhole process at 4K · 3,766 MiB model memory

STAGE 2

Split encoder–decoder

sanskar003/SAM3-Split-Encoder-Decoder

Built from Embedl's INT8 ONNX export (embedl/sam3). The vision encoder runs once per image, the decoder once per prompt. The decoder still carries the 353M-parameter text encoder.

1,872 MiBwhole process · 1,664 MiB model memory

STAGE 3 · THIS RELEASE

Text-free decoder + prompt bank

sanskar003/SAM3-Split-TextFree

The text encoder runs once, offline, on CPU, and its outputs are saved to a prompt bank. The GPU keeps the vision encoder and a 26M-parameter decoder.

1,474 MiBwhole process · 1,262 MiB model memory

Model memory: engines, working memory, I/O buffers CUDA context and runtime

1 · SAM3 base4,028 MiB

3,766

2 · Split1,872 MiB

1,664

3 · Text-free1,474 MiB

1,262

Whole-process VRAM of the running app in nvidia-smi, in MiB: RTX 4070 (16 GB), one 3840×2160 image with 4 prompts. Stage 1: official facebook/sam3 weights in PyTorch 2.6, FP16. Stages 2 and 3: TensorRT 10.3 engines built --fp16 --int8.

Input frame	1 · SAM3 base	2 · Split	3 · Text-free
1006×679	2,428 MiB	1,872 MiB	1,474 MiB
1920×1080	2,794 MiB	1,872 MiB	1,474 MiB
3840×2160	4,028 MiB	1,872 MiB	1,474 MiB

Why frame size only matters for SAM3 base

The official post-processing resizes every predicted mask to the full frame on the GPU, and PyTorch keeps that peak memory reserved. The split pipelines take a fixed 924×924 input and resize masks on the CPU, so their GPU memory is the same for any frame.

What loads, and when

The production app runs a fixed list of prompts on one image at a time: one vision encode, then one decode per prompt. With that workload every file falls into one of four groups.

Loaded the whole time · production

sam3_encoder.engineVision encoder. Runs once per image. 455.2M params768 MiB VRAM
sam3_decoder_notext.engineDetector head: fusion encoder, DETR decoder, mask head. Runs once per prompt. 26.0M params494 MiB VRAM
prompt_bank.npzPrecomputed text embeddings, read into host RAM at startup. Holds no weights. ~33 KB per prompt0 MiB VRAM

Model memory: 1,262 MiB. Whole process: 1,474 MiB.

Loaded once · offline, on CPU

sam3_text_encoder.onnxText encoder. Runs in ONNX Runtime on CPU to build the prompt bank. 353.4M params1.41 GB file0 MiB VRAM
facebook/sam3 tokenizerTurns each prompt into 32 token ids.

Run again only when the prompt list changes. It takes seconds and needs no GPU.

Build time only

sam3_encoder.onnxSource of sam3_encoder.engine.
sam3_decoder_notext.onnxSource of sam3_decoder_notext.engine.

Engines are tied to a GPU model. Build them on the production GPU.

Not needed in production

sam3_decoder.onnxThe stage-2 decoder, text encoder inside. Re-ran the text encoder for every prompt. 379.4M params896 MiB VRAM as an engine

Kept in the repo for verification and rollback.

Stage 2 → stage 3: how the pipeline changed

The text encoder is shaded orange in both halves. In stage 2 it lived inside the decoder and re-ran for every prompt, although the prompts never change. In stage 3 it runs once, offline, on CPU. Its outputs go to prompt_bank.npz, and production keeps only the vision encoder and a 26M-parameter decoder on the GPU.

Where stage 3 saved 402 MiB

Memory each engine holds once loaded, measured as the drop in free VRAM. The whole saving is weights: the text encoder left the GPU, and nothing else moved.

Vision encoder engine (all of it) Decoder weights Decoder working memory + I/O buffers Freed

Stage 21,664 MiB

768 450 446

Stage 31,262 MiB

768 445 −402 freed

Model memory of the two engines, in MiB, CUDA context excluded. Only the decoder weights changed: 450 → 49 MiB. Every value is also in the table below.

Engine	Params	Weights	Working memory	I/O buffers	Total in VRAM
sam3_encoder both	455.2M	529 MiB	140 MiB	99 MiB	768 MiB
sam3_decoder stage 2	379.4M	450 MiB	303 MiB	143 MiB	896 MiB
sam3_decoder_notext stage 3	26.0M	49 MiB	302 MiB	143 MiB	494 MiB
Stage 2 total	834.6M	979 MiB	443 MiB	242 MiB	1,664 MiB
Stage 3 total	481.2M	578 MiB	442 MiB	242 MiB	1,262 MiB
Saved	353.4M	401 MiB	1 MiB	0 MiB	402 MiB (−24%)

Weights = measured total minus working memory and I/O buffers, so it includes a few MiB of TensorRT overhead.
Working memory is TensorRT's activation scratch space. It barely changed: the mask head (200 queries at 264×264) dominates it, and text activations at 32 tokens are tiny.
I/O buffers are the engines' input and output tensors. pred_masks alone is 200×264×264 float32, 53 MiB.
Adding the CUDA context and runtime (about 210 MiB, the same in both stages) gives the whole-process figures in the journey chart: 1,872 and 1,474 MiB.

Accuracy and speed

Both TensorRT pipelines were compared with the FP32 model: sam3_decoder.onnx run in full precision by ONNX Runtime, on the same image features. 1 test image, 7 prompts, 72 reference detections at score > 0.25. A sanity check, not a benchmark.

Measure	Stage 2	Stage 3
Detections that differ from FP32	8 of 72	1 of 72
Mean score error on detections	0.0135	0.0036
Mean mask IoU vs FP32	0.9855	0.9898
Decode, per prompt	32.2 ms	30.5 ms
1 encode + 4 decodes, with CPU post-processing	270 ms	260 ms
Tokenizer in the production process	yes	no

Why stage 3 is more accurate

In the INT8/FP16 engine, the text encoder's output was about 30% off the FP32 value (mean absolute error 0.24 on features of mean magnitude 0.81). That error reached every prediction. The text encoder now runs only once, so the prompt bank is computed in FP32 and that error is gone.

Why speed barely changed

At 32 tokens the text encoder was cheap to run. Its cost was the memory it held, which is what stage 3 removed.

Every file

Four ONNX files ship in the model repo. You build the two engines and the prompt bank yourself. Disk sizes in MB; VRAM in MiB.

File	Comes from	On disk	Params	What it is	When it's used
sam3_encoder.onnx	model repo	1,821	455.2M	Vision encoder. Image in, 3 feature maps out.	build Source of the encoder engine.
sam3_decoder_notext.onnx	model repo	104	26.0M	Detector head: fusion encoder, DETR decoder, mask head. Feature maps and text embeddings in, detections out.	build Source of the decoder engine.
sam3_text_encoder.onnx	model repo	1,414	353.4M	Text encoder with its 1024→256 projection. Token ids in, `text_features` and `text_padding_mask` out.	offline once Builds the prompt bank, on CPU.
sam3_decoder.onnx	model repo (stage 2)	1,518	379.4M	Stage-2 decoder: text encoder and detector head in one graph.	not in production Verification and rollback.
sam3_encoder.engine	you build it	512	455.2M	TensorRT build (`--fp16 --int8`). Runs once per image.	production Loaded the whole time.
sam3_decoder_notext.engine	you build it	44	26.0M	TensorRT build (`--fp16 --int8`). Runs once per prompt.	production Loaded the whole time.
prompt_bank.npz	you create it	0.033 / prompt	none	For each prompt: the string, its 32 token ids, `text_features` and `text_padding_mask`. Plain arrays, not tied to a GPU.	production Host RAM, read at startup.

Tensor interfaces

Every model's outputs carry the same names as the next model's inputs, so tensors pass straight through by name.

sam3_encoderproduction

inimage float32 [1,3,924,924] out/Add_179_output_0 float32 [1,4356,256] out/fused_229/Conv_output_0 float32 [1,256,264,264] out/fused_232/Conv_output_0 float32 [1,256,132,132]

sam3_decoder_notextproduction

inthe 3 feature maps intext_features float32 [32,1,256] intext_padding_mask bool [1,32] outpred_logits float32 [1,200,1] outpred_boxes float32 [1,200,4], normalized cx, cy, w, h outpresence_logit_dec float32 [1,1] outpred_masks float32 [1,200,264,264]

sam3_text_encoderoffline once

intokenized_text int64 [1,32], zero-padded outtext_features float32 [32,1,256] outtext_padding_mask bool [1,32], true where token id = 0

sam3_decoder (stage 2)not in production

inthe 3 feature maps intokenized_text int64 [1,32] outsame 4 outputs as sam3_decoder_notext

How the text split was made

Found the text-only part.
Every node in sam3_decoder.onnx that is reachable from tokenized_text but not from the image features is text encoder: 353.4M of the decoder's 379.4M parameters (93%).
Found the narrowest cut.
A minimum vertex cut between the token ids and the image-dependent nodes has exactly two tensors: the projected text features and the padding mask. They were renamed text_features and text_padding_mask.
Split the decoder and checked it.
In FP32 ONNX Runtime, text encoder → text-free decoder matches sam3_decoder.onnx bit for bit (largest difference 0.0 on all 4 outputs, 4 prompts).
Built the new decoder engine.
Same --fp16 --int8 recipe: 44 MB, against 465 MB for the stage-2 decoder engine.
Kept the text encoder off TensorRT.
As an INT8/FP16 engine its output was about 30% off FP32. It runs only once, so the prompt bank is computed in FP32 with ONNX Runtime on CPU.
Verified end to end.
1 of 72 detections differs from FP32 (stage 2: 8), and the running process dropped from 1,872 to 1,474 MiB.

How to run it

Full code, including an ONNX Runtime reference pipeline, is on the model card.

Whenever the prompt list changes

CPU only, a few seconds. No engine rebuild.

tok = AutoTokenizer.from_pretrained("facebook/sam3"); tok.pad_token_id = 0
ids = tok(prompts, padding="max_length", max_length=32, truncation=True, return_tensors="np")["input_ids"]
text_encoder = ort.InferenceSession("sam3_text_encoder.onnx")
# run each prompt's ids through text_encoder and save the outputs to prompt_bank.npz

Once per GPU model

Engines only run on the GPU model they were built on.

trtexec --onnx=sam3_encoder.onnx --fp16 --int8 --builderOptimizationLevel=5 --memPoolSize=workspace:4G --saveEngine=sam3_encoder.engine
trtexec --onnx=sam3_decoder_notext.onnx --fp16 --int8 --builderOptimizationLevel=5 --memPoolSize=workspace:4G --saveEngine=sam3_decoder_notext.engine

In production

Load the two engines and the prompt bank once, then for each image:

features = encoder.run({"image": x})                       # once per image
for prompt in prompts:                                     # once per prompt
    outputs = decoder.run({**features, **prompt_bank[prompt]})