Introducing papr-embed: schema-native embeddings, and open-weights embeddings for code retrieval

Today we’re releasing two embedding model series, code-1 with open weights under Apache-2.0:

• papr-embed-code-1 (4B and 0.6B). Retrieval models for code and text. The 4B is #1 on all four of MTEB’s code leaderboards (as of Oct 29, 2026): Code (81.13, #1 of 60), CoIR (80.13, #1 of 65), RTEB Code (71.40, #1, with the 0.6B at #2) and CodeRAG (77.48, #1 of 11). It is also #1 on SciFact. On CoIR it is ahead of F2LLM-v2-14B (79.94) at under a third of its size.

  • Zero-shot where it counts. CodeRAG is 100% zero-shot for us: we never trained on any of its tasks. RTEB Code is 85% zero-shot: Apps is the only task there we trained on (its train split), and DS1000, HumanEval, MBPP, WikiSQL, FreshStack and SWEbench were all held out.

  • In-domain. CoIR, Code (v1) and SciFact are in-domain, since we trained on their train splits. The leaderboard marks this in its zero-shot column (20% on CoIR, 16% on Code). Details below.

• papr-embed-schema-1 (4B and 0.6B). Schema-native embeddings: you describe what matters in your domain as a schema, and the vector carries it as explicit, inspectable dimensions on top of the base embedding.

Both are available through the Papr API, and papr-embed-code-1 is on Hugging Face. Every code-1 number in this post can be reproduced with the replicate_mteb.py script that ships in each model repo.

papr-embed-code-1: code retrieval

nDCG@10 averaged over the 10 CoIR tasks, stock mteb, fp32. Max length 8192, except the five tasks whose corpora have documents over 8k tokens, which we ran at 32,768 (see the per-task table):

* From F2LLM’s submission in the MTEB results repo (mteb 2.6.7), not rerun by us. Scores from older mteb versions don’t always reproduce: stock Qwen3-Embedding-4B’s listed SyntheticText2SQL is 78.21; under our protocol we get 72.52. Base rows are our own runs with the same protocol, prompts and batch settings as the papr rows.

• The 4B is ahead of a model 3.5× its size, by a small margin. 80.12 vs 79.94 for F2LLM-v2-14B. That 0.18 is smaller than run-to-run movement on one task (SyntheticText2SQL, below), so read it as “level with a 14B model”, not as a clear win. The whole lead comes from that task: we score 72.27 on SyntheticText2SQL against F2LLM-v2-14B’s 64.12, and on the other nine tasks F2LLM-v2-14B is ahead, 81.7 to 81.0. It is +1.25 over the Qwen3-Embedding-4B it starts from (78.87), and +1.41 on the nine tasks other than SyntheticText2SQL (80.99 vs 79.58); on SyntheticText2SQL itself it is level with the base (72.27 vs 72.52).

• The 0.6B is 0.50 behind F2LLM-v2-0.6B (77.16 vs 77.66). Its weakest task is Apps (79.31; the 4B gets 92.56). It is +2.49 over Qwen3-Embedding-0.6B (74.67), and +2.61 on the nine tasks other than SyntheticText2SQL, and runs on a single consumer GPU.

• How we ran it. Batch size 32 wherever the task fits in 80 GB in fp32. Five tasks have documents over 8k tokens; we ran those at 32,768 context and batch 4. Only SyntheticText2SQL moves with batch size: its corpus has near-identical SQL answers, so the top hit comes down to floating-point ties, and the same weights have scored 65–72 across setups. For scale, stock Qwen3-Embedding-4B scores 72.52 on it in our runs, against 78.21 on its leaderboard entry from an older mteb. The full protocol, including every change and when we made it, ships with the weights as PROTOCOL.md.

Per-task results:

What we trained on. We want the numbers read correctly:

• No test split was used, anywhere. Only train splits entered training.

• CoIR (10 tasks): the 4B trained on the train splits of 8 of the 10, and the 0.6B on all 10. That is why MTEB lists the 4B as 20% zero-shot on CoIR (2 of 10 tasks never seen: CodeFeedback-ST and CodeTransOcean-Contest) and the 0.6B as 0%.

• SciFact: both models also trained on the SciFact train split. We evaluate on the official test split, so SciFact is an in-domain result, not a zero-shot one.

• Code (MTEB v1) adds two more: both models also trained on lightonai/CodeEditSearchTrain, so CodeEditSearch is in-domain, and MTEB’s CodeSearchNet shares its source with CoIR’s CodeSearchNet, whose train split we used. We declare both. That makes the 4B 16% zero-shot on Code (v1) and the 0.6B 0%. - RTEB (Code) is 85% zero-shot and CodeRAG is 100% zero-shot. Of RTEB (Code)’s seven public tasks (below), only Apps was in our training data; DS1000, HumanEval, MBPP, WikiSQL, FreshStack and SWEbench were not. None of CodeRAG’s tasks were in our training data. These are the leaderboard’s own zero-shot figures for both sizes.

• The leaderboard declares all of this in each model’s training_datasets.

• To check generalization, we also report retrieval tasks we never trained on, compared with the base Qwen3 models under identical settings:

Gains on code don’t come free everywhere. On non-code retrieval both models are flat to worse than their base: they gain about 1 on NFCorpus, lose 2–3 on ArguAna, SciDocs and FiQA (4B), and drop 6 on TREC-COVID. If your workload is mostly prose, use the base Qwen3 model; these are code models.

SciFact (scientific claim retrieval, official MTEB test split, nDCG@10):

Both trained on the SciFact train split only; the test split was never used. Read these as in-domain results. They show what targeted training can do, not zero-shot ability.

Beyond CoIR: the rest of MTEB’s code benchmarks

CoIR is 10 of the 12 tasks in MTEB’s Code (v1) benchmark. We also ran every open task in Code (v1), in RTEB (Code)(MTEB’s held-out code benchmark) and in CodeRAG, and submitted all of them, not a selection. The 4B is #1 on the leaderboard for each one: Code (v1), CoIR, RTEB Code and CodeRAG. The two held-out benchmarks, RTEB Code and CodeRAG, are where it is ranked mostly or entirely on data it never trained on. nDCG@10, mteb 2.26, fp32:

• Code (v1), 12 tasks: #1 of 60. The 4B averages 81.13, ahead of F2LLM-v2-14B and C2LLM-7B (both 80.75); the 0.6B averages 77.43. We lead on the average without winning any single task: our best per-task ranks are 4th on CodeFeedback-MT and 4th on CoIR’s CodeSearchNet, and our weakest is CodeTransOcean-DL (27th).

• RTEB (Code), 85% zero-shot: #1, with the 0.6B at #2. The 4B averages 71.40 over the seven public tasks and the 0.6B 67.34. We’re the only models on the leaderboard with a score for every public task, which is what puts us at #1 and #2. Other models such as voyage-4-large, Nemotron-3-Embed-8B and Octen-Embedding-8B are listed but don’t yet have a full score. On the six tasks most models share (Apps, DS1000, HumanEval, MBPP, WikiSQL, FreshStack; Apps is the only one in-domain for us), those three are ahead: the 4B averages 81.99 against about 86 for them (voyage-4-large 86.15, Nemotron-3-Embed-8B 86.00, Octen-Embedding-8B 85.56). Our weak spots are WikiSQL (91.79; the best model scores 99.50) and FreshStack (46.96 vs 57.76), plus DS1000. The 0.6B beats the 4B on WikiSQL (96.08 vs 91.79), which says our text-to-SQL training didn’t scale with size. RTEB’s two private tasks, Code1 and JapaneseCode1, have been requested from the MTEB maintainers.

• SWEbench retrieval needs long context. 21% of its 58,058 documents exceed 8k tokens, and truncating at 8k would drop 44% of all corpus tokens. Both models support 32k positions, so we ran it at 32,768. Scores are low for every model on this task: the 4B’s 7.89 is level with the best entries in the results repo (bekko-embedding-v1-a25m 8.41, BM25 7.76).

• Long documents elsewhere. Five CoIR tasks also have documents over 8k tokens (Apps, StackOverflowQA, CodeFeedback-MT and both CodeSearchNet variants), so we reran those at 32,768 too, as MTEB’s maintainers asked. It changed almost nothing: the largest move was CodeSearchNet-CC on the 0.6B, 92.23 → 91.91.

CodeRAG, 100% zero-shot: #1 of 11. We never trained on any CodeRAG task. Only 11 models have scores, so read the field as small:

The 4B ranks #1 on the leaderboard and the 0.6B #3, ahead of every model its size. The leaderboard ranks models by how many tasks each wins, not by average. The 4B wins two of the three tasks, while splade-code-8B’s mean is 0.23 higher because of LibraryDocumentation. Library documentation, long API reference pages, is our weak spot here, as it is on FreshStack and DS1000. We skipped CodeRAG’s fourth task, StackoverflowPosts (47M documents); no model in the results repo has run it.

Reference precision is fp32. The published numbers are fp32, and so is what we serve. Running in fp16 is faster but costs about 0.8 points on the 4B. We’d rather you get the model we measured.

papr-embed-schema-1: embeddings that know your schema

Most embeddings give you one opaque vector. papr-embed-schema-1 gives you a vector with structure:

• the base block: a strong general embedding;

• 14 schema bands: one per field of a schema you define. For a scientific-claims domain those might be claim stance, study type and organism; for a codebase, they might be API surface, language and intent;

• an evidence block.

The parts are combined with calibrated weights, so cosine similarity decomposes exactly into “how similar is the text” plus “how similar is it on each thing you said matters”. Use any schema we ship or bring your own.

Results on SciFact (official MTEB, nDCG@10):

• The 4B with a schema beats Qwen3-Embedding-8B at half the parameters.

• The domain-pinned 0.6B adds more than 3 points to its base.

• papr-embed-schema-1 is about control as much as score. The schema is an input you own, so you can change what “similar” means for your data without retraining.

Use it

Hugging Face (open weights): download papr-embed-code-1-4b or papr-embed-code-1-0.6b from huggingface.co/papr-ai.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("papr-ai/papr-embed-code-1-4b")
queries = model.encode(["how do I reverse a list in python"], prompt_name="query")
docs = model.encode(["def reverse(xs):\n    return xs[::-1]"])
print(model.similarity(queries, docs))

Task-specific instructions are in task_prompts.json in each repo; those are what the leaderboard numbers use. papr-embed-schema-1 is served through the Papr API and SDK at platform.papr.ai; its weights are not released yet.

Papr API:

curl https://memory.papr.ai/v1/embeddings \
  -H "X-API-Key: $PAPR_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "papr-embed-code-1-4b", "input": ["how do I reverse a list in python"],
       "input_type": "query",
       "instruction": "Given a question about coding, retrieve code or passages that solve it"}'

Run the evals yourself. The public papr-embed-evals repo reproduces both series: scripts/run_code1.py runs the open code-1 weights from Hugging Face on CoIR, SciFact or any other mteb task, and the schema-1 scripts show how to score and use schemas through the API.

Verify our numbers:

hf download papr-ai/papr-embed-code-1-4b --local-dir papr-embed-code-1-4b
python papr-embed-code-1-4b/replicate_mteb.py --model papr-embed-code-1-4b --tasks coir --out results --batch 32 --efficient-attn
# mteb==2.10.0, fp32. On one 80 GB GPU, Apps, StackOverflowQA, CodeFeedback-MT/ST and both CodeSearchNet tasks need --batch 16.

What’s next

• Publish a research paper for both papr-embed-schema and papr-embed-code models.

• Open weights for papr-embed-schema-1.

• Google Cloud availability (Vertex AI Model Garden), once we have confirmed the listing.

Links: Hugging Face (download the code-1 weights) · Evals repo · MTEB leaderboard · Developer docs and API · research paper (coming)