LinerAI/snowflake-arctic-embed-m-v2.0-academic for Academic Search

This is a fine-tuned version of Snowflake/snowflake-arctic-embed-m-v2.0 optimized for academic and scientific literature search. The model has been trained using contrastive learning with hard negative mining, specifically curated for academic search scenarios.

Highlights

• Optimized for Academic Search: Fine-tuned on datasets specifically designed for academic literature retrieval
• Hard Negative Mining: Trained with carefully mined hard negatives to improve discrimination between similar academic papers
• Matryoshka Representation Learning (MRL): Supports flexible embedding dimensions (768, 512, 256, 128) for efficiency
• Efficient & Fast: Medium-sized model offering excellent speed-performance trade-off
• Long Context: Supports up to 4096 tokens

Model Description

Attribute	Value
Base Model	Snowflake/snowflake-arctic-embed-m-v2.0
Architecture	GTE
Embedding Dimension	768
MRL Dimensions	768, 512, 256, 128
Max Sequence Length	4096
Pooling	CLS token
Precision	float16

Evaluation Results

Model	Avg.	SciFact: Recall@10	TRECCOVID: Recall@10	NFCorpus: Recall@10	SCIDOCS: Recall@10	LitSearch: Recall@10	QASA: Recall@10
snowflake-arctic-embed-m-v2.0-academic	0.3729	0.8609	0.0219	0.177	0.2129	0.6435	0.321
snowflake-arctic-embed-m-v2.0	0.3654	0.8353	0.0224	0.1669	0.2122	0.6508	0.3046

Training Details

Training Configuration

Parameter	Value
Learning Rate	2e-5
Batch Size	8192 (effective)
Per-Device Batch Size	32
Warmup Steps	100
Weight Decay	0.1
Precision	fp16
Max Length	4096
Loss Function	InfoNCE (Contrastive)
Temperature (τ)	0.02

Training Data

The model was trained on LEAD (Liner Embedding Academic Dataset), a combination of ~55,560 samples tailored for academic search:

• MS MARCO (49%): General passage retrieval dataset with hard negatives
• Academic Search Dataset (51%): Custom dataset built specifically for academic literature search, with two-stage hard negative mining

Matryoshka Representation Learning (MRL)

This model supports Matryoshka Representation Learning. You can truncate embeddings to smaller dimensions (512, 256, 128) for faster computation and reduced storage.

# Full dimension (768)
full_embedding = embeddings[:, :768]

# MRL dimensions
embedding_512 = embeddings[:, :512]
embedding_256 = embeddings[:, :256]
embedding_128 = embeddings[:, :128]

Usage

Using Transformers

import torch
from transformers import AutoModel, AutoTokenizer

model_path = "LinerAI/snowflake-arctic-embed-l-v2.0-academic"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, torch_dtype=torch.float16, trust_remote_code=True)
model.eval()

# For queries
def encode_query(text):
    input_text = f"query: {text}"
    inputs = tokenizer(input_text, return_tensors="pt", max_length=4096, truncation=True)
    with torch.no_grad():
        outputs = model(**inputs)
        embeddings = outputs.last_hidden_state[:, 0]  # CLS token
        embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
    return embeddings

# For passages
def encode_passage(text):
    inputs = tokenizer(text, return_tensors="pt", max_length=4096, truncation=True)
    with torch.no_grad():
        outputs = model(**inputs)
        embeddings = outputs.last_hidden_state[:, 0]  # CLS token
        embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
    return embeddings

# Example: Academic search
query = "transformer models for protein structure prediction"
abstract = "We introduce AlphaFold, a deep learning system that predicts protein structures..."

query_emb = encode_query(query)
passage_emb = encode_passage(abstract)

similarity = torch.nn.functional.cosine_similarity(query_emb, passage_emb)
print(f"Similarity: {similarity.item():.4f}")

Input Format

Query Format

query: {your_query_text}

Passage Format

{your_passage_text}

Intended Use

• Academic Paper Search: Finding relevant research papers given a research query
• Literature Review: Discovering related work in academic literature
• Scientific Document Retrieval: Retrieving scientific documents, abstracts, and articles
• Research Question Answering: Finding papers that address specific research questions

Limitations

• Maximum sequence length is 4096 tokens
• Best performance achieved when using the specific input formats described above
• The model uses asymmetric encoding (query prefix for queries, no prefix for passages)
• Requires trust_remote_code=True for loading

License

This model is released under the Apache 2.0 license.