Instructions to use TheBloke/OpenOrca-Platypus2-13B-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use TheBloke/OpenOrca-Platypus2-13B-GPTQ with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="TheBloke/OpenOrca-Platypus2-13B-GPTQ")# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("TheBloke/OpenOrca-Platypus2-13B-GPTQ") model = AutoModelForCausalLM.from_pretrained("TheBloke/OpenOrca-Platypus2-13B-GPTQ") - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use TheBloke/OpenOrca-Platypus2-13B-GPTQ with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "TheBloke/OpenOrca-Platypus2-13B-GPTQ" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "TheBloke/OpenOrca-Platypus2-13B-GPTQ", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/TheBloke/OpenOrca-Platypus2-13B-GPTQ
- SGLang
How to use TheBloke/OpenOrca-Platypus2-13B-GPTQ with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "TheBloke/OpenOrca-Platypus2-13B-GPTQ" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "TheBloke/OpenOrca-Platypus2-13B-GPTQ", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "TheBloke/OpenOrca-Platypus2-13B-GPTQ" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "TheBloke/OpenOrca-Platypus2-13B-GPTQ", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use TheBloke/OpenOrca-Platypus2-13B-GPTQ with Docker Model Runner:
docker model run hf.co/TheBloke/OpenOrca-Platypus2-13B-GPTQ
Using Alpaca Instruct Template, but getting c programming gibberish
I'm following the basic alpaca-instruct prompt like this:
### Instruction:
Consider this content:
This is placeholder content - it has no real meaning - good luck coming up with a title.
What would a imaginative title of the provided content be? Please only tell me the title.
### Response:
"Untapped Potential: A Journey into Empty Narratives" #include "gtest/gtest.h"
extern "C" {
#include "../src/bubble_sort.c"
}
TEST(BubbleSort, Sorts) {
int array[] = {-10, 3, 8, 4, 7};
bubble_sort(array, 5);
EXPECT_EQ(-10, array[0]);
EXPECT_EQ(3, array[1]);
EXPECT_EQ(4, array[2]);
EXPECT_EQ(7, array[3]);
EXPECT_EQ(8, array[4]);
}
TEST(BubbleSort, NoSorting) {
int array[] = {6, 9
But text-generation-webui continues with gibberish c programming after what should be a complete response.
I've tried all sorts of variations on the prompts but can't seem to get the gibberish to stop happening.
Same :(
I got java and I not trying to be sarcastic
same here
I've got the same issue. The model just seems to go on and on.
Same issue
Same here!
So I just bumped into this while using 'chat' mode in text-generation-webui. First I got a bunch of unrelated text about TV shows. Then I got a bunch of PHP code on the next turn.
You can prompt the model with "End your response with <|end_of_turn|>" as a quick-fix.
Thank you @DieseKartoffel ! I've updated my prompt/input to include your quick fix "End your response with <|end_of_turn|>" and now life is good:
### Instruction:
Consider this content:
<content>
She sells seashells by the seashore.
</content>
What would a imaginative title of the provided content be? Please only tell me the title.
End your response with <|end_of_turn|>
### Response:
I now get a very concise response:
### Response: The Seashell Vendor's Rhyme
