Groq
batchling is compatible with Groq through any supported framework
The following endpoints are made batch-compatible by Groq:
/openai/v1/chat/completions/openai/v1/audio/transcriptions/openai/v1/audio/translations
Check model support and batch pricing
Before sending batches, review the provider's official pricing page for supported models and batch pricing details.
The Batch API docs for Groq can be found on the following URL:
https://console.groq.com/docs/batch
Example Usage
API key required
Set GROQ_API_KEY in .env or ensure it is already loaded in your environment variables before running batches.
Here's an example showing how to use batchling with Groq:
groq_example.py
import asyncio
import os
from dotenv import load_dotenv
from groq import AsyncGroq
from batchling import batchify
load_dotenv()
async def build_tasks() -> list:
"""Build Groq requests."""
client = AsyncGroq(api_key=os.getenv(key="GROQ_API_KEY"))
questions = [
"Who is the best French painter? Answer in one short sentence.",
"What is the capital of France?",
]
return [
client.chat.completions.create(
model="llama-3.1-8b-instant",
messages=[
{
"role": "user",
"content": question,
}
],
)
for question in questions
]
async def main() -> None:
"""Run the Groq example."""
tasks = await build_tasks()
responses = await asyncio.gather(*tasks)
for response in responses:
print(f"{response.model} answer:\n{response.choices[0].message.content}\n")
async def run_with_batchify() -> None:
"""Run `main` inside `batchify` for direct script execution."""
async with batchify():
await main()
if __name__ == "__main__":
asyncio.run(run_with_batchify())
Output: