Dashboard

All AI Ask Public API Documentation

Welcome to the All AI Ask Public API. This API allows you to programmatically compare multiple LLMs and evaluate their outputs using AI judges.

Endpoint

POST https://api.allaiask.com/api/v1/prompt

Authentication

Include your API key in the request headers: Authorization: Bearer <YOUR_API_KEY>

Request Body

FieldTypeRequiredDescription
promptstringYesThe prompt text to send to the models.
modelsstring[]YesArray of model IDs to generate responses from.
judgesstring[]NoArray of model IDs to use as judges for comparing the outputs.
judge_promptstringNoCustom grading instructions for judges (e.g. "Focus on technical accuracy").
judge_weightsobjectNoWeights for the judge (accuracy, speed, cost). Default: 1, 0, 0.
attachmentsobject[]NoArray of base64 encoded files (images/PDFs depending on model support).
system_promptstringNoCustom instructions to set model behavior/persona.
generation_settingsobjectNoOptional parameters like temperature, maxTokens, topP.
response_schemaobjectNoOptional JSON schema to force a shared structured output across all models.
model_tuningobjectNoPer-model fine-tuning options (thinking budget, reasoning effort). See below.
zero_data_retentionbooleanNoPrevent prompt text, system prompt, model outputs, and judge reasoning/metadata from being logged persistently in database records. Default: false (or user profile preference).

Attachments Object

{
  "name": "image.png",
  "type": "image/png",
  "base64": "..."
}

Judging Weights Object

Balance the priorities for the judge evaluation. All values are 0.0 to 1.0.

{
  "accuracy": 1.0,
  "speed": 0.5,
  "cost": 0.2
}

Generation Settings Object

Fine-tune how models generate responses.

{
  "temperature": 0.7,    // 0.0 to 2.0 (Lower = deterministic, Higher = creative)
  "maxTokens": 1024,     // Limit the length of the generated response
  "topP": 1.0            // Nucleus sampling threshold
}

Structured Outputs (Response Schema)

Force all models to output valid JSON matching a specific schema. Native schema enforcement is used for OpenAI, Gemini, Groq (select models), and Amazon Bedrock/Nova (via Converse forced tool use); other providers receive a strengthened prompt instruction. Using a shared schema allows for consistent comparison and automated evaluation.

{
  "type": "object",
  "properties": {
    "summary": { "type": "string" },
    "confidence": { "type": "number" }
  },
  "required": ["summary", "confidence"],
  "additionalProperties": false
}

[!IMPORTANT] To ensure compatibility with OpenAI's strict mode, always include "additionalProperties": false in your schema objects.

Model Tuning Object

Pass per-model fine-tuning parameters keyed by model ID. This object must be assigned to the "model_tuning" field in your request payload. Only set values you want to override — omitted models use defaults.

{
  "model_tuning": {
    "claude-sonnet-4-6": { "thinkingBudget": 8000 },
    "gemini-3.5-flash":   { "thinkingBudget": 8192, "webSearch": true },
    "o3-mini":            { "reasoningEffort": "high" },
    "grok-4.3":           { "reasoningEffort": "medium" }
  }
}
ModelParameterTypeDescription
claude-sonnet-4-6, claude-opus-4-7, claude-opus-4-6thinkingBudgetintegerExtended thinking token budget (1024–32000). Set 0 to disable.
gemini-3.5-flash, gemini-3.1-pro, gemini-3.1-flash, gemini-3.1-flash-litethinkingBudgetintegerGemini thinking token budget (0–24576). Set 0 to disable.
gemini-3.5-flash, gemini-3.1-pro, gemini-3.1-flash, gemini-3.1-flash-litewebSearchbooleanEnable Google Search grounding (web search).
o3-mini, o3, o4-mini, gpt-5.4-pro, gpt-5.4, gpt-5.4-mini, gpt-5.4-nano, gpt-5, gpt-5-mini, gpt-5-nanoreasoningEffortstring"low", "medium", or "high". Controls reasoning compute spend.
grok-4.20-0309-reasoning, grok-4.3reasoningEffortstring"none", "low", "medium", or "high". Controls reasoning compute spend.
cerebras-gpt-oss-120b, cerebras-glm-4.7reasoningEffortstring"low", "medium", or "high". Controls reasoning compute spend on Cerebras.

[!NOTE] Amazon Nova models (nova-pro, nova-lite, nova-micro) and Mistral models do not expose reasoning/thinking tuning parameters — omit them from model_tuning.

Example Request

curl -X POST https://api.allaiask.com/api/v1/prompt \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "What are the benefits of side-by-side LLM comparison?",
    "models": ["gpt-4.1", "claude-sonnet-4-6", "gemini-3.1-pro"],
    "judges": ["o3-mini"],
    "judge_weights": { "accuracy": 1, "speed": 0.5 },
    "judge_prompt": "Evaluate the clarity and technical depth of the explanations.",
    "zero_data_retention": true,
    "model_tuning": {
      "claude-sonnet-4-6": { "thinkingBudget": 4000 },
      "gemini-3.1-pro": { "webSearch": true }
    },
    "generation_settings": {
      "temperature": 0.7,
      "maxTokens": 1024
    },
    "response_schema": {
      "type": "object",
      "properties": {
        "answer": { "type": "string" }
      }
    }
  }'

Supported Models

You can use any of these IDs in the models array.

[!IMPORTANT] Models are routed by modelId, not by a separate provider field. Each modelId is globally unique and the gateway maps it to the correct provider internally. You never specify a provider — just the ID.

The same underlying model served by more than one provider has a distinct modelId. The provider is encoded in the ID (usually as a prefix), so there is never a collision. For example, OpenAI's gpt-oss-120b open-weight model is available on two providers:

modelIdProviderNotes
gpt-oss-120bGroqLow time-to-first-token (~60–120ms)
cerebras-gpt-oss-120bCerebras~2–3× higher throughput on long outputs

To compare the same model across providers, simply list both IDs: "models": ["gpt-oss-120b", "cerebras-gpt-oss-120b"]. They run as independent entries and appear as separate results in the response.

Reasoning & Flagship Models

The most capable models for complex problem solving, reasoning, and advanced logic.

  • gpt-5.4 (OpenAI)
  • gpt-5.4-pro (OpenAI)
  • gpt-5 (OpenAI)
  • gpt-4.1 (OpenAI)
  • gpt-4o (OpenAI)
  • o3-mini (OpenAI)
  • grok-4.20-0309-reasoning (xAI)
  • claude-opus-4-6 (Anthropic)
  • claude-sonnet-4-6 (Anthropic)
  • gemini-3.1-pro (Google)
  • gpt-oss-120b (Groq/OpenAI)
  • cerebras-gpt-oss-120b (Cerebras/OpenAI)
  • cerebras-glm-4.7 (Cerebras/Z.ai)
  • deepseek-v4-pro (DeepSeek)
  • mistral-large (Mistral)
  • nova-pro (Amazon Bedrock)

Fast & Efficient Models

Optimized for high-speed, cost-effective multimodal chat and generation.

  • gpt-5.4-mini (OpenAI)
  • gpt-5.4-nano (OpenAI)
  • gpt-5-mini (OpenAI)
  • gpt-5-nano (OpenAI)
  • grok-4.20-0309-non-reasoning (xAI)
  • grok-4.3 (xAI)
  • claude-haiku-4-5 (Anthropic)
  • gemini-3.5-flash (Google)
  • gemini-3.1-flash-lite (Google)
  • llama-4-scout (Groq)
  • gpt-oss-20b (Groq/OpenAI)
  • qwen3-32b (Groq)
  • deepseek-v4-flash (DeepSeek)
  • mistral-medium (Mistral)
  • mistral-small (Mistral)
  • ministral-8b (Mistral) — cheapest flat-rate option
  • codestral (Mistral) — code-specialised
  • nova-lite (Amazon Bedrock)
  • nova-micro (Amazon Bedrock) — cheapest, fastest budget model

Legacy / Dated Models

These models are still fully supported by the API but have been superseded by newer generations.

  • gpt-4o-mini (OpenAI)
  • gpt-4-turbo (OpenAI)
  • claude-sonnet-4.5 (Anthropic)
  • claude-opus-4.5 (Anthropic)
  • claude-opus-4.1 (Anthropic)
  • claude-sonnet-4 (Anthropic)
  • claude-opus-4 (Anthropic)
  • gemini-3.1-flash (Google)
  • gemini-2.5-flash (Google)
  • gemini-2.5-flash-lite (Google)
  • llama-3.3-70b (Groq)
  • llama-3.1-8b (Groq)
  • grok-3 (xAI)
  • grok-3-mini (xAI)

Supported Judges

While most models can technically judge, we recommend using these IDs in the judges array for the most reliable evaluations:

  • Primary Choice: gpt-5.4-pro or gpt-4.1
  • Alternative: claude-opus-4-6 or gemini-3.1-pro
  • Specialized Reasoning: grok-4.20-0309-reasoning or deepseek-v4-pro

Note: Judging involve a "position swap" test (running the evaluation twice) to ensure consistency and eliminate bias. When judges are requested in the API, the response will wait for these evaluations to complete before returning.

Response Body

FieldTypeDescription
promptLogIdstringUnique identifier for the request.
resultsobject[]Array of outputs from each requested model.
evaluationsobject[]Array of judging results (pairwise comparisons).
creditsUsednumberTotal credits deducted from your balance for this request.

Result Object

{
  "modelId": "gpt-4o-mini",
  "status": "completed",
  "output": "The response text...",
  "tokensIn": 150,
  "tokensOut": 320,
  "latencyMs": 850,
  "providerCost": 0.00015 // Raw cost from the provider
}

Context & Prompt Caching

To optimize performance and reduce API usage costs, several providers supported by the All AI Ask platform offer context/prompt caching. This allows you to cache large, repetitive inputs (such as system instructions, file contents, or extensive chat history).

1. Automatic Caching

For the following providers, caching is fully automatic at the infrastructure level. You do not need to make any API request changes; the platform will automatically reuse cache segments when matching prefixes are sent:

  • OpenAI: Automatically caches identical prompt prefixes $\ge$ 1,024 tokens. Match segments receive a 50% discount on input tokens.
  • DeepSeek: Automatically caches identical prompt prefixes in multiples of 64 tokens. Matches receive a 90% discount on input tokens.
  • Mistral: Automatically caches identical prompt prefixes at their gateway.

2. Explicit Caching: Anthropic (Claude)

Anthropic allows explicit prompt caching for large system instructions, history, or attachments.

  • Requirement: Minimum prompt length of 1,024 tokens (Sonnet/Opus) or 2,048 tokens (Haiku). Up to 4 breakpoints can be declared.
  • Implementation: Specify "cache_control": { "type": "ephemeral" } within the system_prompt array or on specific history elements.

Example Request (Anthropic Caching)

{
  "prompt": "Evaluate the new code edits.",
  "models": ["claude-sonnet-4-6"],
  "system_prompt": [
    {
      "type": "text",
      "text": "You are a senior code reviewer with 20+ years of experience...",
      "cache_control": { "type": "ephemeral" }
    }
  ],
  "history": [
    {
      "role": "user",
      "content": "Here is the codebase: [insert massive codebase files...]",
      "cache_control": { "type": "ephemeral" }
    },
    {
      "role": "assistant",
      "content": "Understood. I have reviewed the codebase structure."
    }
  ]
}

3. Explicit Caching: Google (Gemini)

Gemini supports explicit context caching for extremely large files, videos, or prompt chains.

  • Requirement: Minimum of 32,768 tokens (often equivalent to ~1 hour of audio/video or a massive codebase).
  • Implementation:
    1. Create a cache resource directly via Google's Gemini API /cachedContents endpoint using your API credentials.
    2. Pass the returned cache name string (e.g., cachedContents/1234abcd) in the "model_tuning" field under "cachedContent" for your Gemini model.

Example Request (Gemini Caching)

{
  "prompt": "Summarize key findings from this video.",
  "models": ["gemini-3.1-pro"],
  "model_tuning": {
    "gemini-3.1-pro": {
      "cachedContent": "cachedContents/1234567890abcdef"
    }
  }
}

Error Codes

  • 400: Missing prompt or invalid model IDs.
  • 401: Invalid or missing API Key.
  • 402: Insufficient Balance.
    • Response includes estimatedCreditsRequired.
  • 429: Rate limit exceeded.