Skip to main content
Preview
prompt_jev is in preview. The function name, parameters, and return types may change. Avoid workloads with regulated or sensitive data while in Preview.

Triage classifications by confidence

Every prompt_jev answer comes with calibrated probabilities. Use this guide to split a classified table into three groups: rows confident enough to act on, rows worth a second pass with a language model, and rows a person should read. You're done when each row has a disposition and you know what fraction of the table each bucket holds.

Before you start​

  • A table with a prompt_jev result column. See Classify text with prompt_jev, which builds the my_db.job_postings table these examples use and fills its role_family column from the job postings dataset.
  • A rough sense of what an error costs you. That's what sets the threshold, not a default number.

Step 1: Know which number to read​

The three question types expose certainty differently.

TypeReadMeaning
noulThe returned DOUBLEProbability the statement is true. Values near 0.5 are the uncertain ones.
choice.confidence, and .probabilities for the runner-upHow certain the model is about the winning label.
score.confidence, and .score for the positionHow certain the model is about where on the scale the row sits.

For choice, the gap between the top two probabilities often separates cases better than confidence alone. A row at 0.45 against 0.43 is a genuine tie; a row at 0.55 against 0.12 is not.

Read the confidence and runner-up
SELECT
  title,
  role_family.choice AS role_family,
  role_family.confidence,
  list_sort(
    list_transform(role_family.probabilities, p -> p.probability),
    'desc'
  ) AS ranked
FROM
  my_db.job_postings
LIMIT
  10;
SQL Editor loading...
Login to connect

An analytics engineering posting is the classic tie: it splits between analytics and data_engineering and lands around 0.5 on both. A posting titled "Staff Machine Learning Engineer" does not.

Step 2: Look at the distribution before picking a threshold​

Bucket the confidence values to estimate how many rows each cutoff would send for review. A confidence distribution describes the model's certainty, not its error rate. Choose the threshold using labelled examples and the cost of wrong decisions.

Over 200 postings classified into five role families, the mass sits at the top and the tail is thin but real:

Bucket the confidence values
SELECT
  floor(role_family.confidence * 10) / 10 AS confidence_bucket,
  count(*) AS rows_in_bucket
FROM
  my_db.job_postings
GROUP BY
  confidence_bucket
ORDER BY
  confidence_bucket;
SQL Editor loading...
Login to connect
PreviewLogin for live results

Two thirds of the postings land at 0.9 or above, and 30 sit below 0.6. For illustration, a cut at 0.85 sends 136 rows straight through, 34 to a second model pass, and 30 to a person. Validate these cutoffs on your own data before using them.

Create a representative labelled evaluation set and measure accuracy per confidence bucket. Include high-confidence rows so confident mistakes remain visible. Compare errors across labels and relevant input groups, such as languages or document sources, rather than relying on overall accuracy alone. Keep evaluation examples separate from the examples you use to refine the instructions and criteria.

Measure accuracy per bucket
SELECT
  floor(role_family.confidence * 10) / 10 AS confidence_bucket,
  count(*) AS rows_in_bucket,
  round(avg((role_family.choice = human_label)::INT), 3) AS accuracy
FROM
  my_db.job_postings
WHERE
  human_label IS NOT NULL
GROUP BY
  confidence_bucket
ORDER BY
  confidence_bucket;
SQL Editor loading...
Login to connect

Step 3: Assign a disposition​

Write the buckets as a column so downstream queries agree on the split.

Assign a disposition
CREATE OR REPLACE TABLE my_db.role_family_triage AS
SELECT
  job_id,
  title,
  description,
  role_family.choice AS role_family,
  role_family.confidence,
  CASE
    WHEN role_family IS NULL THEN 'failed'
    WHEN role_family.confidence >= 0.85 THEN 'auto'
    WHEN role_family.confidence >= 0.60 THEN 'escalate'
    ELSE 'review'
  END AS disposition
FROM
  my_db.job_postings;

SELECT
  disposition,
  count(*) AS postings
FROM
  my_db.role_family_triage
GROUP BY
  disposition
ORDER BY
  postings DESC;
SQL Editor loading...
Login to connect

Handle NULL explicitly. A failed request and a low-confidence answer need different follow-ups: the first is a retry, the second is a judgement call.

Step 4: Send the uncertain rows to a language model​

prompt_jev is cheap per row, prompt is not. Running the cheap pass over everything and the expensive pass over the remainder keeps the cost close to the cheap one.

Send the uncertain rows to a language model
CREATE TABLE my_db.escalated_role_family AS
SELECT
  job_id,
  title,
  prompt (
    'Which kind of data role does this job posting describe? Reply with the label only.' || E'

Posting: ' || description,
    struct := {role_family: 'VARCHAR', reasoning: 'VARCHAR'},
    struct_descr := {
      role_family: 'one of: analytics, data_engineering, data_science, machine_learning, other',
      reasoning: 'one sentence explaining the choice'
    }
  ) AS second_pass
FROM
  my_db.role_family_triage
WHERE
  disposition = 'escalate';
SQL Editor loading...
Login to connect

Merge the two passes into one column:

Merge the two passes
SELECT
  t.job_id,
  coalesce(e.second_pass.role_family, t.role_family) AS final_role_family,
  CASE
    WHEN e.job_id IS NULL THEN 'jev'
    ELSE 'llm'
  END AS decided_by
FROM
  my_db.role_family_triage t
  LEFT JOIN my_db.escalated_role_family e USING (job_id)
WHERE
  t.disposition IN ('auto', 'escalate');
SQL Editor loading...
Login to connect

The escalation pass is also where a reasoning field earns its place: prompt_jev gives you a number, not an explanation, so the rows a person reviews are the ones worth spending tokens on an explanation for. The second model's explanation is its own assessment, not a record of how Jev reached its decision.

Step 5: Queue the rest for a person​

Queue the rest for a person
SELECT
  job_id,
  title,
  role_family AS suggested_role_family,
  confidence
FROM
  my_db.role_family_triage
WHERE
  disposition = 'review'
ORDER BY
  confidence ASC;
SQL Editor loading...
Login to connect

Sorting ascending puts the least certain rows first, so a reviewer working top-down clears the highest-value items first.

Step 6: Retry the failures​

NULL results are requests that failed after retries or ran past the AI function timeout. Rerun them against the source table.

Retry the failures
UPDATE my_db.job_postings
SET
  role_family = prompt_jev (
    description,
    'Which kind of data role does this job posting describe?',
    choice := [
      'analytics',
      'data_engineering',
      'data_science',
      'machine_learning',
      'other'
    ]
  )
WHERE
  role_family IS NULL
  AND description IS NOT NULL;
SQL Editor loading...
Login to connect

Tuning the split​

  • A large review bucket points at the criteria, not the threshold. Overlapping options or a missing other option force the model to spread probability across several labels. Fix the criteria and rerun.
  • High confidence and wrong answers can reflect unclear criteria, a task the model handles poorly, or model error. Inspect the mistakes against Jev's known limitations rather than assuming a higher threshold will remove them.
  • Thresholds are per question, not per table. A role-family question and a seniority question on the same postings will sit at different cut-offs.
  • Recheck after any wording change. Editing the instructions or criteria shifts the distribution, so the old threshold no longer means what it meant.