prompt_jev() now works on images

Try it in one query

Hugging Face hosts lots of public image datasets as Parquet, with each image stored as bytes. Read them with hf:// and hand the bytes straight to Jev:

Copy code
SELECT image.path AS file,
       prompt_jev([{name: 'dish', image: image.bytes}], 'Is `dish` a dessert?') AS is_dessert
FROM 'hf://datasets/ethz/food101/data/validation-00000-of-00003.parquet'
LIMIT 5;

Each row gets a probability between 0 and 1. These five photos are beignets, so they score high.

How fast, how accurate, how cheap?

We used the same dataset: Food-101, labeled photos of 101 dishes. One question per photo, "Which dish is shown?", with all 101 dishes as the allowed answers, asked to Jev, Gemini, GPT and Claude on the same 1,010 photos (10 per dish).

We can score accuracy here because Food-101 has the right answer for every photo. On your own data you won't have that, so the real question is which answers you can trust.

The big difference between Jev and a standard LLM is the confidence score. Every answer comes with one, and on this benchmark it reflects how often Jev is actually right. If you keep only the answers at 0.8 or more (WHERE jev.confidence >= 0.8), Jev is right 97.7% of the time, on 80% of the photos.

This of course begs the question - what if we combine Jev with another model to a cheap first pass and then a secondary cleanup loop?

Notice how we can achieve big gains in accuracy for relatively small increases cost compared to using a System Two LLM for the entire job.

Getting your images in

If your images sit in S3, one query loads them into a BLOB column (with an S3 secret in place):

Copy code
CREATE TABLE images AS
SELECT filename AS path, content AS image
FROM read_blob('s3://my-bucket/photos/*.jpg');

Jev needs each side of an image between 280 and 1,568 pixels. Bigger images need a resize first: a Flight (Python that runs on MotherDuck compute) with Pillow does it. The docs have a ready-to-use example that reads from S3, shrinks anything too big and writes the result back as a BLOB.

Or skip the code and ask the MotherDuck MCP server from Claude or your agent of choice:

Load the JPEGs under s3://my-bucket/photos/ into a new table photos.main.images, one row per file with the image as a BLOB. Resize anything longer than 1568px on its longest side, skip images under 280px on their shortest side, and run it as a MotherDuck Flight. Tell me how many images landed and how many were skipped.

Five centuries of art in one GROUP BY

We took the Metropolitan Museum of Art's open access collection, half a million CC0 catalog records, and kept every public-domain painting, print, drawing and photograph with an image: about 77k works, loaded as 600px BLOBs. Then we asked 14 questions about every image in one SQL query. Here are 4 of them:

Copy code
SELECT object_id,
       prompt_jev(
         [{name: 'art', image: image}],
         questions := {
           smile: {type: 'choice',
                   instructions: 'If a human face is clearly visible in `art`, what is the expression of the most prominent face?',
                   criteria: ['no_clear_face', 'neutral_or_serious', 'closed_mouth_smile', 'smile_showing_teeth', 'laughing']},
           dog: {type: 'noul', instructions: 'Is there a dog anywhere in `art`?'},
           dog_box: {type: 'box', instructions: 'the dog in `art`'},
           skull: {type: 'noul', instructions: 'Does `art` show a human skull or skeleton?'}
           -- plus 10 more: main subject, weather, sunset, drink, food, cat, ...
         }) AS jev
FROM images;

Once the answers are stored as columns, they join with the Met's catalog like any other data. When did people start smiling in art? One GROUP BY:

Copy code
SELECT (floor(o.begin_year / 50) * 50)::INT AS half_century,
       round(100 * count(*) FILTER (WHERE f.smile IN ('closed_mouth_smile', 'smile_showing_teeth', 'laughing'))
                 / count(*) FILTER (WHERE f.smile <> 'no_clear_face')) AS pct_smiling
FROM features f  -- Jev's answers, one column per question
JOIN objects o USING (object_id)
WHERE o.begin_year BETWEEN 1400 AND 1949
  AND o.end_year - o.begin_year <= 25  -- only works the Met dates within 25 years
  AND NOT f.rejected
GROUP BY ALL
ORDER BY half_century;

Almost nobody smiles before 1600! Many reasons for that, but that's for another blog.

Explore the full dive below, or open it in the dive gallery:

Get started

Images are now just another column in MotherDuck. Load your product photos, scans, screenshots or camera-trap pictures as BLOBs, ask your questions, and join the answers with the rest of your data.

Everything you need is in the prompt_jev() docs.