From a product photo to structured candidates

• They cannot name it. A component, a fitting, a fabric, a plant pot. The shopper knows exactly what they want and cannot type it.
• They saw it somewhere. A screenshot from social media, a photo in a friend’s kitchen, a still from a video.
• They have the physical object. A replacement filter, a cartridge, a bulb. The product is in their hand and the model number has worn off the label.
• They want something similar but different. “Like this, but cheaper” or “like this, in blue” — a visual starting point plus a constraint.

Notice that only the second of these is really about visual similarity. The others are identification problems where the image is the input format, not the goal. That distinction shapes everything downstream: if your interface stops at “here are things that look alike”, you have answered a question only one of these four shoppers asked.

From image input to candidate retrieval

The simlir image endpoint takes a hosted HTTPS image URL and returns products in the same structured shape as text search, with a visual_similarity_score added. It costs 3 credits, and market is required.

Why a hosted URL rather than raw bytes. It keeps the request text-only, which matters in an agent workflow: with the MCP tool simlir_search_products_by_image, the model does not need to inspect the image itself before deciding to call the tool. It passes a URL along. That also means your application controls image hosting, retention and privacy — you decide what is stored, for how long, and under what policy.

Practically, this puts one step in front of your search call: get the shopper’s photo to a hosted HTTPS URL first. For a mobile app that usually means uploading to your own storage and passing the resulting URL. Plan for the upload latency in your interface, because it happens before any search work begins.

You can narrow the search at the same time. category, brand, min_price and max_price all apply to image search, which is how you serve the “like this, but under £50” case — visual input for the shape, filters for the constraint.

Confidence and ambiguity

Image search is inherently less certain than text search, and pretending otherwise is the main way these features lose trust. Three properties of the response deserve respect:

Beyond the response itself, some ambiguity is inherent to photographs and no scoring model removes it:

The last row is the one teams forget. An empty image result often means “not in this market’s catalogue”, not “this product does not exist”. Those are very different messages to a shopper.

Structured fields after the match

This is the point of the whole exercise. Once a photo has resolved to candidates, the shopper gets everything text search would have given them:

Which means the follow-up questions all become answerable without another round of image work:

• “Is it cheaper anywhere else?” — the record carries a dated price snapshot with the retailer it was seen at.
• “What size is it?” — spec, dimensions and weight, with units.
• “Is it any good?” — review_score with its review_count.
• “Where do I buy it?” — links.retailer, which may be null when there is no verified retailer product URL.
• “Is this definitely the right one?” — gtin and model_number, which the shopper can check against the object in their hand.

That last one is underrated. When a shopper is holding the product, showing the identifier lets them verify the match themselves, which is far more convincing than any confidence score you could display.

The visual object is also useful in the interface: a short summary of what the hero image shows, labels, any clearly visible detected text, and dominant colours. It is derived from the product’s own hero image and is safe to show directly or pass to an agent. It is null when a row has not been visually enriched, so give that case a rendering.

Market and retailer context

Visual matching is global; buying is local. A product that looks identical in a photograph may be sold under a different model designation, in a different pack size, at a different price, in another country.

simlir requires market on image search and echoes it back on every product. Compare the value you sent against the value returned, and surface the market in your interface. “3 matches in the UK catalogue” is a materially more honest heading than “3 matches”.

Interface patterns for multiple candidates

Text search can defensibly show one strong answer. Image search rarely should.

• Show a small set, not one result. Three to five candidates with visible distinguishing attributes lets the shopper resolve the ambiguity in a second, using knowledge you do not have.
• Lead with what distinguishes them. If two candidates differ by capacity, put capacity on the card. A grid of near-identical photographs is the least useful way to present a visual match.
• Show the shopper’s own image alongside. It makes comparison immediate and makes a wrong match obvious rather than authoritative.
• Offer refinement, not just rejection. “None of these” should lead somewhere: a text query, a category filter, a prompt for a clearer photo.
• Say what was searched. Market, and any filters you applied on the shopper’s behalf.
• Design the empty state first. It will be seen more often than in text search, and it is where trust is won or lost.

Testing false matches

Standard relevance testing will not surface the failures that matter here, because it uses clean catalogue images. Build an evaluation set from photographs that look like the ones your users will actually send.

• Real-world photosHeld in hand, poor lighting, awkward angle, partial occlusion, cluttered background.
• ScreenshotsCropped from social media and video, with overlaid text and compression artefacts.
• Near-identical productsSame brand, different variant. The hardest and most commercially damaging case.
• Products outside the catalogueConfirm the system degrades honestly rather than returning a confident wrong answer.
• Products outside the marketCheck the message distinguishes “not here” from “does not exist”.
• Non-product imagesPeople, scenery, documents. Confirm you return nothing rather than something.
• Empty and short responsesVerify the interface handles a meta.count of zero without breaking.

Score two things separately: how often the correct product appears anywhere in the returned set, and how often it is ranked first. The gap between those numbers tells you whether to invest in ranking or in presentation. If recall is high but top-one accuracy is low, showing five candidates solves your problem far more cheaply than any model change.

Frequently asked questions