Building a computer-use agent supercharged by RAG

All of the code is on GitHub.

The problem: the question is on screen, not in the prompt

Let’s say you want an agent to file a quarterly compliance report in a web application. Page 2 asks which NCPDP field identifies a partial fill for a Schedule II drug, and the date by which compliance is required. A model answering those from memory will likely hallucinate.

The standard answer is retrieval: look the facts up, put the passages in the prompt, answer out of them. That assumes you know the question when the run starts. Here you don’t — the questions are behind a Continue button, and the agent only finds them by operating the software. So retrieval stops being a step that happens before the run and becomes a tool the model calls during it: it writes the query from the page in front of it, and what comes back carries the filename and page number it came from.

This project is about building this mechanism in a repeatable, consistent way.

What we’re building

A realistic demo form. A 3-step FastAPI portal with server-side validation: identifying information, 4 regulatory attestations, then review and submit.

A corpus of federal PDFs. 3 US publications on HIPAA Administrative Simplification, embedded on CPU with BAAI/bge-small-en-v1.5 into 384 dimensions, in Postgres with pgvector.

Two tools, one request. computer drives mouse and keyboard against a virtual display; search_regulations returns the nearest passages with their filename and page number. No router decides between them.

A grader that never talks to the agent. deskwork verify reads the row that landed in Postgres and checks it purely against the source documents.

It runs from a compose file:

git clone https://github.com/ashnkumar/deskwork && cd deskwork
cp .env.example .env          # then put your key in ANTHROPIC_API_KEY

docker compose up -d --build
docker compose exec agent deskwork ingest   # 173 chunks from 3 PDFs
docker compose exec agent deskwork run      # 22 or 23 steps, about 2 minutes
docker compose exec agent deskwork verify   # did it file a correct report?

While run is going, open http://localhost:6080/vnc.html and watch.

The computer tool runs on our side, not Anthropic’s

Computer use isn’t a hosted environment. The API returns actions — {"action": "left_click", "coordinate": [512, 400]} — and your own code performs them. The docs are explicit: “Your application must explicitly run the computer use tool; Claude cannot run it directly. You are responsible for implementing the screenshot capture, mouse movements, keyboard inputs, and other actions…” Elsewhere on the same page: “Computer use is a client-side tool.”

Here that means an Xvfb display at 1024×768 with Firefox on it, scrot for screenshots and xdotool for input, inside the agent container. Every action becomes a subprocess we spawn.

Lesson 1: Retrieval before the full loop can result in lower performance

The first thing we built is the usual RAG pipeline: embed the request, fetch the top passages, insert them in the prompt, let the run proceed with the evidence already in context.

But that won’t work here — we don’t know the questions yet, because they’re on pages the agent hasn’t reached. Running the retrieval based on the first thing it sees means it retrieves the same generic passages on every run, whatever the form turns out to ask.

The agent reaches the attestations at step 9, and at step 10 it says:

Four regulatory questions. Let me look each up in the corpus rather than relying on memory.

then passes the question it read off the screen a step earlier to search_regulations, almost verbatim — we couldn’t have done this earlier in the run.

So retrieval is declared as an ordinary tool and goes out alongside the computer tool:

        response = client.beta.messages.create(
            model=config.model,
            max_tokens=MAX_TOKENS,
            betas=[BETA_FLAG],
            system=system_prompt(config.display_width, config.display_height),
            tools=[tool.to_params() for tool in by_name.values()],
            thinking={"type": "adaptive"},
            output_config={"effort": config.effort},
            messages=messages,
        )

In an ordinary RAG app your code decides when to retrieve. Here it can’t: knowing a turn needs a lookup means knowing what’s on screen, and the screen is whatever the form did last. So there’s no routing logic in agent.py at all — both tools go out on every request, and the model decides which one to use.

2 things in that trace belong on the model’s side: the query text, which we can’t predict ahead of time and that the model needs to pull off the page it sees, and the number of searches — 4 questions, 2 calls, in every one of 11 graded runs, with nothing telling it to batch them.

Lesson 2: Ask for it in the prompt, then check it deterministically

Simply having the tool doesn’t mean the model will predictably use it. The first layer is a soft one — a system prompt that draws the line in as few words as possible. Its first rule, trimmed:

**1. Never state a regulatory fact from memory.** Rule identifiers, field names and \
numbers, effective dates, compliance deadlines, CFR citations — every one of these comes \
from `search_regulations` and nowhere else.

The tool description repeats it and so does the task prompt. But an instruction is a request, not a guarantee: nothing in the loop checks that a string on its way into a form came from a retrieved passage, and the demo’s questions are answerable from training data anyway — so a run that skipped the search and typed a remembered answer would look identical to one that did the work.

That’s why the second layer has no relationship to the agent at all:

docker compose exec agent deskwork verify --report-id QI-2025-014

verify reads the submissions row out of Postgres and grades it against the corpus to make sure each citation refers to an actual document and page in the database, and that the page contains the value given in the answer. Importantly, it never sees the transcript, and exits non-zero when the filed report is wrong.

This was tricky to tune. In the first attempt it performed 3 substring tests and looked for .pdf somewhere in the citation field, which meant a report citing invented.pdf passed, and so did an answer that stated the correct date under a negation. Checking the citation against the corpus is what closed that, and it still doesn’t amount to fabrication being prevented — nothing here knows what a sentence means. What it rules out is an answer attributed to a page that can’t support it.

Here is what we measured: 11 runs, 11 correct reports. That is not the same as a 100% success rate. 11 trials can’t distinguish a reliable agent from a lucky one — the exact one-sided 95% lower bound on 11-for-11 is 76%, so the true rate could be roughly 1 run in 4 failing and this sample wouldn’t know.

Lesson 3: ~1K character chunks yielded just 60% accuracy

A grader only helps if the tool could have found the answer at all, and the first version could not.

We started at 1100-character chunks with 150 of overlap so the chunks are viewable to us on a single page if they’re printed. Against the 5 questions the demo depends on, it retrieved the right passage for 3. Nothing errored. The tool returned 4 plausible passages every time, the agent typed a confident answer out of the best of them, and 2 of 5 were wrong.

The fix was smaller windows: 500 characters with 90 of overlap allowed it to answer all 5. The reason is specific to this corpus — Federal Register PDFs are multi-column, so an 1100-character window pulls in a column boundary, a heading, and half an unrelated paragraph, and the sentence that answers the question competes with several hundred characters that have nothing to do with it.

The same PDFs cost us a second bug first, one layer down — text extracted from a two-column government PDF arrives hard-wrapped, with words broken across line endings:

# Federal Register PDFs are multi-column and extract with hard-wrapped lines and hyphenated
# line breaks. Left as-is, "adminis-\ntrative" never matches a search for "administrative".
_HYPHEN_BREAK = re.compile(r"(\w)-\n(\w)")
_SOFT_WRAP = re.compile(r"(?<![\n.:;!?])\n(?![\n•])")

We wouldn’t have known this since the ingest succeeds, the row count looks right, the index builds, and search returns 4 passages with plausible cosine distances. The only thing that surfaces either is an evaluation set with known answers — and where you put that eval set decides whether it ever runs. In a notebook somebody opens when retrieval feels wrong, it only runs once you already suspect a problem. In the ordinary test tier it runs on every commit, and fails the build the day the number moves.

The shipped numbers are a measurement, so a test re-derives them just to be sure.

    tuned = score(ingest.CHUNK_CHARS, ingest.CHUNK_OVERLAP)
    assert tuned == len(CORPUS_EVAL), f"configured chunk size answers only {tuned}/5"
    baseline = score(1100, 150)
    assert baseline == 3, f"the 1100-char baseline now answers {baseline}/5, not the 3/5 documented"

Pinning the baseline at 3 rather than asserting the tuned size is the difference between a number the prose quotes and a number the suite owns.

The oracle took a second attempt too, and the first version was the more instructive mistake:

# The filename is part of the oracle deliberately. A bare substring check passes when
# "460" turns up in an unrelated number, or when the generic word "electronic" appears
# anywhere at all — so it can go green without the answer-bearing passage ever being
# retrieved. Requiring the right document makes the assertion mean what its name claims.

Testing retrieval purely with regex would pass while retrieval quietly degrades, which makes it worse than no test: it reports success without measuring anything.

Lesson 4: The screenshots are the context window

In a computer-use loop the screenshots are the context window. Every action returns one — the alternative is the model spending a turn asking for a screenshot after each click — and 20-odd steps of PNGs is most of what a run costs. An image from 15 turns ago is worth nothing; the screen has moved on.

We tried just dropping the old images and found that this breaks because thinking blocks live in the same list, each carries a signature, and the docs are clear: “Pass every thinking block back to the API complete and unmodified,” and “Modified thinking blocks are rejected with a 400 error.” It fails closed — no warning, no fallback, just a 400 on the next request. Which makes a general-purpose “clean up the message list” pass a time bomb: it works until the run is long enough to reach a thinking block.

What runs now touches exactly one kind of content. This lives in our loop, not in the prompt — once per turn, before we build the request, prune_images() rewrites the message list we’re about to send. The model has no say in it and never sees it happen. Its docstring says why:

    """Drop all but the `keep` most recent screenshots, in place.

    Screenshots dominate token spend in a computer-use loop, and an image from fifteen
    turns ago is nearly worthless — the screen has moved on. Removing them keeps a long run
    affordable. The image block is replaced with a short text note rather than deleted, so
    the transcript still reads coherently and no tool_result is left with empty content.

    Only tool_result content is touched. Thinking blocks carry signatures and must be
    echoed back byte-identical, so they are never rewritten.
    """

But before you write your own: (1) Claude Opus 4.5 and later Opus models keep all prior thinking in context, where it’s billed as input tokens like any other history. Earlier models keep only the last turn and strip the rest for you. So make sure to know the nuances of the models you’re working with. (2) The API will prune the screenshots for you too — context editing, the clear_tool_uses_20250919 strategy behind the context-management-2025-06-27 beta, clears old tool results server-side as one request parameter. (3) The same goes for the harness; the SDK ships a Tool Runner that owns the request → execute → repeat cycle, the result formatting and the iteration limit, and this is enough for most use cases. In our case, we had to hand-write this because our demo rewrites image blocks between turns — reaching back into messages we’ve already sent and swapping the screenshot out — and prints every step. That’s a reason this repo keeps a loop but start with the Tool Runner and iterate from there.

We had to hand-write a few lines because context editing clears a whole tool result and substitutes a placeholder, whereas prune_images keeps the tool_result block and swaps only the image inside it:

    for block, position in image_positions[:removable]:
        block["content"][position] = {
            "type": "text",
            "text": "[screenshot removed to save context]",
        }
    return removable

The transcript then still shows that a screenshot was taken at that step and what came back, which is what the model reasons about later. For a production loop, use context editing — it’s one request parameter and it runs server-side. The ~30 lines here buy a transcript that reads end to end, which is the point of a repository someone is meant to read.

One more rule from the same neighborhood, because it outranks your error handling: every tool_use must come back with a tool_result.

                try:
                    result = tool(**payload)
                except Exception as exc:
                    # Every tool_use block must get a tool_result or the conversation is
                    # malformed and the next request is rejected. A tool that raises —
                    # a dropped Postgres connection, an embedding failure — would
                    # otherwise take the whole run down with it. Report and continue;
                    # the model can retry or route around it.
                    result = ToolResult.error(f"{type(exc).__name__}: {exc}")

A raising tool isn’t an error path in an agent loop, it’s a protocol violation — and the request that fails is the one after it, which makes it unpleasant to diagnose from the traceback you get.

Lesson 5: Understand your screen resolution

All of that assumes the agent clicks where it meant to.

The implementation this was rebuilt from ran the agent against a Retina MacBook screen, reported a 1024×768 display to the API, and converted between the two with a scale_coordinates() helper. Every conversion in it is a place to be wrong by a scale factor, and one wrong conversion puts the click near the button instead of on it — what you see in the transcript is a model that looks bad at using a computer.

The API docs bring this up:

The API downscales oversized images before Claude sees them, and Claude returns coordinates for the image it sees, so relying on the server-side downscale leaves you without the scale factor you need to map those coordinates back to your screen.

Only images past a much larger threshold — “more than 8,000 px on a side” — are rejected outright. Below that, an oversized screenshot is silently resized and the coordinates come back in a resolution you never chose.

What we run instead is a display created at exactly the size we report, so no scaling exists in either direction, with the assumption asserted instead of assumed:

        image = Image.open(io.BytesIO(data))
        if image.size != (self.width, self.height):
            raise ComputerToolError(
                f"Display is {image.size[0]}x{image.size[1]} but the tool declares "
                f"{self.width}x{self.height}. Coordinates would not line up. "
                "Set DESKWORK_DISPLAY_WIDTH/HEIGHT to match the Xvfb geometry."
            )

1024×768 sits under every current model’s limit (the latest Anthropic models accept up to 2576 pixels on the long edge), so the downscale path is never reached. If your target really is a 4K desktop, you’ll definitely have to implement rescaling.

What this doesn’t solve

• 11 out of 11 is a small sample. 11 trials can’t distinguish a reliable agent from a lucky one. One task, one form, one machine.

• The demo’s questions ship with the demo, and they were written from the corpus. Every one is answerable, and one asks outright for a document and page number. That shaping fixes what is findable; it does nothing about when the agent learns what it’s being asked.

• The corpus is 3 documents — enough to make retrieval meaningful and a wrong answer detectable, not enough to say anything about retrieval at scale. It’s also trusted input: the API runs classifiers over computer-use screenshots and steers the model to ask for confirmation when one trips, and nothing equivalent runs over a passage from your own database.

Try it

git clone https://github.com/ashnkumar/deskwork && cd deskwork
cp .env.example .env          # then put your key in ANTHROPIC_API_KEY

docker compose up -d --build
docker compose exec agent deskwork ingest
docker compose exec agent deskwork run
docker compose exec agent deskwork verify

Source: https://github.com/ashnkumar/deskwork