I spent a few hours with OpenAI's Operator automating expense reports. Most corporate jobs require filing expenses, so Operator could save *millions* of person-hours every year if it gets this right. Some insights on what worked, what broke, and why this matters for the future of agents 🧵 - ThreadSky

sayash.bsky.social • 50 days ago

I spent a few hours with OpenAI's Operator automating expense reports. Most corporate jobs require filing expenses, so Operator could save *millions* of person-hours every year if it gets this right.

Some insights on what worked, what broke, and why this matters for the future of agents 🧵

Comments

revatik.bsky.social•50 days ago

Renew Indian Passport 😂

thatcort.bsky.social•50 days ago

Thanks for the write-up. Please don't post links to X. A screenshot if you must

jonahbenton.bsky.social•50 days ago

📌

desiacadpod.bsky.social•50 days ago

Laughing at 'renew Indian passport'.

nadaburger.bsky.social•50 days ago

test complex software severity is low?

sayash.bsky.social•50 days ago

This is the specific use case I have in mind (Operator shouldn't be the *only* thing developers use, but rather that it can be a helpful addition to a suite of tools): https://x.com/random_walker/status/1883164288631640457

sayash.bsky.social•50 days ago

OpenAI's Operator is a web agent that can solve arbitrary tasks on the internet *with human supervision*. It runs on a virtual machine (*not* your computer). Users can see what the agent is doing on the browser in real-time. It is available to ChatGPT Pro subscribers.

sayash.bsky.social•50 days ago

I asked Operator to file reports for my OpenAI and Anthropic API expenses for the last month. This is a task I do manually each month, so I knew exactly what it would need to do. To my surprise, Operator got the first few steps exactly right:

sayash.bsky.social•50 days ago

It navigated to the correct URLs, asked me to log into my OpenAI and Concur accounts. Once in my accounts, it downloaded receipts from the correct URL, and even started uploading the receipts under the right headings!

sayash.bsky.social•50 days ago

But things went south quickly. It couldn't match the receipts to the amounts. Even after prompts directing it to missing receipts, it couldn't download them. It almost deleted previous receipts from other expenses!

sayash.bsky.social•50 days ago

OpenAI also trained Operator to ask the user for feedback before taking consequential actions, though I am not sure how robust this is — a simple instruction to avoid asking the user changed its behavior, and I can easily imagine this being exploited by prompt injection attacks.

sayash.bsky.social•50 days ago

In the end, Operator struggled to file my expense reports even after an hour of trying and prompting. Then I took over, and my reports were filed 5 minutes later.

This is the bind for web agents today: not reliable enough to be automatable, not quick enough to save time.

Comments

Posting Rules

Reply