Welcome to the Abundant Take-Home Project!

Your task is to create a new Terminal Bench task that challenges state-of-the-art AI models. You'll start with an existing example, modify it to increase difficulty, and ensure all tests pass.

• Study Terminal Bench thoroughly at tbench.ai
  • Understanding what makes a good benchmark task
  • Pay attention to difficulty levels and evaluation criteria
• Read our review guidelines

• Fork an existing task from /tasks
  • Choose a domain you're experienced with (e.g. fullstack, devops, data science, etc.)
  • Or if you prefer to start from scratch, here's a list of ideas
• Modify it to increase difficulty while maintaining clarity
• Update the solution and unit tests to match your changes

• Create a pull request with your new task
• Review the CI/CD checks carefully to ensure that
  • Your solution and unit tests are set up correctly by verifying that Oracle Solution passes and NOP Agent test fails
    • Note that not all sample tasks have those setup correctly, in which case you'll probably want to fix those first
  • Your tasks are sufficiently difficult so that at least 2 out of the 3 agent tests failed (see Run tests for modified tasks section of Test with Terminus (Claude | Gemini | GPT))
    • Each agent failed for the right reasons in agent.log, in the downloaded session artifact (see Upload session recordings section)
• IMPORTANT: In the PR description, make sure to include
  • Which task did you take inspiration from, if any?
    • How is your task different?
  • The results of each agent test
    • Why did the agent fail? Make sure it's not due to poor unit test setup
  • BONUS: What other variations could you create based on this task?
• Send PR back for review (do not merge the PR before the review is done)

Leverage your expertise: Take inspiration from your previous work and come up with tasks that resemble real-world engineering tasks.

Open-ended approach: Feel free to use any AI tools or resources to help you achieve the outcome. We're interested in seeing your problem-solving process!

Start simple: Begin with a task similar to existing examples, then gradually increase complexity.

Quality over quantity: One well-crafted, challenging task is better than multiple simple ones.

Don't hesitate to reach out if you have questions! We're here to support you throughout the process.

Contact: ke@abundant.ai for technical questions or repository access 🤝

We're excited to see what you create! This project will give you valuable insight into our work while helping us evaluate your technical skills and creativity.