Agent Systems: Goal Setting, Prioritization, and Termination

Problem

Autonomous execution can drift from the user’s objective, spend resources on low-value actions, or continue after meaningful progress has stopped.

Intent and structure

goal → milestones → prioritized action → progress check → continue, redirect, or stop

A goal needs a success criterion, scope, budget, priority, dependencies, and termination policy. Monitor artifacts and state transitions rather than conversational confidence.

Termination conditions

Stop when the goal is achieved, a quality threshold is met, the budget is exhausted, no progress occurs after bounded attempts, or a human interrupts. Distinguish semantic completion (“the agent says done”) from verified completion (“the required artifact passed its checks”).

Forces and failure modes

Aggressive stopping saves cost but can miss a solution; permissive stopping can run indefinitely. Make the stopping decision observable and allow escalation or replanning when the goal changes.

Figure

Figure: goals become operational through priorities, bounded actions, progress checks, and termination.