Papers — Animesh Alang

Preprint, not peer reviewed

Capability Manifests Are Not Enough: Parameter Scopes for Default-Deny Tool Brokers

Animesh Alang, Adithya Pradeep, Paras Gandhi, Giwin Vincent Edwin Omesh · University of California, Santa Cruz

Tool-using agents fail open when a planner invents an action name, or when an allowed tool is called with a malicious path, destination, or command. ClawForge— Cloud Track winner at NVIDIA Hack-a-Claw (UCSC, May 2026)—routes every tool call through an ActionEnvelope checked against an AgentCapabilityManifest with default deny. We ask what that broker actually stops. On an author-constructed suite of 151 tool envelopes, adapted from InjecAgent and AgentDojo attack categories to ClawForge’s SentinelClaw tools, the shipped action-name broker blocks all out-of-manifest and disabled-tool calls and never auto-executes high-risk tools, but it auto-allows 26.3% of attack cases—almost all of them parameter abuse on tools the manifest marks allowed (unsafe-allow 0.857 on that category). Adding parameter scopes (path prefixes, forbid keys, command deny-patterns, destination allowlists) cuts unsafe-allow to 0 on the design suite (McNemar vs. action-only, p=1.9×10⁻⁹ ) with no false denies on benign reads/writes. A held-out residual set written after scopes were frozen still sees 30% unsafe-allow (symlink-style paths, unexpected side-channel keys). A static deny-list and a disclosed LLM-judge proxy both underperform and, for the proxy, flip under jailbreak framing the broker ignores. All trajectories are synthetic; we do not claim live prompt-injection robustness of a planner.

Data disclosure: All test trajectories (151 tool envelopes and a held-out residual set) are synthetic and author-constructed; no live prompt-injection robustness is claimed.

PDF

Preprint, not peer reviewed

One Database, Three Retrievers: When Hybrid and Graph Help a Local-Knowledge Agent—and When They Hurt

Animesh Alang · University of California, Santa Cruz

Local questions mix place type, neighborhood, live hours, and “places like X.” Regular is a deployed SMS/iMessage agent for Santa Cruz County whose retrieval stack lives in one PostgreSQL 17 instance: exact dense search (pgContext) over local bge-small embeddings, multi-field full-text lists fused by reciprocal rank fusion (RRF), and a property graph (pgGraph) with schema and embedding-derived edges. We ask which of these arms help on compositional local queries. On 366 simulated queries across four shapes, equal-weight hybrid RRF hurts similarity-style asks (Recall@5 0.583 → 0.438 on similar_open) but helps type and award asks; upweighting the dense list (weight 4 vs. 1 per lexical list) recovers much of the loss and significantly beats equal hybrid overall. Adding graph traversal on top of dense search (dense+graph) is the best overall configuration (Hit@1 0.475, MRR 0.585), with a significant Hit@1 gain over dense-only on bare “somewhere like X” queries (+0.092, 95% CI [+0.008, +0.175]). We report a failure taxonomy with paraphrased examples, recorded end-to-end behavioral results (we do not re-run the agent harness because it writes throwaway users), and position the design against hybrid fusion analysis and LLM-extracted GraphRAG.

Data disclosure: All 366 evaluation queries are simulated, and relevance labels come from database rules rather than human judgment.

PDF

Preprint, not peer reviewed

When Personalization Moves Latent Mastery but Not the Test: An LLM-Simulated Study of Knowledge Tracing and Study-Method Selection

Animesh Alang · University of California, Santa Cruz

All learners in this paper are LLM-simulated; nothing here is evidence about real students. I ask a narrow question about AI tutors of the kind I am building in dino.study: if an LLM tutor keeps a per-student mastery estimate and picks a study method from that estimate, does anything measurable improve over a non-personalized LLM tutor? I run three arms (N=30 simulated students each) on a fixed introductory-probability item bank: (i) generic explain-then-practice, (ii) knowledge tracing with adaptive pacing, (iii) tracing plus method selection among retrieval practice, spaced repetition, worked examples, and Socratic questioning. On 8-item pre/post/delayed tests, normalized gains do not differ (all near zero; permutation p>0.25). On the simulator’s latent mastery, they do: total mastery gain is 0.070 (control), 0.156 (trace), 0.229 (trace+method), all pairwise p<0.001. LLM mastery estimates track latent state (r=0.85) but next-item AUC is chance (≈ 0.50); BKT is no better. A sticky method policy locks onto one method in 90% of students; an anti-lock-in fix removes lock-in without changing latent gains. Equalizing method gain constants and swapping in random/oracle methods barely moves outcomes. The usable claim is therefore modest and process-level: under this disclosed simulator, tracing changes latent mastery; short MC tests cannot see it; method selection adds a further latent bump that is partly built into the gain table.

Data disclosure: All learners are LLM-simulated; results are not evidence about real students.

PDF

Preprint, not peer reviewed

Generate-then-Verify Is Not Enough: Routing Beats Verifiers on a Simulated Receptionist Suite

Animesh Alang · University of California, Santa Cruz

Closed-knowledge AI receptionists still invent prices and comply with jailbreaks. We reimplement the tiffani.app stack—dense retrieval over owner docs, a separate generate-then-verify pass, and deterministic pre-model routing for handoff and danger—and evaluate it on five fully simulated businesses and 180 simulated customer queries (no production traffic). Against retrieval-only, single-pass RAG, and same-model self-check baselines on Llama-3.1-8B-Instruct-FP8, a verifier does not significantly cut overall hallucination vs single-pass (4.4%→3.3%, McNemar p=0.69 on seed 42; p=0.25 on seed 43). The clear win is routing: correctness on handoff+danger jumps from 28% to 89% (p=0.001). Verify does beat self-check on correctness (80.6%→86.1%, p=0.021) at lower latency. We report category breakdowns, error types, a Claude agreement sample (κ=0.77 on hallucination), and remaining gaps.

Data disclosure: All five businesses and all 180 customer queries are simulated; no production tiffani.app traffic or real users are included.

PDF