Small models for security playbooks
Building a benchmark for security playbook generation, with the goal of training small language models to write workflows that meet task requirements and policies.
- 01 IN DEVELOPMENT
Build the benchmark
Security scenarios, source documents, and checks for correct workflow behavior.
- 02 PLANNED
Train small models
Learn to generate playbooks using separate training data and reinforcement learning.
- 03 PLANNED
Test generalization
Compare with baselines on held-out scenario families kept out of training.
A security playbook is an automated response workflow: it gathers information, makes decisions, and takes actions. The benchmark asks a model to generate that workflow from a short security objective and case facts, using available documentation to find the procedures, tools, and policies it needs.
I am constructing scenarios grounded in existing security playbooks and documentation, together with criteria for judging generated workflows. Each answer specifies the situation, what must happen, and what must not happen. Different workflows can be correct; copying one reference graph is not the goal.
For example, if approval is required before blocking an address, including both actions is not enough. Blocking must depend on approval and remain unreachable when approval is denied. The evaluator must detect missing work, incorrect branches, and bypassed prerequisites—not just valid output formatting.
The current phase is benchmark and evaluator development: reviewing source evidence, checking that tasks are answerable, and finding scoring gaps that let incorrect workflows pass. Evaluation currently covers platform-neutral playbook outlines; running a complete playbook against real service integrations is a separate step. Small-model generation training, including reinforcement learning, is the next research objective.