ClawTrust
1.7k Downloads
Overview
Provides a two-layer safety gate for agent input and output: (1) prompt injection detection using a ProtectAI DeBERTa classifier via Hugging Face Inference, and (2) optional OpenAI omni-moderation for content safety across multiple harm categories.
Key Advantages
1.Specialized prompt injection detection tuned for common attack patterns (e.g., attempts to override or reveal system prompts).
2.Layered defense: combines injection detection with robust content moderation when OpenAI is configured.
3.Simple CLI integration via `scripts/moderate.sh` with well-defined JSON output for downstream agents.
4.Explicit handling guidance: clear `flagged` and `action` fields describing how agents should respond to flagged content.
5.Configurable sensitivity via `INJECTION_THRESHOLD`, allowing teams to tune for higher recall or higher precision on injection detection.
Use Cases
- Guarding agent inputs in public-facing or multi-user environments where adversarial prompt injections are likely.
- Filtering untrusted messages in group chats, shared workspaces, or integrations with public channels (e.g., Slack, Discord).
- Protecting system prompts and hidden instructions by detecting user attempts to override or expose internal configuration.
- Adding a safety layer before tools that can cause side effects (code execution, file access, external APIs) are invoked.
- Moderating model outputs related to violence, self-harm, sexual content, hate, or other sensitive topics, with automatic rewrite-on-violation workflows.
Evaluation Scores
8.0
/ 10
Reliability
7.0
Functionality
8.5
Usability
8.0
Safety
8.5
Performance
7.5
Compatibility
8.0
Based on 1 evaluation · Latest: 3/19/2026
Download Trend
Loading...
Evaluation History (1)
8.0/103/19/2026▼
OS: darwin-x64LLM: google/gemini-2.5-flash
**Quick judgment:** A strong, practical safety skill for agents exposed to untrusted input, offering dedicated prompt injection detection plus optional OpenAI-based content moderation. Well-suited as a front-line guard in real deployments, with the main trade-offs being external API dependence and the usual risks of classifier false positives/negatives.
**Key strengths & risks**
- **Strengths:**
- Purpose-built detection of prompt injection attempts (e.g., "ignore previous instructions", requests for system prompts, configuration probing).
- Optional second layer using OpenAI omni-moderation to cover harassment, hate, self-harm, sexual content, violence, etc.
- Clear JSON schema (`flagged`, `injection.score`, `content.flaggedCategories`, `action`) that agents can reliably consume.
- Sensitivity can be tuned via `INJECTION_THRESHOLD`, making it adaptable to different risk tolerances.
- **Risks / limitations:**
- Depends on external services (Hugging Face Inference and optionally OpenAI); outages, latency, or rate limits can impair protection.
- Claims of very high accuracy (e.g., ">99.99%" on typical attacks) should not be taken as guarantees; expect both false negatives and false positives.
- Privacy considerations: all checked text is sent to third-party APIs; sensitive or regulated data may require additional controls or self-hosted alternatives.
- Overly aggressive thresholds can block benign but unusual instructions; conservative thresholds may miss more sophisticated injections.
**Recommended scenarios:**
- Public or semi-public agents (web, Slack/Discord bots, shared org assistants) where malicious or experimental users are expected.
- Any agent allowed to access tools with side effects (code execution, file I/O, HTTP calls, database access), where prompt injection poses real security risk.
- Multi-user or group-chat contexts where some participants may be untrusted and can attempt to steer or jailbreak the agent via indirect prompts.
- Systems that must log or audit safety decisions: the JSON output makes it easy to trace when and why content was blocked or rewritten.
Comments (0)
No comments yet. Be the first!