Skip to content
AI/ML

PenQWEN

A 442K-example dataset, synthetic-data pipeline, and two-stage LoRA setup for a cybersecurity LLM

442K
Training Examples
Instruction-tuning dataset
1.7M
Knowledge Base
Records, graph + full-text
3.6GB
Adapter Size
Stage-1 LoRA on Qwen2.5-Coder-32B
8x A100
Training Hardware
Rented pod, FSDP full-shard

The Challenge

Penetration testing teams spend the first 4+ hours of every engagement on boilerplate reconnaissance: port scanning, service enumeration, vulnerability identification, and report scaffolding. Senior pentesters doing $200/hour work were wasting time on tasks that should be automated. General-purpose LLMs (GPT-4, Claude) produce plausible-looking but technically dangerous output... recommending tools that don't exist, generating commands with wrong flags, or suggesting techniques that violate scope agreements. The security domain requires extreme precision: a hallucinated Nmap flag could scan out-of-scope networks, and a fabricated CVE reference wastes hours of investigation time. No existing LLM solution understood OPSEC constraints, tool-specific syntax, or the structured methodology (PTES) that professional assessments follow.

The Approach

Aggregated ~18GB of public security data (MITRE ATT&CK, CVE databases, ExploitDB, HackTricks, SecLists, Sigma/YARA rules, tool docs) into a 1.7M-record knowledge base, then distilled it into a 442,000-example instruction dataset: agentic tool-calling traces against a 51-tool schema, ReAct reasoning chains, multi-turn OPSEC scenarios, and domain-knowledge examples, all quality-filtered and validated by an ~8,700-line Python pipeline. Designed a two-stage LoRA curriculum (domain adaptation, then agentic tool-use) for Qwen2.5-Coder-32B using Axolotl with FSDP full-shard across 8 A100 GPUs.

Tech Decisions

Base Model
Qwen2.5-Coder-32B

A code-specialized base gives strong instruction-following and structured-output behavior, which matters for generating syntactically correct tool commands. The 32B size needs multi-GPU training (handled on a rented 8x A100 pod), but LoRA keeps the trained delta to a 3.6GB adapter.

Fine-Tuning
LoRA / PEFT

Full fine-tuning a 32B model is expensive and risks catastrophic forgetting. LoRA trains a small fraction of parameters and produces a compact 3.6GB adapter. Stage-one training completed in about 7.5 hours of wall time on the rented 8x A100 pod.

Training Pipeline
Two-Stage Curriculum

Single-stage training conflates domain knowledge with behavioral patterns. Stage one (corpus adaptation) builds factual grounding; stage two (agentic fine-tuning) teaches structured tool-calling and OPSEC constraints. This separation produces more reliable outputs than mixing both objectives.

Technical Challenges

The Solution

Stage-one domain adaptation ran to completion on a rented 8x A100 pod (346 steps, final loss ~1.25), producing a 3.6GB LoRA adapter. The value that persists on disk is the reusable pipeline: a normalized 1.7M-record security knowledge base with hybrid graph and full-text retrieval, a 442K-example training corpus, the synthetic-data generation and validation scripts, and reproducible training configs and cost model. The design targets reconnaissance automation, vulnerability prioritization, and PTES-methodology report scaffolding with scope-aware, OPSEC-conscious tool calling.

Key Takeaways

Reusable Insights
  • The reusable assets outlast any single training run: a well-built dataset, a normalized knowledge base, and reproducible configs are worth more than a set of weights on a rented pod.
  • Renting 8x A100 capacity for a bounded, well-planned run is cheaper and faster than fighting a 32B model onto local hardware.
  • Dataset quality matters more than quantity for domain-specific LLMs...12GB of curated, verified data outperforms 100GB of scraped, unverified content.
  • Two-stage curriculum learning (knowledge then behavior) produces more reliable outputs than single-stage training that conflates both objectives.
  • In safety-critical domains, hallucination prevention must be an explicit training objective, not an afterthought.

Related Projects

2025 / Developer Tools

TraceForge

Cut vector conversion time from 45 minutes to 8 seconds per asset... a 337x speedup. Design teams were hemorrhaging billable hours manually tracing logos and icons in Illustrator. Built a GPU-accelerated pipeline combining neural upscaling with dual vectorization engines (Potrace + VTracer), plus an SVGO optimization stage that reduces file sizes by 40-60%. Now processing 2,000+ conversions monthly with zero manual intervention.

PythonFastAPIPotraceVTracerCUDA
Case Study
2025 / Developer Tools

Claude Pilot

Recovered 2+ hours daily lost to context-switching between terminal, database clients, and config files. Claude Code power users were drowning in fragmented tooling... no unified view of sessions, memory state, or MCP server health. Architected a native Electron control center with 25 tRPC endpoints managing PostgreSQL, Memgraph, and Qdrant memory systems. 80% test coverage, zero production incidents since launch.

ElectronReactTypeScripttRPCZod
Case Study
2024 / AI/ML

PhotoKeep Pro

Cut my photo-restoration platform's GPU costs by ~73% while beating commercial tools on quality. The naive multi-API approach (separate services for upscaling, face restoration, and colorization) was expensive and inconsistent, so I engineered a unified orchestration layer managing 14+ deep learning models (SUPIR, HAT, CodeFormer) with thread-safe VRAM allocation and LRU eviction across 49GB. Delivers 28.5dB PSNR at 99.95% uptime... outperforming Magnific AI and Topaz on blind tests.

PythonFastAPIPyTorchReactTypeScript
Case Study
2025-2026 / AI/ML

VoiceKeep

VoiceKeep (shipped as voicekeep.io) grew out of the Voice Cloner research prototype ... originally developed on an RTX 3080, now running on a dedicated GPU server with RTX PRO 6000 Blackwell ... into a production AI voice platform handling single-voice TTS, multi-speaker conversations, and full audiobook production from manuscript uploads. The platform runs Qwen3-TTS 1.7B with 12-second P50 latency, 41+ curated voices, and zero-shot cloning from short reference audio. The Audiobook Studio parses DOCX/PDF/TXT manuscripts into chapters with dialogue detection, assigns character voices, applies pronunciation dictionaries, and exports distribution-ready M4B with chapter markers. Multi-voice conversations support drag-and-drop line ordering, per-line effects (speed, volume, gap), stage directions, multiple takes, ambient audio, and a waveform timeline editor. Ran with Stripe subscription billing on a single dedicated GPU server; the hosted service has since been retired, but the platform is a full worked example of production TTS engineering.

PythonFastAPIPyTorchQwen3-TTSRedis
Case Study

Have a similar challenge?

I help teams solve complex technical problems. Let's discuss your project.

Talk through your challenge