AI
The Engagement Lifecycle Before Any Testing Starts
An engagement at Digital Boundary Group doesn’t begin when testing starts. It begins at scoping, where a questionnaire goes to the client to estimate effort and surface quirks. That leads to a requirements document for enterprise clients with specific delivery needs. Once a contract is signed, an intake doc collects test targets and credentials. A tester then runs an intake review checklist to validate it. Testing, reporting, review, and delivery follow. Every step generates its own document, all archived in a folder.
What the Legacy Tooling Actually Looked Like
Word with VBA macros and a SmartDocs plugin handles reporting. The macro parses findings to build summary tables and CSV exports, both requiring manual fixes. SmartDocs connects to a SharePoint list of over 400 template findings stored as Word attachments. Updating an OWASP ASVS reference means downloading every snippet and editing each one manually. Checklists govern every stage, but they’re general enough that experienced testers skip steps. Armstrong said he doesn’t follow his own checklists because he wrote most of them.
Rebuilding the Finding Structure
Armstrong reviewed publicly available application pentest reports before redesigning DBG’s finding structure. Reports vary substantially, partly because application security reporting grew from network and infrastructure templates with no standard for what to include. The old DBG finding used a catchall section mixing reproduction steps, impact analysis, and technical detail. The redesign splits this into a background section, discrete steps to reproduce, a risk assessment section capturing severity justification, and separate fields for CWE, ASVS, and OWASP Top 10 references. Each section is narrow enough that reviewers can find what they need without reading everything.
Evaluating and Choosing a Reporting Platform
DBG formed a task force and evaluated every reporting platform on the market, using a 12-question form to assess key features. Collaboration ranked highest: technical reviews are DBG’s primary mechanism for junior testers to get feedback from senior ones. The team chose Plextrac. It uses CKEditor for rich text, letting DBG store finding content as HTML independent of any specific vendor. That independence was deliberate. Migrating away from SmartDocs was painful enough that they wanted to avoid ever being locked in again.
Migrating 400 Findings into a Typed Data Model
The 400+ template findings lived in Word documents attached to a SharePoint list. A co-op named James wrote scripts to parse all that Word content, produce a CSV, and upload it to a Microsoft List. The List stores findings with proper typed fields, not attachment files. This lets DBG track metadata that wouldn’t fit in a Word document, like bug bounty eligibility and approximate payout for each finding type. A one-click CSV export feeds that content directly into Plextrac. DBG built a parallel engagement data model in the same platform to track clients, scoping data, and test results centrally.
Automating the Scoping Handoff
With scoping data stored centrally in Microsoft List, a Power Automate flow sends it to sales and the project management platform in one click. Both contacts receive the same structured information, removing the manual email step that previously required the tester to copy data from static documents. The low-code interface means anyone on the team can update the email template. Challenges include splitting attention between billable engagements and the rollout, and living in an in-between state where neither system is fully replaced. Armstrong plans to use tester surveys as the primary measure of whether the new processes are working.
Notable Quotes
the least breaking uh breaker talk uh Ryan Armstrong · ▶ Watch (0:09)
like oh no word that’s awful it was Ryan Armstrong · ▶ Watch (9:39)
I don’t comply with my own checklists Ryan Armstrong · ▶ Watch (18:54)
Key Takeaways
- Manual steps at every engagement stage create error and inconsistency, not just inefficiency.
- Storing template findings as typed fields rather than Word documents makes bulk updates and metadata tracking possible.
- Iterative migration while maintaining billable work means accepting an in-between state that slows adoption.
About the Speaker(s)
Ryan Armstrong is the Manager of Application Security Services at Digital Boundary Group (DBG). He joined DBG as an application penetration tester after completing his PhD in Biomedical Engineering at Western University in 2016, and has spent several years managing the application security team alongside active testing work.