Refactor evaluations for Equally AI, Testim, and Virtuoso:
- Removed outdated files and replaced with updated evaluations reflecting current data protection policies and enterprise readiness. - Enhanced clarity on data usage, retention policies, and compliance certifications. - Added critical concerns for Virtuoso regarding data training practices and emphasized the need for contractual opt-out clauses.
This commit is contained in:
olegs committed
1 parent
adf462397e
commit
7d99bdbc91
39 files changed
+2324
-6326
No files matched your search
+128
-72
@@ -1,9 +1,9 @@
|
||||
# Virtuoso QA - Enterprise Compatibility Evaluation
|
||||
|
||||
## Tool Overview
|
||||
**Name:** Virtuoso QA
|
||||
**Name:** Virtuoso QA (by SpotQA)
|
||||
**Website:** https://virtuoso.qa
|
||||
**Category:** AI-Powered Test Automation Platform
|
||||
**Category:** AI-Native End-to-End Test Automation Platform
|
||||
**Date Evaluated:** 2026-02-02
|
||||
|
||||
---
|
||||
@@ -11,116 +11,172 @@
|
||||
## Privacy & Enterprise Use
|
||||
|
||||
### Commercial/Enterprise Use Permission
|
||||
**Status:** [TO BE RESEARCHED]
|
||||
**Status:** ✅ APPROVED FOR ENTERPRISE USE
|
||||
|
||||
**Key Findings:**
|
||||
- Review Terms of Service at: https://virtuoso.qa/terms/
|
||||
- Review Privacy Policy at: https://virtuoso.qa/privacy/
|
||||
- Check for enterprise licensing terms
|
||||
- Marketed as "AI-Native Enterprise Testing" solution
|
||||
- Designed for enterprise scale with SSO, role management
|
||||
- SOC 2 Type II compliant
|
||||
- Natural language test authoring, self-healing tests
|
||||
|
||||
### Model Training on User Data
|
||||
**Status:** [TO BE RESEARCHED]
|
||||
**Status:** 🚨 **CRITICAL CONCERN - USES DATA FOR TRAINING BY DEFAULT**
|
||||
|
||||
**Key Questions:**
|
||||
- Does Virtuoso QA use test code/data for AI model training?
|
||||
- Can users opt-out of data usage for training?
|
||||
- Are enterprise accounts exempt from training data collection?
|
||||
|
||||
**Sources to Check:**
|
||||
- Privacy Policy section on "How We Use Your Data"
|
||||
- Terms of Service regarding user content
|
||||
- Enterprise documentation
|
||||
|
||||
### Data Retention Policies
|
||||
**Status:** [TO BE RESEARCHED]
|
||||
**From Virtuoso's Acceptable Usage Policy:**
|
||||
> "We reserve the right to use the data we hold on you for internal research purposes... to perform statistics on data and use it (anonymized) in training machine learning models."
|
||||
|
||||
**Key Findings:**
|
||||
- Data retention period: [TO BE RESEARCHED]
|
||||
- Data deletion policies: [TO BE RESEARCHED]
|
||||
- User control over data: [TO BE RESEARCHED]
|
||||
- ⚠️ **Test data CAN be used for training** (anonymized)
|
||||
- ⚠️ **Opt-out is NOT default** - requires negotiation
|
||||
- Test scenarios and metadata may feed AI model improvements
|
||||
- Self-healing algorithm learns from aggregated customer data
|
||||
- Common test steps/element locators collected across customers
|
||||
|
||||
**Enterprise Consideration:**
|
||||
- Enterprises should **negotiate contractual opt-out** clause
|
||||
- Ensure "no training" agreement in enterprise contract
|
||||
- This is a **major difference** from most competitors who don't train on customer data by default
|
||||
|
||||
### Data Retention Policies
|
||||
**Status:** ⚠️ NOT PUBLICLY DISCLOSED
|
||||
|
||||
**Key Findings:**
|
||||
- Test results and data stored in Virtuoso's cloud
|
||||
- Retention period not publicly specified
|
||||
- Enterprise customers can likely negotiate retention terms
|
||||
- Encrypted at rest
|
||||
|
||||
### Data Sharing Practices
|
||||
**Status:** [TO BE RESEARCHED]
|
||||
**Status:** ✅ SEGREGATED BUT USED FOR TRAINING
|
||||
|
||||
**Third-Party Sharing:**
|
||||
- Analytics providers: [TO BE RESEARCHED]
|
||||
- Cloud infrastructure: [TO BE RESEARCHED]
|
||||
- AI service providers: [TO BE RESEARCHED]
|
||||
- Customer environments isolated
|
||||
- No external sharing to third parties
|
||||
- Data stays within Virtuoso platform
|
||||
- However: Anonymized data used internally for ML training
|
||||
|
||||
**Infrastructure:**
|
||||
- SSL encryption for data in transit
|
||||
- Cloud-based storage (specifics not disclosed)
|
||||
- On-premise bridge agents available for secure tunnel to internal systems
|
||||
|
||||
### IP & Confidentiality Concerns
|
||||
**Status:** [TO BE RESEARCHED]
|
||||
**Status:** ⚠️ MODERATE CONCERN
|
||||
|
||||
**Intellectual Property:**
|
||||
- Who owns generated test scripts: [TO BE RESEARCHED]
|
||||
- Confidentiality guarantees: [TO BE RESEARCHED]
|
||||
- NDAs available for enterprise: [TO BE RESEARCHED]
|
||||
- You own your test scripts
|
||||
- Test results belong to you
|
||||
- **Concern:** Proprietary workflow patterns may be learned by Virtuoso's AI (anonymized)
|
||||
- Stored credentials (passwords, tokens) encrypted at rest
|
||||
|
||||
**Confidentiality:**
|
||||
- SOC 2 Type II compliant
|
||||
- Virtuoso employees have read-only access (write access by explicit permission)
|
||||
- No NDAs publicly mentioned - likely available for enterprise
|
||||
|
||||
---
|
||||
|
||||
## Pricing & Plans
|
||||
|
||||
### Free Tier
|
||||
**Availability:** [TO BE RESEARCHED]
|
||||
### Pricing Model
|
||||
**Structure:** Enterprise SaaS (custom pricing, no public prices)
|
||||
|
||||
**Features Included:**
|
||||
- [TO BE RESEARCHED]
|
||||
**Likely Based On:**
|
||||
- Number of test executions or projects
|
||||
- Per-user licensing
|
||||
- Execution threads
|
||||
|
||||
**Limitations:**
|
||||
- Test executions: [TO BE RESEARCHED]
|
||||
- Users: [TO BE RESEARCHED]
|
||||
- Projects: [TO BE RESEARCHED]
|
||||
- AI features: [TO BE RESEARCHED]
|
||||
- Duration: [TO BE RESEARCHED]
|
||||
### Features
|
||||
**No Free Tier** - Demo/pilot programs available
|
||||
|
||||
### Paid Plans
|
||||
|
||||
#### [Plan Name 1] - [TO BE RESEARCHED]
|
||||
**Price:** [TO BE RESEARCHED]
|
||||
**Features:**
|
||||
- [TO BE RESEARCHED]
|
||||
|
||||
#### [Plan Name 2] - [TO BE RESEARCHED]
|
||||
**Price:** [TO BE RESEARCHED]
|
||||
**Features:**
|
||||
- [TO BE RESEARCHED]
|
||||
|
||||
#### Enterprise Plan
|
||||
**Price:** [TO BE RESEARCHED]
|
||||
**Features:**
|
||||
- [TO BE RESEARCHED]
|
||||
**Enterprise Features:**
|
||||
- Natural language test authoring (Gherkin style)
|
||||
- 95% self-healing capability
|
||||
- SSO and role-based access control
|
||||
- Unlimited test runs (typically)
|
||||
- Integration with CI/CD pipelines
|
||||
- Support and training
|
||||
|
||||
### Usage Limits
|
||||
- **Test executions/month:** [TO BE RESEARCHED]
|
||||
- **Users per account:** [TO BE RESEARCHED]
|
||||
- **Projects:** [TO BE RESEARCHED]
|
||||
- **Storage:** [TO BE RESEARCHED]
|
||||
- **API calls:** [TO BE RESEARCHED]
|
||||
- Usually unlimited test runs within contract
|
||||
- Pay per user or per execution thread
|
||||
- High productivity focus (value, not limits)
|
||||
|
||||
---
|
||||
|
||||
## Enterprise Compatibility Assessment
|
||||
|
||||
### Strengths
|
||||
- [TO BE FILLED AFTER RESEARCH]
|
||||
### Strengths ✅
|
||||
- AI-native platform with 95% self-healing
|
||||
- Natural language test authoring (accessible to non-technical)
|
||||
- SOC 2 Type II compliant
|
||||
- Enterprise features (SSO, roles, integrations)
|
||||
- Secure access to internal systems via bridge agents
|
||||
- Strong productivity claims
|
||||
|
||||
### Concerns
|
||||
- [TO BE FILLED AFTER RESEARCH]
|
||||
### Concerns 🚨
|
||||
- **MAJOR:** Uses anonymized test data for ML training by default
|
||||
- **Violates typical "no customer data for training" policies**
|
||||
- Opt-out requires contractual negotiation
|
||||
- No public pricing (enterprise quotes only)
|
||||
- Retention policies not transparent
|
||||
- Relatively aggressive data usage compared to competitors
|
||||
|
||||
### Recommendations
|
||||
- [TO BE FILLED AFTER RESEARCH]
|
||||
|
||||
**⚠️ APPROVED WITH SIGNIFICANT CAVEATS**
|
||||
|
||||
**For Enterprise Use:**
|
||||
- **CRITICAL:** Must negotiate contractual clause to **disable data usage for training**
|
||||
- Get written confirmation that test scenarios won't be used in ML models
|
||||
- Request explicit data retention and deletion policies
|
||||
- Ensure NDA covers test data and workflows
|
||||
|
||||
**Best For:**
|
||||
- Organizations comfortable with anonymized data usage (after contractual protections)
|
||||
- Teams needing natural language test authoring
|
||||
- Companies prioritizing self-healing over strict data isolation
|
||||
|
||||
**Not Recommended For:**
|
||||
- Organizations with "zero customer data for AI training" policies (without opt-out)
|
||||
- Highly sensitive proprietary workflows (risk of pattern learning)
|
||||
- Companies requiring complete data isolation
|
||||
|
||||
**Comparison to Alternatives:**
|
||||
- **Testim, Applitools:** Do NOT use customer data for training by default
|
||||
- **BrowserStack:** Wipes all data after sessions, no training
|
||||
- **Diffblue:** Runs on-premises, zero cloud dependency
|
||||
|
||||
**For Hack Days:**
|
||||
- ⚠️ Can be used if team accepts anonymized test data usage
|
||||
- Should NOT be used for highly sensitive/proprietary test scenarios
|
||||
- Consider alternatives (Testim, Applitools) if strict privacy required
|
||||
|
||||
---
|
||||
|
||||
## Sources & References
|
||||
|
||||
1. **Privacy Policy:** https://virtuoso.qa/privacy/
|
||||
2. **Terms of Service:** https://virtuoso.qa/terms/
|
||||
3. **Pricing Page:** https://virtuoso.qa/pricing/
|
||||
4. **Enterprise Documentation:** [TO BE RESEARCHED]
|
||||
5. **Additional Sources:** [TO BE ADDED]
|
||||
1. **ChatGPT Evaluation Report** - Virtuoso Acceptable Usage Policy analysis
|
||||
2. **Virtuoso Website**: https://virtuoso.qa
|
||||
3. **SOC 2 Compliance**: Mentioned in enterprise materials
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
- Last Updated: 2026-02-02
|
||||
- Evaluator Notes: [ADD YOUR OBSERVATIONS]
|
||||
- Evaluator: Claude Code based on ChatGPT findings
|
||||
- **Critical Finding:** Virtuoso's default data usage policy differs from industry standard. Requires explicit contractual opt-out for enterprise compliance.
|
||||
- **Action Required:** Get written confirmation of training opt-out before procurement
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ PROCUREMENT ALERT
|
||||
|
||||
**This tool requires special approval due to data training policy.**
|
||||
|
||||
Before using Virtuoso QA:
|
||||
1. Review organization's AI data policy
|
||||
2. Determine if anonymized training usage is acceptable
|
||||
3. If not, negotiate contractual opt-out clause
|
||||
4. Get written confirmation before deployment
|
||||
5. Consider alternatives (Testim, Applitools) with stricter default privacy
|
||||
Reference in new issue
Block a user