Building the Ideal CI-CD Pipeline_ Achieving Visual Perfection

Brandon Murray
Technical Customer Success Manager
Building the Ideal
CI/CD Pipeline:
Achieving Visual Perfection

TRADITIONAL CI/CD TESTING PIPELINE
Unit Testing
Api Testing
Functional Testing
Performance Testing
Exploratory Testing
Manual Testing
Production Monitoring
Development QA Production

Problems with traditional functional testing
1. Time to Develop/ Maintenance
2. Does not validate the UI
3. Requires manual testing
4. Flaky tests
5. Execution time (compounded by multiple browser environments)

Ideal World
● Faster test creation
● Low test maintenance
● Full UI coverage
● Reduce manual testing
● More stable tests
● Faster execution
● Allow shift left
● Immediate PR feedback
● Release faster

Unit Testing
Api Testing
Visual AI Functional
Testing
Functional Testing
Performance Testing
Exploratory Testing
Manual Testing
(drastically reduced
or eliminated)
IDEAL CI/CD TESTING PIPELINE

How can we address traditional function
testing limitations?

4. Flaky tests
5. Execution time (compounded by
multiple browser environments)
Solution: Visual AI Testing

A Simple, One Page, Single Browser Example
8

Traditional Functional Test Example
18
lines of code
21
locators & labels
9
Navigation
Button Click
Validation
60
minutes to create

10
Please enter username
and password
Traditional Functional Test Example
Escaped bug
Escaped bug
Escaped bug
Text change caught
Text change caught
New feature missed
2 broken locators, false positives
Original UI New UI
NEW
NEW
NEW

11
Bugs Still Escape
Traditional automated testing frameworks are not built for modern apps.

12
Bugs Still Escape
Traditional automated testing frameworks are not built for modern apps.

Now, Let’s Provide Initial Coverage Using Visual Testing
2 Minutes to Create
Vs. 60 Minutes
1 Locator
Vs. 21 Locators
and Labels

14
Please enter username
and password
Visual AI Testing ﬁnds functional bugs that traditional functional testing can’t.
Escaped bug
Escaped bug
Escaped bug
Text change caught
Text change caught
New feature missed
2 broken locators, false positives
Original UI New UI
NEW
NEW
NEW

● Do I still need Visual AI Testing if I use a css
framework like tailwind?
● Can I drop all of my assertions?

Visual AI Is 99.9999% Accurate
Not fooled by Config Changes or Browser Updates
View by human in Chrome 67
Pixel Diffs
View by human in Chrome 68
Visual AI Diffs

What about adjusting the threshold for
pixel matching algorithms?

This is why Applitools
invented Visual AI
Visual AI mimics how the
human eye and brain
analyzes computer
screens
Trained on >2b images to deliver
99.9999% accuracy
19

21
Ignoring Displaced Diﬀerences - Only available with Visual AI while bugs go
unnoticed with pixel matching

22
Ignoring Displaced Diﬀerences - Only available with Visual AI while bugs go
unnoticed with pixel matching

4. Flaky tests
Solutions: Self-Healing, Visual AI, Lazy Loading,
and assertions

Flakiness
● Failures due to missing or changed locators.
● Timing issues
● False positives (pixel matching)

Self-healing
● Self-Heal tests that fail due to missing or changed
locators.

Record and generate with self healing selectors
for Cypress

Flakiness
● Failures due to missing or changed locators.
○ Self Healing and Cypress Record and Generate
● Timing issues
○ SDK LazyLoading
○ Assertions
● False positives (pixel matching)
○ Visual AI

4. Flaky tests
Solution: Dom capture/snapshot with render grid
(Ultrafast Test Cloud)

Execution Time
● Dom capture/snapshot with render grid (Ultrafast
Test Cloud)
● Parallel Execution - Run your tests completely in the
cloud in parallel for faster CI/CD pipelines.
○ Execution cloud or any other grid solution
○ Alternative - Chromedriver execution in parallel
○ Parallel CI agents

Dom Capture/snapshot vs Image Capture

Dom Capture/snapshot
Capture the entire UI
with a single snapshot
DOM & CSS sent to
render grid
Applitools Eyes
Visual AI analysis
renders DOM
snapshots in parallel
Login to Eyes to
view test results
.
Firewall
33
Applitools
Ultrafast Test Cloud
Images
rendered for
each
environment

How does Applitools know what “Golden
Image” to compare against?

How do we use these enhancements to
complement our developer workﬂow?

Pull Request Worﬂow integration
● Branching support for developers to work in parallel
● Pull request check - real time check without the need
to rerun pipeline
● Automatic merging of images (baselines)

Ideal World
● Faster test creation ✅
● Low test maintenance ✅
● More stable tests ✅
● Faster execution ✅
● Full UI coverage ✅
● Allow shift left ✅
● Immediate PR feedback ✅
● Reduced manual testing ✅
● Release faster ✅

IDEAL CI/CD TESTING PIPELINE
Unit Testing
Api Testing
Visual AI Functional
Testing
Functional Testing
Performance Testing
Exploratory Testing
Manual Testing
(drastically reduced
or eliminated)

Building the Ideal CI-CD Pipeline_ Achieving Visual Perfection

Recommended

Recommended

More Related Content

Similar to Building the Ideal CI-CD Pipeline_ Achieving Visual Perfection

Similar to Building the Ideal CI-CD Pipeline_ Achieving Visual Perfection (20)

More from Applitools

More from Applitools (20)

Recently uploaded

Recently uploaded (20)

Building the Ideal CI-CD Pipeline_ Achieving Visual Perfection