CASE STUDY

C O M PA N Y
CASE STUDY
WatchDox
INDUSTRY
Communication Technology
TESTING TYPES
Functional Test Case Execution
Exploratory Testing
Usability
APP TYPES
Mobile apps
CHALLENGES
Expanding product line and increasing
maturity of quality standards
Small QA team unable to provide
full range of testing coverage
Multitude of mobile device,
geography and network combinations
R E S U LT S
Delivery of higher-quality solutions at a
rapid pace
Cost-effective expansion of necessary
layers of QA without the need to slow
release schedule
Confidence to make bold business
decisions and position WatchDox as
a market leader
C
ustomer
WatchDox, founded in 2008 and acquired by Blackberry in 2015, is an Israeli-based
tech company whose SaaS document-centric security platform enables organizations to
control their critical documents via mobile devices. WatchDox’s platform is used to securely
sync files, share documents and collaboratively work with files wherever they need to go,
even on devices beyond IT control. Their data-centric security solution enables organizations
to control synced and shared files across any tablet, smartphone or PC.
Fortune 500 companies - from private equity firms and hospitals to federal agencies and major
film studios - rely on WatchDox. Customers trust WatchDox because they provide the highest
quality of security when sharing sensitive documentation, especially on mobile networks and
devices. Whether the client is trying to send a private patient file or a confidential document
to their board members through WatchDox, they get the control they need without needing
to jump through hoops.
WatchDox created an enterprise-grade system that provides safety and security surrounding
document management. Their mission is to enable enterprise mobility for employees at
an executive level. The system eases the pains of sharing both internally and externally
from anywhere while allowing the customer to have complete control over the documents.
From the start, rapid innovation was important to WatchDox, and they built a cutting edge
development organization with world-class processes to continue delivering agile solutions.
C hallenges
When WatchDox was first starting out, they faced the same challenges that many startups
face. Budgets were tight and developing new features was the priority for the development
team.
1
WatchDox sold their solution to companies of all sizes, but they quickly realized that larger
enterprise customers found greater value in what they were doing and these customers produced
better long term revenue streams.
However, enterprise customers have higher expectations for
the solutions they bring into their technology stacks and this forced WatchDox to re-evaluate
how they ensure quality.
In the early days, WatchDox would prioritize their testing needs by trying to balance the tradeoffs
between speed to release new features and the potential risk each new feature might have on
the platform. As their business evolved, their product line expanded and the company matured,
their approach to solution quality also matured.
However, even as they added professional QA people to their team, it was impossible to cover
every aspect of testing in-house. They did not have a large team to test the platform, they
needed to continue to move fast and there was no way they could test on all the devices their
mobile platform needed to support. Device coverage across OS’s and form factors is a strategic
value of the WatchDox solution. With the explosion in mobile device options available globally,
device coverage was (and continues to be) a moving target.
In this environment, they found themselves being forced to accept a certain level of risk in
each release, choosing only certain build aspects to test and acknowledging that end-to-end
testing was not an option. However, their expanding enterprise clientele expected no less than
the best, most reliable, and highest quality service. As a result, in 2011 WatchDox came to the
realization that they needed to evolve how they approached quality.
WatchDox began by regression testing every new build, regardless of its size, to ensure that
the whole platform continued to work as expected.
Unfortunately, this type of testing, while
extraordinarily important, can be repetitive and time consuming – potentially delaying release
cycles and, due to its repetitive nature, leading to a bored QA team.
WatchDox needed to
keep their QA people engaged while at the same time guaranteeing that their testing was
all-encompassing and that no builds went unchecked. Any solution they chose needed to easily
fit with their existing QA team and provide an all-inclusive service with a strong focus on
quality, as well as providing as-needed testing regardless of the release type.
103 Unique Android
Device/OS/Carrier Combinations
A8-50 A5500
A8-50 A5500
Arrows NX
F-02G
Arrows
NX F-02G
Desire 816
Desire 816
G4
G4
Galaxy Core
Prime
Galaxy
Core Prime
Galaxy Grand
GalaxyPrime
Grand Prime
Galaxy Note
3 Note 3
Galaxy
Galaxy S
2
Galaxy
S2
Galaxy S
3 I9300
Galaxy
S 3 I9300
Galaxy S
4
Galaxy
S4
Galaxy S
6
Galaxy
S6
Galaxy Tab
2 (7")
Galaxy
Tab 2 (7")
Galaxy Tab
3 (7.0)
Galaxy
Tab 3 (7.0)
Galaxy Tab
S2 (8")
Galaxy
Tab S2 (8")
K3 NoteK3 Note
Moto G Moto
(2013)G (2013)
Nexus 4Nexus 4
Nexus 6Nexus 6
Nexus 7Nexus
(2012)7 (2012)
Nexus 7Nexus
(2nd Gen)
7 (2nd Gen)
Nexus SNexus S
OptimusOptimus
L7 II
L7 II
Xperia M4
AquaM4 Aqua
Xperia
Arrows F-02E
Arrows F-02E
Ascend Ascend
G630 G630
G3
G3
Galaxy Ace
S5830
Galaxy
Ace S5830
Galaxy Fame
Galaxy(S6810)
Fame (S6810)
Galaxy Note
10.1
(2014
Galaxy
Note
10.1Edition)
(2014 Edition)
Galaxy Note
4 Note 4
Galaxy
Galaxy S
3
Galaxy
S3
Galaxy S
3 miniS(I8190)
Galaxy
3 mini (I8190)
Galaxy S
5
Galaxy
S5
Galaxy Tab
2 (10.1)
Galaxy
Tab 2 (10.1)
Galaxy Tab
3 (7.0)
Galaxy
Tab 3 (7.0)
Galaxy Tab
Pro Tab
(10.1)
Galaxy
Pro (10.1)
Galaxy Young
GalaxyS6310
Young S6310
L80 DualL80 Dual
Moto G Moto
(2014)G (2014)
Nexus 5Nexus 5
Nexus 7Nexus 7
Nexus 7Nexus
(2013)7 (2013)
Nexus 9Nexus 9
OptimusOptimus
L5
L5
Voyager2
DG310 DG310
Voyager2
Xperia PXperia P
A8-50 A5500
Arrows F-02E
Arrows NX F-02G
109 Unique iOS
Device/OS/Carrier Combinations
iPad (Early 2012) 3G
iPad 2 3G
iPad Air 2 Wi-Fi
iPad Air Wi-Fi
iPad Mini Wi-Fi
iPhone 4
iPhone 5
iPhone 5S
iPhone 6 Plus
iPad (Early 2012) 3G
iPad 2 3G
iPad Air 2 Wi-Fi
iPad Air Wi-Fi
iPad Mini Wi-Fi
iPhone 4
iPhone 5
iPhone 5S
iPhone 6 Plus
iPad (Early 2012) Wi-Fi
iPad 2 Wi-Fi
iPad Air 3G/4G
iPad Mini 2 Wi-Fi
iPad Pro
iPhone 4S
iPhone 5C
iPhone 6
iPhone 6S
iPad (Early 2012) Wi-Fi
iPad 2 Wi-Fi
iPad Air 3G/4G
iPad Mini 2 Wi-Fi
iPad Pro
iPhone 4S
iPhone 5C
iPhone 6
iPhone 6S
2
S olution
“W E ’ V E F O U N D S O M E
SIGNIFICANT ISSUES
BEFORE THEY WERE
RELEASED TO CLIENTS”
“ THE LEVEL OF CONFIDENCE
WE GET WITH APPLAUSE
INCREASES OUR ABILIT Y TO
MAKE BOLD DECISIONS AND
POSITION WATCHDOX AS A
MARKET LEADER”
Amir Rapson
VP R&D & Israel Site Manager, BlackBerry
To get the speed, flexibility and device coverage they needed, WatchDox set out to find a
third-party service. Their suite of products was growing and completing the necessary testing
for every build was time consuming. They needed to find a fast, secure and reliable way of
testing, because they couldn’t risk a major flaw finding its way into the hands of top executives
at major firms. This led WatchDox to add Applause to their testing regimen.
“In the early stages we always concentrated our manual QA on the new stuff in the next release.
In the earlier days with the regression coverage, we had timeline pressures, so we had to make
decisions about what area to test. Since we started working with Applause, we stopped having
to make those tradeoff decisions. We could cover everything when we started working with
Applause,” says Oren Geva, Director of R&D Operations, BlackBerry. “Using Applause services
allows us to get the full regression testing done in a timely manner. What would have taken
us over a week with only a minimum of device coverage, now takes just a few days by using
10-15 testers per cycle, using several phone and tablet models each.” This coverage has allowed
WatchDox to grow in a scalable way. From the beginning WatchDox has released a build every
two months per product. Today, WatchDox has multiple products and Applause conducts manual
regression testing for all of them.
WatchDox was concerned about the long-term effect repetitive regression testing would have
on the commitment and morale of their team. “QA professionals are very intelligent people, but
they also need to do very repetitive work. This causes them to move to different projects and
different jobs frequently. Having Applause testers allows our team to work on more interesting
projects. It keeps the QA engineers very happy and that’s definitely important to us,” according
to Amir Rapson, VP R&D & Israel Site Manager, BlackBerry.
In addition to rapid regression test case execution, WatchDox takes advantage of Applause’s
exploratory and localization testing services. WatchDox has moved beyond using Applause to
ensure their products work as intended for baseline functionality to leveraging Applause testing
to improve their products and provide enriched customer experiences.
For example, WatchDox
uses Applause-supplied in-country native speakers to test product language variations including
Japanese, Spanish, and French.
“A driving factor for working with Applause was a desire to increase our overall quality,”
explained Oren Geva, “We still had bugs popping up even after we expanded our in-house
testing practices. Applause gave us a cost effective way to add necessary layers of QA. We added
33
reviews to test plans, regression testing, and exploratory testing. Whatever we don’t cover in
an automation test, we cover with full manual coverage from Applause.”
“Applause testers are more similar to external users, therefore do not have the unavoidable
‘internal’ bias of our in-house testers. As a result, the exploratory testing they do is therefore
highly valuable”. Exploratory cycles are a great way to challenge the system. People do unexpected
things, and we have many people testing in many different environments. We find lots of things
that we would never have been able to find in our limited lab situation. The diversity, creativity
and inventiveness that Applause testers bring to test cycles uncovers some really interesting
issues,” according to Oren Geva.
WATCHDOX IS TESTED
IN 44 COUNTRIES
Applause also provides the device coverage that WatchDox needs to support its global clientele.
WatchDox maintains an in-house device laboratory, but they are challenged by the limited
availability of certain devices in Israel. Additionally, new devices are released and OS changes
happen constantly so attempting to keep up with the device landscape is a constantly moving
target. The Applause testing community is so large and diverse that device coverage is not an
issue. WatchDox prioritizes the devices, OSes, browsers and releases they want to test on and
Applause matches those requirements with the testers in each cycle.
A large part of the value WatchDox provides is connected to secure sharing. WatchDox never
found a reason to worry about any of the Applause testers. They were often given access to new
features and the Applause Project Managers and Test Team Leads were given access to create
test cases.
R esults
As WatchDox has grown as a company, holding tight to their key strengths and differentiators
has remained critical.
As they moved from working with small early adopter clients to larger
clients, there was the potential to slow down, but WatchDox maintains its startup mentality and
continues to launch new releases with the same sense of urgency.
That sense of urgency has not diminished as the solution functionality and device coverage have
expanded. In their early days they would risk quality issues, but those approaches could not
sustain them. Working with Applause has enabled WatchDox to deliver higher quality solutions
without sacrificing speed. According to Oren Geva, “we have held reviews to look at the severity
and number of issues found in the field, and its safe to say we’ve found some significant issues
before they were released to clients. I don’t think we would have caught those bugs without
“ONE OF THE KEY SUCCESS
FACTORS TO OUR LONG
TERM PARTNERSHIP IS THE
PROFESSIONALISM OF THE
APPLAUSE PROGRAM MANAGERS
AND APPLAUSE’S EXECUTIVE
MANAGEMENT OVERSIGHT”
Oren Geva
Director of R&D Operations, BlackBerry
4
Applause,” says Oren Geva.
WatchDox has worked with over 100 testers from the Applause community on a regular basis.
They have held trainings with them and have even created on-going relationships with some of
them. Applause testers are an extension of the WatchDox engineering team.
WatchDox trusts
their Applause testers. They are often given access to new features and the Applause Project
Managers and Test Team Leads write many of the test cases. In the long run WatchDox didn’t just
find a company to outsource certain testing needs to, they found a partnership with Applause.
According to Oren, “one of the key success factors to our long term partnership is the
professionalism of the Applause program managers and Applause’s executive management
oversight – ever since we started, Applause has worked hard to help us succeed. They help us
with guidance and work flows. Applause has adapted to what we required and they’ve put in
the effort over the course of our multi-year relationship to make this work.”
Today, a typical test cycle provides about 150 hours of professional testing conducted over a 72
hour period. And, Applause has conducted more than 170 test cycles for WatchDox. Furthermore,
during that time the WatchDox solution portfolio has expanded from one app to five.
A bout A pplause
Applause empowers companies of all sizes to
Amir Rapson sums up the WatchDox – Applause relationship this way, “Applause is part of our
development process and release cycle. We’ve relied on them for years. The level of confidence we get
with Applause increases our ability to make bold decisions and position WatchDox as a market leader.”
deliver great digital experiences (DX) – across
web,
mobile
and
IoT
as
well
as
brick-and-
mortar – spanning every customer touchpoint.
Applause delivers unmatched in-the-wild testing,
user feedback and research solutions by utilizing
its DX platform to manage communities around
the world. This provides brands with the real-world
insights
they
need
to
achieve
omni-channel
success across demographics, locations, devices
and operating systems that match their user base.
Americas Inquiries
Europe Inquiries
Israel Inquiries
Applause U.S. HQ
Applause Europe HQ
11 Galgaley Haplada
100 Pennsylvania Ave
Kopenicker Str. 154
1st Entrance – 2nd Floor
Suite 500
10997 Berlin, Germany
Herzliya, Israel
Framingham, MA 01701
+49 30 57700400
+972.74.701.4240
1-844-300-2777
Thousands
Google,
of
FOX,
Runkeeper
–
Best
rely
companies
Buy,
on
–
BMW,
Applause
including
PayPal
to
and
ensure
great digital experiences for their customers.
Learn more at www.applause.com.
55