Scaling Training Data for AI Applications

Editor's Notes

1. Sarah runs through the agenda
2. If possible, mention that Applause can source data
3. Click for animation. Kristin to start from the end of this slide.
4. The size, breadth, and quality of our community is what enables us to deliver immense value to our clients.Our community has several hundred thousand testers. Each member of the community is carefully vetted (profile, nda, assessments, courses) to make sure feedback is provided in a detailed and concise manner. The community is diverse, with QA professionals, usability experts, and people with no technical background (average joe off the street) – so you get the right type of feedback. If you need access to someone in England with a certain type of credit card, we can do that In the past year, the community submitted a million pieces of feedback (bug reports, test cases, completed usability surveys, etc.) – that’s over 2,700 a day. We are doing this at-scale for the worlds largest brands.
5. On the right-hand side of this slide, you can see the ‘Testing Output’ portion. This is something that Applause has been doing for years and years. In the last couple years, we’ve identified another area that really only Applause, and a globally managed vetted community, can help with, and that’s providing quality training data at scale. So you can see the different types of data that Applause can source from the global community, from handwriting and text to speech and video. We’ll talk a lot more in this presentation about some of the work we’ve done with sourcing handwriting to train an AI algorithm for how to read handwriting.
6. When we’re talking about sourcing data, there are some major challenges out in the market. A lot of organizations start to go down this path, and then realize it’s actually much more challenging than they might’ve realized. Sourcing training data on your own is, to be frank, extremely challenging at best, and possibly outright impossible. You might not have access to the # of people you need. Even if you do have access to that # of people, you need to ensure you’re getting quality data. If you do get quality data, you need a team to annotate and label the data. And even if you do all that, you need to think about diversity and being able to evolve over time. It’s a massive challenge of logistics and overhead.
7. The “Wheel” of challenges include: Data quality (bias, errors) Lack of quantity Diversity You need thousands of artifacts to properly train an AI algorithm. For example, we recently did some work with BBC to train their voice assistant, Beeb, and the algorithm required over 105,000 voice utterances, which Applause provided for BBC. But of course, having a lot of data is pretty much worthless if it’s poor quality. If your data isn’t labeled correctly, or if it’s not in the right format to begin with, it can delay your project and sometimes is completely useless, depending on the data type Diversity is the third element. If you’re building an AI algorithm, you don’t want to rely on 1 single person to provide the artifacts. That’s not going to lead to a strong AI output. So getting data from not just a lot sources, but a wide variety of sources, is impactful And of course, these projects evolve, and you need to be able to evolve with them
8. So we talk about the challenges – how can Applause solve for them? Our system can produce usable data and follow some pretty strict requirements. Limited overhead  we’re a white-glove service and our internal teams manage the recruitment of data providers, we thoroughly evaluate the data artifacts we get from a source, and we train our testers to follow your requirements strictly. And we haven’t mentioned yet, but Privacy and Security is a major element to consider. There are compliance laws to consider, such as GDPR. Applause works within those confines to ensure confidentiality while also providing useful data for a customer And then elasticity and scalability, our model can shift as a customer’s requirements change, which can happen quite a bit with AI projects
9. Let’s look at an example of a real customer, and how Applause sourced training data for them. Here are some of the challenges that come up with this kind of project.
10. Want to share an example. We worked with an organization to build an algorithm that can read handwritten documents. So the idea is that you could scan a handwritten document, and the AI algorithm could read and understand the document. The software scans the form and identifies the keys and values. It detects the form field name. The content is the value -- even if it was filled in with a typewriter field, it might not be in the same place on every form. The software needs to understand the difference between the key and the value So how do you acquire the data that you need to train that algorithm? And what are some of the challenges that come up there? For one, we needed to source documents globally to acquire different: Handwriting styles Languages Other critical factors Example 1 is Amazon Amazon
11. This was a project where the customer was looking for handwriting samples, to teach an algorithm to read handwriting. It needed thousands of handwriting samples to work, so again, there’s the quantity aspect coming up. But add this couldn’t just be 1 person submitting 500 or 1000 samples – that would’ve made this a lot easier to execute. For this project to work, each person could only submit 1 handwriting sample – in other words, we needed thousands of unique handwriting samples. Since each document had to come from a unique person, Applause had to recruit thousands of people – this is the kind of project that really only an organization like Applause can satisfy. We sourced well over 1000 folks in our community who were willing to provide handwritten documents. Why is an essay or letter valuable? It’s about handwriting recognition, so Applause asked our community for handwritten essays and letters. We had folks who were digging in their closet from 10 years ago that someone had written. We were looking for unique handwriting samples. We even had someone once ask for SAT and ACT essays, but obviously this wasn’t something we could provide
12. In addition to getting a lot of people, Applause was being asked to produce a lot of different types of documents. So you can see, the tax forms, we needed to provide a lot of different types. We had a team that could manage this and ensure we were bringing in the diversity of documents we needed to help the algorithm. We needed at least 50 W-2 forms, 50 IRS 1098-T forms, etc. And then the requirements of the documents themselves, this gets into the “quality” of the data  no deformations on the page, the page can have no more than 1 single folded margin in the middle, and there were several specifications for that. And you’re scanning these documents, so they need to be in good light conditions or the flash is used in dark settings Redactions  the testers had to put in their own dummy data to protect PII There’s a lot of overhead that comes with this project, especially at scale. So having a team that can not only manage this project, but knows what to look for, is really crucial to success.
13. And finally, privacy is a major concern here. We’ve got GDPR, HIPAA that you need to consider. So by giving a company a completed tax document or healthcare form, you could be violating some laws or opening yourself up to a lawsuit. So this is something that if you’re trying to do this on your own, there’s a lot of overhead  you can imagine a team of 10+ people having to work around the clock for weeks if not months to remove PII and ensure confidentiality. Here at Applause, we have processes in place where we can protect confidentiality. And in this case, we instructed testers to fill out the forms with dummy data. That way, the organization is still getting the handwriting sample, but there’s no sacrifice of PII.
14. Quantity Need hundreds, if not thousands of individuals to make this work Diversity Requires many different types of data (geographic, document type, etc.) to properly train the algorithm Privacy and confidentially Need process, dedicated resources to ensure privacy and not violating GDPR, exposing PII Sourcing often needs to be blind and account for the nature of the product Process and sustainable model Sourcing training data for AI at scale is a major undertaking  you need a team that is wholly dedicated to delivering on this project