Vndrew Kritzer and Hammond Guerin stared at the screen and then at each other. I
ID: 367148 • Letter: V
Question
Vndrew Kritzer and Hammond Guerin stared at the screen and then at each other. It was June 30, 2014— six weeks since they had graduated from the Darden School Of Business. ‘I’he ad-serving algorithm Kritzer and Guerin had spent six months developing for Vungle, a mobile advertising company, seemed to be outperforming the company’s current algorithm. But they did not •vant to start celebrating too soon. Could their algorithm really deliver the type of improvement they had promised Vungle’s CEO? Would Install rates of advertised apps really increase? Would Vungle see an increase in ad-serving efficiency as a result?
Neither Kritzer nor Guerin could afford for the algorithm to disappoint. Now that he had graduated, Kritzer was headed to Linkedln, having left a legend among iMBA students for his appreciation of data scrence, tech, and media and ralslng expectations for what Darden students knew and could Icarn about data science, analytics, and the ever-growing world of big data. I lis work on the Vunglc proicct during his second year had received a lot of attention, and he was looking forward to having the results support the effort.
Guerin’s data science capabilities were also legendary among his NIBA peers. He won every school forecasting competition, and his data mining algorithms even beat those Of the professlonal consultants who did classroom visits. Late in his second year, Guerin decided to turn down a generous offer from a well-known consulting firm in favor of an Offer from Vungle for an annual salary of and stock options to serve as the head of Vungle’s brand new data science team out in the company’s San Francisco headquarters. The job was a dream for the computer scientist turned MBA. He and his wife were already house hunting in the Bay Area, looking for the right place to raise their baby daughter,
Company Overview
Vungle was an in-app video advertising company. With 70 employees and $25.5 million from three rounds of investments, Vungle was routinely listed as one of the most promising start-ups operating in Silicon Valley.l The three year-old Company offered a platform that embedded video ads in mobile apps to encourage users to download and install additional apps. It was estimated that more than million people saw an advertisement enabled by ungle each month.2
‘I’lus field-based case prepared Yacl Grushka-Cockayne, Assistant Professor of Business Administration, and Kenneth C. l,tchlendahl Jr., Associate Professor of Business Administration. both from the School of Business; and bv Bert I Reyck, Professor and I lead of the Department of lanagemenl Science and Innovation. and Ioannis Fragkos, Research Associate, both College lnndon. It w.’lS Wi•ltten bas’S for class discussion rather than to Illustrate effective or ineffectn•c handling Of an administrative situation.
Vungle was founded in 2011 by two young entrepreneurs from the United Kingdom, Zain Jaffer and Jack Smith, during their graduate studies at University College London. Initially a video ad production firm, Vungle’s expenses In its first year were running too high and revenue was not reaching the founders’ expectations. Late in 2011, Jaffer borrowed funds from his then girlfriend (and future wife) and his business professor, Bert Dc Reyck. Each invested $15,000 and the company rematned afloat. 3
The turning point for Vungle came In 2012, when the two founders creatively used their own video production technology to get the attention of the San Francisco—based start-up incubator AngelPad. In doing so, they beat 2,000 applicants for the final slot in the incubator program. ‘I’his opportunity provided Vungle $120,000 in seed funding. Jaffer moved to San Francisco to serve as the firm’s CEO and rematned in that position. He was profiled In a “35 Under 35” list by Inc. magazine in 2014.4
The Mobile Advertising Ecosystem: Market, Operations, and Pricing
In 2013, ihe average U.S. consumer spent two hours and 42 minutes on mobile devices per day; 860/0 Of that was spent in apps, the clear dominant form of mobile usage.5 The growth in the mobile market and the extensive tune Spent in apps introduced a new advertislng channel. According to the iM0bile Nlarketing Association, 75% of ads setved to mobile consumers in 2013 were served while they were using apps/’ Mobile in-app ads experienced a 600/0 annual growth in 2013 and were expected to surpass PC online ad revenues by 2017.7
By 2014, in-app video advertising was replacing mobile banner ads—the latter offered a lower-quality user experience and were typically clicked on accidentally. The in app video ads were typically 15 seconds long and promoted a app or product. Apple’s iOS system accounted for 809/0 of ads berng served. Video ads peaked during prime-time TV hours.8
Four parties participated in the In-app mobile advertisement channel. The user of the mobile device (Vser), the owner of the app being used (published, the sponsor of the video ad the user was exposed to (advertised, and the platform that matched the choice of ad to a specific user (e.g., V’ung/e). In the mobile advertislng domain, supply was considered to be the slots available for showing ads, and demand consisted of the advertisers willing to buy the supply by placing ads.
When the user launched an app, his or her device would send a request to Vungle for an ad. For instance, suppose user Chris was playing Sonic Dash by the publisher Sega. Vungle’s platform would then determine the best ad to serve to Chris while he played Sonic Dash. Assume Vungle decided to serve Chris an ad for the game Hay Day (the advertiser; see Exhibit 1 for a schematic of this process). Assuming Chris was stlll playing Sonic Dash when the ad was served, then Chris would see the video for Hay Day. If Chris Was interested in learning more about I lay Day, he would click on the ad and be redirected to the app store: Chris might then decide to install Hay Day.
In most cases, payment was made by the advertiser upon installation. Publishers typically received 600/0 of the revenues and the ad provider the remaining 40%. See Figure 1 for the conversion funnel depicting how an install is achieved. Of all ad requests, most were served and became impressions. When at least 80% of a video ad was watched, it was considered complete. When the user clicked on the ad to get more information, It was counted as a click. The process could then result in an install.
Figure I, Mobile in-app advertising funnel.
Requests
Ads were monetized at all different points along the funnel—swhcther CPI (cost per install), CPC (cost per click), CPCV (cost per completed view), or CPM (cost per 1,000 views). The vast majority of ads were CPI.
On a typical day, using its current ad-serving algorithm, Vungle experienced a 98% fill rate, 88% completion rate, 5% click-through rate, and 0.5% conversion rate. The funnel for Vungle narrowed substantially at the end. Small improvements in the click-through or conversion rates could have a large effect on Vungle’s revenue. The effectiveness of an app-promotion campaign and the success of the serving platform were typically measured by eRPM, or effective revenue (for both publisher and Vunglc) per 1,000 impressions,’ which could vary from $2 to as high as $7 per campaign.
A/B Testing and the Data Science Project
Kritzer and Guerin were tasked with developlng an ad-serving learning algorithm. Their data science approach would use historical information about users, publishers, and install rates to detcrmine which ad campaign to serve In order to increase the chance of a conversion and, more specifically, eRPNl. If the system proved successful, implementing it would require regular updates to the model by a data scientist, most likely Guerin himself.
Jaffer consulted with Vungle’s chief technology officer, Wayne Chan, on how best to test the developed algorithm. Chan planned to test the developed method in parallel with the existing Vungle algorithm. As was typical in such experiments, the two Conditions, (Vunglc’s existing algorithm) and B (the data science approach) would be evaluated in parallel on randomly assigned users. Since Kritzer and Guerin’s algorithm was
new and unproven, Chan’s team thought it would make sense to direct only 1/ 16th of the users to the B
condition. *Ille other randomly assigned 15/ 16ths of users would receive an ad based on the existing algorithm (i.e., the A condition).
Users were assigned to the A or B algorithm using a process called MD5 hashing. An MD5 hash transforms each user ID into a unique 32-character hexadecimal string. Each character of the hexadecimal string could be 0—9, A, B, C, D, E, or F—16 options in total. Each character occurred with equal likelihood, making it simple for Vungle to direct traffic in 1/ 16th increments using a logic statement (assuming that the original string was random).
The parallel run Of the two algorithms began on June l, 2014. Jaffer was excited to see if B would outperform A and, if so, what the financial benefits would be. He also wondered how long Chan would have to wait to declare a winner. Would a few days be enough time? Or would he need to wait longer? After two weeks, B Was looking pretty good. Its daily eRPNf was on average $0.131 higher than A’s. Would this translate into annual revenues worthy of the necessary data science investment? Exhibits 2 and 3 provide the daily results Of the A/B test.
fiinking about his new role at Vungle, Guerin was curious to see how the superior condition would be chosen. How would one conclude that B was better than A? If he could be confident about such a conclusion, he would bc able to develop a robust testing platform for many future experiments.
QUESTIONS
The following learning outcomes are tested during this assessment of case study assignment
Apply research and analytical skills to problem solving
Demonstrate critical thinking ability and use of highly developed cognitive skills to solve business problems.
The case is
A/B Testing at Vungle case examines results of parallel run (A/B test)of two algorithms during June 2014. You need to make a detailed analysis for the case
Use the case to answer the following questions:
Assignment Questions ( the Answers should be computerized - in Details - Please do Not Copy and Paste )
How does Vungle makes money?
What challenges faces Jaffer?
What would you advise Jaffer regarding the performance of the new data science algorithm ?
Which assumptions underlie your analysis? How does the assumption of normality come into play?
How might Jaffer conclude that B better than A? And if it is, what would the financial benefit be?
Explanation / Answer
How does Vungle makes money?
Vungle is a 3 year old start-up operating in Silicon Valley.It orivudes a platform that embedded video ads in mobile apps which encourage app users to download and install other apps. Vungle earn money from these adds as more than million people saw an advertisement enabled by Vungle each month. Main source of revenue is Click per Install where a viewer install some app through a link which appears with viseo he is watching. Vungle gets 60% of revenue from $2 to as high as $7 per campaign.
Related Questions
Navigate
Integrity-first tutoring: explanations and feedback only — we do not complete graded work. Learn more.