Randall A. Lewis
Curriculum Vitae

Click for an up-to-date (2021) resume/CV: Randall Lewis, PhD

Click for an Incomplete List of Papers Available Online

---

CONTACT INFORMATION

PERSONAL STATEMENT

I am a causal measurement expert. I design causal measurement systems to guide business decision-making. In today’s digital economy, this requires inventing new methodologies and integrating them into products to create causal machine learning analyses at scale. I have 13+ years of experience focusing my skills on measuring the causal impact of advertising, estimating price-elasticities of demand, developing advanced AB-testing platform analytics, and measuring the value of entertainment media in order to help humans and automated decision systems make better choices.

---

RESEARCH INTERESTS

Industrial-scale applied econometrics (econinformatics), distributed statistical computing, advertising effectiveness & measurement system design, large-scale field experiment (treatment effect) analysis and system design (A/B testing), social networks, causal machine learning (exogeneity / instrumental variables combined with LASSO, ridge, clustering, random forests, neural nets, deep learning), endogenous bandits, nonparametrics, Bayesian statistics, probabilistic modeling, adaptive/reinforcement learning, dynamic / longitudinal causal models, item-response theory, industrial organization / strategic game theory of firms, demand estimation, pricing, structural modeling, user-value / latent-utility estimation, discrete-choice modeling, behavioral economics, advertising externalities, ad-quality assessment, ad wear-out, ad valuation, asset valuation, potential outcomes, platform economics / multi-sided markets

---

PROFESSIONAL

Nanigans Feb 2019 – Current

• Chief Scientist

Netflix Sept 2015 – Feb 2019

• Director of Economic, Content Valuation & Marketing, Science & Analytics
• Economic Research Scientist, Growth & Targeting, Science & Algorithms

Google, Inc. May 2012 – Sept 2015

• Senior Economic Research Scientist, Economics and Strategic Technologies

Yahoo! Research Labs June 2008 – May 2012

• Economic Research Scientist, Microeconomics and Social Systems

---

EDUCATION

Massachusetts Institute of Technology
Ph.D. in Economics Sept 2006 – Sept 2010
Fields of Concentration: Industrial Organization, Econometrics

• Dissertation Title: “Measuring the Effects of Online Advertising on Human Behavior Using Natural and Field Experiments” (advisers: Glenn Ellison, Jerry Hausman)

Brigham Young University
B.A. in Economics (Valedictorian), B.S. in Mathematics Sept 2001 – June 2002, Jan 2005 – Aug 2006

---

HONORS, SCHOLARSHIPS, AND FELLOWSHIPS

2017   JMR Journal Annual Paul E. Green Best Paper Award
2017   Gary L. Lilien ISMS-MSI Practice Prize Finalist
2017   John D. C. Little Award Finalist: “Less is More”
2015 QME Journal Annual Dick Wittink Best Paper Award
2011, 2010   Yahoo! Superstar Runner-Up, Nominee
2006   MIT Presidential Fellow
2006   BYU Dept. of Economics Valedictorian
2001   BYU Hinckley Presidential Scholar

SKILLS

Expertise: Python, TQL, R, SQL, Presto/Hive, Matlab, Stata, Bash/Unix, H2O, Vowpal Wabbit, git, LaTeX, LyX
Experience: Perl, Pig, PHP, C, C++, Javascript, HTML, CSS, TensorFlow, R-Keras
Languages: English (native), Spanish (fluent)

---

PEER-REVIEWED PUBLICATIONS

• Garrett A. Johnson, Randall A. Lewis, and Elmar Nubbemeyer. “Ghost Ads: Improving the Economics of Measuring Ad Effectiveness.” Journal of Marketing Research: December 2017. (+2017 Best Paper, Finalist)
• Garrett A. Johnson, Randall A. Lewis, and David H. Reiley. “When Less is More: Data and Power in Advertising Experiments.” Marketing Science, July 2016. (+2017 Finalist)
• Randall A. Lewis and Justin M. Rao. “The Unfavorable Economics of Measuring the Returns to Advertising.” Quarterly Journal of Economics, July 2015.
• Randall A. Lewis and Dan T. Nguyen. “Display Advertising’s Competitive Spillovers to Consumer Search.” Quantitative Marketing and Economics, June 2015.
• Randall A. Lewis and David H. Reiley, “Online Ads and Offline Sales: Measuring the Effects of Retail Advertising via a Controlled Experiment on Yahoo!” Quantitative Marketing and Economics, 12(3): 235-266 (September 2014).  (+2015 Best Paper)
• Anja Lambrecht, Avi Goldfarb, Alessandro Bonatti, Anindya Ghose, Dan Goldstein, Randall Lewis, Anita Rao, Navdeep Sahni, and Song Yao, “How Do Firms Make Money Selling Digital Goods Online?” Marketing Letters, 9th Triennial Choice Symposium, September 2014.
• Randall A. Lewis, Justin M. Rao, and David H. Reiley, “Measuring the Effects of Advertising: The Digital Frontier,” Economic Analysis of the Digital Economy, May 2015.
• Randall A. Lewis and David H. Reiley, “Advertising Effectively Influences Older Users: A Yahoo! Experiment Measuring Retail Sales,” Review of Industrial Organization, 44(2): 147-159 (March 2014).
• Randall A. Lewis and David H. Reiley, “Down-to-the-Minute Effects of Super Bowl Advertising on Online Search Behavior,” Proceedings of the 14th ACM Conference on Electronic Commerce (EC’2013), E. Tardos, R.P. McAfee, eds., June 2013.
• Randall A. Lewis and James B. McDonald. “Partially Adaptive Estimation of the Censored Regression Model,” Econometric Reviews (January 2013).
• Hector Garcia-Molina, Prabhakar Krishnamurthy, Randall Lewis, Panagiotis Papadimitriou, and David Reiley, “Display Advertising Impact: Search Lift and Social Influence.” 17th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD17), August 2011.
• Randall A. Lewis, Justin M. Rao, and David H. Reiley, “Here, There, and Everywhere: Correlated Online Behaviors Can Lead to Overestimates of the Effects of Advertising.” 20th International World Wide Web Conference (WWW20), April 2011.
• Hausman, Jerry A., Randall A. Lewis, Konrad Menzel, and Whitney K. Newey. 2011. “Properties of the CUE estimator and a modification with moments.” Journal of Econometrics, 165(1): 45-57.
• Nathan E Lewis, Gunnar Schramm, Aarash Bordbar, Jan Schellenberger, Michael P Andersen, Jeffrey  Cheng, Nilam Patel, Alex Yee, Randall A Lewis, Roland Eils, Rainer König & Bernhard Ø Palsson. “Large-scale in silico modeling of metabolic interactions between cell types in the human brain.” Nature Biotechnology 28, 1279–1285 (November 2010).
• David H. Reiley, Sai-Ming Li, and Randall A. Lewis, “Northern exposure: a field experiment measuring externalities between search advertisements.” Proceedings of the 11th ACM Conference on Electronic Commerce (EC’2010), D. Parkes, C. Dellarocas, and M. Tennenholtz, eds., pp. 297-304.

---

WORKING PAPERS

• “You Only Compress Once: Optimal Data Compression for Estimating Linear Models,” with J Wong, E Forsell, T Mao, M Wardrop.
• “Efficient Computation of Linear Model Treatment Effects in an Experimentation Platform,” with J Wong, M Wardrop.
• “Incrementality Bidding & Attribution,” with Jeffrey Wong.
• “The Online Display Ad Effectiveness Funnel & Carry-Over: A Meta-Study of Ghost Ad Experiments,” with Garrett A. Johnson and Elmar Nubbemeyer.
• “Cost Per Incremental Action: Efficient Pricing of Advertising,” with Garrett A. Johnson.
• “Brand Engagement Lift: Causal Effects of Skippable Video Ads on YouTube,” with Alejandro Veen, Jiangyong Yin, and Yi-Hsin Yeh.
• “Econometrics of Sampled Networks,” with Arun G. Chandrasekhar; submitted to Econometrica.
• “Location, Location, Location: Repetition and Proximity Increase Advertising Effectiveness,” with Garrett A. Johnson and David H. Reiley.
• “Worn-Out or Just Getting Started? The Impact of Frequency in Online Display Advertising,” revise & resubmit at Marketing Science.
• “Ad Attributes and Attribution: Large-Scale Field Experiments Measure Online Customer Acquisition” with David H. Reiley & Taylor A. Schreiner; revise & resubmit at Quantitative Marketing and Economics.
• “When Less is More: The Smallest Superset of Identification.”

---

CONFERENCE PRESENTATIONS & INVITED TALKS

• Invited Keynote at eBay Digital Marketing, San Jose, CA (remote). June 16, 2021. “Incrementality: In Practice.”
• Invited Marketing Seminar, CU Boulder Leeds School of Business, Boulder, CO (remote). March 19, 2021. “Incrementality Bidding & Attribution.”
• Invited Marketing Seminar, Columbia University, New York, NY (remote). October 15, 2020. “Incrementality Bidding & Attribution.”
• Invited Data Science & Marketing Seminar, Etsy, San Francisco, CA (remote). June 24, 2020. “Incrementality Bidding & Attribution.”
• Invited Keynote at Google Incrementality on Air, Tel Aviv, Israel (remote). June 2020. “Incrementality: In Practice.”
• Data Science Tech Talk, Apple Maps, Cupertino, CA. February 20, 2020. “Incrementality Bidding & Attribution.”
• MSBA Symposium on Causality and Explainability, U of Santa Clara, Santa Clara, CA, October 25, 2019. “Incrementality Bidding & Attribution.”
• Simons Institute Workshop: Information Design & Data Science, UC Berkeley, Berkeley, CA, October 1, 2019. “Incrementality Bidding & Attribution.”
• 11th Triennial Choice Symposium, Cambridge, MD, May 2019. “Incrementality Bidding & Attribution.”
• Carnegie Mellon University’s Heinz College Seminar, Pittsburg, PA, April 1, 2019. “Incrementality Bidding & Attribution.”
• ARF: Best Practices for Proving Advertising ROI Using Experimental Design, New York, NY, January 24, 2019. “Incrementality at Netflix.”
• Quantitative Marketing and Economics Conference, Chicago, IL, August 24-25, 2018. “Incrementality Bidding & Attribution.”
• Invited Keynote at Machine Learning in the Real World Workshop, Criteo, Paris, France, June 28, 2018. “Incrementality Bidding & Attribution.”
• Invited Marketing Seminar at Kellogg School of Business, Boston, MA, February 28, 2018. “Incrementality Bidding & Attribution.”
• American Marketing Association Conference, New Orleans, LA, February 24, 2018. “Incrementality Bidding & Attribution,” “Ghost Ads Meta Study.”
• Invited Seminar at Wayfair, Boston, MA, January 30, 2018. “Incrementality Bidding & Attribution.”
• Microsoft Digital Marketing Meetup, Redmond, WA, November 16, 2017. “Incrementality Bidding & Attribution.”
• NABE Tech Economics Conference, Seattle, WA, November 15-16, 2017. Data Science Panel & “Incrementality Bidding & Attribution.”
• HBS in SF: Experiments in Tech Symposium hosted by Uber, San Francisco, CA, November 10, 2017. “Ghost Ads.”
• Incrementality in Advertising MeetUp hosted by AdRoll, San Francisco, CA, October 24, 2017. “Incrementality Bidding & Attribution.”
• Microsoft Digital Economics Conference, Redmond, WA, October 19-20, 2017. “Incrementality Bidding & Attribution.”
• Ford Behavioral & Experimental Economics Workshop, Dearborn, MI, September 26, 2017. “Incrementality Bidding & Attribution.”
• Conference on Computational Challenges (C3) in the Social Sciences, Napa, CA, September 11-12, 2017. “Incrementality Bidding & Attribution.”
• Montana State Applied Economics Workshop and Seminar, Bozeman, MT, September 8, 2017.  “Ghost Ads,” “Incrementality Bidding & Attribution.”
• AdKDD Workshop, Halifax, NS, Canada, August 14, 2017. “Incrementality Bidding & Attribution.”
• Entertainment Analytics Conference, Venice, CA, August 3-4, 2017. “Incrementality Bidding & Attribution.”
• Santa Clara University Digital Marketing Lecture, Santa Clara, CA, June 15 & 22, 2017. “Ghost Ads.”
• Stanford Graduate School of Business Lectures, Stanford, CA, April 27 and May 8, 2017. “Near Impossibility,” “Ghost Ads.”
• Digital Marketing Conference, San Francisco, CA, April 5, 2017. “Ghost Ads.”
• Mini-conference on Big Data at UC Irvine, Irvine, CA, March 10, 2017. “Ghost Ads.”
• Conference on Data Science in the Entertainment Industries, Santa Monica, CA, August 18, 2016. “Ghost Ads.”
• 9th Workshop on the Economics of Advertising and Marketing, Vilnius, Lithuania, July 2016. “Ghost Ads.”
• NABE Big Data Conference, Boston, MA. June 2016. “Ghost Ads.”
• IDB Invited Seminar, Washington, D.C., June 2016. “Silicon Valley Experimentation” + “Ghost Ads” + “Econinformatics.”
• NABE Conference of Tech Economists, San Francisco, CA. April 2016. “Ghost Ads.”
• NBER SIEPR Digitization Conference, Stanford, CA. March 4, 2016. “Ghost Ads.”
• DataLead Workshop on Machine Learning and Economics, Paris, France, November 2015. “Ghost Ads.”
• AMA Advanced Research Technologies Forum, San Diego, CA, June 2015. “Unfavorable Economics.”
• ASSA Electronic Commerce and Big Data Session, Boston, MA, January 2015. “Worn-Out.”
• INFORMS Annual Conference, San Francisco, CA. November 2014. “Unfavorable Economics.”
• International Symposium on Recent Developments in Econometric Theory with Applications in honor of Jerry A. Hausman, Xiamen, China. June, 2014. “Econinformatics.”
• Marketing Science Conference, Atlanta, GA. June 2014. “Near Impossibility.”
• Fung Institute, Berkeley, CA. February-March 2014. Lectures on Quantitative Marketing.
• Econ-CS Seminar, Berkeley, CA. February 2014. “Activity Bias, Near Impossibility.”
• NBER Winter IO, Stanford, CA. January 2014. “Near Impossibility.” Discussant presented.
• ASSA Economics of Digital Advertising Session, Philadelphia, PA, January 2014. “Near Impossibility” (session chair).
• Facebook Tech Talk, Menlo Park, CA. October 2013. “Near Impossibility.”
• Joint Statistical Meetings, Montreal, Quebec, Canada. August 2013. “Near Impossibility.” (Invited session: “Statistics: The Secret of Successful Web Giants.”)
• Economics of Advertising and Marketing @ Recanati, Tel Aviv, Israel. June 2013. “Near Impossibility.”
• Electronic Commerce 2013 (EC’13), Philadelphia, PA. June 2013. “Econometrics for CS,” “Super Bowl.”
• 9th Triennial Choice Symposium, Netherlands. June 2013. “Near Impossibility.”
• NBER Economics of Digitization Park City, UT. June 2013. “Digital Frontier” (attendee).
• Wharton CAI Measuring Ad Effectiveness Panel, Philadelphia, PA. May 16, 2013. “Location/Frequency”
• Wharton OPIM Seminar, Philadelphia, PA. April 23, 2013. “Near Impossibility.”
• Data, Society, and Inference Seminar Series @ UC Berkeley, Berkeley, CA. April 15, 2013. “Near Impossibility.”
• NBER SIEPR Digitization Conference, March 8, 2013. “Location/Frequency.”
• ASSA Digitization Session, San Diego, CA, January 5, 2013. “Northern Exposure.”
• FTC Microeconomics Conference, Washington, D.C., November 15-16, 2012. “Display & Search/Spillovers.”
• FTC Microeconomics Seminar, Washington, D.C. October 26, 2012. “Wear-Out” and “Location.”
• NBER SI 2012 Economics of IT and Digitization Workshop, Cambridge, MA, July 19-20, 2012. “Display & Search.”
• NBER Economics of Digitization: Preview Conference, Chicago, IL, June 28-29, 2012. “Digital Frontier.”
• Marketing Science Conference, Boston, MA, June 7-9, 2012. “Activity Bias.”
• Innovations and Big Data, San Jose, CA, March 1, 2012. “Econinformatics: Economics and Big Data.”
• UC Berkeley Shansby Marketing Seminar, Berkeley, CA, February 27, 2011. “Worn-Out” & “Cross-Media/Spillovers.”
• Yahoo! TechPulse (“Best Of” Session), December 2011. “Location.”
• BYU Economics and IDeA Labs Invited Seminars, Brigham Young University, Provo, UT, October 14, 2011. “Econinformatics Workshop,” “Near Impossibility,” “Econinformatics,” “Location.”
• Advances with Field Experiments, University of Chicago, Chicago, IL, October 29, 2011. “Near Impossibility.”
• Joint Statistical Meetings, Miami Beach, FL, August 2011. “Location.” (Invited session: “Statistics: The Secret of Successful Web Giants.”)
• Electronic Commerce 2011 (EC’11) Tutorial, San Jose, CA, June 6, 2011. “Measuring Online Ad Effectiveness.” (Joint with Deepak Kumar, David Reiley, and Taylan Yildiz.)
• SISHOO conference (Caltech/Yahoo!), Huntington Beach, CA, December 2010. “Frequency.”
• Yahoo! TechPulse, Santa Clara, CA, October 2010. “Frequency.”
• International Industrial Organization Conference, Vancouver, BC, May 2010.  “Ad Attribution” & “Northern Exposure.”
• SISHOO conference (Caltech/Yahoo!), Huntington Beach, CA, December 2009. “Ad Attribution.”
• Economic Science Association regional meeting, Tucson, AZ, November 2009.  “Ad Attribution.”
• Yahoo! TechPulse, San Jose, CA, October 2009. “Ad Attribution.”

---

PATENTS

• 2015 Isolated Simulations for Accurate Predictions of Counterfactual Events (filed)
• 2015 Ad Ranking Utilizing Bids and Adjustment Factors Based on the Causal Contribution of Ads (filed)
• 2014 Multi-treatment Experiments (filed)
• 2014 Determining Effectiveness of Content Items in a Computer Network Environment (filed)
• 2013 Time Slicing in Online Advertising Experimentation (filed)
• 2011 Audience Browser: A GUI that Displays Characteristics of an Advertising Audience Selected by Specifying Targeting Constraints (filed)
• 2011 Measuring Distance Based on Convenience in Online Advertising (filed)
• 2010 Online and offline advertising campaign optimization (filed)
• 2010 Offline metrics in advertisement campaign tuning (filed)
• 2010 Placeholder bids in online advertising (filed)

---

REFERENCES

PERSONAL

• Missionary, San Jose, Costa Rica Nov 2002 – Nov 2004
• USA Triathlon Junior National Champion September 2000
• Utah State Math Contest Winner April 1998, 1999, 2000