Randall A. Lewis
121 Albright Way
randall [AT] econinformatics.com
I am a “Big Data” measurement expert. As an applied econometrician, I use causal statistics to extract valuable insights from large data sets. In today’s digital economy, this requires inventing new types of measurement systems and cleverly adapting econometric algorithms to efficiently perform advanced analyses at scale (causal machine learning). I work with a range of econometric models from social networks to large-scale field experiments. I have primarily focused my skills on measuring the impact of online display and search advertising on important business outcomes such as clicks, page views, searches, survey outcomes, and both online and offline (in-store) sales in order to help human and automated machine learning decision systems make better choices.
Industrial-scale applied econometrics (econinformatics), statistical computing on MapReduce, advertising effectiveness & measurement system design, large-scale field experiments (A/B testing), behavioral & experimental economics, statistical inference on social networks, causal statistics, causal machine learning (LASSO, Ridge, Clustering, Neural Nets, Deep Learning via instrumental variables), nonparametric statistics, Bayesian statistics, adaptive learning mechanisms, item-response theory, industrial organization, user value estimation, advertising externalities, ad quality assessment, ad wear-out
Netflix Sept 2015 – Current
- Economic Research Scientist, Growth & Targeting, Science & Algorithms
Google, Inc. May 2012 – Sept 2015
- Senior Economic Research Scientist, Economics and Strategic Technologies
Yahoo! Research Labs June 2008 – May 2012
- Economic Research Scientist, Microeconomics and Social Systems
Massachusetts Institute of Technology
Ph.D. in Economics Sept 2006 – Sept 2010
Fields of Concentration: Industrial Organization, Econometrics
- Dissertation Title: “Measuring the Effects of Online Advertising on Human Behavior Using Natural and Field Experiments” (advisers: Glenn Ellison, Jerry Hausman)
Brigham Young University
B.A. in Economics (Valedictorian), B.S. in Mathematics Sept 2001 – June 2002, Jan 2005 – Aug 2006
HONORS, SCHOLARSHIPS, AND FELLOWSHIPS
2015 QME Journal Annual Dick Wittink Best Paper Award
2006 BYU Dept. of Economics Valedictorian
Expertise: SQL, Hive, Stata, Shell/Bash, Unix Core Utilities, R, Matlab, Perl, Google Apps & MS Office, LaTeX, LyX
Languages: English (native), Spanish (fluent)
- Garrett A. Johnson, Randall A. Lewis, and Elmar Nubbemeyer. “Ghost Ads: Improving the Economics of Measuring Ad Effectiveness.” Journal of Marketing Research (forthcoming).
- Garrett A. Johnson, Randall A. Lewis, and David H. Reiley. “When Less is More: Data and Power in Advertising Experiments.” Marketing Science, July 2016.
- Randall A. Lewis and Justin M. Rao. “The Unfavorable Economics of Measuring the Returns to Advertising.” Quarterly Journal of Economics, July 2015.
- Randall A. Lewis and Dan T. Nguyen. “Display Advertising’s Competitive Spillovers to Consumer Search.” Quantitative Marketing and Economics, June 2015.
- Randall A. Lewis and David H. Reiley, “Online Ads and Offline Sales: Measuring the Effects of Retail Advertising via a Controlled Experiment on Yahoo!” Quantitative Marketing and Economics, 12(3): 235-266 (September 2014).
- Anja Lambrecht, Avi Goldfarb, Alessandro Bonatti, Anindya Ghose, Dan Goldstein, Randall Lewis, Anita Rao, Navdeep Sahni, and Song Yao, “How Do Firms Make Money Selling Digital Goods Online?” Marketing Letters, 9th Triennial Choice Symposium, September 2014.
- Randall A. Lewis, Justin M. Rao, and David H. Reiley, “Measuring the Effects of Advertising: The Digital Frontier,” Economic Analysis of the Digital Economy, May 2015.
- Randall A. Lewis and David H. Reiley, “Advertising Effectively Influences Older Users: A Yahoo! Experiment Measuring Retail Sales,” Review of Industrial Organization, 44(2): 147-159 (March 2014).
- Randall A. Lewis and David H. Reiley, “Down-to-the-Minute Effects of Super Bowl Advertising on Online Search Behavior,” Proceedings of the 14th ACM Conference on Electronic Commerce (EC’2013), E. Tardos, R.P. McAfee, eds., June 2013.
- Randall A. Lewis and James B. McDonald. “Partially Adaptive Estimation of the Censored Regression Model,” Econometric Reviews (January 2013).
- Hector Garcia-Molina, Prabhakar Krishnamurthy, Randall Lewis, Panagiotis Papadimitriou, and David Reiley, “Display Advertising Impact: Search Lift and Social Influence.” 17th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD17), August 2011.
- Randall A. Lewis, Justin M. Rao, and David H. Reiley, “Here, There, and Everywhere: Correlated Online Behaviors Can Lead to Overestimates of the Effects of Advertising.” 20th International World Wide Web Conference (WWW20), April 2011.
- Hausman, Jerry A., Randall A. Lewis, Konrad Menzel, and Whitney K. Newey. 2011. “Properties of the CUE estimator and a modification with moments.” Journal of Econometrics, 165(1): 45-57.
- Nathan E Lewis, Gunnar Schramm, Aarash Bordbar, Jan Schellenberger, Michael P Andersen, Jeffrey Cheng, Nilam Patel, Alex Yee, Randall A Lewis, Roland Eils, Rainer König & Bernhard Ø Palsson. “Large-scale in silico modeling of metabolic interactions between cell types in the human brain.” Nature Biotechnology 28, 1279–1285 (November 2010).
- David H. Reiley, Sai-Ming Li, and Randall A. Lewis, “Northern exposure: a field experiment measuring externalities between search advertisements.” Proceedings of the 11th ACM Conference on Electronic Commerce (EC’2010), D. Parkes, C. Dellarocas, and M. Tennenholtz, eds., pp. 297-304.
- “The Online Display Ad Effectiveness Funnel & Carry-Over: A Meta-Study of Ghost Ad Experiments,” with Garrett A. Johnson and Elmar Nubbemeyer.
- “Cost Per Incremental Action: Efficient Pricing of Advertising,” with Garrett A. Johnson.
- “Brand Engagement Lift: Causal Effects of Skippable Video Ads on YouTube,” with Alejandro Veen, Jiangyong Yin, and Yi-Hsin Yeh.
- “Econometrics of Sampled Networks,” with Arun G. Chandrasekhar; submitted to Econometrica.
- “Location, Location, Location: Repetition and Proximity Increase Advertising Effectiveness,” with Garrett A. Johnson and David H. Reiley.
- “Worn-Out or Just Getting Started? The Impact of Frequency in Online Display Advertising,” revise & resubmit at Marketing Science.
- “Ad Attributes and Attribution: Large-Scale Field Experiments Measure Online Customer Acquisition” with David H. Reiley & Taylor A. Schreiner; revise & resubmit at Quantitative Marketing and Economics.
- “When Less is More: The Smallest Superset of Identification.”
CONFERENCE PRESENTATIONS & INVITED TALKS
- Conference on Data Science in the Entertainment Industries, Santa Monica, CA, August 18, 2016. “Ghost Ads.”
- 9th Workshop on the Economics of Advertising and Marketing, Vilnius, Lithuania, July 2016. “Ghost Ads.”
- NABE Big Data Conference, Boston, MA. June 2016. “Ghost Ads.”
- IDB Invited Seminar, Washington, D.C., June 2016. “Silicon Valley Experimentation” + “Ghost Ads” + “Econinformatics.”
- NABE Conference of Tech Economists, San Francisco, CA. April 2016. “Ghost Ads.”
- NBER SIEPR Digitization Conference, Stanford, CA. March 4, 2016. “Ghost Ads.”
- DataLead Workshop on Machine Learning and Economics, Paris, France, November 2015. “Ghost Ads.”
- AMA Advanced Research Technologies Forum, San Diego, CA, June 2015. “Unfavorable Economics.”
- ASSA Electronic Commerce and Big Data Session, Boston, MA, January 2015. “Worn-Out.”
- INFORMS Annual Conference, San Francisco, CA. November 2014. “Unfavorable Economics.”
- International Symposium on Recent Developments in Econometric Theory with Applications in honor of Jerry A. Hausman, Xiamen, China. June, 2014. “Econinformatics.”
- Marketing Science Conference, Atlanta, GA. June 2014. “Near Impossibility.”
- Fung Institute, Berkeley, CA. February-March 2014. Lectures on Quantitative Marketing.
- Econ-CS Seminar, Berkeley, CA. February 2014. “Activity Bias, Near Impossibility.”
- NBER Winter IO, Stanford, CA. January 2014. “Near Impossibility.” Discussant presented.
- ASSA Economics of Digital Advertising Session, Philadelphia, PA, January 2014. “Near Impossibility” (session chair).
- Facebook Tech Talk, Menlo Park, CA. October 2013. “Near Impossibility.”
- Joint Statistical Meetings, Montreal, Quebec, Canada. August 2013. “Near Impossibility.” (Invited session: “Statistics: The Secret of Successful Web Giants.”)
- Economics of Advertising and Marketing @ Recanati, Tel Aviv, Israel. June 2013. “Near Impossibility.”
- Electronic Commerce 2013 (EC’13), Philadelphia, PA. June 2013. “Econometrics for CS,” “Super Bowl.”
- 9th Triennial Choice Symposium, Netherlands. June 2013. “Near Impossibility.”
- NBER Economics of Digitization Park City, UT. June 2013. “Digital Frontier” (attendee).
- Wharton CAI Measuring Ad Effectiveness Panel, Philadelphia, PA. May 16, 2013. “Location/Frequency”
- Wharton OPIM Seminar, Philadelphia, PA. April 23, 2013. “Near Impossibility.”
- Data, Society, and Inference Seminar Series @ UC Berkeley, Berkeley, CA. April 15, 2013. “Near Impossibility.”
- NBER SIEPR Digitization Conference, March 8, 2013. “Location/Frequency.”
- ASSA Digitization Session, San Diego, CA, January 5, 2013. “Northern Exposure.”
- FTC Microeconomics Conference, Washington, D.C., November 15-16, 2012. “Display & Search/Spillovers.”
- FTC Microeconomics Seminar, Washington, D.C. October 26, 2012. “Wear-Out” and “Location.”
- NBER SI 2012 Economics of IT and Digitization Workshop, Cambridge, MA, July 19-20, 2012. “Display & Search.”
- NBER Economics of Digitization: Preview Conference, Chicago, IL, June 28-29, 2012. “Digital Frontier.”
- Marketing Science Conference, Boston, MA, June 7-9, 2012. “Activity Bias.”
- Innovations and Big Data, San Jose, CA, March 1, 2012. “Econinformatics: Economics and Big Data.”
- UC Berkeley Shansby Marketing Seminar, Berkeley, CA, February 27, 2011. “Worn-Out” & “Cross-Media/Spillovers.”
- Yahoo! TechPulse (“Best Of” Session), December 2011. “Location.”
- BYU Economics and IDeA Labs Invited Seminars, Brigham Young University, Provo, UT, October 14, 2011. “Econinformatics Workshop,” “Near Impossibility,” “Econinformatics,” “Location.”
- Advances with Field Experiments, University of Chicago, Chicago, IL, October 29, 2011. “Near Impossibility.”
- Joint Statistical Meetings, Miami Beach, FL, August 2011. “Location.” (Invited session: “Statistics: The Secret of Successful Web Giants.”)
- Electronic Commerce 2011 (EC’11) Tutorial, San Jose, CA, June 6, 2011. “Measuring Online Ad Effectiveness.” (Joint with Deepak Kumar, David Reiley, and Taylan Yildiz.)
- SISHOO conference (Caltech/Yahoo!), Huntington Beach, CA, December 2010. “Frequency.”
- Yahoo! TechPulse, Santa Clara, CA, October 2010. “Frequency.”
- International Industrial Organization Conference, Vancouver, BC, May 2010. “Ad Attribution” & “Northern Exposure.”
- SISHOO conference (Caltech/Yahoo!), Huntington Beach, CA, December 2009. “Ad Attribution.”
- Economic Science Association regional meeting, Tucson, AZ, November 2009. “Ad Attribution.”
- Yahoo! TechPulse, San Jose, CA, October 2009. “Ad Attribution.”
- 2015 Isolated Simulations for Accurate Predictions of Counterfactual Events (filed)
- 2015 Ad Ranking Utilizing Bids and Adjustment Factors Based on the Causal Contribution of Ads (filed)
- 2014 Multi-treatment Experiments (filed)
- 2014 Determining Effectiveness of Content Items in a Computer Network Environment (filed)
- 2013 Time Slicing in Online Advertising Experimentation (filed)
- 2011 Audience Browser: A GUI that Displays Characteristics of an Advertising Audience Selected by Specifying Targeting Constraints (filed)
- 2011 Measuring Distance Based on Convenience in Online Advertising (filed)
- 2010 Online and offline advertising campaign optimization (filed)
- 2010 Offline metrics in advertisement campaign tuning (filed)
- 2010 Placeholder bids in online advertising (filed)
*Email & Phone Numbers
- Missionary, Church of Jesus Christ of Latter-day Saints, San Jose, Costa Rica Nov 2002 – Nov 2004
- USA Triathlon Junior National Champion September 2000
- Utah State Math Contest Winner April 1998, 1999, 2000
Most of my research papers are available at my SSRN Author Page or via my Google Scholar Author Page.
“Advertising Especially Influences Older Users: A Yahoo! Experiment Measuring Retail Sales” (with Randall A. Lewis) Read the abstract. Download the manuscript. Download the supporting online materials. Last revised: January 2011.
“Online Ads and Offline Sales: Measuring the Effects of Online Advertising via a Controlled Experiment on Yahoo!” (with Randall A. Lewis) Read the abstract. Download the manuscript. Last revised: August 2013
“Add More Ads? Experimentally Measuring Incremental Purchases Due To Increased Frequency of Online Display Advertising” (with Garrett A. Johnson and Randall A. Lewis) Read the abstract. Download the manuscript. Last revised:April 2013.
“Ad Attributes and Attribution: Large-Scale Field Experiments Measure Online Customer Acquisition” (with Randall A. Lewis and Taylor A. Schreiner) Read the abstract. Download the manuscript. Last revised: March 2011.
On the Near Impossibility of Measuring the Returns to Advertising – Alternative Link (Old Version) joint with Justin Rao. New version 12/2013. (very dated) slides available here (very dated) video available under Talk Videos at justinmrao.com.
Here, There and Everywhere: Correlated Online Behaviors Can Lead to Overestimates of the Effects of Advertising (local copy) (joint with Randall Lewis and David Reiley). Proceedings of World Wide Web Conference 2011 Research Papers
How Do Firms Make Money Online? – Lambrecht, Anja and Goldfarb, Avi and Bonatti, Alessandro and Ghose, Anindya and Goldstein, Daniel G. and Lewis, Randall Aaron and Rao, Anita and Sahni, Navdeep and Yao, Song, How Do Firms Make Money Online? (September 29, 2013).