Skip to main content
Springer Nature Link
Log in

Flexible estimation of time-varying effects for frequently purchased retail goods: a modeling approach based on household panel data

  • Regular Article
  • Published:
OR Spectrum Aims and scope Submit manuscript

Abstract

In most previous applications of brand choice models, possible time-varying effects in consumer behavior are ignored by merely imposing constant parameters. However, it is very likely that trends or short-term variations in consumers’ intrinsic brand utilities or sensitivities to marketing instruments occur. For example, preferences for specific brands or price elasticities in product categories such as coffee or chocolate may vary in the run-up to festive occasions like Easter or Christmas. In this paper, we employ flexible multinomial logit models for estimating time-varying effects in brand choice behavior. Time-varying brand intercepts and time-varying effects of covariates are modeled using penalized splines, a flexible, yet parsimonious, nonparametric smoothing technique. The estimation is data driven; the flexible functions, as well as the corresponding degrees of smoothness, are determined simultaneously in a unified approach. Our model further allows for alternative-specific time-varying effects of covariates and can mimic state-space approaches with random walk parameter dynamics. In an empirical application for ground coffee, we compare the performance of the proposed approach to a number of benchmark models regarding in-sample fit, information criteria, and in particular out-of-sample fit. Interestingly, the most complex P-spline model with time-varying brand intercepts and brand-specific time-varying covariate effects outperforms all other specifications both in- and out-of-sample. We further present results from a sensitivity analysis on how the number of knots and other P-spline settings affect the model performance, and we provide guidelines for the model building process about the many options for model specification using P-splines. Finally, the resulting parameter paths provide valuable insights for marketing managers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. We focus on brand choice here because most of the effects of marketing variables (e.g., price, promotion activities) on demand can be attributed to changes in secondary demand (i.e., brand switching) as opposed to primary demand (i.e., purchase incidence and purchase quantity). See in particular Gupta (1988) and Bell et al. (1999).

  2. Please note that even though the latent utility is a linear function of the covariates, the choice probability is a nonlinear function of the latent utility in the MNL model [see Eq. (3)]. Therefore, marginal effects response variables (e.g., demand or market share) are nonlinear functions of the covariates (see Train 2009, pp. 57–59).

  3. Using normally distributed error terms would lead to the multinomial probit (MNP) model (Train 2009; Paetz and Steiner 2017). The probit model, if correlations are allowed, does not exhibit the IIA assumption. However, it lacks a closed-form solution for choice probabilities, and thus, estimation is much more complicated and time-consuming.

  4. From our experience, \( M = 40 \) generally works well for datasets with weekly data and a time span of 52–78 weeks. For datasets with longer time windows, we suggest starting with at least \( M = 0.3 \cdot {\rm T} \) as a default value to ensure sufficient flexibility.

  5. Note that in case of brand choice data only discrete responses are available and the continuous utility values are latent.

  6. We focus on continuous parameter evolutions instead of particular discrete states and consequently do not consider hidden Markov models or regime-switching models (see, e.g., Poulsen 1990; Netzer et al. 2008; Park and Gupta 2011).

  7. Note that retailers have nowadays access to similar datasets at the disaggregated level given the rise of loyalty card programs in which individual transactions can be associated with households.

  8. We divided the brands into two classes (brands 1, 2, and 4 versus brands 3 and 5) to test whether estimation results stay robust as compared to the full set of brands. We estimated both MNL models with constant parameters and MNL models with time-varying parameters for the different subsets of brands. Our results indicate that differences between estimated parameters and parameter paths are not overly sensitive to the particular sets of brands, implying that the IIA property inherent to models of the MNL type is not a major concern in our application. Detailed results can be obtained from the authors upon request.

  9. Since observed prices and the use or non-use of promotions are strongly correlated, and as a result, the correlations between promotions and related behavioral price terms like reference prices, gains, and losses (see Sect. 3.2) also turn out substantial, we do not explicitly consider information on promotions for model estimation to prevent multicollinearity problems.

  10. Note that purchases of multiple packs (even of the same brand) are considered in our dataset as multiple single brand choices, however we do not account for quantity effects (in the sense that consumers may have chosen different pack sizes) in our market share calculations. Given that most coffee purchases are single-unit purchases of the most popular package size of 500 g, we argue that our market share calculations are (fairly) representative. On the other hand, we acknowledge that this is a limitation of our empirical study and incorporation of primary demand effects would be desirable for applications in a retailing context. We thank one reviewer for pointing this out.

  11. Via the smoothing parameter, we can continuously vary the effective number of parameters between the total number of basis functions (smoothing parameter equal to zero) and the dimension of the polynomial that is left unpenalized by the difference order (corresponding to a total of r parameters for rth-order differences). For example, with 50 basis functions and a second-order difference penalty, the effective number of parameters can vary continuously in the range between 2 and 50. The effective number of parameters can then be determined as the trace of the product between the unpenalized Fisher information and the inverse of the penalized Fisher information, see Gray (1992) for details.

  12. All models based on P-splines were estimated using the free software package BayesX (Belitz et al. 2015), which also has a convenient R-interface R2BayesX (Umlauf et al. 2015). We ran the models on a Windows computer with Intel Xeon Processor E5-1650 v3 (15 M Cache, 3.50 GHz) and 32 GB RAM.

  13. We thank one reviewer for suggesting this model extension.

  14. We thank one reviewer for suggesting this sensitivity analysis. The full results are available from the authors upon request.

  15. We have further estimated corresponding model versions with time-varying brand intercepts only and with time-varying covariate effects only. However, these versions showed a worse performance (in- and out-of-sample), thus replicating the order of models as in Sect. 3.4.1. Results are available from the authors upon request.

  16. We thank one reviewer for pointing us to this model extension.

References

  • Abe M (1998) Measuring consumer, nonlinear brand choice response to price. J Retail 74:541–568

    Article  Google Scholar 

  • Abe M (1999) A generalized additive model for discrete choice data. J Bus Econ Stat 17:271–284

    Google Scholar 

  • Ailawadi KL, Gedenk K, Neslin SA (1999) Heterogeneity and purchase event feedback in choice models: an empirical analysis with implications for model building. Int J Res Mark 16:177–198

    Article  Google Scholar 

  • Baltas G, Doyle P (2001) Random utility models in marketing research: a survey. J Bus Res 51:115–125

    Article  Google Scholar 

  • Baumgartner B (2003) Measuring changes in brand choice behavior. Schmalenbach Bus Rev 55:242–256

    Article  Google Scholar 

  • Belitz C, Brezger A, Klein N, Kneib T, Lang S, Umlauf N (2015) BayesX: software for Bayesian inference in structured additive regression models. http://www.BayesX.org/. Accessed 28 Feb 2017

  • Bell DR, Lattin JM (2000) Looking for loss aversion in scanner panel data: the confounding effect of price response heterogeneity. Market Sci 19:185–200

    Article  Google Scholar 

  • Bell DR, Chiang J, Padmanabhan V (1999) The decomposition of promotional response: an empirical generalization. Market Sci 18:504–526

    Article  Google Scholar 

  • Brezger A, Steiner WJ (2008) Monotonic regression based on bayesian p-splines: an application to estimating price response functions from store-level scanner data. J Bus Econ Stat 26:90–104

    Article  Google Scholar 

  • Briesch R, Krishnamurthi L, Mazumdar T, Raj SP (1997) A comparative analysis of reference price models. J Consum Res 24:202–214

    Article  Google Scholar 

  • Chandukala SR, Kim J, Otter T, Rossi PE, Allenby GM (2008) Choice models in marketing: economic assumptions, challenges and trends. Found Trends Mark 2:97–184

    Article  Google Scholar 

  • Chintagunta PK, Dubé JP, Goh KY (2005) Beyond the endogeneity bias: the effect of unmeasured brand characteristics on household-level brand choice models. Manag Sci 51:832–849

    Article  Google Scholar 

  • De Boor C (2001) A practical guide to splines. Springer, New York (Revised edition)

    Google Scholar 

  • Dubé JP, Hitsch GJ, Rossi PE (2009) Do switching costs make markets less competitive? J Market Res 46:435–445

    Article  Google Scholar 

  • Eilers PHC, Marx BC (1996) Flexible smoothing with B-splines and penalties (with comments and rejoinder). Stat Sci 11:89–121

    Article  Google Scholar 

  • Elshiewy O, Guhl D, Boztug Y (2017) Multinomial logit models in marketing—from fundamentals to state-of-the-art. Mark ZFP 39:32–55

    Article  Google Scholar 

  • Erdem T, Keane MP, Sun B (2008) The impact of advertising on consumer price sensitivity in experience goods markets. Quant Mark Econ 6:139–176

    Article  Google Scholar 

  • Fahrmeir L, Kneib T, Lang S (2004) Penalized structured additive regression for space-time data: a Bayesian perspective. Stat Sinica 14:731–761

    Google Scholar 

  • Fahrmeir L, Kneib T, Lang S, Marx B (2013) Regression: models, methods and applications. Springer, Heidelberg

    Book  Google Scholar 

  • Foekens EW, Leeflang PSH, Wittink DR (1999) Varying parameter models to accommodate dynamic promotion effects. J Econ 89:249–268

    Article  Google Scholar 

  • Gneiting T, Raftery AE (2007) Strictly proper scoring rules, prediction, and estimation. J Am Stat Assoc 102:359–378

    Article  Google Scholar 

  • Gordon BR, Goldfarb A, Li Y (2013) Does price elasticity vary with economic growth? A cross-category analysis. J Market Res 50:4–23

    Article  Google Scholar 

  • Gray RJ (1992) Flexible methods for analyzing survival data using splines, with application to breast cancer prognosis. J Am Stat Assoc 87:942–951

    Article  Google Scholar 

  • Guadagni PM, Little JDC (1983) A logit model of brand choice calibrated on scanner data. Market Sci 2:203–238

    Article  Google Scholar 

  • Guhl D, Baumgartner B, Kneib T, Steiner W (2018) Estimating time-varying parameters in brand choice models: a semiparametric approach. Int J Res Mark 35:394–414

    Article  Google Scholar 

  • Gupta S (1988) Impact of sales promotions on when, what, and how much to buy. J Market Res 25:342–355

    Article  Google Scholar 

  • Haupt H, Kagerer K, Steiner WJ (2014) Smooth quantile-based modeling of brand sales, price and promotional effects from retail scanner panels. J Appl Econ 29:1007–1028

    Article  Google Scholar 

  • Heilman CM, Bowman D, Wright GP (2000) The evolution of brand preferences and choice behaviors of consumers new to a market. J Market Res 37:139–155

    Article  Google Scholar 

  • Jain DC, Vilcassim NJ, Chintagunta PK (1994) A random-coefficients logit brand-choice model applied to panel data. J Bus Econ Stat 12:317–328

    Google Scholar 

  • Jedidi K, Mela CF, Gupta S (1999) Managing advertising and promotion for long-run profitability. Market Sci 18:1–22

    Article  Google Scholar 

  • Kahneman D, Tversky A (1979) A prospect theory: an analysis of decisions under risk. Econometrica 47:263–291

    Article  Google Scholar 

  • Kalyanaram G, Little JDC (1994) An empirical analysis of latitude of price acceptance in consumer package goods. J Consum Res 21:408–418

    Article  Google Scholar 

  • Kalyanaram G, Winer RS (1995) Empirical generalizations from reference price research. Market Sci 14(Part 2 of 2):G161–G169

    Article  Google Scholar 

  • Kamakura WA, Russel GJ (1989) A probabilistic choice model for market segmentation and elasticity structure. J Market Res 26:379–390

    Article  Google Scholar 

  • Kim JG, Menzefricke U, Feinberg FM (2005) Modeling parametric evolution in a random utility framework. J Bus Econ Stat 23:282–294

    Article  Google Scholar 

  • Kneib T, Fahrmeir L (2006) Structured additive regression for categorical space-time data: a mixed model approach. Biometrics 62:109–118

    Article  Google Scholar 

  • Kneib T, Baumgartner B, Steiner WJ (2007) Semiparametric multinomial logit models for analysing consumer choice behaviour. Adv Stat Anal 91:225–244

    Article  Google Scholar 

  • Kopalle PK, Mela CF, Marsh L (1999) The dynamic effect of discounting on sales: empirical analysis and normative pricing implications. Market Sci 18:317–332

    Article  Google Scholar 

  • Kopalle PK, Kannan PK, Boldt LB, Arora N (2012) The impact of household level heterogeneity in reference price effects on optimal retailer pricing policies. J Retail 88:102–114

    Article  Google Scholar 

  • Lachaab M, Ansari A, Jedidi K, Trabelsi A (2006) Modeling preference evolution in discrete choice models: a Bayesian state-space approach. Quant Mark Econ 4:57–81

    Article  Google Scholar 

  • Lang S, Steiner WJ, Weber A, Wechselberger P (2015) Accommodating heterogeneity and nonlinearity in price effects for predicting brand sales and profits. Eur J Oper Res 246:232–241

    Article  Google Scholar 

  • Lattin JM, Bucklin RE (1989) Reference effects of price and promotion on brand choice behavior. J Market Res 26:299–310

    Article  Google Scholar 

  • Leeflang PSH, Bijmolt THA, van Doorn J, Hanssens DM, van Heerde HJ, Verhoef PC, Wieringa JE (2009) Creating lift versus building the base: current trends in marketing dynamics. Int J Res Mark 26:13–20

    Article  Google Scholar 

  • Leeflang PSH, Wieringa JE, Bijmolt THA, Pauwels KH (2015) Modeling markets—analyzing marketing phenomena and improving marketing decision making. Springer, New York

    Google Scholar 

  • McFadden D (1974) Conditional logit analysis of qualitative choice behavior. In: Zarembka P (ed) Frontiers in econometrics. Academic Press, New York, pp 105–142

    Google Scholar 

  • Mela CF, Gupta S, Lehmann DR (1997) The long-term impact of promotion and advertising on consumer brand choice. J Market Res 34:248–261

    Article  Google Scholar 

  • Netzer O, Lattin JM, Srinivasan V (2008) A hidden Markov model of customer relationship dynamics. Market Sci 27:185–204

    Article  Google Scholar 

  • Paetz F, Steiner WJ (2017) The benefits of incorporating utility dependencies in finite mixture probit models. OR Spectr 39:793–819

    Article  Google Scholar 

  • Papatla P, Krishnamurti L (1996) Measuring the dynamic effects of promotions on brand choice. J Market Res 33:20–35

    Article  Google Scholar 

  • Park S, Gupta S (2011) A regime-switching model of cyclical category buying. Market Sci 30:469–480

    Article  Google Scholar 

  • Petrin A, Train K (2010) A control function approach to endogeneity in consumer choice models. J Market Res 47:3–13

    Article  Google Scholar 

  • Poulsen CS (1990) Mixed Markov and latent Markov modelling applied to brand choice behaviour. Int J Res Mark 7:5–19

    Article  Google Scholar 

  • Rossi PE (2014) Even the rich can make themselves poor: a critical examination of IV methods in marketing applications. Market Sci 33:655–672

    Article  Google Scholar 

  • Ruppert D, Wand MP, Carroll RJ (2003) Semiparametric regression. Cambridge University Press, Cambridge

    Book  Google Scholar 

  • Rutz OJ, Sonnier GP (2011) The evolution of internal market structure. Market Sci 30:274–289

    Article  Google Scholar 

  • Sriram S, Balachander S, Kalwani MU (2007) Monitoring the dynamics of brand equity using store-level data. J Marketing 71:61–78

    Article  Google Scholar 

  • Steiner WJ, Brezger A, Belitz C (2007) Flexible estimation of price response function using retail scanner data. J Retail Consum Serv 14:383–393

    Article  Google Scholar 

  • Train K (2009) Discrete choice models with simulation, 2nd edn. Cambridge University Press, New York

    Book  Google Scholar 

  • Umlauf N, Adler D, Kneib T, Lang S, Zeileis A (2015) Structured additive regression models: an R interface to BayesX. J Stat Softw 63:1–46

    Article  Google Scholar 

  • van Heerde HJ, Leeflang PSH, Wittink DR (2004) Decomposing the sales promotion bump with store data. Market Sci 23:317–334

    Article  Google Scholar 

  • Villas-Boas JM, Winer RS (1999) Endogeneity in brand choice models. Manage Sci 45:1324–1338

    Article  Google Scholar 

  • Weber A, Steiner W (2012) Zur Berücksichtigung von Heterogenität versus funktionaler Flexibilität in Absatzreaktionsmodellen: Eine empirische Studie auf Basis von Handelsdaten. Z Betriebswirt 82:1337–1365

    Article  Google Scholar 

  • Weber A, Steiner WJ, Lang S (2017) A comparison of semiparametric and heterogeneous store sales models for optimal category pricing. OR Spectrum 39:403–445

    Article  Google Scholar 

  • Winer RS (1986) A reference price model of brand choice for frequently purchased products. J Consum Res 13:250–256

    Article  Google Scholar 

  • Wood SN (2017) Generalized additive models: an Introduction with R. Chapman and Hall/CRC, London

    Book  Google Scholar 

Download references

Acknowledgements

We thank two anonymous reviewers for their insightful comments and suggestions that helped to improve the paper considerably.

Author information

Author notes

  1. All authors contributed equally to this research.

Authors and Affiliations

  1. Department of Marketing, University of Osnabrück, Rolandstraße 8, 49069, Osnabrück, Germany

    Bernhard Baumgartner

  2. School of Business and Economics, Institute of Marketing, Humboldt University Berlin, Spandauer Straße 1, 10178, Berlin, Germany

    Daniel Guhl

  3. Department of Statistics and Econometrics, Georg-August-Universität Göttingen, Humboldtallee 3, 37073, Göttingen, Germany

    Thomas Kneib

  4. Department of Marketing, Clausthal University of Technology, Julius-Albert-Straße 2, 38678, Clausthal-Zellerfeld, Germany

    Winfried J. Steiner

Authors
  1. Bernhard Baumgartner
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Daniel Guhl
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Thomas Kneib
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Winfried J. Steiner
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Winfried J. Steiner.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Baumgartner, B., Guhl, D., Kneib, T. et al. Flexible estimation of time-varying effects for frequently purchased retail goods: a modeling approach based on household panel data. OR Spectrum 40, 837–873 (2018). https://doi.org/10.1007/s00291-018-0530-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00291-018-0530-6

Keywords