This study analysed 700 million grocery store transactions. Here’s what we learned about consumer behavior

In recent decades, the choice of product varieties available to consumers has exploded, even though the retail sector has become dominated by few very large firms. This column shows that this changing environment has led households to concentrate their spending on a few preferred products, which often differ from those consumed by their neighbours. Using data from 700 million grocery store transactions and a novel economic framework, it shows that this development is driven by households selecting products that better match their particular tastes, with large welfare gains.

In recent decades, important changes have taken place in how goods are designed, produced, marketed, and delivered. Numerous mergers and acquisitions have led to an increasing concentration in the retail sector, and production throughout the economy has become dominated by a small number of ‘superstar’ firms (Autor et al. 2017). This has led to concerns about rising market power and mark-ups (DeLoecker and Eeckout 2017).

At the same time, anecdotal experience suggests an explosion of consumer product choice. Decades ago, the mustard market was dominated by French’s yellow mustard, while today the marketplace has hundreds of products in a plethora of styles (Gladwell 2004). Yoplait and Dannon now share the marketplace with a range of Greek and probiotic yogurt products. While Tostitos was originally offered in a single style, these now range from “Scoops” to “Cantina” in a variety of flavours.

In Neiman and Vavra (2019), we use data from 700 million individual household grocery store transactions from 2004 to 2016 to explore how the rapidly changing retail environment has manifested itself in individual household product spending patterns. We show that, over this time period, the typical household has increasingly concentrated its spending on a declining number of products. However, this is not driven by superstar products capturing larger aggregate market shares or by more limited selection of product availability. Instead, each household increasingly concentrates its spending on its own preferred products, but households also increasing differ in which products they consume. As a result, the concentration of aggregate spending on these same products, calculated by pooling expenditures across all households, has actually decreased over this same period. Figure 1 shows this divergence between household and aggregate concentration trends and the implied increase in the fragmentation of the product space, which we refer to as a rise in niche consumption.

Figure 1 Product spending concentration

We illustrate this more concretely with one of the previous examples. Tostitos now complements its “Traditional” line of chips with products such as “Scoops” and “Cantina”, but most households tend to concentrate purchases in one of these varieties, rather than spreading evenly over the three. This means that individual households concentrate their spending on a smaller number of products within the tortilla chips market, even as total spending has become more fragmented. This example suggests that the simultaneous rise in household spending concentration and decline in aggregate spending concentration is driven in part by the introduction of new products which cater to heterogeneous preferences. Consistent with this observation, we find that niche consumption has risen most rapidly in the categories and in the particular retail stores where product availability has increased most rapidly.

However, this empirical evidence does not rule out other possible explanations for these trends, such as changing household demand or search costs. Furthermore, it is difficult to assess the consequences of rising niche consumption for welfare and market power from this empirical evidence alone. In the second half of Neiman and Vavra (2019), we thus develop a formal model to explore these issues. Many standard models which assume identical households, symmetric preferences, or that households purchase a single product are unable to speak to the empirical trends we document. In our model, households choose how many products to consume, spend different amounts on each good, and differ from other households in their choice of which products to buy. Nevertheless, our model delivers simple closed-form analytical expressions for both household and aggregate concentration in terms of easily interpretable structural parameters.

Through the lens of the model, we can then precisely characterise the underlying structural changes required to match the rise of niche consumption. Consistent with the suggestive empirical evidence, our model implies a crucial role for increases in product variety in driving the rise of niche consumption. Furthermore, we find that this increase in product variety has led to substantial welfare gains. Overall, between 2004 and 2016, greater product availability has increased welfare by roughly 8%. Interestingly, the underlying source of these welfare gains is quite distinct from the typical ‘gains from variety’, which arise in standard models that abstract from heterogeneity. In our model, differences across households in their tastes for different products is key to the welfare gains from variety. An expansion of the set of products available lets households purchase a set of goods which is more tailored to their idiosyncratic preferences. That is, the availability of a greater variety of products leads to greater ability to select products well-suited to individual tastes. These variety gains are absent from standard models, and they are generally not captured by government growth statistics. Our results imply that these effects are large. An 8% increase from 2004-2016 translates to roughly 0.6% unmeasured growth per year.

Our model also has interesting implications for mark-ups and market power. It features heterogeneous mark-ups because producers of popular products care more about maximising profits from existing customers, while producers of smaller niche products care more about expanding their customer base. Surprisingly, however, our model can match the observed trends in household and aggregate concentration without any resulting change in aggregate market power. This reflects two opposing forces. First, households are willing to pay higher prices for new niche products, which better cater to their individual preferences, leading to an increase in market power and mark-ups. Second, at the same time, greater competition from a wider range of products reduces overall market power and mark-ups. In our model, these forces exactly offset, so aggregate market power remains unchanged, even though there are dramatic changes in both household and aggregate product concentration. In the spirit of Syverson (2019) and Berry et al. (2019), therefore, our model cautions against making quick inference about trends in market power from trends in concentration.

Furthermore, we show that mark-ups will vary across products as well as across households, since different households consume very different products. The existing literature has shown empirical evidence that mark-ups vary across households with broad characteristics related to shopping behaviour (e.g. Stroebel and Vavra 2019). However, our model suggests that mark-ups likely vary across households even within narrow demographic groups. This suggests that there may be interesting interactions between household characteristics, product entry and mark-ups. How do firms choose which products to introduce? When is it preferable to target particular household niches as compared to introducing more mass-market products? How do these decisions interact with advertising and marketing, and how will they interact with the continued growth of the online retail sector? We think these are interesting questions that we hope to pursue in future work.