- Link to scholarly article: Giuliani M. & Madama I. (2022) What if? Using counterfactuals to evaluate the effects of structural labor market reforms: evidence from the Italian Jobs Act, Policy studies, DOI: 10.1080/01442872.2022.2039385
"I wonder how it would have turned out if I hadn't....". How many times have you had this thought?
It applies to people and their individual choices (getting married, enrolling for a doctorate, accepting a job offer, moving abroad). It applies to political systems and interstate relations (reforming an electoral system, making an international agreement, imposing economic sanctions, conferring the right to bear arms). And it also applies to public policies such as the one examined here: the Jobs Act introduced in Italy by the Renzi government.
The problem with this reasoning is that it is not possible to know what would have happened if we had refused that job, if the UK had remained in the European Union or if NATO had established a no-fly zone over the skies of Ukraine. This is the curse of the counterfactual. To understand the effects of a decision, we need to know what would have happened if we had not made it.
A perfect example of a counterfactual is what happens to Gwyneth Paltrow in the film Sliding Doors. A minor mishap in descending the stairs to catch the subway train causes two entirely different storylines to start. In the first, the actress misses the train, and her personal story continues on a track already laid out. In the second, she manages to board the train with one last attempt. But she finds a surprise waiting for her at home, and a new storyline unfolds. Same person, same past. Only that one small event makes the difference. It is John Stuart Mill’s "method of difference". It makes it possible, on the one hand, to identify the cause of a given phenomenon and – on the other –– by comparing two otherwise perfectly identical worlds – to assess the effects of given events, choices, or policies.
Rewinding the tape of history or politics, however, is not possible. The counterfactual remains inaccessible. This is why opinions so often diverge on the effectiveness of a public measure, whether it is a foreign policy decision, a labor market intervention, or an anti-Covid restriction.
Take the Italian Jobs Act, for example.
The Italian labor market before the reform
This was a policy of flexibilization of labor relations centered on the so-called ‘contratto a tutele crescenti’ (literally, a contract with increasing protection) introduced by the Renzi government at the turn of 2014-2015. One of its main goals was to reduce labor market segmentation and thereby boost employment. And to do so, especially for young people and women, categories whose employment levels have historically lagged much behind those in Italy’s European partners.
As regards the employment rate, around 2015, there was an increase in the percentage of employed people, which grew steadily thereafter until 2020, when it halted with the onset of the Covid-19 pandemic. Several politicians, observers, and even international organizations have commented on this success by connecting it with the new strategy introduced by the Jobs Act.
The method: artificial parallel worlds, a ‘synthetic’ counterfactual case
But have they been correct? To answer this question, we need a counterfactual case: an Italy without a Jobs Act that would retrace those years and thus be able to give us a measure of the difference due to Renzi's strongly desired reform. In the absence of a counterfactual, the comparative method suggests selecting the country most similar to Italy and using it as a mirror capable of highlighting the differences we are looking for. What is that country? Does it exist? In some ways, Italy can be considered similar to other southern European countries: it is comparable to Spain, for example, in total employment trends, but not in the share of fixed-term contracts; it resembles Portugal in the incidence of unemployment, but not in female employment rates; and again, it is comparable to Greece in the dynamics of female employment, but not in the incidence of youth unemployment. Even in these cases, these are always imprecise similarities, valid only for certain stages and specific aspects.
A ‘paired comparison’ – a comparison between pairs of somewhat similar things – is not always feasible: nor is it the best answer to the comparison problems raised by the search for a counterfactual. Other worlds are not quite sufficiently similar to our own. It is difficult to reject the suspicion that it is precisely such differences that make the difference: that is, which give rise to specific labor market dynamics.
Instead of identifying a priori a country that is ‘sufficiently similar’, a viable option is to be guided by the data and artificially construct a counterfactual that results from combining different countries. The best combination will be the one best able to approximate the Italian dynamics of the variable of interest in the pre-reform phase, thus proving to be an effective counterfactual. Such a ‘synthetic’ counterfactual – indeed, this technique is called the ‘synthetic control method’ – will then be the benchmark used to evaluate the effects of the policy in subsequent years. In our case, these are the effects of the Jobs Act.
Young people, women, employment, and fixed-term contracts
We applied this methodology to several labor market indicators that reflect the main expectations associated with the introduction of the Jobs Act: the employment rate, the percentage of fixed-term contracts, the employment rate among young people in the 15-24 age group, and the employment rate among women. The countries from which we started constructing the synthetic counterfactual were all those currently in the European Union.
The graphs (Figure 1) show the dynamics of the indicators just listed between 2003 and 2019. The blue solid line depicts the actual figure for Italy recorded by Eurostat, while the dark red long-dashed line represents its synthetic counterfactual, i.e., the best possible approximation to the pre-Jobs Act dynamics resulting from the weighted linear combination of the data recorded for the other EU countries.
As can be seen, before the Renzi government’s measure, depicted by the vertical line, the two lines are close and have very similar trends, signaling the good ability of the synthetic counterfactual to represent Italy (technically, to minimize the mean square error of the RMSPE prediction). This furnishes sufficient guarantees to interpret the same line in the years from 2015 onward as what would have happened in Italy if the Jobs Act had not been introduced: i.e. it is the counterfactual we were looking for in order to assess the effects of the policy.
The results: at best, it didn't work
Italy's employment rate line does indeed grow from 2015, but – as can be seen in Figure 1 in the top left graph – so too does the counterfactual line. Indeed, the latter rears up with a growth rate higher than the real one. Hence it can be inferred that the Jobs Act not only failed to promote recovery but, on the contrary, slowed it down. As regards fixed-term contracts, which enable us to assess the dualism of the labor market, the situation is even worse. The actual figure shows a growth in fixed-term contracts when the Jobs Act was instead intended to promote permanent ones, while without it the spread of temporary workers would have remained fairly constant. As regards the youth employment rate, Renzi's policy had no effect, as shown by the proximity before and after 2015 of the two lines in the bottom left graph. Finally, female employment grew in the second part of the decade, but without the Jobs Act, it would probably have grown even more.
The gaps shown between the actual dynamics and those of the synthetic counterfactual could be statistical artifacts of the procedure used, which is why the methodology includes a whole series of so-called ‘placebo tests’ in order to rule out that possibility. Our checks confirmed that 4 years after the introduction of the Jobs Act, disparities were statistically significant on the first two indicators – employment rates and percentage of fixed-term contracts – while they were not so for the second two – youth and female employment. Even in regard to the latter, however, it must be categorically ruled out that the Jobs Act could have had any positive effect. This evidence can only fuel further reflections on labor market flexibilization strategies, starting with divergent hypotheses in the literature in this regard (for those interested, more details can be found in the scholarly article).
For Italy, the resulting landscape is anything but comforting, especially when compared with statements, expectations, and early encouraging comments – and with the research expectations of at least one of the authors. This is the joke that counterfactuals can pull.
