A genuinely difficult question is this: does the idea of a rigorous "social science" really make sense, given what we know of the nature of the social world, the nature of human agency, and the nature of historical change?
There are of course large areas of social inquiry that involve genuine observation and measurement: demography, population health statistics, survey research, economic activity, social statistics of various kinds. Part of science is careful observation of a domain and analysis of the statistical patterns that emerge; so it is reasonable to say that demography, public health, and opinion research admit of rigorous empirical treatment.
Second, it is possible to single out complex historical events or processes for detailed empirical and historical study: the outbreak of WWI, the occurrence and spread of the Spanish influenza epidemic, the rise of authoritarian populism in Europe. Complex historical events like these admit of careful evidence-based investigation, designed to allow us to better understand the sequence of events and circumstances that made them up. And we can attempt to make sense of the connections that exist within such sequences, whether causal, cultural, or semiotic.
Third, it is possible to identify causal connections among social events or processes: effective transportation networks facilitate the diffusion of ideas and germs; price rises in a commodity result in decreases in consumption of the commodity; the density of an individual's social networks influences the likelihood of career success; etc. It is perfectly legitimate for social researchers to attempt to identify these causal connections and mechanisms, and further, to understand how these kinds of causal influence work in the social world. A key goal of science is explanation, and the kinds of inquiry mentioned here certainly admit of explanatory hypotheses. So explanation, a key goal of science, is indeed feasible in the social realm.
Fourth, there are "system" effects in the social world: transportation, communication, labor markets, electoral systems -- all these networks of interaction and influence can be seen to have effects on the pattern of social activity that emerge in the societies in which they exist. These kinds of effects can be studied from various points of view -- empirical, formal, simulations, etc. These kinds of investigation once again can serve as a basis for explanation of puzzling social phenomena.
This list of legitimate objects of empirical study in the social world, resulting in legitimate and evidence-based knowledge and explanation, can certainly be extended. And if being scientific means no more than conducting analysis of empirical phenomena based on observation, evidence, and causal inquiry, then we can reasonably say that it is possible to take a scientific attitude towards empirical problems like these.
But the hard question is whether there is more to social science than a fairly miscellaneous set of results that have emerged through study of questions like these. In particular, the natural sciences have aspired to formulating fundamental general theories that serve to systematize wide ranges of natural phenomena -- the theory of universal gravitation or the theory of evolution through natural selection, for example. The goal is to reduce the heterogeneity and diversity of natural phenomena to a few general theoretical hypotheses about the underlying reality of the natural world.
Are general theories like these possible in the social realm?
Some theorists have wanted to answer this question in the affirmative. Karl Marx, for example, believed that his theory of the capitalist mode of production provided a basis for systematizing and explaining a very wide range of social data about the modern social world. It was this supposed capacity for systematizing the data of the modern world that led Marx to claim that he was providing a "science of society".
But it is profoundly dubious that this theory, or any similarly general theory, can play the role of a fundamental theory of the social world, in the way that perhaps electromagnetic theory or quantum mechanics play a fundamental role in understanding the natural world.
The question may seem unimportant. But in fact, to call an area of inquiry "science" brings some associations that may not be at all justified in the case of study of the social world. In particular, science is often thought to be comprehensive, predictive, and verifiable. But knowledge of the social world falls short in each of these ways. There is no such thing as a comprehensive or foundational social theory, much as theorists like Marx have thought otherwise. Predictions in the social realm are highly uncertain and contingent. And it is rare to have a broad range of social data that serves to "confirm" or "verify" a general social theory.
Here is one possible answer to the question posed above, consistent with the points made here. Yes, social science is possible. But what social science consists in is an irreducible and pluralistic family of research methods, observations, explanatory hypotheses, and mid-level theories that permit only limited prediction and that cannot in principle serve to unify the social realm under a single set of theoretical hypotheses. There are no grand unifying theories in the social realm, only an open-ended set of theories of the middle range that can be used to probe and explain the social facts we can uncover through social and historical research.
In fact, to the extent that the ideas of contingency, heterogeneity, plasticity, and conjuncturality play the important role in the social world that I believe they do, it is difficult to avoid the conclusion that there are very narrow limits to the degree to which we can aspire to systematic or theoretical explanation in the social realm. And this in turn suggests that we might better describe social inquiry as a set of discrete and diverse social studies rather than unified "social science". We might think of the domain of social knowledge better in analogy to the contents of a large and diverse tool box than in analogy to an orrery that predicts the "motions" of social structures over time.