Whenever I design a statistical analysis, I always struggle with determining the number of subjects to include in validation studies of measurement scale. I know that too small a sample can lead to erroneous conclusions drawn but including too many subjects is a waste of time and resources. Hence it would be good to understand if there is a minimum to guarantee an acceptable level of precision and stability of results.
You are specifically asking about internal validation of scales and hence this response is for that specific approach alone. It would be safe to assume that you already know about the assumptions that has to be fulfilled for any statistical approach you wish to use, as it appears you frequently design statistical analyses.
One of the most important approach decisions when designing a study is the number of subjects to include. In inferential statistics, the sample size is based on the power of a statistical hypothesis testing. In descriptive studies, however, sample size is usually determined by the range of the confidence interval of a given parameter. This is the case in internal validation of measurement scales where two types of parameters are of interest: Cronbach’s alpha coefficient which is a measure of reliability, and factor analysis loadings which gives an indication of the dimensional structure of the scale. I won’t go into further details on this as you can easily read up if you want to know more.
The bottom line: the general rule that the ratio of subjects to variables (N/p) to calculate the sample size required in internal validation of measurement scales has been recommended without strict theoretical or empirical basis provided. The most widely used rule uses the ratio of the number of subjects (N) to the number of items (p), and this varies from three to 10 depending on authors (Cattell, 1978; Everitt, 1975; Gorsuch, 1983; Nunnaly, 1978).
What we do have is years of researches on the consequences of using factor analysis on insufficient sample sizes. The findings explicitly state that the validation of short scales does not warrant a smaller sample size.
If one’s aim is to reveal the factor structure, under the hypothesis that the underlying common factor model is true, a minimum of 300 subjects is generally acceptable in the conditions encountered in the field of psychiatry. This sample size needs, however, to be larger when the expected number of factors within the scale is large (>10). Furthermore, to obtain more accurate solutions, researchers should choose Exploratory Factor Analysis as the method for factor extraction. (Rouquette, Falissard, 2010)