Stadia and fitness centers are an important power in conducting the national fitness campaign. The implementation of the “Healthy Chinese” strategy and promotion of the construction of stadia and fitness centers were achieved by exploring the regional differences and examining the factors that affect stadia and fitness centers. This study discusses the spatial differences between stadia and fitness centers at provincial, urban agglomeration, and prefecture-level city scales. In this study, GDI (Global differentiation index) was used to measure the number of sports and fitness space differentiation. Moran’s I index and hotspot analysis prove the accumulation characteristics of sports and fitness places. The Pearson correlation coefficient and Grey correlation are used to determine the linear relationship and degree of closeness between the influencing factors of stadia and fitness centers, and the Geo-Detector can determine the explanatory power of influencing factors on stadia and fitness centers. The results are shown as follows. First are the number of stadia and fitness centers and the number of sports and fitness places per 10000 people. They are mainly distributed in the eastern region and less in western China, except in the core cities of the Chengdu-Chongqing urban agglomeration. Moran’s I index shows that there is a significant and positive spatial autocorrelation at the urban agglomeration scale and prefecture-level city scale, but it is not obvious at the provincial scale. Moreover, hotspot analysis shows that the smaller the scale, the more obvious the spatial agglomeration of the number of stadia and fitness centers and the number of sports and fitness places per 10000 people. The hotspots are mainly distributed in the eastern coastal region, while the cold spots are mostly distributed in the northwest, southwest, and Qinghai-Tibet regions. Second are the GDI of stadia and fitness centers and the number of stadia and fitness centers per 10000 people, which expands with the reduction in the scale. On the scale of urban agglomerations, the difference between developing and expanding urban agglomerations is the smallest, while the difference between optimizing and upgrading is the largest. Third are the economic aggregate and population, which are important driving factors for concentrating stadia and fitness centers, and the population has the strongest interaction with the proportion of the urban population. At the scale of the prefecture-level city and urban agglomeration, the size of the cities is an important factor influencing the number of stadia and fitness centers. Finally, the following policy suggestions are provided. As GDI expands with scaling down, there are obvious differences between east and west in the hotspot analysis at the prefecture-level city scale. Therefore, in the future, the government should invest in the construction of sports and fitness places in western China. The results of the geodetector show that GDP and population are important driving factors. Therefore, the government can actively build commercial sports venues by attracting investment. Additionally, while building fitness venues, the government should research the fitness needs of different groups to create a local fitness culture and fitness concepts.