This paper introduces a new procedure for clustering a large number of financial time series based on high-dimensional panel data with grouped factor structures. The proposed method attempts to capture the level of similarity of each of the time series based on sensitivity to observable risk factors as well as to the unobservable factor structure, which is group specific. The proposed method allows for correlations between observable and unobservable factors and also allows for cross-sectional and serial dependence and heteroskedasticities in the error structure, which are common in financial markets. In addition, theoretical properties are established for the procedure. We apply the method to analyze the returns for over 6,000 international stocks from over 100 financial markets.