Fibrous and particulate environmental substances such as asbestos fibers and silica particles cause not only lung fibrosis but also various health disturbances. Asbestos induce malignant tumors such as pleural mesothelioma and lung cancer. Silicosis patients exposed to silica particles show complications of various autoimmune diseases such as rheumatoid arthritis, systemic lupus erythematosus, systemic sclerosis, and antineutrophil cytoplasmic antibody (ANCA)-related vasculitis/nephritis. The causative alteration of immune cells exposed to these environmental substances may form baseline modification of human immune system not only localized pulmonary lesions, alteration of alveolar macrophages, and others but also general immune system and changes of function in effector, regulatory, and cytotoxic T cells and natural killer cells. In this review, both (localized and generalized) immune alterations caused by environmental fibrous and particulate substances are summarized and reported.