Fossil fuels account for more than 80% of the world's energy consumption. Constituents of the atmosphere have changed perceptibly due to the increased use of fossil fuels. Therefore, many researchers have tried to relate their effect on society. In Pakistan, fossil fuel consumption and its CO2-based emission factor have been significantly correlated to economic growth. However, it needs further attention to study the adverse effects of fossil fuels. This study is an attempt to assess the cost of fossil fuels to society. Damages caused by fossil fuels are evaluated for the years of 2005-2009, using local pollution factors based on CO2 emission. Results show that the market price of fossil fuels increases after adding up the cost of damage caused by the final use of the fuel. People pay a huge amount of PKR 133 billion per year for taxes, health services, insurance premiums, and low living standards. Accordingly, it is suggested that we must shift from fossil fuels to other alternative clean types of energy.