62.6% of all Americans participate in the labor force, leaving 37.4% who do not. There are a number of reasons that people don't work. Having said that, one must look at the current participation rate historically to determine "how bad things are".
There are different reasons for this. One is the system that the nation has put in place, which encourages an underground employment wherein people work but they are not paid through the traditional reporting system. They work "under the table", for cash. It benefits both the employer and the employee because neither has the burden of Social Security, income tax, Medicare or employer matching costs. Often the employee can claim unemployment and thus, qualify for food stamps, aid for dependent children, housing subsidies and pull in tens of thousands in earned income credit.