DIRT POOR

The floor was dirt. Only the wealthy had something other than dirt. Hence, the saying, "dirt poor".