Poor, deluded fools. Because their skin's as white as the rich man's, they believe that they might someday be rich themselves. But without the Negro, Owen, these men would be forced to see that, in fact, they have no more chance of becoming rich than do the very slaves they despise and trample on. They'd see how close they are to being slaves themselves. Thus, to protect and nurture their dream of becoming someday, somehow, rich, they don't need actually to own slaves, so much as they need to keep the Negro from ever being free.