Free black males lost the right to vote in Connecticut and Rhode Island in 1818 and 1822 respectively. They got it back around the time of the Civil War. I have added many new entries to the American Racial History Timeline. Check it out.
More below on the racial heritage of the Democratic Party:
The party proclaimed itself the tribune of the common white man, as against all other groups in the society, whether of class, race, or gender. In particular, it defined itself, even in the North, as the protector of slavery. (Howe, 524)
Today, the Democratic Party is the perverted tribune of non-whites, women, homosexuals, non-Christians and other social deviants, but exactly the opposite was true in the early nineteenth century. It was an unabashed white racialist organization. In contrast, Northern Federalists and Whigs opposed Indian Removal in the Old Southwest, were for black suffrage, and generally opposed westward expansion (the Louisiana Purchase, the Mexican War, purchase of Cuba, expansion into Latin America and the Caribbean).