James Milton Smith began his career in law, admitted to the bar in 1846 and establishing his practice in Columbus, Georgia. He served in the Confederate Army during the Civil War, rising to the rank of colonel and participating in significant battles, including Gettysburg. Post-war, Smith entered politics, elected as a Democratic delegate to the Confederate Congress and later to the Georgia Legislature, where he became Speaker of the Georgia House of Representatives. He was elected Governor of Georgia in 1872, focusing on fiscal responsibility and leaving office with a surplus. Smith also held judicial positions, including the first chairman of the Georgia Railroad Commission and a judgeship in the Chattahoochee Circuit of the Superior Court.

James Milton Smith's political identity is marked by a blend of fiscal conservatism and racially charged rhetoric. His tenure as Governor saw him restore Georgia's credit rating and reduce expenditures, yet his public comments about African-Americans reveal a deeply entrenched racial bias. Smith's approach to governance juxtaposes his fiscal prudence with troubling racial views, creating a complex and troubling ideological profile.