Kazmir (1-1), a two-time AL All-Star who pitched in an independent league last season, allowed two runs in six innings. It was his first win since beating Tampa Bay on Sept. 19, 2010, while with the Angels.
The left-hander, who allowed eight earned runs in 8 1-3 innings in his first two starts, struck out seven.
Swisher, who hadn't played since Monday, homered off Kevin Correia (3-2) in the first. Jason Kipnis, who had three hits, also homered in the inning and added an RBI single in the second.
Cleveland has won six straight for the first time since winning seven in a row April 26-May 3, 2011.
Rookie outfielder Aaron Hicks, the Twins' top prospect, hit his first major league home run to lead off the fifth, clearing the 19-foot wall in left field.
Kazmir on 53 games for the Rays from 2005-09 before his career his career was derailed by arm injuries. He was 9-15 for the Angels in 2010 and appeared in only one game in 2011 before being released.
Kazmir pitched for Sugar Land in the Atlantic League last season, going 3-6 with a 5.34 ERA in 14 games. He signed a minor league contract with the Indians in January.
Correia failed to go seven innings for the first time in six starts this season.
Kipnis, who has six RBIs in his last two games, and Swisher both homered to right in the first. Swisher, who entered the game in a 4-for-28 slump, missed the two-game series against Philadelphia and didn't play Friday.
The Twins scored in the second on Pedro Florimon's RBI single, but his misplay in the field helped the Indians add two in the bottom of the inning. After Michael Brantley's single gave Cleveland a 3-1 lead, Florimon was unable to catch Kipnis' popup that fell for a run-scoring single. The shortstop ranged into shallow left field, but the ball drifted away from him and dropped in.
Kipnis tripled in the seventh and scored on Asdrubal Cabrera's double. Mark Reynolds scored Cabrera with a two-out single.
Ryan Doumit's RBI single off Joe Smith in the eighth cut the lead to 6-3, but Chris Parmelee flied out with two on to end the inning. Ezequiel Carrera had an RBI single in the bottom of the inning.
Hicks, who is batting .123, homered on Kazmir's first pitch of the fifth. Joe Mauer had a hit and a walk in three at-bats and is in a 5-for-42 slump.
NOTES: Indians setup man Vinnie Pestano was unavailable because of a "cranky" right elbow, according to manager Terry Francona. Pestano threw a bullpen session before the game. ... Hicks leads AL rookies in walks with 13. ... Twins OF Oswaldo Arcia didn't start Saturday after starting 12 consecutive games. He had a pinch-hit single in the seventh. ... Mauer was the DH after catching 10 innings Friday. ... Indians OF Michael Bourn (lacerated right index finger) took batting practice Friday for the first time since being injured on April 14. Bourn could begin a minor league rehabilitation assignment in the next few days. ... Francona said RHP Brett Myers (sore elbow) threw off flat ground Saturday and is making progress to throwing off the mound . ... The Indians' April 23 rainout in Chicago has been rescheduled as part of a traditional doubleheader on June 28. .... The series ends Sunday when RHP Mike Pelfrey (2-3) faces RHP Corey Kluber (2-0).