Iverson Molinar added 19 points and Shakeel Moore scored 10 points for the 10th-seeded Bulldogs (18-14), who will face second-seeded and ninth-ranked Tennessee in the quarterfinals Friday night.
Erik Stevenson scored 18 points and Devin Carter added 10 for the seventh-seeded Gamecocks (18-13).
Mississippi State led by just two points at halftime, but dominated the second half by outscoring South Carolina 37-17.
The Bulldogs led by just three points early in the second half, but finished the game on a 26-7 run. The Gamecocks made just one field goal in the last 14:29 and none in the last 10:29.
South Carolina's final five points came on free throws, but they came up short there as well, making 11 of 19 foul shots compared to 20 of 27 for the Bulldogs.
Molinar scored six points as Mississippi State began the second half with an 11-3 run that gave it a 47-37 lead. Stevenson had a 3-pointer and an assist as South Carolina responded with a 7-0 run.
Smith's layup started an 11-2 run that gave the Bulldogs a 58-46 lead as the Gamecocks went ice cold. Six more points from Smith helped Mississippi State extend its lead to 66-48 with four minutes remaining.
The Bulldogs scored the first six points of the game and led 8-4 before going cold.
Stevenson made two 3-pointers and Jermaine Couisnard added one during an 11-0 run that gave the Gamecocks a 15-8 lead.
Molinar made a 3-pointer to end a 4 1/2-minute field-goal drought and Moore added a 3-pointer that pulled the Bulldogs within 20-18.
Couisnard's 3-pointer pushed the lead back to six, but Smith had four points during a 10-0 run that gave the Bulldogs a 30-26 lead.
South Carolina briefly regained the lead on Carter's layup, but Mississippi State scored the final four points to take a 36-34 halftime lead.