The Swans suffered a second consecutive defeat as Southampton made the most of Wilfried Bony's first-half dismissal with a 1-0 win at the Liberty.
Bony, who saw an earlier header kept off the line, was shown red after receiving a second booking in the 39th minute, with the Swans then having to deal with heavy Southampton pressure.
The Swans survived until the break in a first half in which Wayne Routledge had struck the crossbar.
Despite some heroic defending, the Swans couldn't contain the Saints for the duration of the second half as sub Victor Wanyama's 80th-minute strike beat Lukasz Fabianski.
It proved the difference, with Southampton maintaining their unbeaten record against the Swans in the Premier League.
This defeat also ended the Swans' perfect record at home, and it was a consecutive loss following the previous result at Chelsea.
Next up for the Swans is a Capital One Cup home tie with Everton on Tuesday.
The Swans made two changes to the starting XI as Federico Fernandez made his first Barclays Premier League start in place of the injured Jordi Amat while Wilfried Bony returned as Bafe Gomis moved to the bench.
Garry Monk was also without Dwight Tiendalli, who had a slight groin problem, and Leon Britton.
Lukasz Fabianski, Angel Rangel, Federico Fernandez, Ashley Williams (capt), Neil Taylor, Ki Sung-Yueng (Marvin Emnes, 85), Jonjo Shelvey, Gylfi Sigurdsson, Nathan Dyer (Jefferson Montero, 59), Wayne Routledge, Wilfried Bony.
Gerhard Tremmel, Jazz Richards, Kyle Bartley, Tom Carroll, Bafe Gomis.
Fraser Forster, Nathaniel Clyne, Maya Yoshida (Florin Gardos, ht), Morgan Schneiderlin, Jose Fonte (capt), Shane Long, Dusan Tadic, James Ward-Prowse (Victor Wanyama, 68), Jack Cork (Steven Davis, 64), Graziano Pelle, Ryan Bertrand.
Kelvin Davies, Emmanuel Mayuka, Matthew Targett, Sam McQueen.
Prior to kick off, the new inductees of the Robbie James Wall of Fame were introduced on the Liberty pitch and they received a rapturous applause.
The Swans created the early chances with Gylfi Sigurdsson driving straight at Fraser Forster from 20 yards after latching onto Wilfried Bony's flick on before Wayne Routledge's powerful volley flew just wide.
Both teams were evenly matched, but the Swans were getting more on top as the half went along, and after 28 minutes they went within inches of opening the scoring.
Wilfried Bony and Maya Yoshida were then booked within minutes of each other, but the game continued to flow.
After Dyer found Sigurdsson on the edge of the box, the midfielder's first-time pass picked out the onrushing Routledge, whose thunderous effort beat Forster but not the woodwork as it cannoned off the crossbar.
Southampton's first attempt came just past the half hour as Graziano Pelle's low drive was easily dealt with by Lukasz Fabianski.
But the Swans were knocking on the door once again just a minute later as Bony rose to meet Sigurdsson's corner, but the Ivorian's goal-bound header was kept off the line by Ryan Bertrand.
That was Bony's least involvement, however, as the striker was shown a second yellow card for a challenge on Yoshida in the 39th minute.
But the Swans saw out the remaining minutes to go into the break with the scores level.
Half-time: Swans 0 Southampton 0
Southampton made a change at the break as Florin Gardos replaced Yoshida in the heart of the visitors' defence.
The Saints were enjoying the better possession and territory as they looked to make the most of their man advantage.
And they went close after 55 minutes as Dusan Tadic cut inside before his effort deflected off Nathan Dyer and looped over Fabianski and onto the top netting for a corner.
Pelle then got onto the end of Nathan Clyne's cross, but Fabianski reacted well to gather low down at his near post.
The Swans then made a change as Jefferson Montero replaced Dyer just before the hour mark.
Ronald Koeman made his final substitutions - one enforced - as Steven Davis and Victor Wanyama came on for Jack Cork and the injured James Ward-Prowse.
The Saints were continuing to probe, and again Fabianski did well to deal with a dipping Tadic free-kick from 20 yards out.
And, with ten minutes left to go, they got their goal as the Swans' stern resistance was breached.
Sub Wanyama worked his way into the box after latching onto Pelle's pass and he fired into the far corner.
Monk made his final change with five minutes to go as Marvin Emnes replaced Ki Sung-Yueng.
But the visitors comfortably held out to claim victory in SA1.