The Swans went down to a 1-0 defeat to Southampton as sub Charlie Austin's strike proved the difference at St Mary's.
Austin - on for Shane Long - had already struck the crossbar before thundering home the winner after 63 minutes to earn Southampton their first win of the league season.
The Swans went closest through Gylfi Sigurdsson late on, only for his low shot to be kept out by the diving Fraser Forster.
Defeat leaves the Swans - who handed record signing Borja his debut off the bench - on four points from five games.
The Swans made just one change to the starting XI.
From the side that drew 2-2 with Chelsea last weekend, Gambian winger Modou Barrow was the only fresh face in Francesco Guidolin's starting plans as he took the place of Neil Taylor.
Kyle Naughton made the 100th Premier League start of his career, while Jack Cork again captained the Swans against a former club, having led the side against Chelsea last time out.
Record signing Borja was included in the matchday squad for the first time since his move from Atletico Madrid, with the striker among the substitutes.
He was joined on the bench by Wayne Routledge, who returned to the squad after missing out against Chelsea, while Nathan Dyer was ruled out as he was due to undergo an ankle operation.
SOUTHAMPTON: Fraser Forster, Cedric Soares, Jose Fonte (capt), Shane Long (Charlie Austin, 53), Steven Davis, Dusan Tadic (Cuco Martina, 81), Jordy Clasie (Pierre-Emile Højbjerg, 53), Oriol Romeu, Virgil Van Dijk, Ryan Bertrand, Nathan Redmond.
SUBS: Alex McCarthy, Maya Yoshida, Lloyd Isgrove, Harrison Reed.
SWANS: Lukasz Fabianski, Kyle Naughton, Federico Fernandez, Jordi Amat, Stephen Kingsley (Angel Rangel, ht), Jack Cork (capt, Borja, 81)), Ki Sung-Yueng (Jefferson Montero, 65), Leroy Fer, Gylfi Sigurdsson, Modou Barrow, Fernando Llorente.
SUBS: Kristoffer Nordfeldt, Mike van der Hoorn, Leon Britton, Wayne Routledge.
There was little to choose between the teams in the opening half hour at St Mary's, with both teams creating a few half chances.
Kyle Naughton was the first to strike an effort on goal, but his sixth-minute shot was saved low down by Fraser Forster.
Fernando Llorente was just beaten to the ball by the keeper when Jack Cork flicked on a Gylfi Sigurdsson free-kick into the box.
Then the Saints, who were enjoying the better possession, then went close as Shane Long's shot was spilled by Lukasz Fabianski and Dusan Tadic and Steven Davis - both flagged for offside - got in each other's way for the loose ball.
The Swans replied with an inswinging Barrow cross, which saw Jose Fonte snuff out Llorente's close-range effort.
Back at the other end, Virgil Van Dijk got onto the end of a corner, but headed straight at Fabianski while Nathan Redmond skied his effort after racing into the box.
Then came the game's best chance as the clock ticked towards the end of the 45 minutes. Tadic used his trickery and pace to beat Stephen Kingsley, and when he found Redmond eight yards out, the winger looked odds on to score. But, when he turned inside past Jordi Amat, Redmond skied his effort over the crossbar.
That was the end of the action in the first half as both teams went into the break on level terms.
Half-time: Southampton 0 Swans 0
Francesco Guidolin made a change at the break, with Angel Rangel - making his 150th Premier League appearance - on for Stephen Kingsley, who was struggling with a stomach bug.
The Saints raised their game after the break and they were enjoying the better territory and possession.
They were quick out of the blocks, and it took some quick thinking from Kyle Naughton to prevent the hosts from going ahead three minutes after the break as he cleared Long's cross off his own line.
The Swans looked to work on the counter, and Leroy Fer headed wide from a Barrow cross while Gylfi Sigurdsson fired off target from 25 yards after a near flick on by Llorente.
But the Saints were looking the more likely to get the opener.
Ryan Bertrand saw his cross into the box evade everyone before bouncing past the far post while Oriol Romeu headed into the ground and over the crossbar when unmarked eight yards out.
The warning signs were there, and none more so when sub Charlie Austin - a replacement for Long - got on the end of Redmond's cross before seeing his shot clatter off the woodwork.
But, a minute later, he did take his chance as Tadic's cross deflected off Jordi Amat into the path of Austin, who chested the ball down before firing into the net from close range.
It was a deserved lead, and it could have been doubled soon after only for Romeu to head wide.
The Swans needed a response, and Guidolin introduced Jefferson Montero in place of Ki Sung-Yueng.
But it was Barrow who probed next, with the winger cutting into the box before firing low which Forster initially parried before collecting.
Guidolin made his final change with ten minutes to go as record-signing Borja came on for his debut in place of captain Jack Cork.
The Swans pressed for an equaliser, and they almost got it when Llorente found Sigurdsson inside the box, but his low effort was tipped wide by the fully-stretch Forster.