Joey Barton's first half penalty earned Stuart Pearce's side a win in what was the upset of the day in the Premiership.
City, who had picked up only one point from their opening two League matches produced a battling performance against last season's Champions League finalists.
Though City had luck on their side - Arsenal were twice denied by the woodwork - Pearce's team deserved reward for their efforts.
And it was a notable victory, their first in 19 Premiership meetings and City's first win against the Gunners since 1991.
City made three changes following their midweek goalless draw against Portsmouth as Ben Thatcher was dropped for disciplinary reasons Claudio Reyna was injured and striker Georgios Samaras relegated to the substitutes' bench.
Stephen Jordan replaced Thatcher at left-back, Ousmane Dabo was drafted into midfield, while there was a home debut for Italian striker Bernardo Corradi.
Arsenal fielded a strong line-up, though Ashley Cole, Phillipe Senderos, Jose Antonio Reyes and Freddie Ljungberg were absentees.
After a luke-warm opening the matched sparked to life in the 14th minute when Thierry Henry beat the offside trap but was denied by the legs of keeper Nick Weaver.
City might have stolen into the lead moments later when Trevor Sinclair set up Barton who rolled his shot agonisingly wide when he looked certain to score.
Henry wasted another glorious chance shooting wide from the far side of the penalty area.
But City had the best chance of the early stages when Paul Dickov's cross was met by Corradi whose goal bound header was brilliantly kept out by Jens Lehmann diving low to his left.
A goal eluded Henry for the third time in the opening half after he was let in following a terrible error by Sylvain Distin, but Weaver came to the rescue again blocking his shot with his legs.
Arsenal were also denied a penalty when replays showed Richard Dunne clearly took the legs Cesc Fabregas after the ball had gone.
Robin van Persie crashed a shot against the upright as Arsenal were frustrated again in front of goal.
Then, against the run of play, City were awarded a spot-kick when Justin Hoyte clattered into Sinclair and Barton stepped up to score with a shot which went in off the underside of the crossbar.
Arsenal were denied by the woodwork on a second occasion in stoppage time when Kolo Toure headed Fabregas's corner against the crossbar.
Arsenal cranked up the tempo in the second half as they pulled out all the stops to get an equaliser. But despite the considerable pressure they exerted they were repelled time and time again by City's well marshalled defence.
And City restricted Arsenal to only a couple of clear-cut chances, notably when Fabregas and Van Persie both shot disappointingly wide from great goal scoring positions.
Arsenal brought on Theo Walcott and Emmanuel Adebayor as they switched to a 4-4-2 in pursuit of that elusive goal.
Van Persie ought to have equalised in the 83rd minute but somehow contrived to head Fabregas's corner wide from five yards, and that was to prove their last chance of the match.