Gunners skipper Thomas Vermaelen missed the crucial penalty on a difficult night for the Premier League side, who have won only once in the last six matches in all competitions.
Garry Thompson volleyed the hosts ahead, before Vermaelen headed home a Santi Cazorla cross at the back post with with only two minutes remaining of normal time.
Defeat is another setback in a disappointing domestic season so far for manager Arsene Wenger and his club, who last won a major title in 2005.
Victory for the Bantams means they have now won their last nine penalty shoot-outs and are the first team from the bottom tier of English football to reach the semi-final of the competition since Wycombe in 2006-07.
Wenger had shown his intent with a starting XI that included just three changes from the side that beat West Brom on Saturday and 10 from the one that began the epic 7-5 win over Reading in the last round. French midfielder Francis Coquelin was the only survivor from that game.
The last time the teams met was in the Premier League in January 2001, but they started Tuesday's game 65 places apart in the league pyramid
Arsenal had the better of the opening exchanges, and both Vermaelen and Lukas Podolski spurned presentable headed opportunities.
They were left to rue their wastefulness when Vermaelen brought down the lively Nahki Wells and Thompson volleyed home the resulting Gary Jones free-kick from close range to spark jubilation amongst nearly 24,000 supporters in the packed-out home stands.
Coquelin came close to an equaliser for the two-time champions when his low volley beat Bantams keeper Matt Duke but not the post, before Gervinho managed to totally miss the ball when presented with an open goal from Gibbs's low centre.
But the first half belonged to Bradford, who have not lost a Cup game since a 4-2 defeat by Watford in the FA Cup third round in January. Wells nearly doubled their lead on the stroke of half-time but his low half-volley trickled just wide.
The second half started in a scrappy fashion, with the hosts continuing to frustrate Arsenal with their energetic display.
The top-flight side dominated possession, but passing errors and a lack of cutting edge continued to plague their game.
Just when it looked like the team from the fourth tier were going to make the semi-final for the first time in their history, Vermaelen headed in a pinpoint cross from Cazorla to send the game into an extra 30 minutes.
Neither side could find a winner so it was down to penalties and successful efforts from Nathan Doyle, Gary Jones and Alan Connell allied to misses from Cazorla, Chamakh and Vermaelen gave the Bantams a famous win.
BBC
Gunners Stunned by Bradford
