Oscar Cardozo slotted home the winning kick after Yuichi Komano had smashed his effort against the bar in the first shootout of the 2010 tournament.
Japan, bidding to become the first Asian team to beat a South American side in the World Cup history, threatened Paraguay 25 seconds into action as Yoshito Okubo connected a wayward pass in the Paraguay midfield and blasted a low shot from 25 meters which went wide one meter of Justo Villar's goal.
At the 28th minute, Paraguay's Claudio Morel swung in a corner to cause a goalmouth melee in which Roque Santa Cruz fired wide.
Both sides fought hard and created some chances but squandered all of them in the regular and extra time.
Paraguay coach Gerardo Martino said the game decided by penalties was not fair.
"I know it's unfair to decide the match by penalty shoot-out but that's the way it is," he said.
Japan coach Takeshi Okada claimed his side didn't have regrets.
"I am proud of my players who also played for Asia until the end," he said.
The two-day quarterfinals starting on July 2 will pit Uruguay against Ghana, Argentina against Germany, the Netherlands against Brazil, and Paraguay against Spain.
Interestingly, four South American giants will each face a European powerhouse plus an African team.