The average American driver drives something like 15,000 miles in a year. That represents at least 250 hours of driving (based on an average speed of 60 miles per hour). The average car has a range of around 300 miles, so that 15,000 miles represents 50 trips to the pump, which at 5 minutes per fill up would be a little longer than 4 hours. So overall the average commuter is spending the equivalent of about 31 eight hour days driving every year... a full months worth.
The average plug-in hybrid driver probably only needs gas 10 times a year or so, or 50 minutes at gas stations in a year. However, they need to plug the car in, which might take a minute or so, and unplug it before they leave, which could take another minute, so over the course of a year there's 12 hours of plugging in and out. There's some experimental "park over" chargers that might help eliminate that, if you take the time to install that... in both your garage floor and in your car. And there is the time for having the plug-in charger installed, though that will no doubt be done by someone else.
So time wise it's about 254 hours for a gas car, and 263 hours for a plug-in electric hybrid car. So it costs 9 hours more, or 3.5% more time to plug-in. The average hourly wage is something like $19.50, though it fluctuates, so overall driving time represents a loss of productivity of around $5,000 per person. $175 more if you drive a plug-in hybrid.
The workforce participation rate is something like 65%, so this $5000 per person represents about $1.17 Trillion in lost productivity. If all workers switch to plug-in hybrids (this isn't likely), the productivity lost will increase to $1.2 Trillion. There's that 3.5% again... so $35 Billion more lost since GDP is something like $15 Trillion that would be only a 0.25% decrease in the GDP. All based on a few minutes plugging in every day.