Ye Standard Disclaimer: IANAL and IANAA
Based on the information I can find regarding the European Council's Sixth Directive you may have to pay the VAT:
TITLE XII PERSONS LIABLE FOR PAYMENT FOR TAX
Persons liable to pay tax to the authorities
The following shall be liable to pay value added tax:
under the internal system:
(a) taxable persons who carry out taxable transactions other than those referred to in Article 9 (2) (e) and carried out by a taxable person resident abroad. When the taxable transaction is effected by a taxable person resident abroad Member States may adopt arrangements whereby tax is payable by someone other than the taxable person residing abroad. Inter alia a tax representative or other person for whom the taxable transaction is carried out may be designated as such other person. The Member States may also provide that someone other than the taxable person shall be held jointly and severally liable for payment of the tax;
(b) persons to whom services covered by Article 9 (2) (e) are supplied and carried out by a taxable person resident abroad. However, Member States may require that the supplier of services shall be held jointly and severally liable for payment of the tax;
(c) any person who mentions the value added tax on an invoice or other document serving as invoice;
- on importation : the person or persons designated or accepted as being liable by the Member States into which the goods are imported.
From other source like Wikipedia you may not be liable to collect the VAT if your sales to EU are < 100,000 euro during any rolling 12 month period.
In either case I would contact an accountant to get a clearer picture because aside from the EU regulations under Sixth Directive you could be facing existing treaties between US and member states which may impact VAT collection.