You can drink the tap water in most places in Iceland without boiling it (especially in Reykjavik), but if you're further into the countryside then I would check first!