Back draught shutter ratling

Hi

I installed a circular flap on my ducting for my bathroom vent. This was because outside air was spilling into the bathroom. I've installed a round flap on the ducting which is meant to sit closed unless the the extractor fan pushes air against it to open. It does work but also air from outside seems to swing it open, so I can hear it flap open and close during windy spells.

It's located in the loft about 20cm after the extractor unit, with the remaining 1m og ducting going into the sophet fascia before exiting downwards outside.

Is this something as silly as putting the flap on backwards? I assume not as the flap would be closed when the fan is on and there hasn't been any extractor issues.