You're not the one for me

there still is something fishy about it though

For sure! You should still be honest with your partners about the nature of the relationship