I see two concerns, humanitarian and ecological. The ecological concern is only a problem with overconsumption. The humanitarian concern I don't think applies to fish since they are dumb.
What about bycatch? That's a factor in both. Also ecological: the fact that most of the plastic waste in the sea (and on earth in general) is from fishing. Also, not all seafood is dumb. Octopus is one of the smartest things out there and we eat that. Lobsters have been shown to feel pain and we boil them alive.
So you're only addressing the dumb point... despite some of these not being considered intelligent, they still have central nervous systems and therefore feel pain. That's a very basic ethical concern. Why is intelligence the only concern? Should we eat all the dumber humans?
Bycatch is also an issue in this point because innocent and more intelligent creatures like dolphin and octopus will be caught, killed and not even eaten.