Software engineering for data scientists
Software engineering for data scientists
Hi all! Data scientist here, trying professionalise a group of hobby programmers who've somehow found ourselves doing it for a living. The programming we know; it's the infrastructure we're lacking. None of us knows how to organise a programming team, myself very much included.
Can anyone recommend resources, books, courses on software engineering suitable for data scientists?
I'm not sure what you're asking. It kinda seems like your looking for advice on managing a team?
Not exactly managing a team -- it's more a question of best practices around pull requests, version control, testing, code reviews, pair programming etc. I'm not interested in management, but I do I want to know what a well managed team ought to be doing!
They ought to have a manger/team lead interested in managing. An ad-hoc approach can only get you so far.
Though, you seem to be asking more about Devops than management. If you're making a startup, just cobble an MVP together using as little custom-built tools as possible and redo it if/when you get enough investment to be able to take the time and pay for the necessary resources to do it right and maintain it. You're going to want to redo your first version anway, better to not waste time and effort on complex solutions even if they're more appealing. Saas is your friend.
You could look at the development practices parts of extreme programming.
Also remember software development is a craft to be mastered. It takes a lifetime of continuous improvement to get there.
I always recommend the edX/Harvard CS50 course to anyone who has not had academic computer science experience.
Thanks, I'll have a look!
CS50 is a great rec. I believe the CS109 course material is available online too, which is a course specific to data science. The lessons are in Jupyter notebooks and are fun. But sounds like you or your group is already familiar with data science specific engineering so CS50 might be more helpful.
If you are looking for how to use AWS I’d recommend investing the time to learn Terraform. If you’re looking to learn how to organize as a software team and don’t have work experience in that, I’d recommend checking out Linear, Monday, or Jira and just sticking with simple kanban. Brief meetings at some cadence to discuss what focus time is used for and the encapsulation of work. Use Notion or Confluence to organize thoughts and share decision records or the results of technical investigations (spikes).
Just some thoughts.
I really like The Mythical Man-Month by Fred Brooks. It's originally from 1975, based on his experience in managing a team that wrote the operating system for a series of IBM computers. So it doesn't talk about modern tooling. But I do like the way it gives the lay of the land, so to speak. Lots of interesting ideas, and quite a lot of wonderful illustrations and diagrams too :)
This looks really interesting! I'll give it a go. Thanks!