Skip Navigation
NixOS @infosec.pub ComradeKhoumrag @infosec.pub

how to / best practices for setting up and tracking a Python project with nix flakes

I want to start a new project, and I want to try to handle all the reproducibility / "containerization" in nix instead of dockerfiles. I see some examples online but I think they're including more uncommon procedures and/or don't do things the "nix" way.

What's the right way to manage a simple python project? Should I just make a derivation.nix for use in nix-shell -p and have the ephemeral shell be my container? Can/should I do it with nix flakes instead? What would a simple nix flake look like that:

pulls an initial python repo from github

possibly executes whatever build instructions might be included

extends other system packages or other versions of the same python package,

has local area network access,

and GPU access

2
2 comments

  • There's two ways to create a dev-environment, which is how you should start. Don't worry about building a project with nix until you get a dev shell with all your (runtime) dependencies. You can do this by creating a shell.nix or use flakes and have an output for devShells.default. You can use mkShell to accomplish this. I would give you a reference config, but I can't do that at the moment. If you need help with this later reach out and I'll get something together.

    After that you can work on building it. This involves making a derivation that contains the runtime dependencies of your code, as well as the source itself. If its python, this would include, likely, the entirety of your project. This could be a little tricky, but a lot of people package python code, so there's things in nixpkgs meant for packages python projects. This article might be helpful, or out-dated: https://nixos.wiki/wiki/Python but it should point to some useful things in nixpkgs to build python projects.

    Then, you can focus on containerization, if you really need it. I don't know how much you know about nix, but building in nix inherently isolates dependencies away from one another, so if you get the process above working you know it is reproducible. But unlike a container, the above build structure only works on a machine with nix. So if you want to distribute your project as a container, while utiltizing your nix build process, you can look into something like dockerTools from nixpkgs: https://nix.dev/tutorials/nixos/building-and-running-docker-images.html

    Let me know if you have more questions.

    • Thanks, I think I finally got it, but I think I'll need to switch to flakes if I want to be able to install say numpy==1.10.0 instead of latest (is there a way to specify which python package version should be installed in mkShell?)

      Does this explanation make sense?

      { pkgs ? import  {} }:
pkgs.mkShell {
  packages = with pkgs; [
    (python310Full.withPackages 
                        (python-pkgs: with python-pkgs;
                          [numpy pandas requests]))];
  #inputsFrom = [ pkgs.python310Full ];
}

      To derive this ephemeral shell...

      use a package manager defaulted from nix packages: {pkgs ? import {} }:

      Make an ephemeral shell with programs located in the package manager: pkgs.mkShell {

      Define the set of installed packages in the ephemeral shell

       packages = with pkgs; [
    (python310Full.withPackages # Include python310 in the set of installed packages
                        (python-pkgs: with python-pkgs; # not entirely sure what's going on here, is python-pkgs a predefined arg? fails with nativeBuildInputs if this isn't here 
                          [numpy pandas requests]))]; # the pip packages to include