Tabby: bump 0.7.0 -> 0.8.3 and add systemd service

[ghthor]

Description of changes

1. followup to tabby: init at 0.7.0 take 2 #291744
2. Upgraded tabby to release 0.8.3
3. Replaced static linking of vendored llama-cpp with dynamic link to llama-cpp from nixpkgs
4. Add systemd service for [tabby, tabby-scheduler] with timer configuration for tabby scheduler

Upgraded to upstream 0.8.3 release

Added support for rocm AMD acceleration.

• tested rocm acceleration locally with AMD Radeon RX 560 Series
• tested cuda acceleration locally with NVIDIA Geforce RTX 2080 SUPER
• tested darwin metal on an M1 macbook

Both tests used the new systemd service to run tabby

Dynamic Linking w/ llama-cpp

I was having compile errors with the vendored copy of llama-cpp that tabbyml was statically compiling, so I switched to dynamic linking with the llama-cpp that is within nixpkgs.

Systemd service

Example of the configuration required here.

https://github.com/ghthor/shrc/blob/18109599c526e580e54b11a16b6c1ef017777918/nix/cryptnix/configuration.nix#L434

  services.tabby = {
    enable = true;
    acceleration = "cuda";
    model = "TabbyML/StarCoder-3B";

    settings = {
      repositories = [
        {
          # Has to be created/updated manually, and permissions corrected
          git_url = "file:///var/lib/tabby/source_repo";
        }
      ];
    };
  };

Using the service makes it easy to enable a specific type of acceleration.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[ghthor]

@sersorrel I was able to test the rocm support on my local workstation, still pinging you if you'd like to test as well.

[ghthor]

@happysalada ready for review

[happysalada]

This is looking good, thank you !
I have a couple of questions / comments

[happysalada]

Would you mind if we took the defaults from config.supportCuda and supportRocm. I dont think we need to have this manually configurable. If the user has cuda enabled use cuda should be the easy default.

[ghthor]

Yep, I think that makes sense, will fix this

[ghthor]

I added some defaulting of the acceleration type in the presence of either rocmSupport or cudaSupport on nixpkgs.config. Even with that defaulting I have left acceleration as a service and package configuration value to handle the case when both of the above are present, as we can't make a clear decision and we do have to pick a single option. See this comment I've added to the code where the llama-cpp package is overriden.

274d9b9#diff-abdbe2b562428d2ed485d33ee478cf9e126faae74d5cc60b48284c7c254ddf33R86

[keysmashes]

Looks good! I built with nix-build -E 'with import ./. {}; tabby.override { acceleration = "rocm"; }', and ./result/bin/tabby serve --model TabbyML/StarCoder-1B --device rocm appeared to work fine.

What's the purpose of the modelDownload config? I didn't explicitly run tabby download, it just downloaded it automatically.

[ghthor]

What's the purpose of the modelDownload config? I didn't explicitly run tabby download, it just downloaded it automatically.

Yep, I can remove this extra configuration. I started using tabby before June 1, 2023 when they added this commit[1], didn't realize

[1] TabbyML/tabby@ca077a3

[happysalada]

This looks good to me, happy to leave this open 1 more day to give some more time to other people to take a look.

[de11n]

Extremely well done! A few small comments.

[de11n]

It feels backward to mention NixOS module configuration in the warning of a specific package. Perhaps this validation deserves to be in the module instead. A shorter, less-verbose warning could be here instead.

[ghthor]

Yeah that makes sense, I'll move this around

[de11n]

Looks better. Thanks. It looks like you have a trailing whitespace somewhere, causing the CI to fail.

[happysalada]

pkgs/by-name/ta/tabby/package.nix:
70: Trailing whitespace

[happysalada]

I just noticed the darwin x86_64 build is broken. Would you mind marking the build as broken for that platform ?

[ghthor]

Alright, little searching I found this #140325 for reference on how to mark the package as broken which I think points to it being a boolean flag in the meta object.

I don't have hardware to test linux-aarch64, but I will soon as that's what I'd like to try and run this on at work.

[ghthor]

Fixed commits to follow https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md#commit-conventions

[happysalada]

Perfect thanks ! I plan on merging this when i go home today.