Link-Time Optimization (LTO) by default for Rust packages in Ubuntu

Project Discussion
1

Hi!

Ubuntu already enables LTO by default for C and C++ packages (and due to issues disables it for some - https://launchpad.net/ubuntu/+source/lto-disabled-list/ ). However, for Rust packages LTO is not enabled by default. I think it’s a missing optimization opportunity for packages that definitely leads to smaller Rust binaries (it’s a common concern for Rust apps) and can improve CPU performance for some packages due to more aggressive compiler optimizations with enabled LTO.

Unfortunately, in other ecosystems, LTO for Rust is not enabled too:

I guess Debian case is the most critical one for Ubuntu, but I think it could be good opportunity to lead “Rust optimization“ story across distros. Especially since Ubuntu is already at least trying to be a “Rust pioneer“ (FMPOV).

I want to propose an idea for the Ubuntu community/maintainers to consider enabling LTO (and possibly `codegen-units = 1` since it’s a kinda similar optimization technique in Cargo) for Rust packages. Regarding LTO mode - I think it should be the same to C and C++ packages (I guess FatLTO mode is enabled for them). Enabling LTO for Rust will/should be MUCH easier thing to do comparing to C or C++ since LTO in Rust uncovers almost no issues with projects (comparing for LTO-shown “surprises“ like UB in C or C++).

I am talking only and only about enabling LTO for Rust-only part of Rust applications (aka regular LTO with Cargo). I am not talking about cross-lang LTO (doing LTO cross language boundaries like a Rust app with C dependencies) - this one will be too difficult to achieve in practice at scale (but we can consider it later ofc).

I’ve read community guidelines, tried to find a right category for such question/proposal - hopefully it’s the right place for such things. If I am wrong - please redirect me to a right place :slight_smile: If there are already some discussions about the topic - please let me know.

Thank you.

1 Like
2

I do think this is in general a good idea. My first concern though is on the impact on compile times and memory usage – someone™ should probably do some experiments to try to measure these and the impact on performance so we can make some kind of intelligent decision rather than just trying stuff because it sounds good. I can talk to the Rust team about this I guess!

2 Likes
3

My first concern though is on the impact on compile times and memory usage – someone™ should probably do some experiments to try to measure these and the impact on performance so we can make some kind of intelligent decision rather than just trying stuff because it sounds good.

I definitely have better motivation for enabling LTO for Rust apps than “it sounds good” :smiley:

Actually, motivation and benefits from enabling LTO for Rust apps will be completely the same as we already have for C and C++ apps (ToolChain/LTO - Ubuntu Wiki and corresponding links from other distributions like OpenSUSE, ArchLinux, etc.): faster code and smaller binaries. C and C++ benefits is applicable here since Rust is kinda the same technology (compiled to native binaries, Rustc is based on LLVM as well as Clang, and LTO technology in GCC and LLVM ecosystem is semantically the same thing). Since motivation from faster code and smaller binaries was good enough for enabling LTO for C and C++ binaries - it should be good for Rust binaries too. Providing faster Rust binaries by default leads to better UX with such tools, and smaller binaries makes Ubuntu-based images smaller, that is useful for space-constrained scenarios (embedded-like cases and cloud containers workloads (imagine workloads with a LOT of containers).

Regarding your concerns regarding compile times and memory usage increase during the build process - they will be the same as Ubuntu already have with LTO for C and C++ packages (since LTO for Rust and LTO for C and C++ is the same thing in general). Rustc compiler (the main compiler for Rust), as well as Clang for C and C++, supports two major LTO kinds - FatLTO (aka FullLTO) and ThinLTO. I believe that Ubuntu right now uses FatLTO for C and C++. So if Ubuntu chooses FatLTO for Rust too, the build time will be increased for Rust packages twice (my personal observation based on many LTOed Rust projects - links will be provided below), and memory usage during the build process will be increased too but how much - it depends. However, I want highlight, that such change was already introduced for C and C++ packages with LTO - so it was okay. Another thing is ThinLTO - build time overhead will be much smaller (since ThinLTO allows to perform LTO in a multithreaded way) but in the price of much smaller binary size improvements and, probably, a bit less efficient code. However, choosing LTO kinds - FatLTO vs ThinLTO - is a technical decision, that we can discuss separately (but the choice should be made anyway). If some build platform are overloaded or heavily memory-constrained (things like MIPS or whatever else is supported by Ubuntu) - we can try to disable LTO for such platforms.

Some possibly useful links:

  • Issue search results · GitHub - here you can find a LOT of LTO-related issues regarding enabling LTO in Rust projects. There you can see binary size improvements for a large amount of open-source Rust projects with actual numbers about binary size and compile time increase. Note that I mostly tested only FatLTO and Codegen-Units = 1 config (since it’s the most aggressive config)
  • Regarding performance benches - it heavily depends on a project. E.g. if a project is IO-bound and not CPU-bound, you will see no performance improvements from LTO since LTO helps to perform more aggressive compiler optimizations targeted for CPU-heavy workloads (like better inlining). I can share with you results like this - Enable Link-Time Optimization (LTO) and codegen-units = 1 · Issue #1 · tsowell/wiremix · GitHub - but once again - it depends on the application kind. Completely the same situation as we already have with C and C++ packages.

Recommending enabling LTO and Codegen-units = 1 is not something unique for the Rust ecosystem. It’s already proposed by multiple resources, including articles/documentation from Ubuntu:

I hope I answered at least partially your questions/concerns. I don’t want to introduce LTO for Rust in some non-intelligent and blind way. It’s really a useful and powerful optimization technique that improves Rust-based binaries for Ubuntu users.

I can talk to the Rust team about this I guess!

Thanks a lot! I would appreciate if you could raise attention to this topic with right people.

1 Like
4

@mwhudson Hi! Just curious - have you had a chance to discuss this topic with the Ubuntu Rust team? :slight_smile: