library introduction
A library in Linux (or programming in general) is a collection of pre-compiled code that can be reused in a programs. programmers don’t have to write complex and common functions that have already been written by others. So programmers link to libraries to save the time, reduce errors and focus on unique sides of their code instead of reinventing the wheel.
For example, libc is a library that has functions for I/O operations and memory management, and it is used in almost every program in Linux
different types of libraries
We have two types of libraries:
- static library
- dynamic linked library
static library: The code of the library is merged into the program in compilation process. The code of a static library is embedded directly into the executable file.
- Advantages: Self-contained executables, no dependency issues.
- Disadvantages: Larger binary size, harder to update.
dynamic linked library: The program doesn’t contain the library code. Instead, the program contains metadata and links to the required libraries. At the runtime, dynamic linker in Linux loads these libraries into the memory.
- Advantages: Smaller executables, easier updates, memory efficiency.
- Disadvantages: Dependency management, potential version conflicts.
note: dynamic linked libraries can also be used and shared by other programs at the same time, so they also known as shared libraries.
Understanding the differences between these two types of libraries, their advantages, and their use cases is essential for any developer working in Linux. In this document, we will explore the concepts, creation, usage, and management of both static and dynamic libraries in detail.
library locations in Linux
Almost every libraries are located in three locations :
- /lib/
- /usr/lib/
- /usr/local/lib/
/lib/ : Necessary libraries for booting the system and running basic commands in the root filesystem (for binaries in /bin or /sbin).
This directory also contains kernel modules.
/usr/lib/ : Libraries for user-installed software and non-essential system programs (for binaries in /usr/bin or /usr/sbin) .
note:
1- In modern sytems /lib/ are linked to /usr/lib/ and the content of both is the same.
2- You might see /lib64/ for 64-bits systems or /lib32/ for 32-bits systems.
/usr/local/lib : Contains libraries for software installed locally by the system administrator or users. These libraries are typically associated with programs compiled and installed manually (outside of the system’s package manager).
shared library naming conventions
Shared libraries are named in two ways:
- soname
- real name
soname
It is named in following format:
lib<name>.so.<major>
note: the major version of library is the main version that is in-compatible and programs linked against one major version of the library may not work with another major version due to potential incompatibilities.
Soname is used to keep the compatibility between the shared library and the binaries which are linked with the shared library.
When listing required shared libraries, binaries should include the sonames of those libraries.
real name
The real name is the actual library file that contains the library code. It is named like soname appended with .<minor>.<patch>.
minor and patch are compatible versions of a library. Programs linked against one minor version of the library can work with another minor version.
Once installation of a shared library is complete, soname is a symbolic link to the real name (i.e. actual library file).
example:
root@ubuntu:/lib/x86_64-linux-gnu# ls -l | grep libread
lrwxrwxrwx 1 root root 18 Nov 26 2017 libreadline.so.6 -> libreadline.so.6.3
-rw-r--r-- 1 root root 282392 Feb 4 2016 libreadline.so.6.3
In this example the libreadline.so.6 is soname that is linked to libreadline.so.6.3 (the real name).
ldd
This is a utility that lists the shared library dependencies for a binary program.
root@ubuntu:/# ldd /usr/bin/ping
linux-vdso.so.1 (0x00007ffd7d9ba000)
libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007fc44379b000)
libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0 (0x00007fc443779000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc443400000)
libunistring.so.5 => /lib/x86_64-linux-gnu/libunistring.so.5 (0x00007fc443253000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc4437d4000)
locations for finding libraries
When you run a program, the dynamic linker (you will see in next section) should find and loads the requires libraries. But where should it look for these libraries?
/etc/ld.so.conf
This is a configuration file where some places to find libraries are defined.
root@ubuntu:/# cat /etc/ld.so.conf
include /etc/ld.so.conf.d/*.conf
As you can see this file doesn’t contain any location for libraries, instead it includes other config files in /etc/ld.so.conf.d directory.
root@ubuntu:/# ls /etc/ld.so.conf.d
fakeroot-x86_64-linux-gnu.conf libc.conf x86_64-linux-gnu.conf
root@ubuntu:/# cat /etc/ld.so.conf.d/x86_64-linux-gnu.conf
# Multiarch support
/usr/local/lib/x86_64-linux-gnu
/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu
As you can see in ld.so.conf.d directory there are some config files that has been specified some path to find libraries.
standard locations
As we said before there are several locations where libraries are located. /lib, /usr/lib, /usr/lib64, /usr/local/lib and so on.
/etc/ld.so.cache & ldconfig
If dynamic linker wants to search all these locations to find a specific library file, it will take a long time. Isn’t there a better way? It definitely is.
ldconfig is a utility that searches all the locations which we mentioned before and makes a cache file in /etc/ld.so.cache. This cache file contains the names and paths of the most recent shared libraries.
So instead of searching all these places, dynamic linker can find a specific library so fast in this cache.
To lists of directories and candidate libraries stored in the current cache:
root@ubuntu:/# ldconfig -p
1112 libs found in cache `/etc/ld.so.cache'
libzvbi.so.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/libzvbi.so.0
libzvbi-chains.so.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/libzvbi-chains.so.0
libzstd.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libzstd.so.1
libzmq.so.5 (libc6,x86-64) => /lib/x86_64-linux-gnu/libzmq.so.5
libzip.so.4 (libc6,x86-64) => /lib/x86_64-linux-gnu/libzip.so.4
libzip.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libzip.so
libz.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libz.so.1
libz.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libz.so
And you can see the full cache, which I only showed a part of here.
note: When you install/upgrade a library or create your own you should run sudo ldconfig command to update your cache and create necessary symbolic links.
LD_LIBRARY_PATH
This is an environment variable that is possible to set directories in which the dynamic linker will look for a library sooner than anywhere else.
root@ubuntu:/# export LD_LIBRARY_PATH=/home/mahdi/lib/:/usr/lib/mahdi/ #to set this env variable
root@ubuntu:/# echo $LD_LIBRARY_PATH #to see the value of this env variable
/home/mahdi/lib:/usr/lib/mahdi/
dynamic linker
dynamic linker/loader: This is a program that is responsible to load required libraries for a program. ld-linux is the dynamic linker/loader in Linux systems. This program may have different names in different distributions, but in Ubuntu it is known as ld.so.
root@ubuntu:/# ld.so /usr/bin/ls -l
total 4194400
lrwxrwxrwx 1 root root 7 Apr 22 2024 bin -> usr/bin
drwxr-xr-x 2 root root 4096 Feb 26 2024 bin.usr-is-merged
drwxr-xr-x 3 root root 4096 Feb 2 21:37 boot
dr-xr-xr-x 2 root root 4096 Apr 24 2024 cdrom
drwxr-xr-x 19 root root 4440 Feb 24 11:03 dev
drwxr-xr-x 159 root root 12288 Feb 20 08:33 etc
drwxr-xr-x 7 root root 4096 Dec 16 21:54 home
lrwxrwxrwx 1 root root 7 Apr 22 2024 lib -> usr/lib
lrwxrwxrwx 1 root root 9 Apr 22 2024 lib64 -> usr/lib64
drwxr-xr-x 2 root root 4096 Apr 8 2024 lib.usr-is-merged
drwx------ 2 root root 16384 May 2 2024 lost+found
drwxr-xr-x 3 root root 4096 May 2 2024 media
drwxr-xr-x 5 root root 4096 Oct 10 12:58 mnt
drwxr-xr-x 2 root root 4096 Apr 24 2024 opt
dr-xr-xr-x 361 root root 0 Feb 24 11:02 proc
drwx------ 12 root root 4096 Feb 20 19:57 root
drwxr-xr-x 42 root root 1060 Feb 24 11:26 run
lrwxrwxrwx 1 root root 8 Apr 22 2024 sbin -> usr/sbin
drwxr-xr-x 2 root root 4096 Mar 31 2024 sbin.usr-is-merged
drwxr-xr-x 16 root root 4096 May 9 2024 snap
drwxr-xr-x 2 root root 4096 Apr 24 2024 srv
-rw------- 1 root root 4294967296 May 2 2024 swap.img
dr-xr-xr-x 13 root root 0 Feb 24 12:07 sys
drwxrwxrwt 21 root root 4096 Feb 24 12:00 tmp
drwxr-xr-x 14 root root 4096 May 2 2024 usr
drwxr-xr-x 15 root root 4096 Sep 6 22:01 var
In this example ld.so (dynamic linker) found and loaded required libraries for ls then ran the program.
So why we do run different programs without ld.so and they still work properly?
Because binary programs and Utilities are in Executable and Linkable Format (ELF) and when you run a ELF binary file, it loads the dynamic linker first.
