The LDAP service becomes increasingly important as more networked systems begin to depend on it. In such an environment, it is standard practice to build redundancy (high availability) into LDAP to prevent disruption should the LDAP server become unresponsive. This is done through LDAP replication.
Replication is achieved via the Sync replication engine, syncrepl. This allows changes to be synchronised using a Consumer - Provider model. A detailed description of this replication mechanism can be found in the OpenLDAP administrator’s guide and in its defining RFC 4533.
There are two ways to use this replication:
-
Standard replication: Changed entries are sent to the consumer in their entirety. For example, if the
userPasswordattribute of theuid=john,ou=people,dc=example,dc=comentry changed, then the whole entry is sent to the consumer. -
Delta replication: Only the actual change is sent, instead of the whole entry.
The delta replication sends less data over the network, but is more complex to set up. We will show both in this guide.
Important:
You must have Transport Layer Security (TLS) enabled already before proceeding with this guide. Please consult the LDAP with TLS guide for details of how to set this up.
Provider configuration - replication user
Both replication strategies will need a replication user, as well as updates to the ACLs and limits regarding this user. To create the replication user, save the following contents to a file called replicator.ldif:
dn: cn=replicator,dc=example,dc=com
objectClass: simpleSecurityObject
objectClass: organizationalRole
cn: replicator
description: Replication user
userPassword: {CRYPT}x
Then add it with ldapadd:
$ ldapadd -x -ZZ -D cn=admin,dc=example,dc=com -W -f replicator.ldif
Enter LDAP Password:
adding new entry "cn=replicator,dc=example,dc=com"
Now set a password for it with ldappasswd:
$ ldappasswd -x -ZZ -D cn=admin,dc=example,dc=com -W -S cn=replicator,dc=example,dc=com
New password:
Re-enter new password:
Enter LDAP Password:
The next step is to give this replication user the correct privileges, i.e.:
- Read access to the content that we want replicated
- No search limits on this content
For that we need to update the ACLs on the provider. Since ordering matters, first check what the existing ACLs look like on the dc=example,dc=com tree:
$ sudo ldapsearch -Q -Y EXTERNAL -H ldapi:/// -LLL -b cn=config '(olcSuffix=dc=example,dc=com)' olcAccess
dn: olcDatabase={1}mdb,cn=config
olcAccess: {0}to attrs=userPassword by self write by anonymous auth by * none
olcAccess: {1}to attrs=shadowLastChange by self write by * read
olcAccess: {2}to * by * read
What we need is to insert a new rule before the first one, and also adjust the limits for the replicator user. Prepare the replicator-acl-limits.ldif file with this content:
dn: olcDatabase={1}mdb,cn=config
changetype: modify
add: olcAccess
olcAccess: {0}to *
by dn.exact="cn=replicator,dc=example,dc=com" read
by * break
-
add: olcLimits
olcLimits: dn.exact="cn=replicator,dc=example,dc=com"
time.soft=unlimited time.hard=unlimited
size.soft=unlimited size.hard=unlimited
And add it to the server:
$ sudo ldapmodify -Q -Y EXTERNAL -H ldapi:/// -f replicator-acl-limits.ldif
modifying entry "olcDatabase={1}mdb,cn=config"
Provider configuration - standard replication
The remaining configuration for the provider using standard replication is to add the syncprov overlay on top of the dc=example,dc=com database.
Create a file called provider_simple_sync.ldif with this content:
# Add indexes to the frontend db.
dn: olcDatabase={1}mdb,cn=config
changetype: modify
add: olcDbIndex
olcDbIndex: entryCSN eq
-
add: olcDbIndex
olcDbIndex: entryUUID eq
#Load the syncprov module.
dn: cn=module{0},cn=config
changetype: modify
add: olcModuleLoad
olcModuleLoad: syncprov
# syncrepl Provider for primary db
dn: olcOverlay=syncprov,olcDatabase={1}mdb,cn=config
changetype: add
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: syncprov
olcSpCheckpoint: 100 10
olcSpSessionLog: 100
Customisation warning:
The LDIF above has some parameters that you should review before deploying in production on your directory. In particular –olcSpCheckpointandolcSpSessionLog.
Please see the slapo-syncprov(5) man page. In general,olcSpSessionLogshould be equal to (or preferably larger than) the number of entries in your directory. Also see ITS #8125 for details on an existing bug.
Add the new content:
sudo ldapadd -Q -Y EXTERNAL -H ldapi:/// -f provider_simple_sync.ldif
The Provider is now configured.
Consumer configuration - standard replication
Install the software by going through the installation steps. Make sure schemas and the database suffix are the same, and enable TLS.
Create an LDIF file with the following contents and name it consumer_simple_sync.ldif:
dn: cn=module{0},cn=config
changetype: modify
add: olcModuleLoad
olcModuleLoad: syncprov
dn: olcDatabase={1}mdb,cn=config
changetype: modify
add: olcDbIndex
olcDbIndex: entryUUID eq
-
add: olcSyncrepl
olcSyncrepl: rid=0
provider=ldap://ldap01.example.com
bindmethod=simple
binddn="cn=replicator,dc=example,dc=com" credentials=<secret>
searchbase="dc=example,dc=com"
schemachecking=on
type=refreshAndPersist retry="60 +"
starttls=critical tls_reqcert=demand
-
add: olcUpdateRef
olcUpdateRef: ldap://ldap01.example.com
Ensure the following attributes have the correct values:
provider: Provider server’s hostname –ldap01.example.comin this example – or IP address. It must match what is presented in the provider’s SSL certificate.binddn: The bind DN for the replicator user.credentials: The password you selected for the replicator user.searchbase: The database suffix you’re using, i.e., content that is to be replicated.olcUpdateRef: Provider server’s hostname or IP address, given to clients if they try to write to this consumer.rid: Replica ID, a unique 3-digit ID that identifies the replica. Each consumer should have at least onerid.
Note:
A successful encrypted connection viaSTART_TLSis being enforced in this configuration, to avoid sending the credentials in the clear across the network. See LDAP with TLS for details on how to set up OpenLDAP with trusted SSL certificates.
Add the new configuration:
sudo ldapadd -Q -Y EXTERNAL -H ldapi:/// -f consumer_simple_sync.ldif
Now you’re done! The dc=example,dc=com tree should now be synchronising.
Provider configuration - delta replication
The remaining provider configuration for delta replication is:
- Create a new database called
accesslog - Add the
syncprovoverlay on top of theaccessloganddc=example,dc=comdatabases - Add the
accesslogoverlay on top of thedc=example,dc=comdatabase
Add syncprov and accesslog overlays and DBs
Create an LDIF file with the following contents and name it provider_sync.ldif:
# Add indexes to the frontend db.
dn: olcDatabase={1}mdb,cn=config
changetype: modify
add: olcDbIndex
olcDbIndex: entryCSN eq
-
add: olcDbIndex
olcDbIndex: entryUUID eq
#Load the syncprov and accesslog modules.
dn: cn=module{0},cn=config
changetype: modify
add: olcModuleLoad
olcModuleLoad: syncprov
-
add: olcModuleLoad
olcModuleLoad: accesslog
# Accesslog database definitions
dn: olcDatabase={2}mdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcMdbConfig
olcDatabase: {2}mdb
olcDbDirectory: /var/lib/ldap/accesslog
olcSuffix: cn=accesslog
olcRootDN: cn=admin,dc=example,dc=com
olcDbIndex: default eq
olcDbIndex: entryCSN,objectClass,reqEnd,reqResult,reqStart
olcAccess: {0}to * by dn.exact="cn=replicator,dc=example,dc=com" read by * break
olcLimits: dn.exact="cn=replicator,dc=example,dc=com"
time.soft=unlimited time.hard=unlimited
size.soft=unlimited size.hard=unlimited
# Accesslog db syncprov.
dn: olcOverlay=syncprov,olcDatabase={2}mdb,cn=config
changetype: add
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: syncprov
olcSpNoPresent: TRUE
olcSpReloadHint: TRUE
# syncrepl Provider for primary db
dn: olcOverlay=syncprov,olcDatabase={1}mdb,cn=config
changetype: add
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: syncprov
olcSpCheckpoint: 100 10
olcSpSessionLog: 100
# accesslog overlay definitions for primary db
dn: olcOverlay=accesslog,olcDatabase={1}mdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcAccessLogConfig
olcOverlay: accesslog
olcAccessLogDB: cn=accesslog
olcAccessLogOps: writes
olcAccessLogSuccess: TRUE
# scan the accesslog DB every day, and purge entries older than 7 days
olcAccessLogPurge: 07+00:00 01+00:00
Customisation warning:
The LDIF above has some parameters that you should review before deploying in production on your directory. In particular –olcSpCheckpoint,olcSpSessionLog.
Please see the slapo-syncprov(5) manpage. In general,olcSpSessionLogshould be equal to (or preferably larger than) the number of entries in your directory. Also see ITS #8125 for details on an existing bug.
ForolcAccessLogPurge, please check the slapo-accesslog(5) manpage.
Create a directory:
sudo -u openldap mkdir /var/lib/ldap/accesslog
Add the new content:
sudo ldapadd -Q -Y EXTERNAL -H ldapi:/// -f provider_sync.ldif
The Provider is now configured.
Consumer configuration
Install the software by going through the installation steps. Make sure schemas and the database suffix are the same, and enable TLS.
Create an LDIF file with the following contents and name it consumer_sync.ldif:
dn: cn=module{0},cn=config
changetype: modify
add: olcModuleLoad
olcModuleLoad: syncprov
dn: olcDatabase={1}mdb,cn=config
changetype: modify
add: olcDbIndex
olcDbIndex: entryUUID eq
-
add: olcSyncrepl
olcSyncrepl: rid=0
provider=ldap://ldap01.example.com
bindmethod=simple
binddn="cn=replicator,dc=example,dc=com" credentials=<secret>
searchbase="dc=example,dc=com"
logbase="cn=accesslog"
logfilter="(&(objectClass=auditWriteObject)(reqResult=0))"
schemachecking=on
type=refreshAndPersist retry="60 +"
syncdata=accesslog
starttls=critical tls_reqcert=demand
-
add: olcUpdateRef
olcUpdateRef: ldap://ldap01.example.com
Ensure the following attributes have the correct values:
provider: Provider server’s hostname –ldap01.example.comin this example – or IP address. It must match what is presented in the provider’s SSL certificate.binddn: The bind DN for the replicator user.credentials: The password you selected for the replicator user.searchbase: The database suffix you’re using, i.e., content that is to be replicated.olcUpdateRef: Provider server’s hostname or IP address, given to clients if they try to write to this consumer.- rid: Replica ID, a unique 3-digit ID that identifies the replica. Each consumer should have at least one
rid.
Note:
Note that a successful encrypted connection viaSTART_TLSis being enforced in this configuration, to avoid sending the credentials in the clear across the network. See LDAP with TLS for details on how to set up OpenLDAP with trusted SSL certificates.
Add the new configuration:
sudo ldapadd -Q -Y EXTERNAL -H ldapi:/// -f consumer_sync.ldif
You’re done! The dc=example,dc=com tree should now be synchronising.
Testing
Once replication starts, you can monitor it by running:
$ ldapsearch -z1 -LLL -x -s base -b dc=example,dc=com contextCSN
dn: dc=example,dc=com
contextCSN: 20200423222317.722667Z#000000#000#000000
On both the provider and the consumer. Once the contextCSN value for both match, both trees are in sync. Every time a change is done in the provider, this value will change and so should the one in the consumer(s).
If your connection is slow and/or your LDAP database large, it might take a while for the consumer’s contextCSN match the provider’s. But, you will know it is progressing since the consumer’s contextCSN will be steadily increasing.
If the consumer’s contextCSN is missing or does not match the provider, you should stop and figure out the issue before continuing. Try checking the slapd entries in /var/log/syslog in the provider to see if the consumer’s authentication requests were successful, or that its requests to retrieve data return no errors. In particular, verify that you can connect to the provider from the consumer as the replicator BindDN using START_TLS:
ldapwhoami -x -ZZ -D cn=replicator,dc=example,dc=com -W -h ldap01.example.com
For our example, you should now see the john user in the replicated tree:
$ ldapsearch -x -LLL -b dc=example,dc=com -h ldap02.example.com '(uid=john)' uid
dn: uid=john,ou=People,dc=example,dc=com
uid: john