Zookeeper\Solr - Linux Installation Reference

Below are the steps to install Solr and Zookeeper on Debian 12.7.

I have these servers, and I will install Solr and Zookeeper on them

Solr Nodes:
hqew1ww-rg-p04-solr-1.contoso.com:8983/solr/
hqew1ww-rg-p04-solr-2.contoso.com:8983/solr/
hqew1ww-rg-p04-solr-3.contoso.com:8983/solr/

Update System

$sudo apt update && sudo apt upgrade -y 

Install Java JDK

$sudo apt install default-jdk 

Check Java

$java -version

Download Solr, Install, Check Services

$cd /opt/
$sudo wget https://archive.apache.org/dist/lucene/solr/8.11.2/solr-8.11.2.tgz
$sudo tar xzf solr-8.11.2.tgz
$sudo bash solr-8.11.2/bin/install_solr_service.sh solr-8.11.2.tgz
$sudo systemctl status solr

Download Zookeeper

$sudo wget https://downloads.apache.org/zookeeper/stable/apache-zookeeper-3.8.4-bin.tar.gz
$sudo tar -xvzf apache-zookeeper-3.8.4-bin.tar.gz

Rename Directory

$sudo mv apache-zookeeper-3.8.4-bin zookeeper

Create a Zookeeper User and Group

$sudo useradd -m -r -d /var/lib/zookeeper -s /bin/false zookeeper

Set Ownership and Permissions

$sudo mkdir -p /var/lib/zookeeper

$sudo chown -R zookeeper:zookeeper /opt/zookeeper 

$sudo chown -R zookeeper:zookeeper /var/lib/zookeeper

 

Create a Config File

$sudo vim /opt/zookeeper/conf/zoo.cfg

Config file content:

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/lib/zookeeper
clientPort=2181
server.1=hqew1ww-rg-p04-solr-1.contoso.com:2888:3888
server.2=hqew1ww-rg-p04-solr-2.contoso.com:2888:3888
server.3=hqew1ww-rg-p04-solr-1.contoso.com:2888:3888

Create my ID file (For each server)

echo "1" | sudo tee /var/lib/zookeeper/myid > /dev/null
sudo chown zookeeper:zookeeper /var/lib/zookeeper/myid

echo "2" | sudo tee /var/lib/zookeeper/myid > /dev/null
sudo chown zookeeper:zookeeper /var/lib/zookeeper/myid

echo "3" | sudo tee /var/lib/zookeeper/myid > /dev/null
sudo chown zookeeper:zookeeper /var/lib/zookeeper/myid

The Command Below Starts the Zookeeper (If you want to test before run as a service)

$/opt/zookeeper/bin/zkServer.sh start

The Command Below Connects to Zookeeper

$/opt/zookeeper/bin/zkCli.sh -server 127.0.0.1:2181

To Stop

$/opt/zookeeper/bin/zkServer.sh stop

Create a System Service

$sudo vim /etc/systemd/system/zookeeper.service

[Unit]
Description=Zookeeper Daemon
Documentation=http://zookeeper.apache.org
Requires=network.target
After=network.target

[Service]
Type=forking
WorkingDirectory=/opt/zookeeper
User=zookeeper
Group=zookeeper
ExecStart=/opt/zookeeper/bin/zkServer.sh start /opt/zookeeper/conf/zoo.cfg
ExecStop=/opt/zookeeper/bin/zkServer.sh stop /opt/zookeeper/conf/zoo.cfg
ExecReload=/opt/zookeeper/bin/zkServer.sh restart /opt/zookeeper/conf/zoo.cfg
TimeoutSec=30
Restart=on-failure

[Install]
WantedBy=default.target

Reload System Daemon

$sudo systemctl daemon-reload

 Start the Zookeeper Service and Enable It to Start After System Reboot Using the Following Commands:

$sudo systemctl start zookeeper 

$sudo systemctl enable zookeeper

Error Encountered

I found an error when I tried to start the service due to permission issues. The reason is that I ran the command /opt/zookeeper/bin/zkServer.sh start using my root account. The directory /opt/zookeeper/logs/ did not have permission for the zookeeper user, and the directories were created when I tried to run the service to test using bash.

These Were the Errors:

hqew1ww-rg-p04-solr-1 zkServer.sh[3724919]: /opt/zookeeper/bin/zkServer.sh: line 164: /opt/zookeeper/bin/../logs/zookeeper-zookeeper-server-hqew1ww-rg-p04-solr-1.out: Permission denied
hqew1ww-rg-p04-solr-1 zkServer.sh[3724920]: /opt/zookeeper/bin/zkServer.sh: line 175: /var/lib/zookeeper/zookeeper_server.pid: Permission denied

You Can Check the Permissions:

$ls -lha /opt/zookeeper/logs/
$ls -lha /var/lib/zookeeper

Solution: 

Simply delete the directories or configure the ownership. The owner should be the zookeeper user.

###

 Solr Cloud (Cluster)

### 

$sudo vim /etc/default/solr.in.sh 

ZK_HOST="hqew1ww-rg-p04-solr-1.contoso.com:2181,hqew1ww-rg-p04-solr-2.contoso.com:2181,hqew1ww-rg-p04-solr-3.contoso.com:2181" 

SOLR_MODE="solrcloud"

 

Edit the Zookeeper configuration file and add the permissions:

 

$sudo vim /opt/zookeeper/conf/zoo.cfg
 

4lw.commands.whitelist=mntr,conf,ruok

Restart Solr and Zookeeper

I had an issue with ZooKeeper when I opened Solr and checked the nodes. I received the following message: 

'Only one zk allowed in standalone mode'

The issue was related to the configuration file.

server.1=hqew1ww-rg-p04-solr-1.contoso.com:2888:3888
server.2=hqew1ww-rg-p04-solr-2.contoso.com:2888:3888
server.3=hqew1ww-rg-p04-solr-3.contoso.com:2888:3888

You must use 'server' at the beginning of the line; any other value will cause your ZooKeeper to fail.

Upload the configuration to Zookeeper

Follow this article:

https://doc.sitecore.com/xp/en/developers/latest/platform-administration-and-architecture/walkthrough--setting-up-solrcloud.html#upload-the-configuration-to-zookeeper

1. In one of the Solr nodes you have set up, locate the configset folder under the server\solr\configsets and copy the _default  folder to a new sitecore_configs folder.

2. In the sitecore_configs/solrconfig.xml file , set the autoCreateFields setting to false (update.autoCreateFields:false).

3. In this new folder, open the managed-schema file in the conf folder and do the following:

   • Set the value in <uniqueKey>id</uniqueKey> to _uniqueid.

   • In the fields section, add the following field configuration for _uniqueid:

     <field name="_uniqueid" type="string" indexed="true" required="true" stored="true"/>

 Important: You should upload the configuration to Zookeeper.

To add the Solr command to youe Path.

$sudo vim ~/.bashrc

In the final of the file add this: 

export PATH=$PATH:/opt/solr/bin

Reload:

$source ~/.bashrc

$solr zk upconfig -d sitecore_config -n sitecore -z hqew1ww-rg-p04-solr-1.contoso.com:2181,hqew1ww-rg-p04-solr-2.contoso.com:2181,hqew1ww-rg-p04-solr-3.contoso.com:2181

Important: The configuration for xDB is different; it does not use the same config set. If you use the wrong config set, you will start seeing errors like this in the Sitecore logs:

"Sitecore.Xdb.Collection.Search.Solr.Failures.SolrResponseException at Sitecore.Xdb.Collection.Search.Solr.SolrClient.EnsureSolrSuccessStatusCode"

"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"undefined field id",
"code":400

Upload your config set based on _default.

$solr zk upconfig -d _default -n xdb -z hqew1ww-rg-p04-solr-1.contoso.com:2181,hqew1ww-rg-p04-solr-2.contoso.com:2181,hqew1ww-rg-p04-solr-3.contoso.com:2181

https://doc.sitecore.com/xp/en/developers/latest/platform-administration-and-architecture/walkthrough--using-solrcloud-for-xconnect-search.html#create-an-xdb-collection

Create a collection

For all non xDB you should use the sitecore config set

For xDB use xDB config set.

 

Enable Solr SSL

I will convert my PFX to the correct format to allow SSL to be used on Solr

$openssl pkcs12 -in SolrProd.pfx -nocerts -out solrprod-key.pem -nodes
$openssl pkcs12 -in SolrProd.pfx -clcerts -nokeys -out solrprod-cert.pem


$openssl pkcs12 -export -in solrprod-cert.pem -inkey solrprod-key.pem -out solr-prod.p12 -name solprod-ssl

Using Jave keyTool

$keytool -importkeystore -destkeystore solrprod-ssl.keystore.jks -srckeystore solr-prod.p12 -srcstoretype PKCS12 -alias solrprod-ssl

sudo vim /etc/default/solr.in.sh

SOLR_SSL_KEY_STORE=/var/solr/solrprod-ssl.keystore.jks
SOLR_SSL_KEY_STORE_PASSWORD=yourSSLpasswordXXX
SOLR_SSL_TRUST_STORE=/var/solr/solrprod-ssl.keystore.jks
SOLR_SSL_TRUST_STORE_PASSWORD=yourSSLpasswordXXX
SOLR_SSL_NEED_CLIENT_AUTH=false
SOLR_SSL_WANT_CLIENT_AUTH=false

Tip: 

$sudo chown solr:solr /var/solr/solrprod-ssl.keystore.jks 

 The Solr user must have permission to read the file solrprod-ssl.keystore.jks 

Restart Solr

I had an issue on issue on the logs:

INFO (MetricsHistoryHandler-20-thread-1) [ ] o.a.s.c.s.i.SolrClientNodeStateProvider$ClientSnitchCtx Error on getting remote info, trying again: IOException occurred when talking to server at: http://serverIP:8983/solr

Add this to your Solr config file:

SOLR_OPTS="$SOLR_OPTS -Dsolr.ssl.checkPeerName=false"

Note: solr.ssl.checkPeerName=false in Solr, you can add this setting in the Solr configuration file to disable peer name checking for SSL connections. 

I don't have the IPs in my SSL certificate. If you include the IP or hostname (if you are using a hostname in the config file) in the certificate, you can avoid using -Dsolr.ssl.checkPeerName=false

Your Solr config file should be like this (/etc/default/solr.in.sh
):

SOLR_OPTS="$SOLR_OPTS -Dsolr.ssl.checkPeerName=false"
SOLR_SSL_KEY_STORE=/var/solr/solrprod-ssl.keystore.jks
SOLR_SSL_KEY_STORE_PASSWORD=yourSSLpasswordXXX
SOLR_SSL_TRUST_STORE=/var/solr/solrprod-ssl.keystore.jks
SOLR_SSL_TRUST_STORE_PASSWORD=yourSSLpasswordXXX
SOLR_SSL_NEED_CLIENT_AUTH=false
SOLR_SSL_WANT_CLIENT_AUTH=false

If you're running SolrCloud and have collections that were created prior to enabling SSL, you'll need to modify the cluster properties to enable HTTPS.

curl -k "https://hqew1ww-rg-p04-solr-1.contoso.com:8983/solr/admin/collections?action=CLUSTERPROP&name=urlScheme&val=https"

Bonus

In fact, I found two issues with this installation, and below are some of my notes that may be useful for someone 

I had an issue with SSL:

caused by javax.net.ssl.sslhandshakeexception no subject alternativenames matching

The only way for me, at least, to fix this was by using an SSL certificate with the DNS name in the subject. I tried using the option -Dsolr.ssl.checkPeerName=false, but it did not work.

Another issue was related to how Solr registers with SolrCloud. I wanted the registration to be based on the FQDN instead of the IP address, as using IPs can cause SSL issues (if you don't have the IP in the subject name) due to how collections replicate between Solr nodes. To achieve this, I tried using the SOLR_HOST setting as described in the Solr documentation, but it did not work. The solution I found was to edit the solr.xml configuration file and add the Solr server name there, like this:

$sudo vim /var/solr/data/solr.xml

  <solrcloud>

    <str name="host">hqew1ww-rg-p04-solr-1.contoso.com</str>
    <int name="hostPort">${solr.port.advertise:0}</int>
    <str name="hostContext">${hostContext:solr}</str>

    <bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool>

    <int name="zkClientTimeout">${zkClientTimeout:30000}</int>
    <int name="distribUpdateSoTimeout">${distribUpdateSoTimeout:600000}</int>
    <int name="distribUpdateConnTimeout">${distribUpdateConnTimeout:60000}</int>
    <str name="zkCredentialsProvider">${zkCredentialsProvider:org.apache.solr.common.cloud.DefaultZkCredentialsProvider}</str>
    <str name="zkACLProvider">${zkACLProvider:org.apache.solr.common.cloud.DefaultZkACLProvider}</str>

  </solrcloud>

I added the hostname for each server. The example above is for node 1 (hqew1ww-rg-p04-solr-1.contoso.com).

--

You should increase the heap memory for your Solr. To do this, edit the configuration file and add the SOLR_HEAP variable. In my example, I allocated 4 GB of memory:

$sudo vim /etc/default/solr.in.sh
SOLR_HEAP=4096m