HCL Domino 14 – tika-server.jar NSD

TL:DR – Delete the Full Text Index of Names.nsf if you see a tika server failure

One of the best features /selling points of Notes and Domino is the ability to switch off a Domino server / Domain, archive the data, for many years, and then when required restore the server, update the server to the latest version and use the archived data without any issues, or hardly any issues.

We had to restore a full Domino system from 2001 this week to access important data for a project. The previous system was running as an IBM Domino 6.0 partitioned server on Windows Server 2000 and contained about 600GB of precious data. The data was stored on magnetic tape, initially, and then transferred to a cheap hard disk at some point, and then put into a closet to be forgotten.

The restore process:

1. Mount the ATA drive with the old data and transfer the data to a file server.
2. Use Ansible to create some new Domino 14 servers on Proxmox.
3. Use autoconf with the relevant response files to recreate the old servers.
4. Stop the new servers and transfer the relevant old data.
5. Start the new servers and update the files to the latest versions.

Once complete you can access the files with the correct IDs and passwords and the cross certify the old domain with the working domains and resecure the domain.

Apart from a few security access issues and local access control issues all the data was accessible and the domain just worked. Amazing when you think about it, 23 years later and the database still just work.

However, one of the servers failed the next day with an NSD and it took a bit of investigation to find the issue and solve it.

The NSD (extract):

    0 S notes     151606    4181  1  80   0 - 669391 futex_ 07:00 ?       00:00:06 /opt/hcl/domino/notes/latest/linux/jvm/bin/java -Djt=/local/notesdata -Dll=/local/notesdata/IBM_TECHNICAL_SUPPORT/ndtika4181 -Dcf=/opt/hcl/domino/notes/latest/linux/log4jTika.xml -jar /opt/hcl/domino/notes/latest/linux/tika-server.jar -c /opt/hcl/domino/notes/latest/linux/dtikacfg.xml
    0 Z notes     151618    4181  2  80   0 -     0 -      07:00 ?        00:00:08 [java] <defunct>
    0 S notes     151688  151606  2  80   0 - 967184 futex_ 07:00 ?       00:00:08 java -Xdump:none -Xms256m -Xmx2g -Djava.io.tmpdir=/local/notesdata -Dlog4j.configurationFile=/opt/hcl/domino/notes/latest/linux/log4jTika.xml -Dll=/local/notesdata/IBM_TECHNICAL_SUPPORT/ndtika4181 -Djava.awt.headless=true -cp /opt/hcl/domino/notes/latest/linux/tika-server.jar -Dtika.server.id=Domino org.apache.tika.server.core.TikaServerProcess -h localhost -p 9998 -i Domino -c /opt/hcl/domino/notes/latest/linux/dtikacfg.xml -for
---------
<@@ Notes Process Info -> Process: java -> Status for pid 151618 @@>
    Name:	java
    State:	Z (zombie)
    Tgid:	151618

The Java engine also wrote an error file called ndtika…..

"04/15/2024 07:33:27 AM WARN  TikaServerConfig:656 - no system property set for jt, falling back to -Djava.io.tmpdir=${sys:jt}
""04/15/2024 07:33:27 AM WARN  TikaServerConfig:656 - no system property set for cf, falling back to -Dlog4j.configurationFile=${sys:cf}
""04/15/2024 07:33:28 AM ERROR JettyHTTPServerEngine:479 - Could not start Jetty server on port 9,998: Failed to bind to localhost/127.0.0.1:9998
""04/15/2024 07:33:28 AM WARN  TikaServerProcess:159 - exception starting server
" org.apache.cxf.service.factory.ServiceConstructionException: null
	at org.apache.cxf.jaxrs.JAXRSServerFactoryBean.create(JAXRSServerFactoryBean.java:220) ~[tika-server.jar:2.4.1]
	at org.apache.tika.server.core.TikaServerProcess.startServer(TikaServerProcess.java:157) ~[tika-server.jar:2.4.1]
	at org.apache.tika.server.core.TikaServerProcess.main(TikaServerProcess.java:135) ~[tika-server.jar:2.4.1]
Caused by: org.apache.cxf.interceptor.Fault: Could not start Jetty server on port 9,998: Failed to bind to localhost/127.0.0.1:9998
	at org.apache.cxf.transport.http_jetty.JettyHTTPServerEngine.addServant(JettyHTTPServerEngine.java:488) ~[tika-server.jar:2.4.1]
	at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.activate(JettyHTTPDestination.java:187) ~[tika-server.jar:2.4.1]
	at org.apache.cxf.transport.AbstractObservable.setMessageObserver(AbstractObservable.java:53) ~[tika-server.jar:2.4.1]
	at org.apache.cxf.binding.AbstractBindingFactory.addListener(AbstractBindingFactory.java:95) ~[tika-server.jar:2.4.1]
	at org.apache.cxf.jaxrs.JAXRSBindingFactory.addListener(JAXRSBindingFactory.java:91) ~[tika-server.jar:2.4.1]
	at org.apache.cxf.endpoint.ServerImpl.start(ServerImpl.java:130) ~[tika-server.jar:2.4.1]
	at org.apache.cxf.jaxrs.JAXRSServerFactoryBean.create(JAXRSServerFactoryBean.java:208) ~[tika-server.jar:2.4.1]
	... 2 more

It seems the whole Java stack was in a “zombie” state. A reboot later and the server was running again but soon NSD’s again with the Tika issue.

According to Daniel Nashed, “Tika is the new engine used to extract text from attachments for fulltext indexing in Notes/Domino 10 (client and server). It replaces the previous key view package which had a different integration (it was leveraging the kvoop binary).”

Disabling Tika while searching for the issue.

domino stop live
locate tika-server.jar
cd /opt/hcl/domino/notes/14000000/linux
ls
mv tika-server.jar tika-server.jar.backup
domino start live

After some testing the issue was found. The first database that is listed for the Tika server to index is names.nsf, if it has a search index. Best practice is to not have a search index and it turned out only one of the servers, an ancillary server, had names.nsf indexed. The tika server failed with a corrupt index structure and went into a “zombie mode” and when the Domino watchdog task lost access to the process it did a soft shutdown the server and wrote the NSD.

We deleted the index and the server is now stable.