Domino 14.0 on Linux does not work on very old CPUs  

By Daniel Nashed | 3/6/24 4:55 PM | Infrastructure - Notes / Domino | Added by Oliver Busse

Domino 14.0 requires glibc 2.34 because it is build on Redhat Linux 9.1. Glibc is the Linux C run-time which is a core building blog of Linux and can't be changed. This means you can't run it on Linux versions which don't have at least glibc 2.34. Glibc 2.34 itself doesn't work on old CPUs not supporting at least a CPU with microarchitecture level x86-64-v2

Security bulletin: Passwords of Domino Internet users are vulnerable  

By Martijn de Jong | 2/22/24 1:23 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

The official title of the security bulletin is: “HCL Domino is susceptible to a weak cryptography vulnerability (CVE-2023-37495).” The problem is with person documents that were created using the “Add Person” button in the Domino Directory. For people less savvy in Domino: that’s not the usual way to add users to Domino. In Domino, we register users using a certifier file. The only time we add persons to the Domino Directory using the “Add person” button, is when we know that these users will only ever access a Domino application through a web browser. The problem with these “internet users” is that the hash in the Domino Directory for the HTTP password uses a cryptographically weak hash algorithm. If an attacker has access to these hashes, he could determine the user’s password through a brute force attack. You can’t see these hashes from a browser, so the attacker needs to have access to the Domino Directory through a Notes or Nomad client. That limits the potential attackers to all users who are registered as Notes users inside the company.

Auth0 ODIC OpenID with Domino & Some other interesting findings  

By Daniel Nashed | 2/22/24 1:21 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

We are working on a ODIC setup with a German business partner for a larger German customer. Auth0 is one of the major providers. We got it working but only with some tricks for now. It turned out the Auth0 OIDC endpoint has a cache expiration for 15 seconds. This looks like a setting that can't be changed. The Domino OIDC cache uses the expiration header to invalidate the cache. So our cache on the Domino side was constantly reloading and invalid in some cases. You really have to have an expiration that is at least a couple of minutes. Better at least 1 hour. Faking the cache expiration This has been reported to HCL and the team is working on an enhancement. Meanwhile I came up with a work-around setting up a Fake provider on a NGINX server to forward the requests.

Running offline activities on databases with Domino V14   

By Rainer Brandl | 2/22/24 1:16 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Last week I migrated a HCL Domino V11 server to V14 and tried to run the compact task to upgrade the databases to ODS55 but received the following error:HCL Notes: error 0x1F3After some investigations and a very helpful hint of the HCL Support I could modify my existing script because due to the structural change in HCL Domino V14 the NOTES.INI now is located in the Domino\Data directory and so you have to run an offline compact this way: – Stop the HCL Domino Server – Change to the Domino\Data directory – [dominoprogramdirectory]\ncompact.exe -ODS -# 4

Domino Backup/Restore with multiple configurations and targets  

By Daniel Nashed | 2/22/24 1:14 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino Back/Restore is a flexible framework for native Domino backup. The dominobackup.nsf plays an important role for backup and restore operation. It contains the following type of content. Backup/restore/prune configuration Inventory documents for restore operations Restore requests Backup logs You could run backup with different excludes defined on command-line. Or just backup selected databases or incremental backups. But there cannot be different active configurations nor different backup retention in one dominobackup.nsf

Domino autoupdate.nsf for fast internal software downloads  

By Daniel Nashed | 2/22/24 1:13 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino Autoupdate has been introduced in Domino 14.0. It offers automatic downloads from My HCLSoftware download, which has been on early access in parallel and has been released at the same time. My Engage session will go into detail about the functionality with tips and tricks and additional information round both features and the new Domino Download script (https://nashcom.github.io/domino-startscript/domdownload/). But I want already provide some details about options available today with simple integrations.

Easy to use container image providing ICAP support for ClamAV for Domino CScan  

By Daniel Nashed | 2/22/24 1:11 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

My friend and fellow Ambassador Roberto Boccadoro submitted an Engage session about Domino CScan with ICAP. Sadly his session did not make it into the agenda. But he is part of two OpenNTF sessions. This session idea lead to a new OpenSource project I initiated to help with ICAP support. Thanks Roberto for pushing me to get this implemented! :-) The new project provides a simple to build container image, which natively offers ICAP services over TLS with a ClamAV container in the back-end. The container is ready to be consumed with Domino CScan/ICAP (https://help.hcltechsw.com/domino/14.0.0/admin/conf_scanningattachmentsforviruses.html). It comes with a docker-compose file which glues the official ClamAV container with this new image.

Running Domino in LXC containers on Proxmox requires a trick  

By Daniel Nashed | 2/22/24 1:10 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Now that VMware might not be everyone's darling any more because of it's new mother ship, I took another look at Proxmox. I know them for quite a while and they are doing a great job. I rebuilt a Intel NUC with 2 TB NVMe disk with the current version of Proxmox. Proxmox supports full VMs and also LXC -- which is an interesting option for testing in lab environments. You can setup a new Linux test machine in minutes from a template. And there are ready to use templates for all major Linux distributions. I had a post long time ago about Proxmox automation on command-line.

Welcome Domino License Analysis Utility (DLAU) 1.2.2  

By Thomas Hampel | 2/20/24 5:28 AM | Infrastructure - Notes / Domino | Added by Oliver Busse

The new version 1.2.2 of the Domino License Analysis Utility (DLAU) has just been published. The tool allows customers to analyze their current environment to identify the license needs. The new version addresses a number of issues ad improvement requests customers had reported, here's a short list.

Certificate ASN.1 Decoding online  

By Daniel Nashed | 2/15/24 2:35 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Now that I posted the TLS 1.2 interactive information side today, some of you might also want to get details out of certificates. Certificates are usually public information. So it should be OK to paste them into the website https://asn1js.eu/. But there is a GitHub project referenced and you could run it also locally. The inner guts of certificates are presented in ASN.1. When you ever has looked at OpenSSL C code, you will recognize the structures.

The Illustrated TLS 1.2 Connection -- Every byte explained  

By Daniel Nashed | 2/15/24 2:32 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

While debugging a TLS connection issue, I ran into this website -->https://tls12.xargs.org/ It provides more details then most admins ever want to know. But it is a great resource understanding a TLS connection.

Domino Backup customized and centralized logging  

By Daniel Nashed | 2/12/24 2:25 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

This question came up last week in a business partner workshop. The partner wanted to centralize the logging of all Domino backup instances. dominobackup.nsf intended per server. You could configure a global configuration database and local instances for the backup inventory. In theory it could be one database for multiple could be also replicated in smaller environments. This would not be recommended. But there is an easier way for a centralized overview of all your Domino backups.

Domino adding Trusted Roots for Java applications  

By Daniel Nashed | 2/12/24 2:24 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino has different places to store trusted roots depending on the part of the application. Beginning with Domino 12.0.2 HCL started to consolidate root certificates into the new domain wide certstore.nsf. But it will take some time to have all parts of Domino to use the new trusted roots back-end. New callers like OIDC or CScan/ICAP and the certificate URL heath check already use the new back-end including UI integration. JVM trusted roots cacerts overwritten by Domino update Java still uses it's own cacerts file, which is part of the JVM directory. The file is only admin/root writable. Domino release installers replace the cacerts file with the latest cacerts available. But this overwrites custom certificates imported into cacarts. This is a common problem I ran into twice in the last two weeks.

Enable DKIM for Domino  

By Daniel Nashed | 2/12/24 2:22 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Enable DKIM for Domino - DKIM inbound is supported starting with Domino 12.0.1 - DKIM outbound is supported starting with Domino 12.0.2 Now the first providers raise the bar for sending mails. This might not only be relevant for mass mail. Here is a short write up enabling DKIM for RSA and Ed25519 keys.

New default Let’s Encrypt certificate chain with ISRG Root X1 root  

By Daniel Nashed | 2/12/24 2:18 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Let's Encrypt finally changed their default root certificate from DST Root CA X3 to ISRG Root X1. The old root expired already 2 1/2 years ago, but was cross signed with the new chain. Now finally Let's Encrypt uses the new root by default, which results in a shorter chain. They have been using the older, longer chain to specially support older Android devices, which didn't have the X1 root in their trust store. When you are using Let's Encrypt ACME and did not specify an alternate chain, there is nothing to change. The new shorter certificate chain will be automatically used the next time the certificate is renewed. But in case you set specific settings, you might now have to remove those settings, because they flipped the certificate chains. The alternate chain is now the older longer certificate chain. For Domino CertMgr the custom setting is "ACME Alternate Chain Suffix".

Install Traveler 14 on Windows 11  

By Manfred Dillmann | 1/17/24 3:14 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Of course, I am aware that Windows 11 is not an HCL-supported operating system for the Domino or Traveler Server. Nevertheless - for example, I use Windows 11 VMs for my online courses and wanted to install the Traveler Server on this operating system.

How to make private calendar entries marked “Private” in the Resource Database  

By Rainer Brandl | 1/16/24 3:01 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

By default if you create a calendar entry and you mark this entry as "Private" and schedule either a room or a resource the entry in the resource reservation database has no private mark and the subject still is displayed.This can be customized by putting the following entry in the NOTES.INI of the server:RNR_MAKE_TOPIC_PRIVATE=1

The Art of Troubleshooting   

By Daniel Nashed | 1/15/24 3:38 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

In all the years I am involved in troubleshooting, I still see the same patterns. I am planning to start a new initiative this year. To start with, I wrote a short abstract end of the year when looking into this. See this as a beginning of a change from my side to better help on community level and also provide better services as a HCL business partner. It will also include troubleshooting steps for different kind of problem types like crashes, hangs, memory leaks, performance problems. Not all of it can be described in howto material. But raising the awareness on all parts of the support process, can significantly help to solve problems faster. I am in the troubleshooting business for over 25 years, read NSD before breakfast and wrote my own troubleshooting tools.

Installing wireguard on CentOS Stream 9   

By Martijn de Jong | 1/15/24 3:37 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

As I do a lot of my research on new Domino versions, Connections versions and HCL DX on my own server at home and as I’m often not at home, I figured I needed a VPN tunnel to my server, so I can work as if I am home. Wireguard has become kind of the de facto standard for these kind of situations, so I looked into installing it on my CentOS Stream 9 host.

Important to know about Domino 14 Inbound Mail Disclaimer  

By Daniel Nashed | 1/12/24 2:40 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

A partner ask in OpenNTF channel about an issue with the new inbound mail disclaimer in Domino 14. It turned out I was able to reproduce the problem on one of my lab servers and this is critical to know if you enable disclaimers written into the body. There are two different ways to setup the feature. 1. Tag the subject line 2. Write HTML to the body of the message with fall back to subject for signed or encrypted messages. My personal recommendation is always to tag the subject, because modification of the body can lead to all kind of side effects with add-on software and also issues that could happen with incorrect formatted messages.

Running Traveler 14 on Windows 11 for testing  

By Daniel Nashed | 1/8/24 3:57 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

The Windows desktop versions are completely untested and unsupported for all Domino related products. It is not recommended to run any Domino software in production on Windows desktop! There are still a couple of scenarios where Windows desktop can be relevant. For example a local notebook installation, a training environment or the Windows Sandbox -- which is also a Windows desktop environment. Domino 14.0 changed the installer to prompt to confirm the unsupported Windows version. But the installer continues to work with Windows 11 after you confirmed the message. As a German partner discovered last week, Traveler 14.0 doesn't install on Windows 11. InstallAnywhere runs into an error: Windows DLL failed to load during Installation. Windows 10 works unchanged. The issue starts with Windows 11.

Automatically Updating Ubuntu Linux including mail notification  

By Daniel Nashed | 1/8/24 3:51 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

This was on my list for a while. I looked into it, but the mail part wasn't what I wanted. Most admins will just install BSD mailx. But this had way too much dependencies for me. I just wrote a simple tool to replace mailx -> https://github.com/nashcom/nsh-tools/tree/main/nshmailx. But you could just continue to use the standard package. It wasn't just an option for me, because I try to not install other mail server components on a Domino server (like sendmail). The setup for automatic updates are pretty easy and base on scripting.

How to send mails native on Linux and MacOS from command-line  

By Daniel Nashed | 1/8/24 3:49 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

End of last year SMTP Smuggling was a hot topic. It turned out Domino is behaving well as you can read in Thomas Hampel's blog post who is also linking an interesting presentation about this topic. (See details here --> https://blog.thomashampel.com/blog/tomcat2000.nsf/dx/is-hcl-notes-domino-affected-by-smtp-smuggling.htm). Because I didn't find any good tool to craft emails required to check, I wrote a simple command-line tool in C. I used the OpenSSL versions of the network communication to abstract the calls from the standard socket operations. It's a nice show case how SMTP works under the covers -- which didn't change for centuries and works the original standards (RFC 821 and 822) are still what is behind all SMTP traffic. Once that worked, I thought adding STARTTLS would be a good addition. After I was done with my smuggling tests and I could sleep well again, I started thinking about to make it a more useful tool. So I added attachment support creating a MIME message with a Base64 encoded attachment. Here is a link to the proejct in my tools repository --> https://github.com/nashcom/nsh-tools/tree/main/nshmailx

Let’s Encrypt new default chain February 8, 2024  

By Daniel Nashed | 1/8/24 3:47 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Let's Encrypt new default chain February 8, 2024 The change has been announced mid last year and there is probably no action for you to take. It was about time for this move and it is well planned ahead of time. If you didn't change anything on Domino CertMgr side, there is very likely no action to take.

Is HCL Notes / Domino affected by SMTP smuggling?  

By Thomas Hampel | 1/4/24 1:54 AM | Infrastructure - Notes / Domino | Added by Oliver Busse

SMTP Smuggling is a newly discovered attack to a number of mail server products and mail hosting providers. All mail transfer are based on the SMTP protocol that exists for years where server and client, or two servers talk to each other as defined in an internet standard (RFC 5321) Even with perfect antispam checking, SPF, DKIM, DMARC, etc in place, the vulnerability would allow sending spoofed emails, which can result in a huge problem.

OpenNTF - January Webinar: Deep Dive into Debugging Domino with the Divas and “the Dude”  

By OpenNTF | 1/3/24 4:58 PM | Infrastructure - Notes / Domino | Added by Oliver Busse

Debugging Domino has many facets. This session will delve into the do’s and don’ts when debugging Domino, whether it be administration issues or application issues. Don’t panic, instead come to this session. Learn from these industry experts who have spent many years debugging and resolving real-world problems in almost every aspect of Domino you can imagine.

Recovering diskspace Windows  

By Fredrik Norling | 1/2/24 11:28 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Several gigabyte of data can be lost on your domino server but with this amer trick this can easily be recovered. I will continue the theme as the blog post Kim Greene did https://www.dominodiva.com/2023/12/recovering-disk-space-by-deleting-tmp-files/ The major problem for me on windows servers is not actually files that ends with .tmp it’s files that start with EO and ends with TM.

Recovering Disk Space by Deleting TMP files  

By Kim Greene | 1/2/24 11:27 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

When working on a problem for a customer, it was noticed there were several temporary files on the mail server in the /mail directory. This customer’s Domino servers run on the IBM i, so there are multiple Domino servers on the system, a total of four. I decided to do a search to determine how many temporary files existed on the entire IBM i system and where they were located.

HCL Nomad for Domino behind an Apache Reverse Proxy  

By Remco Angioni | 12/28/23 3:04 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Reading the documentation, HCL only gives us the configuration for NGINX. Some companies are using Apache and not NGINX. We encourage the move to NGINX, but for now Apache is still okay.

Domino 14 on Windows & HCL Ambassador  

By Oliver Busse | 12/22/23 10:15 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

I could not believe my eyes but I just received the congratulations from the HCL Ambassador program - yes, somehow I made it again for 2024. Thank you for having me again in this group of remarkable people! One of those amazing people is my friend Roberto Boccadoro who helped to fix my issues when trying to install Domino 14 on a Windows server. Usually I don’t touch this OS if I am not forced, but I run a test machine to explicitly gain those experiences - mostly frustrating. This case was not any different. I won’t describe the solution here but I want to point you to Roberto’s excellent blog post. And yes, the guy mentioned in his post is me.