SOLR-8442: Make security easier WIP by epugh · Pull Request #5107 · apache/solr

epugh · 2026-10-12T11:40:40Z

WIP exploration for SOLR-8442 and related security usability work. This work is intended to be split into smaller, focused PRs once I get some community feedback.

Description

Make it easier to configure and try Solr security from the command line. This branch adds a guided bin/solr start -e secure-setup example that combines HTTPS, HTTP authentication and authorization, and ZooKeeper access control in a two-node SolrCloud cluster using embedded ZooKeeper.

Solution

Add a bin/solr ssl command that imports PEM certificates and private keys into JKS or PKCS12 keystores, builds a truststore, updates the include file, and optionally sets the SolrCloud urlScheme property. Support encrypted private keys and password files. Thanks @janhoy for the suggestion.
Add the Unix/macOS-only secure-setup example set up! Generate a local self-signed certificate, show the generated file locations, and prompt the user to verify HTTPS with a v2 curl request before configuring HTTP accounts. Unfortunantly osx/linux only as I couldn't test effectively on windows
- Start two HTTPS nodes sharing the first node's embedded ZooKeeper. Create a fresh techproducts collection with two shards placed across the nodes, index sample documents as the index user, and exercise distributed updates and queries with security enabled. Preserve existing collections and reuse saved credentials and TLS configuration on restart.
- Separate administrative and document access: the administrator has superadmin and admin; index has indexing and search roles; search has the search role. Use demonstration passwords by default, offer password customization, and retry invalid wizard input using the shared password validation.
- Allow anonymous health and metrics requests while keeping document and administrative APIs protected. Explain the generated security.json, permission order, and v2 monitoring checks in the tutorial.
- Configure ZooKeeper digest authentication and ACLs with separate administrator and read-only accounts. Save HTTP and ZooKeeper credentials in private properties files, print both paths together, and show an authenticated ZooKeeper CLI example when setup finishes.
Add --zk-credentials to the ZooKeeper CLI commands in both CLI implementations, independently of HTTP --credentials. Preserve configured ACL providers when supplying explicit connection credentials. Trying to get away from the weird hardcoded settings in solr.in.sh approach for a cli tool!
Add the securing-Solr tutorial and update the SSL, ZooKeeper, generated CLI, and control-script reference documentation. Keep walkthrough instructions in the tutorial and command behavior in the reference.

Significant implementation, documentation, and test work was assisted by OpenAI Codex.

Tests

Focused secure-example tests for both CLI implementations, including default/custom passwords, credential reuse, authorization, and communication between nodes.
./gradlew iTest --tests test_secure_example.bats: HTTPS verification guidance, two-node setup and restart, anonymous monitoring, HTTP roles, and ZooKeeper ACLs.
./gradlew iTest --tests test_ssl.bats: all seven tests pass, covering HTTPS, Basic Authentication, hostname checks, client truststores, required/optional mTLS, and keystore reload.

Checklist

I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
I have created a Jira issue and added the issue ID to my pull request title.
I have given Solr maintainers access to contribute to my PR branch.
I have developed this patch against the main branch.
I have run ./gradlew check.
I have added tests for my changes.
I have added documentation for the Reference Guide.
I have added a changelog entry for my change.

epugh · 2026-10-29T10:42:47Z

@janhoy you gave me some feedback I think, but now I can't find it ;-(

…ecurity.json" and then uploading it.. we can't use server side apis.

Instead of burying them deep in the solr.include.sh file!

…iring security.

…sed attempt!

…rial

epugh · 2026-11-02T15:29:25Z

I wonder jhow much of this can be pulled out and be it's own refactoring? Like this guy..

epugh · 2026-11-02T15:30:00Z

          .required()
-          .desc("Name of the example to launch, one of: cloud, techproducts, schemaless, films.")
+          .desc(
+              "Name of the example to launch, one of: cloud, techproducts, secure-setup, schemaless, films.")


do we need to be smart and not show secure-setup on windows?

epugh · 2026-11-02T15:36:32Z

      paramLabel = "NAME",
      description =
-          "Name of the example to launch, one of: cloud, techproducts, schemaless, films.")
+          "Name of the example to launch, one of: cloud, techproducts, secure-setup, schemaless, films.")


also needs to be aware of windows!

epugh · 2026-11-02T15:38:26Z

+          changePasswords
+              ? promptPassword(input, "Password for zk-readonly [SolrRocks]: ", "zk-readonly")
+              : "SolrRocks";
+      secureZk.create(zkAdminPassword, zkReadonlyPassword);


i wonder if this should be it's won bin/solr auth tool? or bin/solr zk auth tool call? Probably bin/solr auth?

epugh · 2026-11-02T15:38:54Z

+    Map<String, Object> publicMonitoring = new HashMap<>();
+    publicMonitoring.put("name", "public-monitoring");
+    publicMonitoring.put("collection", null);
+    publicMonitoring.put("path", List.of("/____v2/node/health", "/____v2/metrics"));


ugh, I hate teh ____v2 here!

epugh · 2026-11-02T15:40:26Z

+      String zkPassword = zkCredentials.getProperty("zkDigestPassword");
+      echo("List ZooKeeper data with TLS and digest authentication:");
+      echo(
+          "bin/solr zk ls / --zk-tls-ca-file '"


are we missing --zk-tls ??

epugh · 2026-11-02T15:40:55Z

+
+  private void configureSecureExampleSSL(Scanner input, StartSolrParams start, boolean noPrompt)
+      throws Exception {
+    echo("Generating sample PEM certificate and key files. OpenSSL must be available on PATH.");


we could be all java!

epugh · 2026-11-02T15:42:00Z

+        ("techproducts".equals(params.example()) || "secure-setup".equals(params.example()))
            ? "-Dsolr.modules=clustering,extraction,langid,ltr,scripting,sql -Dsolr.ltr.enabled=true -Dsolr.clustering.enabled=true"
            : "";
+    if ("secure-setup".equals(params.example())) {


i thought we had a constant we added so we didn't need "secure-setup"?

epugh · 2026-11-02T15:43:00Z

          "Unsupported example "
              + exampleType
-              + "! Please choose one of: cloud, schemaless, techproducts, or films");
+              + "! Please choose one of: cloud, schemaless, techproducts, secure-setup, or films");


also need to filter on windows. I think we show the list like three times!

epugh · 2026-11-02T15:43:30Z

+import java.util.Properties;
+
+/** Persists HTTP credentials so the secure example can restart without password prompts. */
+final class SecureExampleCredentials {


this feels like step one towards "solr.properties" ;-)

epugh · 2026-11-02T15:44:18Z

+        "[req]\ndistinguished_name=dn\n[dn]\n[v3]\nsubjectAltName=DNS:localhost,IP:127.0.0.1\n");
+    run(
+        List.of(
+            "openssl",


could be all java! don't we run openssl elsewhere too?

epugh · 2026-11-02T15:44:49Z

+  }
+
+  Path credentialsFile() {
+    return home.resolve("zk-credentials.properties");


I think this should just be solr.properties

epugh · 2026-11-02T15:56:43Z

      picocli.CommandLine commandLine = new picocli.CommandLine(new SolrCLI());
      propagateCommandSettings(commandLine);
-      exit(commandLine.execute(stripEmptyLeadingArg(args)));
+      return commandLine.execute(stripEmptyLeadingArg(args));


interesting, hope this doesn't break something else!

epugh · 2026-11-02T15:57:22Z

+  private Path keystore;
+
+  @CommandLine.Option(
+      names = "--store-type",


do we need two types of store?

epugh · 2026-11-02T15:58:39Z

+    }
+    String type = params.storeType().toUpperCase(Locale.ROOT);
+    if (!type.equals("JKS") && !type.equals("PKCS12")) {
+      throw new IllegalArgumentException("--store-type must be JKS or PKCS12.");


okay, "PKCS12" is apparently better, and also the future... JKS is deprecated.. so let's not sue it anymore.

epugh · 2026-11-02T16:01:10Z

+        trust.setCertificateEntry("solr-" + i, chain[i]);
+      }
+      String original = Files.readString(include);
+      Map<String, String> settings = new LinkedHashMap<>();


this is jsut so messy!

epugh · 2026-11-02T16:01:40Z

  @CommandLine.Option(
      names = {"-e", "--example"},
-      description = "Run an example: cloud, techproducts, schemaless, films")
+      description = "Run an example: cloud, techproducts, secure-setup, schemaless, films")


and one more example ;-)

epugh · 2026-11-02T16:02:52Z

+    assertTrue(enabled.contains(sslSetting));
+    assertTrue(
+        enabled.contains(
+            Constants.WINDOWS ? "set SOLR_AUTH_TYPE=basic" : "SOLR_AUTH_TYPE='basic'"));


ugh, yeah, come on .toml or .yaml or even .properties

epugh · 2026-11-02T16:08:24Z

+        dir.resolve("openssl.cnf"),
+        "[req]\ndistinguished_name=dn\n[dn]\n[v3]\nsubjectAltName=DNS:localhost,IP:127.0.0.1\n");
+    try {
+      Process process =


is this how we test these things?

epugh · 2026-11-02T16:10:48Z

-  (
-    cd "$ssl_dir"
-    rm -f solr-ssl.keystore.p12 solr-ssl.pem
-    keytool -genkeypair -alias solr-ssl -keyalg RSA -keysize 2048 -keypass secret -storepass secret -validity 9999 -keystore solr-ssl.keystore.p12 -storetype PKCS12 -ext SAN=DNS:localhost,IP:127.0.0.1 -dname "CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country"


what is the relationship between keytool and openssl? Is one better? Why did we delete this in vaour of openssl?

nick-boss-tech · 2026-11-02T18:49:03Z

🤖 AI text below 🤖 (reviewed by Nick Shanin)

Prepared fixes for the November 2 review, integrated with your current head: compare 7d193dac0f...F4

The branch merges your head 7d193dac0f with the security-review fixes (both histories preserved, no rebase) and adds four commits on top: the repair itself, the generated start-page synchronization, the removal of the two scoped changelog fragments, and the credentials-option declaration described below. The credentials option needed work at both ends: bin/solr rejected it as unsupported, and RunExampleTool read the value without declaring the option, so both Java parsers rejected it as unknown. The branch teaches the shell script to forward --credentials and -u to example starts as separate arguments, declares the option in RunExampleTool for both parsers and wires the picocli value through, and rejects missing option values and credentials on a plain start before anything starts. The ZooKeeper client-address selection moved back above the logging guard so the log-call validator accepts it again. The secure-example BATS fixture now also covers the credential aliases, password rotation through the API, and a wrong-CA rejection followed by a correct-CA success. The generated CLI start page is synchronized at the head. No changelog fragment rides along; the PR's entry is yours to write.

Verification on Linux at 7aa3f77

Above the gated head 7e531b083182366f8b6e7f00a3781ce3cbb713de, this head adds the fragment removal described above and the credentials-option declaration (RunExampleTool plus its parse test). At this head the new parse test fails without the declaration and passes with it, the Error Prone compile and tidy pass, and on a rebuilt distribution start -e secure-setup --credentials solr:SolrRocks gets past argument parsing into the example wizard in both parser modes. Example starts cannot complete on that host for an unrelated environment reason: its Java localhost connections are intercepted, so the tool's port probe always reports the port as used. The runs below are from the gated head.
:solr:solr-ref-guide:generateCliDocs leaves the tree clean; :solr:core:compileTestJava and :solr:core:validateLogCalls pass; tidy is clean.
With the security manager: SecureExampleCredentialsTest 5 of 5, SecureExampleZkTest 3 of 3, CLIUtilsTest 10 of 10, AuthToolTest and AuthToolPicocliTest 3 of 3 each; SSLToolTest and IncludeFileEditorTest pass with their usual external-process skips under the manager.
Without the security manager: SSLToolTest 9 of 9 with no skips, IncludeFileEditorTest 7 of 7, and both secure-setup example methods pass in each parser class.
Fail-before: with only SSLTool.java and SecureExampleCredentials.java restored from your head, the three defect tests fail with their original signatures (client-auth settings not preserved; the six-character PEM password rejection; an unusable administrator accepted). The same tests pass at this head.
Root check -x test stops at :solr:webapp:verifyUiWasm on a checksum mismatch for the prebuilt UI bundle. The same task fails at your head with the same expected checksum; the downloaded bytes differed between the two runs here, so this is not caused by the branch.

One limit: the packaging BATS suites (test_ssl and test_secure_example in both parser modes) have not been run on this combined head yet, so the end-to-end shell credential flow and the ZK TLS example are verified by the unit suites above, not by BATS. If the No value present startup failure from CI recurs there, the fixture now starts examples with --verbose so the stack is captured. Separately, per the guide, private trust anchors still need restoring after a truststore regeneration.

To take the branch onto your PR head (your head is an ancestor, so this fast-forwards):

git switch SOLR-8442
git fetch https://github.com/nick-boss-tech/solr.git solr-8442-security-review-r2-submit
git merge --ff-only 7aa3f77bed263728a13f361d9f371bf16937672c

You're welcome to take the commits or adapt the changes.

Nick Shanin directed the work and takes responsibility for this contribution.

epugh · 2026-11-02T18:55:48Z

The branch merges your head 7d193dac0f with the security-review fixes (both histories preserved, no rebase) and adds three commits on top: the repair itself, the generated start-page synchronization, and the removal of the two scoped changelog fragments. RunExampleTool already accepts --credentials; bin/solr rejected the option as unsupported, so the repair teaches the shell script to forward --credentials and -u to example starts as separate arguments,

interesting... So, I actually don't have a good sense of if our examples actually work well if you run jthem twice... generally in development I'l do a gradlew clean dev just to reset thigns. I think, for simpliciity just at this point, I'll make it so it explicit that if you do run an example twice, it just steps on things.

so bin/solr start -e secure-setup would just step over any previous bin/solr start -e secure-setup. We really probably should verify that if you run

bin/solr start -e techproducts

#and then

bin/solr start -e films

that process still works... I actually wou'ndt be surprise if you got werid shit, I've never tested that flow.. interested in poking at that?

epugh · 2026-11-04T18:05:46Z

I had a long chat with @Gerlowski and I wanted to capture some notes:

Much easier to have a bin/solr start -e secure that does everything, but doestn' provide extra tooling around certs and sll that users can use. (i.e, don't expose the current bin/solr ssl tooling, jsut hide it.
Onboard with using bouncycastle for all java instead of openssl
a vote for either .yaml or .toml over our current solr.in.sh and solr.in.cmd files!
requiring solr.in.sh is fine for bin/solr commands that aren't about starting/sotpping solr.. that are potentialyly client side.
i think he is onboard with creating a basic security.json pattern???
"most teams have SREs that do all the ssl stuff" ;-). Sigh.

Gerlowski · 2026-11-04T18:19:25Z

Hi! I think you've mentioned the wrong account — I'm not involved in Solr and don't recall any conversation about this. You probably meant @gerlowskija (Jason Gerlowski). Cheers!

epugh added 2 commits October 10, 2026 10:33

Add a simple "secure_solr" example.

8c4d5d9

checking in everything, will break up later.

e8a488e

github-actions Bot added documentation Improvements or additions to documentation start-scripts tests cat:cli labels Oct 12, 2026

epugh changed the title ~~Solr 8442~~ SOLR-8442: Make security easier WIP Oct 12, 2026

epugh added 5 commits October 29, 2026 07:00

Simplify our managing of include files, don't need backups etc

96c59d9

Merge branch 'main' into SOLR-8442

72e848c

Merge remote-tracking branch 'upstream/main' into SOLR-8442

d2670a5

be picocli only

c77c719

this awkward password handling points to the problem of building a "s…

9305200

…ecurity.json" and then uploading it.. we can't use server side apis.

github-actions Bot added tool:build cat:security labels Oct 29, 2026

solr.auth.superuser is super confusing, so trying to clean this up.

023a423

github-actions Bot added the client:solrj label Oct 29, 2026

epugh added 11 commits October 29, 2026 18:46

BAsic pass of secure sorl example...

40c0d5d

compile fix

791f1a4

Use v2 api, explain you need to accept cert.

6d09914

call it "secure-setup" and incorporate zk ACL and credentials

e78bc73

Let's actually support zk credentials!

a90dbe8

Instead of burying them deep in the solr.include.sh file!

Verify wiht multple nodes and hold out health and metrics as not reuq…

d0d19bd

…iring security.

race condition on solr starting up...

3f37692

bug on windows!

da5f596

We don't actually have full windows support, so lets not do a half as…

b51e662

…sed attempt!

Use v2 apis, and make tutorial a bit chattier in line with cloud tuto…

25c88fb

…rial

add in zk server commuincation being over TLS

7d193da

github-actions Bot added jetty-server cat:cloud labels Nov 2, 2026

epugh commented Nov 2, 2026

View reviewed changes

epugh added 6 commits November 2, 2026 14:37

Responding to self review, only support PKCS12 and remove ---zk-tls

99a1850

check if openssl exists, still want bouncy castle ;-)

0baf9a2

simpligfy to one file

a534170

Merge branch 'main' into SOLR-8442

5eca5b6

Merge branch 'main' into SOLR-8442

27449aa

Post adding apache#5214 we get a simplification!

7957aa6

Conversation

epugh commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Solution

Tests

Checklist

Uh oh!

epugh commented Oct 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nick-boss-tech commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

epugh commented Nov 2, 2026

Uh oh!

epugh commented Nov 4, 2026

Uh oh!

Gerlowski commented Nov 4, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

epugh commented Oct 12, 2026 •

edited

Loading

nick-boss-tech commented Nov 2, 2026 •

edited

Loading