Summary
The Apache Solr cluster is obtainable in CDP Public Cloud, utilizing the “Knowledge exploration and analytics” knowledge hub template. On this article we are going to examine how to connect with the Solr REST API operating within the Public Cloud, and spotlight the efficiency affect of session cookie configurations when Apache Knox Gateway is used to proxy the site visitors to Solr servers. Data on this weblog put up will be helpful for engineers creating Apache Solr shopper purposes.
The Apache Solr servers within the Cloudera Knowledge Platform (CDP) expose a REST API, protected by Kerberos authentication. Usually, all of the Solr server situations can deal with site visitors when the Solr cluster is operating in a distributed mode. The given Solr server that’s receiving the request from the shopper will ahead the question to all of the servers dealing with shards for the gathering and mix the outcomes earlier than sending again the response to the shopper. For scalability, it’s best to distribute the queries among the many Solr servers in a round-robin trend.
When Solr is deployed within the public cloud utilizing the “knowledge exploration and analytics” knowledge hub template, there are two methods to succeed in the Solr cluster from a separate shopper host. The primary, simpler strategy is to succeed in Solr utilizing Knox Gateway as a proxy. The Apache Knox Gateway is a system that gives a single level of authentication and entry for Apache Hadoop companies in a cluster. Within the CDP Knowledge Hub cluster Knox accepts HTTP primary authentication, so CDP customers can use their workload or machine person credentials for authentication. Based mostly on these credentials Knox will ahead the requests to Solr servers in round-robin, utilizing Kerberos and Easy and Protected GSSAPI Negotiation Mechanism (SPNEGO) on behalf of the authenticated finish person. (See Determine 1)
Determine 1. Sending Solr queries to the Solr cluster by Knox Gateway
After we connect with Solr by Knox, the Knox Gateway units the KNOXSESSIONID cookie within the HTTPS response. This cookie will be reused and set in every subsequent request, which is able to drastically enhance the efficiency of dealing with Solr requests.
One other strategy is to connect with any Solr server occasion immediately, utilizing HTTPS with SPNEGO authentication. On this case the Knox Gateway just isn’t used. Establishing this connection will be more difficult, as no primary authentication is feasible however Kerberos credentials are required. Additionally, if the Solr shopper host is exterior of the CDP surroundings, then all Solr server ports on the employee hosts have to be uncovered. (See Determine 2)
Determine 2. Sending Solr queries on to a Solr Server occasion
Benchmarking
To measure the efficiency of the Solr API, we developed a small efficiency benchmark script and executed it from a gateway node of the info hub cluster. The benchmark script is obtainable below Apache 2.0 license in this repository.
The next desk and graph current our benchmark outcomes. We executed quick Solr queries on a really small Solr assortment. We different the variety of parallel threads (1..10) and on every thread we executed 100 Solr REST calls utilizing the “curl” command. We examined the Solr API each immediately (connecting to a single given Solr server with out load balancing) and utilizing Knox (connecting to Solr by a Knox Gateway occasion). We repeated the assessments each with and with out reusing the cookies despatched again within the HTTPS responses. In all instances, the benchmark script was operating on the gateway host of the Solr knowledge hub cluster.
Desk 1: Efficiency benchmark outcomes (common response time and throughput) displaying the impact of cookie reuse between subsequent Solr API calls. Colours of the cells correspond to strains visualized in Determine 3.
Determine 3: Efficiency benchmark outcomes (common response) displaying the impact of cookie reuse between subsequent Solr API calls. Colours of the strains correspond to colours utilized in Desk 1.
Our outcomes clearly present how essential it’s to concentrate to make use of the KNOXSESSIONID cookie when connecting to Solr utilizing the Knox Gateway. When the cookie is about, the efficiency is principally the identical, suggesting that the Knox Gateway just isn’t the bottleneck for this specific benchmark. Nonetheless, with out setting KNOXSESSIONID we get a really vital efficiency degradation, which is brought on by the truth that the Knox Gateway must authenticate every HTTPS request one after the other, but when this cookie is about Knox can depend on earlier authentication.
Conclusion
We described two methods to connect with Solr REST API within the CDP Public Cloud; hopefully the knowledge on this weblog put up will enable you to decide on the very best one on your venture. Connecting by Knox is preferable because the Knox Gateway offers load balancing and likewise eases the authentication by eliminating the necessity for shopper aspect Kerberos configuration. Direct connection to the Solr server situations can be doable and is likely to be a superb strategy if Knox gateway turns into a bottleneck or if the additional routing step made by Knox proves so as to add an excessive amount of further latency to the site visitors. Nonetheless, for a lot of the instances we advise beginning the venture through the use of Knox Gateway to succeed in Solr, primarily as a result of organising safe connection and cargo balancing for a direct Solr entry will be more difficult. Utilizing the KNOXSESSIONID cookie can assist to succeed in efficiency much like the direct setup.
