com.github.jcustenborder.kafka.connect:kafka-connect-solr
Advanced tools
A Kafka Connect connector copying data from Kafka to Solr.
The SOLR connector is a high speed mechanism for writing data to Apache Solr.
If you are seeing error messages such as Invalid version (expected 2, but 60) or the data in not in 'javabin' format compare the version of the Solr Server against the version of solrj the connector is compiled with. This error message is most likely due to a version mismatch between the server and solrj. To address this try replacing the solr-solrj-*.jar packaged with the connector with the version that matches the Solr server you are connecting to.
This connector is used to connect to SolrCloud using the Zookeeper based configuration.
The target collection for this connector is selected by the topic name. Transformations like the RegexRouter transformation can be used to change the topic name before it is sent to Solr.
solr.passwordThe password to use for basic authentication.
Importance: High
Type: Password
Default Value: [hidden]
solr.usernameThe username to use for basic authentication.
Importance: High
Type: String
solr.zookeeper.hostsZookeeper hosts that are used to store solr configuration.
Importance: High
Type: List
solr.zookeeper.chrootChroot within solr for the zookeeper configuration.
Importance: High
Type: String
solr.delete.documents.enabledFlag to determine if the connector should delete documents. General practice in Kafka is to treat a record that contains a key with a null value as a delete.
Importance: Medium
Type: Boolean
Default Value: true
solr.commit.withinConfigures Solr UpdaterRequest for a commit within the requested number of milliseconds. -1 disables the commit within setting and relies on the standard Solr commit setting.
Importance: Low
Type: Int
Default Value: -1
In this example each record has an incoming topic name prefixed with solr-. Assuming that our topic is solr-customer the following example will strip the prefix of solr- allowing us to write to the collection named customer. This is accomplished by using the RegexRouter transformation that is bundled with Apache Kafka.
Select one of the following configuration methods based on how you have deployed Kafka Connect. Distributed Mode will the the JSON / REST examples. Standalone mode will use the properties based example.
{
"name" : "cloudSolrSinkConnector1",
"config" : {
"connector.class" : "com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector",
"tasks.max" : "1",
"topics" : "topic1,topic2,topic3",
"solr.zookeeper.hosts" : "zookeeper.example.com:2181",
"solr.username" : "freddy",
"solr.password" : "password12345",
"transforms" : "dropPrefix",
"transforms.dropPrefix.type" : "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.dropPrefix.regex" : "solr-(.*)",
"transforms.dropPrefix.replacement" : "$1"
}
}
connector.class=com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector
tasks.max=1
topics=topic1,topic2,topic3
solr.zookeeper.hosts=zookeeper.example.com:2181
solr.username=freddy
solr.password=password12345
transforms=dropPrefix
transforms.dropPrefix.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.dropPrefix.regex=solr-(.*)
transforms.dropPrefix.replacement=$1
This example will connect to a Solr Cloud cluster without authentication.
Select one of the following configuration methods based on how you have deployed Kafka Connect. Distributed Mode will the the JSON / REST examples. Standalone mode will use the properties based example.
{
"name" : "cloudSolrSinkConnector1",
"config" : {
"connector.class" : "com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector",
"tasks.max" : "1",
"topics" : "topic1,topic2,topic3",
"solr.zookeeper.hosts" : "zookeeper.example.com:2181"
}
}
connector.class=com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector
tasks.max=1
topics=topic1,topic2,topic3
solr.zookeeper.hosts=zookeeper.example.com:2181
This example will connect to a Solr Cloud cluster using basic authentication.
Select one of the following configuration methods based on how you have deployed Kafka Connect. Distributed Mode will the the JSON / REST examples. Standalone mode will use the properties based example.
{
"name" : "cloudSolrSinkConnector1",
"config" : {
"connector.class" : "com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector",
"tasks.max" : "1",
"topics" : "topic1,topic2,topic3",
"solr.zookeeper.hosts" : "zookeeper.example.com:2181",
"solr.username" : "freddy",
"solr.password" : "password12345"
}
}
connector.class=com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector
tasks.max=1
topics=topic1,topic2,topic3
solr.zookeeper.hosts=zookeeper.example.com:2181
solr.username=freddy
solr.password=password12345
In this example we do not care about the incoming topic name. We want to force all topics to a specific collection. This is accomplished by using the RegexRouter transformation that is bundled with Apache Kafka.
Select one of the following configuration methods based on how you have deployed Kafka Connect. Distributed Mode will the the JSON / REST examples. Standalone mode will use the properties based example.
{
"name" : "cloudSolrSinkConnector1",
"config" : {
"connector.class" : "com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector",
"tasks.max" : "1",
"topics" : "topic1,topic2,topic3",
"solr.zookeeper.hosts" : "zookeeper.example.com:2181",
"solr.username" : "freddy",
"solr.password" : "password12345",
"transforms" : "dropPrefix",
"transforms.dropPrefix.type" : "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.dropPrefix.regex" : ".*",
"transforms.dropPrefix.replacement" : "forced-collection"
}
}
connector.class=com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector
tasks.max=1
topics=topic1,topic2,topic3
solr.zookeeper.hosts=zookeeper.example.com:2181
solr.username=freddy
solr.password=password12345
transforms=dropPrefix
transforms.dropPrefix.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.dropPrefix.regex=.*
transforms.dropPrefix.replacement=forced-collection
This connector is used to connect to write directly to a Solr core.
The target collection for this connector is selected by the topic name. Transformations like the RegexRouter transformation can be used to change the topic name before it is sent to Solr.
solr.passwordThe password to use for basic authentication.
Importance: High
Type: Password
Default Value: [hidden]
solr.usernameThe username to use for basic authentication.
Importance: High
Type: String
solr.urlUrl to connect to solr with.
Importance: High
Type: String
solr.queue.sizeThe number of documents to batch together before sending to Solr. See ConcurrentUpdateSolrClient.Builder.withQueueSize(int)
Importance: High
Type: Int
Default Value: 100
solr.delete.documents.enabledFlag to determine if the connector should delete documents. General practice in Kafka is to treat a record that contains a key with a null value as a delete.
Importance: Medium
Type: Boolean
Default Value: true
solr.thread.countThe number of threads used to empty ConcurrentUpdateSolrClients queue. See ConcurrentUpdateSolrClient.Builder.withThreadCount(int)
Importance: Medium
Type: Int
Default Value: 1
solr.commit.withinConfigures Solr UpdaterRequest for a commit within the requested number of milliseconds. -1 disables the commit within setting and relies on the standard Solr commit setting.
Importance: Low
Type: Int
Default Value: -1
This example will connect to a Solr Cloud cluster without authentication.
Select one of the following configuration methods based on how you have deployed Kafka Connect. Distributed Mode will the the JSON / REST examples. Standalone mode will use the properties based example.
{
"name" : "httpSolrSinkConnector1",
"config" : {
"connector.class" : "com.github.jcustenborder.kafka.connect.solr.HttpSolrSinkConnector",
"tasks.max" : "1",
"topics" : "topic1,topic2,topic3",
"solr.url" : "http://solr.example.com:8993/"
}
}
connector.class=com.github.jcustenborder.kafka.connect.solr.HttpSolrSinkConnector
tasks.max=1
topics=topic1,topic2,topic3
solr.url=http://solr.example.com:8993/
This example will connect to a Solr Cloud cluster using basic authentication.
Select one of the following configuration methods based on how you have deployed Kafka Connect. Distributed Mode will the the JSON / REST examples. Standalone mode will use the properties based example.
{
"name" : "httpSolrSinkConnector1",
"config" : {
"connector.class" : "com.github.jcustenborder.kafka.connect.solr.HttpSolrSinkConnector",
"tasks.max" : "1",
"topics" : "topic1,topic2,topic3",
"solr.url" : "http://solr.example.com:8993/",
"solr.username" : "freddy",
"solr.password" : "password12345"
}
}
connector.class=com.github.jcustenborder.kafka.connect.solr.HttpSolrSinkConnector
tasks.max=1
topics=topic1,topic2,topic3
solr.url=http://solr.example.com:8993/
solr.username=freddy
solr.password=password12345
FAQs
Unknown package
We found that com.github.jcustenborder.kafka.connect:kafka-connect-solr demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Socket’s threat research team has detected six malicious npm packages typosquatting popular libraries to insert SSH backdoors.
Security News
MITRE's 2024 CWE Top 25 highlights critical software vulnerabilities like XSS, SQL Injection, and CSRF, reflecting shifts due to a refined ranking methodology.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.