-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
still in dev? #35
Comments
Not sure if it helps, @r2evans, but what I've done is wrap python's kafka client with |
@pedrocava, I had thought of that and have not put much effort into implementing it. Do you have a working gist or repo with working code? I'm sure I could work through it myself (most of the learning is in python's kafka client, with a little bit of learning (My challenge is that I need to deploy it in a k8s/swarm as a programmatic consumer/producer ... ugh.) Thanks. |
Your deployment requirements sound a lot tougher than mine (tested on an EC2 with cronjobs). Can't really share code I've used, but it would go like this: library(reticulate)
producer <- reticulate::import("kafka")$KafkaProducer(bootstrap_servers = something, sasl_plain_username = something_else, ...)
prodeucer$send(topic, message) |
well that was simpler than I expected, thank you @pedrocava ! |
Or you can write the abstraction directly in python: from kafka import KafkaProducer
from kafka.errors import KafkaError
producer = KafkaProducer(
bootstrap_servers = os.environ["kafka-bootstrap-brokers"],
sasl_plain_username = os.environ["kafka-username"],
sasl_plain_password = os.environ["kafka-password"],
security_protocol = "SASL_SSL",
sasl_mechanism = "SCRAM-SHA-512",
retries = 1)
def send_message(message : str, topic : str) -> None:
try:
producer.send(topic, message.encode('utf-8'))
except Exception as e:
print(e)
finally:
print("Message sent to {} topic".format(topic)) and import only higher levels functions in R, which would sure be nicer. |
Do you have the intention of continuing development on this package? It's not easily usable at the moment:
devtools::install_github
does not install it, there are no compile-notes or build instructions, and it is a bit behindlibrdkafka
's version history. This is not a criticism, just an observation if somebody else can pick up the torch and run with it. None of the forks I found showed progress.Thanks.
The text was updated successfully, but these errors were encountered: