have a json topic name "customer-event" and trying to create stream from the below KSQL:
create stream cssc_customer_event_json (description varchar,pageEvent_id varchar)with (kafka_topic='customer-event', value_format='json');
it return below message:
Message
------------------------------------
Statement written to command topic
------------------------------------
after the query run and no stream has been created. Anyone can advise what may be the problem?
Thanks
Regards,
Han
Same thing happens when I try to create table.
create stream cssc_customer_event_json (reportSuite varchar,exclude_id varchar,exclude_value varchar,exclude_description varchar,pageEvent_id varchar)with (kafka_topic='customer-event', value_format='json');
it should create a new stream. But it has no stream created
ksql> show streams;
Stream Name | Kafka Topic | Format
------------------------------------
------------------------------------
The message:
Statement written to command topic
Is generally only seen when the REST endpoints thread writes the message to the command topic, then times out waiting from the engine side to read the message and process it.
There are a couple of reasons this can occur:
There's a misconfiguration in Kafka stopping the engine side reading, e.g. ACLs set incorrectly, so ksqlDB can write, but can't read the data.
There's a stability issue in Kafka - e.g. endangered partitions etc, though this tends to stop the write to Kafka working, not the read side.
The thread reading the command topic has crashed. There were some bugs in earlier versions that could cause this. The ksql application log would report this. Restarting may fix it, or upgrading.
The thread reading the command topic is stuck. There were some bugs in earlier versions that could cause this. The ksql application log would NOT report this. You'd need to do a thread-dump or similar to diagnose. Restarting may fix it, or upgrading.
Related
AuditTo is a Serilog feature that ensures synchronous write to the sink, with an exception thrown if change flushing fails.As the name implies, it is ideal for ensuring the security of audit data to be stored. Right now, I found File, Seq and RabbitMQ sinks supporting AuditTo writes. I couldn't find the SqlLite sink that I'm interested in ... :(
From the other side, we have WriteTo, which batches the log entries and writes them asynchronously. There are no exceptions; it's kind of fire and forget.No one cares whether the log entries are dropped by the connection or the target system's failure or unavailability.
I would like to implement sending the audit logs via AuditTo but also be able to switch the log configuration to WriteTo at the runtime. In the meantime, the app might still write the logs.
I saw that Serilog offers dynamic switching of the logging level via LoggingLevelSwitch.
Any suggestions, ideas, or solutions for such requirements?
We are having a setup where we are using spring-amqp transacted channels to push our messages to RabbitMq. During a testing we found that messages were not even getting published from spring-amqp to rabbitmq;
we suspect metricsCollector.basicPublish(this) in com.rabbitmq.client.impl.ChannelN failure(no exception is thrown).
because we can see that RabbitUtils.commitIfNecessary(channel) in org.springframework.amqp.rabbit.core.RabbitTemplate is not getting called when there is an issue executing metricsCollector.basicPublish(this) for the same code flow.
We have taken TCP dumps and could see that message were written to stream/socket on rabbitmq, but since commit didn't happen due to an a probable amqp api failure the messages were not delivered to corresponding queues.
Jars Version Being used in the setup:-
spring-amqp-2.2.1.RELEASE.jar,
spring-rabbit-2.2.1.RELEASE.jar
amqp-client-5.7.3.jar,
metrics-core-3.0.2.jar
Is anyone facing the similar issue?
Can someone please help.
---edit 1
(Setup) :- We are using same connection Factory for flows with parent transaction and flows not running with parent transactions
On further analyzing the issue , we found that isChannelLocallyTransacted is sometimes showing in-consistent behavior because ConnectionFactoryUtils.isChannelTransactional(channel, getConnectionFactory() is sometimes having a reference to transacted channel (returns true hence expression isChannelLocallyTransacted evaluates to false) due to which tx.commit never happens; so message gets lost before getting committed to RabbitMQ.
I want to terminate a query to drop a table. But i got the below error, and after while the query is terminated, the ksql log don't print any error message. How can i find the root cause?
ksql> terminate CTAS_KSQL1_TABLE_SACMES_PACK_STATS_275;
Could not write the statement 'terminate CTAS_KSQL1_TABLE_SACMES_PACK_STATS_275;' into the command topic.
Caused by: Timeout while waiting for command topic consumer to process command topic
Looks like you may have run into a bug in older versions of ksqlDB. Maybe this one: https://github.com/confluentinc/ksql/issues/4267
The general issue is that the query gets into a state where it can't close down cleanly. What's blocking the shutdown does eventually complete or timeout. In the case of issue #4267 above, the issue was that the sink topic, i.e. the topic ksqlDB is writing to, has been deleted out-of-band, i.e. by something other than ksqlDB, and ksqlDB is stuck trying to get metadata for a non-existent topic. Did you delete the sink topic?
There were others resolved issues too that I can't find.
Bouncing the server after issuing the terminate should clean up the stuck query. Though it's a pretty severe workaround!
Upgrading to a later version, something released after May 2020, the issue should be resolved.
I am looking to run a service that will be consuming messages that are placed into an SQS queue. What is the best way to structure the consumer application?
One thought would be to create a bunch of threads or processes that run this:
def run(q, delete_on_error=False):
while True:
try:
m = q.read(VISIBILITY_TIMEOUT, wait_time_seconds=MAX_WAIT_TIME_SECONDS)
if m is not None:
try:
process(m.id, m.get_body())
except TransientError:
continue
except Exception as ex:
log_exception(ex)
if not delete_on_error:
continue
q.delete_message(m)
except StopIteration:
break
except socket.gaierror:
continue
Am I missing anything else important? What other exceptions do I have to guard against in the queue read and delete calls? How do others run these consumers?
I did find this project, but it seems stalled and has some issues.
I am leaning toward separate processes rather than threads to avoid the the GIL. Is there some container process that can be used to launch and monitor these separate running processes?
There are a few things:
The SQS API allows you to receive more than one message with a single API call (up to 10 messages, or up to 256k worth of messages, whichever limit is hit first). Taking advantage of this feature allows you to reduce costs, since you are charged per API call. It looks like you're using the boto library - have a look at get_messages.
In your code right now, if processing a message fails due to a transient error, the message won't be able to be processed again until the visibility timeout expires. You might want to consider returning the message to the queue straight away. You can do this by calling change_visibility with 0 on that message. The message will then be available for processing straight away. (It might seem that if you do this then the visibility timeout will be permanently changed on that message - this is actually not the case. The AWS docs state that "the visibility timeout for the message the next time it is received reverts to the original timeout value". See the docs for more information.)
If you're after an example of a robust SQS message consumer, you might want to check out NServiceBus.AmazonSQS (of which I am the author). (C# - sorry, I couldn't find any python examples.)
We are using Nagios to monitor our network with great results. There is now a new requirement we are struggling with:
We want to notify Nagios of an non
fatal but critical application errors. The
application does not stop running but
there is some sort of issue that
needs looking into.
Once the issue has been looked into,
we need some way to "unflag" the
issue in Nagios.
We tried using the syslog, but the biggest problem was once an error was logged, the service was put into an error state with no way to recover. Also, while applications would report a critical error to the syslog, most of the time they don't report an "All clear" error.
I've done this using passive checks: http://nagios.sourceforge.net/docs/3_0/passivechecks.html
Basically, you're application is just going to feed the nagios core some data into its external command file. Nagios will eventually read the data and update the alerts, execute event handlers, etc.
Exactly how you set this up will be unique for your case, but if you need any other help just let me know. :)