I have a couple of scripts activities and despite the default timeout setup
It gets a timeout error after 30 minutes.
Any clue?
The Script Activity has a currently undocumented time limit of 30minutes. This cannot be changed and the Activity Timeout setting has no effect.
We can use the Script activity to execute DML (Data Manipulation Language) statements like SELECT, INSERT, UPDATE, as well as DDL (Data Definition Language) statements like CREATE, ALTER, and DROP operations.
Currently, the Script activity supports the following data stores:
• Azure SQL Database
• Azure Synapse Analytics
• SQL Server Database
• Oracle
• Snowflake
The General section is similar to other activities with these items:
• Name - Name of the activity.
• Description - Comments about this activity (optional).
• Timeout - Timeout in D.HH:MM:SS format, the default timeout value is 7 days.
• Retry - Maximum number of retry values.
• Retry interval (sec) - Specify the number of seconds between each retry attempt, the default value is 30.
• Secure output - Output from the log file will not be captured if checked.
• Secure input - Input from the log file will not be captured if checked.
You need to check timeout settings at Source level.
Refer this article by Tech Zero
What is the timeout set on the source ? if it always timeout in 30 mins may be thats what is happening . For example you can set the timeout SQL like
USE AdventureWorks2012 ;
GO
EXEC sp_configure 'remote query timeout', 30 ;
GO
RECONFIGURE ;
GO
https://learn.microsoft.com/en-us/sql/database-engine/configure-windows/configure-the-remote-query-timeout-server-configuration-option?view=sql-server-ver16
Related
I am using terraform resource google_dataflow_flex_template_job to deploy a Dataflow flex template job.
resource "google_dataflow_flex_template_job" "streaming_beam" {
provider = google-beta
name = "streaming-beam"
container_spec_gcs_path = module.streaming_beam_flex_template_file[0].fully_qualified_path
parameters = {
"input_subscription" = google_pubsub_subscription.ratings[0].id
"output_table" = "${var.project}:beam_samples.streaming_beam_sql"
"service_account_email" = data.terraform_remote_state.state.outputs.sa.email
"network" = google_compute_network.network.name
"subnetwork" = "regions/${google_compute_subnetwork.subnet.region}/subnetworks/${google_compute_subnetwork.subnet.name}"
}
}
Its all working fine however without my requesting it the job seems to be using flexible resource scheduling (flexRS) mode, I say this because the job takes about ten minutes to start and during that time has state=QUEUED which I think is only applicable to flexRS jobs.
Using flexRS mode is fine for production scenarios however I'm currently still developing my dataflow job and when doing so flexRS is massively inconvenient because it takes about 10 minutes to see the effect of any changes I might make, no matter how small.
In Enabling FlexRS it is stated
To enable a FlexRS job, use the following pipeline option:
--flexRSGoal=COST_OPTIMIZED, where the cost-optimized goal means that the Dataflow service chooses any available discounted resources or
--flexRSGoal=SPEED_OPTIMIZED, where it optimizes for lower execution time.
I then found the following statement:
To turn on FlexRS, you must specify the value COST_OPTIMIZED to allow the Dataflow service to choose any available discounted resources.
at Specifying pipeline execution parameters > Setting other Cloud Dataflow pipeline options
I interpret that to mean that flexrs_goal=SPEED_OPTIMIZED will turn off flexRS mode. However, I changed the definition of my google_dataflow_flex_template_job resource to:
resource "google_dataflow_flex_template_job" "streaming_beam" {
provider = google-beta
name = "streaming-beam"
container_spec_gcs_path = module.streaming_beam_flex_template_file[0].fully_qualified_path
parameters = {
"input_subscription" = google_pubsub_subscription.ratings[0].id
"output_table" = "${var.project}:beam_samples.streaming_beam_sql"
"service_account_email" = data.terraform_remote_state.state.outputs.sa.email
"network" = google_compute_network.network.name
"subnetwork" = "regions/${google_compute_subnetwork.subnet.region}/subnetworks/${google_compute_subnetwork.subnet.name}"
"flexrs_goal" = "SPEED_OPTIMIZED"
}
}
(note the addition of "flexrs_goal" = "SPEED_OPTIMIZED") but it doesn't seem to make any difference. The Dataflow UI confirms I have set SPEED_OPTIMIZED:
but it still takes too long (9 minutes 46 seconds) for the job to start processing data, and it was in state=QUEUED for all that time:
2021-01-17 19:49:19.021 GMTStarting GCE instance, launcher-2021011711491611239867327455334861, to launch the template.
...
...
2021-01-17 19:59:05.381 GMTStarting 1 workers in europe-west1-d...
2021-01-17 19:59:12.256 GMTVM, launcher-2021011711491611239867327455334861, stopped.
I then tried explictly setting flexrs_goal=COST_OPTIMIZED just to see if it made any difference, but this only caused an error:
"The workflow could not be created. Causes: The workflow could not be
created due to misconfiguration. The experimental feature
flexible_resource_scheduling is not supported for streaming jobs.
Contact Google Cloud Support for further help. "
This makes sense. My job is indeed a streaming job and the documentation does indeed state that flexRS is only for batch jobs.
This page explains how to enable Flexible Resource Scheduling (FlexRS) for autoscaled batch pipelines in Dataflow.
https://cloud.google.com/dataflow/docs/guides/flexrs
This doesn't solve my problem though. As I said above if I deploy with flexrs_goal=SPEED_OPTIMIZED then still state=QUEUED for almost ten minutes, yet as far as I know QUEUED is only applicable to flexRS jobs:
Therefore, after you submit a FlexRS job, your job displays an ID and a Status of Queued
https://cloud.google.com/dataflow/docs/guides/flexrs#delayed_scheduling
Hence I'm very confused:
Why is my job getting queued even though it is not a flexRS job?
Why does it take nearly ten minutes for my job to start processing any data?
How can I speed up the time it takes for my job to start processing data so that I can get quicker feedback during development/testing?
UPDATE, I dug a bit more into the logs to find out what was going on during those 9minutes 46 seconds. These two consecutive log messages are 7 minutes 23 seconds apart:
2021-01-17 19:51:03.381 GMT
"INFO:apache_beam.runners.portability.stager:Executing command: ['/usr/local/bin/python', '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', '/dataflow/template/requirements.txt', '--exists-action', 'i', '--no-binary', ':all:']"
2021-01-17 19:58:26.459 GMT
"INFO:apache_beam.runners.portability.stager:Downloading source distribution of the SDK from PyPi"
Whatever is going on between those two log records is the main contributor to the long time spent in state=QUEUED. Anyone know what might be the cause?
As mentioned in the existing answer you need to extract the apache-beam modules inside your requirements.txt:
RUN pip install -U apache-beam==<version>
RUN pip install -U -r ./requirements.txt
While developing, I prefer to use DirectRunner, for the fastest feedback.
I have a fixed window of 1 minute. I am considering event time.
beam.WindowInto(window.FixedWindows(300))
When I deploy this code ,is the window created instantly even if I have not published any message .suppose I deployed at 6:30 , is it like the windows are automatically created as 6:30 to 6:35, 6:35 to 6:40 and so on ?
If I publish a message to topic having
event timestamp = 6:31 (unix seconds i.e 10,176589653)
when system time = 6:36
..does it mean the watermark for that specific message is at 6:31 and it will miss the window as system time is at 6:36 and allowed lateness=0 and will be rejected.
Windows are always created using UNIX time 0 as a base, meaning, no matter if you start the pipeline at 6:31, 6:32 or 6:35, the windows would always be [6:30, 6:35), [6:35, 6:40).... Note that this also applies for days, the windows would start at 00:00 UTC.
If you want to change this, there's an offset parameter.
Since our test agents are slow some times- i am trying to add some additional time outs for some commands
I did it like using time out value on command as shown below.But its not respecting the value given
My understanding is cypress will wait for "10000" MS for getting the #Addstory element?
Can any one advice is this is the correct way please?
Thank you so much
cy.get('#addstory > .ng-scope').click({ timeout: 10000 })
In cypress.json file, increase the timeout to 10 seconds or what ever timeout you want like this: "defaultCommandTimeout": 10000 and save the file. Now close the app and open it again. Navigate to Settings > Configuration you should be able to see the new value set for defaultCommandTimeout.
I issue was i was adding time out on click not for getting the element when i changed like below -All good waiting for add story to be visible as i expected before click
cy.get('#addstory > .ng-scope',{ timeout: 10000 }).click()
running a Jenkins server with the embedded Jetty, I get errors regarding too big headers in the Jenkins log:
Feb 15, 2017 3:18:15 PM org.eclipse.jetty.util.log.JavaUtilLog warn
WARNING: header full: java.lang.ArrayIndexOutOfBoundsException: 8192
I'd like to increase the Jetty max header size but can't find how to do it, in the case of a Jenkins... I can't find any Jetty config file and don't know if I can set the limit on the Jenkins command line (and what would be the name of the variable to define).
How to achieve this?
If using the built-in Jetty found in the self-running jenkins.war, you cannot adjust that value.
You can only adjust the maximum number of parameters.
--maxParamCount=N = set the max number of parameters allowed in a form submission to protect
against hash DoS attack (oCERT #2011-003). Default is 10000.
Either deploy the war to a full blown container which you can then adjust the value, or change how you use Jenkins to not send excessive URI or HTTP headers (such as using POST vs GET).
To adjust the Jetty 9 header buffer maximum size, you'd adjust the requestHeaderSize in the HttpConfiguration for the ServerConnector that you want that new setting to exist in.
Add this parameter to jenkins config:
JENKINS_ARGS="--requestHeaderSize=258140"
I am new to zabbix. I have a basic requirement of monitoring occurrence of different log messages using zabbix. Say, when there is a log message "server starting", zabbix should show that alert. The idea is that if the server (re)starts 10 times in last 10 minutes, the zabbix dashboard (or at any other place) should display that 10 times.
I have done the following for that :
Created an item under template MyTemplate:
Type : Zabbix Agent (Active)
key : log[/opt/mylog/logs/abc.log,server starting]
Type of information : Log
Update Interval (in sec) : 30
Created a trigger with expression :
{MyTemplate:log[/opt/mylog/logs/abc.log,server
starting].logeventid(1)}=0
With logeventid(1), I am seeing that the alert (trigger) is getting generated only once. It appears only once in the Dashboard --> Last 20 issues. If I go to Monitoring --> Trigger, I see the alert only once, although the log files have 10 entries of the message "server starting" (server restarted 10 times).
Then I set the trigger to following :
{MyTemplate:log[/opt/mylog/logs/abc.log,server
starting].nodata(300)}=0
Now, at Monitoring --> Trigger, I see the alert (trigger) 10 times, but, from the Dashboard --> Last 20 issues it vanishes just after 300 seconds.
My questions are :
What should be the trigger function, I should use? I want to see 10 alerts in zabbix if the same message appears 10 times in the log file within a period of time.
With nodata(300), why does the alert vanish after 300 sec?
Is it ok if I use 30 minutes instead of 300 seconds as an argument of nodata()?
Function logeventid() is normally used for Windows and VMware event logs. In this case, it should probably not be used and it is suspicious that it fires, which might indicate a bug in Zabbix.
Anyway, you can check "Multiple PROBLEM events generation" box in trigger configuration and the trigger will generate a new PROBLEM event every time the condition is true, regardless of its previous value. Instead of logeventid(), you can try using a function that is always true, for instance, strlen()>0.
If you wish the trigger to go into OK state after some time, say, 10 minutes, you can add nodata(10m). Then your trigger will look like this:
{MyTemplate:log[/opt/mylog/logs/abc.log,server starting].strlen()}>0 and
{MyTemplate:log[/opt/mylog/logs/abc.log,server starting].nodata(10m)}=0