My Airflow DAG keeps failing on the only task that I have.
I declared the execution_timeout as 300 seconds, but it keeps crashing after around 37 seconds.
The task consists in scraping a website, without Chromedriver.
I'm on Linux, Raspberry PI.
Here is the code:
from datetime import timedelta
import importlib
import sys
from airflow.operators.bash_operator import BashOperator
from airflow.operators.python_operator import PythonOperator
from airflow.utils.dates import days_ago
from airflow import DAG
from lib.jobs import jobs, linkedin_jobs, glassdoor_jobs
from lib import jobs_and_companies
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'email': ['firstname.lastname#live.fr'],
'email_on_failure': True,
'retries': 0,
'execution_timeout': timedelta(hours=24)
}
dag = DAG(
dag_id='jobs',
default_args=default_args,
description='Collecting jobs from boards.',
concurrency=10,
schedule_interval=timedelta(hours=24),
start_date=days_ago(2),
dagrun_timeout=timedelta(seconds=300),
)
linkedin_jobs_task = PythonOperator(
task_id='linkedin_jobs',
python_callable=linkedin_jobs.scrap_jobs(),
dag=dag,
start_date=days_ago(2),
execution_timeout=timedelta(seconds=300),
)
Can you help me?
Thanks
What error are you seeing when the crash occurs? Also, I notice your Python callable is named linkedin_jobs.scrap_jobs(). Is that spelled correctly or is it supposed to be linkedin_jobs.scrape_jobs()?
Related
Using iforest as described here: https://github.com/titicaca/spark-iforest
But model.save() is throwing exception:
Exception:
scala.NotImplementedError: The default jsonEncode only supports string, vector and matrix. org.apache.spark.ml.param.Param must override jsonEncode for java.lang.Double.
Followed the code snippet mentioned under "Python API" section on mentioned git page.
from pyspark.ml.feature import VectorAssembler
import os
import tempfile
from pyspark_iforest.ml.iforest import *
col_1:integer
col_2:integer
col_3:integer
assembler = VectorAssembler(inputCols=in_cols, outputCol="features")
featurized = assembler.transform(df)
iforest = IForest(contamination=0.5, maxDepth=2)
model=iforest.fit(df)
model.save("model_path")
model.save() should be able to save model files.
Below is the output dataframe I'm getting after executing model.transform(df):
col_1:integer
col_2:integer
col_3:integer
features:udt
anomalyScore:double
prediction:double
I have just fixed this issue. It was caused by an incorrect param type. You can checkout the latest codes in the master branch, and try it again.
I habe a problem wirh the Import of my issues to Bitbucket.
I have 3000 issues, but I can only import 1000. The import give e the following error message:
The error was: string indices must be integers
I was trying to do some action in the inbuilt Phone application, and trying to press 0 key (long press) but every time when I am trying to do, facing the issue:
org.openqa.selenium.WebDriverException: Returned value cannot be converted to WebElement: {ELEMENT=1}
Build info: version: '3.4.0', revision: 'unknown', time: 'unknown'
My code
package mobileapp.com.example;
import java.net.MalformedURLException;
import java.net.URL;
import org.openqa.selenium.By;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
import io.appium.java_client.AppiumDriver;
import io.appium.java_client.TouchAction;
import io.appium.java_client.android.AndroidDriver;
import io.appium.java_client.remote.MobileCapabilityType;
public class Call {
AndroidDriver<WebElement> driver;
#BeforeClass
public void Setup() throws MalformedURLException {
DesiredCapabilities cap= new DesiredCapabilities();
cap.setCapability(MobileCapabilityType.DEVICE_NAME, "Fever");
cap.setCapability("platformVersion", "6.0");
cap.setCapability("platformName", "Android");
cap.setCapability("appPackage", "com.android.dialer");
cap.setCapability("appActivity", "com.android.dialer.DialtactsActivity");
//driver=new AndroidDriver<WebElement>(new URL("http://127.0.0.1:4723/wd/hub"), cap);
driver=new AndroidDriver<WebElement>(new URL("http://127.0.0.1:4723/wd/hub"), cap);
}
#Test
public void simpleTest() {
driver.findElement(By.id("com.android.dialer:id/floating_action_button")).click();
WebElement ele=driver.findElement(By.id("com.android.dialer:id/zero"));
TouchAction action=new TouchAction(driver);
action.longPress(ele);
ele.click();
}
}
I am using Appium version 1.4.16.1 and appium java client version was 5.0.4
The main issue is that you are using very old appium server and latest java client, so your issue is caused by incompatibility of client/server versions.
Update server to the 1.7.x
If locator is correct, it should work after update. Alternately you may use:
driver.pressKeyCode(AndroidKeyCode.KEYCODE_NUMPAD_0);
where driver is instance of AndroidDriver class
I followed the steps described on their Github to install the Object Detection API and then I ran this script:
python object_detection/builders/model_builder_test.py
and the test were successful, so I assumed everything was setup correctly. Next I tried to run the Jupyter Notebook with qtconsole to detect objects in the test images. However it returns this error:
AttributeError Traceback (most recent call last)
<ipython-input-3-be6fe1ba8733> in <module>()
----> 1 from utils import label_map_util
2
3 from utils import visualization_utils as vis_util
4
~\Desktop\Objectdetection\models-master\object_detection\utils\label_map_util.py in <module>()
20 import tensorflow as tf
21 from google.protobuf import text_format
---> 22 from object_detection.protos import string_int_label_map_pb2
23
24
~\Desktop\Objectdetection\models-master\object_detection\object_detection.py in <module>()
114
115
--> 116 label_map = label_map_util.load_labelmap(PATH_TO_LABELS)
117 categories = label_map_util.convert_label_map_to_categories(label_map, max_num_classes=NUM_CLASSES, use_display_name=True)
118 category_index = label_map_util.create_category_index(categories)
AttributeError: module 'utils.label_map_util' has no attribute 'load_labelmap'
Does anyone have an idea what the cause of this problem is?
Thanks.
In the file ~\Desktop\Objectdetection\models-master\object_detection\utils\label_map_util.py
Try changing this:
from object_detection.protos import string_int_label_map_pb2
to this:
from protos import string_int_label_map_pb2
Explanation:
The function load_labelmap in the module label_map_util isn't accessible because the import of string_int_label_map_pb2 is failing.
You can see this if you look at the output of print(dir(label_map_util)).
When using object_detection.protos:
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'logging', 'text_format', 'tf']
After the changing the relative path to protos the function should be accessible:
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_validate_label_map', 'convert_label_map_to_categories', 'create_category_index', 'get_label_map_dict', 'load_labelmap', 'logging', 'string_int_label_map_pb2', 'text_format', 'tf']
I found this very strange and unfortunate behavior in Dart. When I import 'dart:html' in my main file, my Isolate stops working.
With my file "isolate.dart" :
main(){
print('BAM');
}
This prints "BAM":
import 'dart:core';
import 'dart:isolate';
void main() {
Isolate.spawnUri(Uri.parse('isolate.dart'), [], null);
}
but this prints nothing :
import 'dart:core';
import 'dart:isolate';
import 'dart:html';
void main() {
Isolate.spawnUri(Uri.parse('isolate.dart'), [], null);
}
How can I get Isolate to work while using the html import?
UPDATE :
I've found this code https://github.com/TomCaserta/ExampleIsolate and tried to work it to find the problem. It seems like the print() call from the Isolate is causing problems.
This are known bugs/limitations. It is being worked on.
Currently it is not possible to access functionality of the 'dart:html' package in an isolate and 'print()' crashes the isolate probably because there is no package with a 'print' functionality available where the command can be redirected to.
The Dart issue tracker seems not to be available currently.
I try again later to add some references.
Some open issues that I think are related:
Print statement inside isolate crashes isolate.
Isolate.spawn not allowed in Dartium DOM isolate
Support spawn of dom isolates in Dartium
Unable to break on breakpoint in library loaded by spawnUri
isolate in dartium not creating httprequest, works when compiled to javascript