I just started learning Airflow and I'm trying to create a small airflow script to send a notification to slack but somehow it fails to call for Slack API. First task works fine. Here is the code I use:
from airflow import DAG
from airflow.contrib.operators.bigquery_check_operator import BigQueryCheckOperator
from airflow.operators import BashOperator
from slackclient import SlackClient
from airflow.operators.slack_operator import SlackAPIPostOperator
from datetime import datetime, timedelta
default_args = {
'owner': 'test',
'depends_on_past': False,
'start_date': datetime.today(),
'email': ['test#test'],
'email_on_failure': True,
'email_on_retry': False,
'retries': 4,
'retry_delay': timedelta(minutes=5),
}
# Create the DAG
dag = DAG('slack_test',\
default_args=default_args,\
schedule_interval='15 * * * *')
final_check = BigQueryCheckOperator(
task_id='bq_check_agg',
sql='''
#legacy sql
SELECT
max(date) as max_date
FROM
[test:test.test]
WHERE date <= current_timestamp()
''',
dag = dag)
# slack notification
slack_notify = SlackAPIPostOperator(
task_id='slack_notify',
username = 'Airflow',
token='....',
channel='#test-channel',
text='ETL DONE!',
dag=dag
)
slack_notify.set_upstream(final_check)
When I try to test slack_notify task, it fails. Here is the error I receive:
airflow.exceptions.AirflowException: Slack API call failed (%s)
Couldn't find any solutions for that online. Is something wrong with the slack authentication ?
I think you could invite your Airflow bot into your private channel first and then the slack bot can post message in that channel. Otherwise the slack bot cannot find a private channel and it will raise not_in_channel Slack API error.
With public slack channel you don't need to do that and you will receive messages from your Airflow bot.
Hope it helps.
Solved. I was trying to post in a private slack channel. Worked fine with a public one
Related
I am having trouble using Twilio's MessagingResponse class to send and receive messages using a webhook. I am using flask and ngrok to create a temporary URL to test my app, but I am getting a 502 error and 11200 warning with this new implementation.
I can confirm that the number I am attempting to message back is verified in my Twilio account. Here is the new implementation that uses the Twilio MessagingResponse to create the response message instead of sending it directly using the Twilio REST API:
import os
from flask import Flask, request, session, make_response
from twilio.twiml.messaging_response import MessagingResponse
from twilio.rest import Client
from chatbot import ask, append_interaction_to_chat_log
from secret_key import secret_key
app = Flask(__name__)
app.config['SECRET_KEY'] = secret_key
account_sid = os.environ.get('ACCOUNT_SID')
auth_token = os.environ.get('AUTH_TOKEN')
client = Client(account_sid, auth_token)
#app.route('/bot', methods=['POST'])
def bot():
incoming_msg = request.values['Body']
print(incoming_msg)
chat_log = session.get('chat_log')
answer = ask(incoming_msg, chat_log)
session['chat_log'] = append_interaction_to_chat_log(incoming_msg, answer, chat_log)
r = MessagingResponse()
r.message = answer
return make_response(r)
I have been able to successfully send and receive messages using a message object and explicitly stating the phone number I am sending to using this implementation:
import os
from flask import Flask, request, session
from twilio.twiml.messaging_response import MessagingResponse
from twilio.rest import Client
from chatbot import ask, append_interaction_to_chat_log
from secret_key import secret_key
app = Flask(__name__)
app.config['SECRET_KEY'] = secret_key
account_sid = os.environ.get('ACCOUNT_SID')
auth_token = os.environ.get('AUTH_TOKEN')
client = Client(account_sid, auth_token)
#app.route('/', methods=['POST'])
def bot():
incoming_msg = request.values['Body']
print(incoming_msg)
chat_log = session.get('chat_log')
answer = ask(incoming_msg, chat_log)
session['chat_log'] = append_interaction_to_chat_log(incoming_msg, answer, chat_log)
# use the incoming message to generate the response here
message = client.messages.create(
body=answer,
from_='+12232107883', #Twilio number you purchased or verified
to='+19143182181' # The phone number you want to send the message to
)
print(message.sid)
return 'message sent'
Attached is a photo of my implementation of the ngrok URL to configure the webhook.
Essentially, I am trying to implement the new structures in hopes of creating a more secure and scalable bot. Any ideas here? I realize it could be something in my Twilio settings, but I haven't found the solution. Thank you all.
I thought my problem was from not having an ngrok account:
But that did not seem to resolve the issue:
I'm trying to schedule an sms via Twilio. I'm referencing the demo found here: https://www.twilio.com/docs/sms/api/message-resource#schedule-a-message-resource
Per the demo, I set up messaging service via the Twilio Console.
Here is the code, with a dummy destination phone number and authorizations:
from datetime import datetime
import os
from twilio.rest import Client
# Find your Account SID and Auth Token at twilio.com/console
# and set the environment variables. See http://twil.io/secure
account_sid = [sid]
auth_token = [token]
client = Client(account_sid, auth_token)
message = client.messages \
.create(
messaging_service_sid= [msid],
body='This is a scheduled message',
send_at=datetime(2022, 6, 18, 20, 36, 27),
schedule_type='fixed',
# status_callback='https://webhook.site/xxxxx',
to='+15559698489'
)
print(message.sid)
I get the following error:
TypeError: create() got an unexpected keyword argument 'send_at'
What am I missing?
Check that your Twilio Helper Library version is current.
hope you are doing it right these days.
To summarize my problem, I think this is not working becuase I am using a free Twilio account instead of a paid one. But that's just my beginner theory. Now, the issue:
I have tried an official Twilio tutorial (https://www.twilio.com/blog/automating-ngrok-python-twilio-applications-pyngrok, I shared the link in case someone finds it interesting or needs it), which allows us to automate SMS webhook (sms_url) configuration by using Client (twilio) and pyngrok (ngrok).
def start_ngrok():
from twilio.rest import Client
from pyngrok import ngrok
url = ngrok.connect(5000)
print(' * Tunnel URL:', url)
client = Client()
client.incoming_phone_numbers.list(
phone_number=os.environ.get('TWILIO_PHONE_NUMBER'))[0].update(
sms_url=url + '/bot')
I can't explain all the things that I tried in the last 4 days, with no success. I keep getting the same error:
client.incoming_phone_numbers.list(phone_number=os.environ.get('TWILIO_PHONE_NUMBER'))[0].update(sms_url=url + '/bot')
IndexError: list index out of range
Something is not working with the list, it comes empty, although environment variables are working properly. I will work with just one phone_number, so there no need for list, indeed, so I started to change that line to avoid different errors and ended up with this:
def start_ngrok():
from twilio.rest import Client
from pyngrok import ngrok
url = ngrok.connect(5000)
print(' * Tunnel URL:', url)
client = Client()
client.incoming_phone_numbers("my_number").update(sms_url=str(url) + '/bot')
Then I got the final error that I can't solve by my self:
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/twilio/rest/api/v2010/account/incoming_phone_number/__init__.py", line 442, in update
payload = self._version.update(method='POST', uri=self._uri, data=data, )
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/twilio/base/version.py", line 106, in update
raise self.exception(method, uri, response, 'Unable to update record')
twilio.base.exceptions.TwilioRestException:
HTTP Error Your request was:
POST /Accounts/my_account_SID/IncomingPhoneNumbers/+my_number.json
Twilio returned the following information:
Unable to update record: The requested resource /2010-04-01/Accounts/my_account_SID/IncomingPhoneNumbers/+my_number.json was not found
More information may be available here:
https://www.twilio.com/docs/errors/20404
I tried all different phone numbers combinations/formats: nothing works.
Thanks for your time reading all this!
Looks like something changed since the blog was written or there was a mistake.
Try the below:
The only difference is adding .public_url to the url object. Also allowed a GET to /bot for testing.
from dotenv import load_dotenv
from flask import Flask, request
from twilio.twiml.messaging_response import MessagingResponse
load_dotenv()
app = Flask(__name__)
#app.route('/bot', methods=['POST','GET'])
def bot():
user = request.values.get('From', '')
resp = MessagingResponse()
resp.message(f'Hello, {user}, thank you for your message!')
return str(resp)
def start_ngrok():
from twilio.rest import Client
from pyngrok import ngrok
url = ngrok.connect(5000)
print('This is',url)
print(' * Tunnel URL:', url)
client = Client()
client.incoming_phone_numbers.list(
phone_number=os.environ.get('TWILIO_PHONE_NUMBER'))[0].update(
sms_url=url.public_url + '/bot')
if __name__ == '__main__':
if os.environ.get('WERKZEUG_RUN_MAIN') != 'true':
start_ngrok()
app.run(debug=True)
import pandas as pd
from google.cloud import bigquery
import google.auth
# from google.cloud import bigquery
# Create credentials with Drive & BigQuery API scopes
# Both APIs must be enabled for your project before running this code
credentials, project = google.auth.default(scopes=[
'https://www.googleapis.com/auth/drive',
'https://www.googleapis.com/auth/spreadsheets',
'https://www.googleapis.com/auth/bigquery',
])
client = bigquery.Client(credentials=credentials, project=project)
# Configure the external data source and query job
external_config = bigquery.ExternalConfig('GOOGLE_SHEETS')
# Use a shareable link or grant viewing access to the email address you
# used to authenticate with BigQuery (this example Sheet is public)
sheet_url = (
'https://docs.google.com/spreadsheets'
'/d/1uknEkew2C3nh1JQgrNKjj3Lc45hvYI2EjVCcFRligl4/edit?usp=sharing')
external_config.source_uris = [sheet_url]
external_config.schema = [
bigquery.SchemaField('name', 'STRING'),
bigquery.SchemaField('post_abbr', 'STRING')
]
external_config.options.skip_leading_rows = 1 # optionally skip header row
table_id = 'BambooHRActiveRoster'
job_config = bigquery.QueryJobConfig()
job_config.table_definitions = {table_id: external_config}
# Get Top 10
sql = 'SELECT * FROM workforce.BambooHRActiveRoster LIMIT 10'
query_job = client.query(sql, job_config=job_config) # API request
top10 = list(query_job) # Waits for query to finish
print('There are {} states with names starting with W.'.format(
len(top10)))
The error I get is:
BadRequest: 400 Error while reading table: workforce.BambooHRActiveRoster, error message: Failed to read the spreadsheet. Errors: No OAuth token with Google Drive scope was found.
I can pull data in from a BigQuery table created from CSV upload, but when I have a BigQuery table created from a linked Google Sheet, I continue to receive this error.
I have tried to replicate the sample in Google's documentation (Creating and querying a temporary table):
https://cloud.google.com/bigquery/external-data-drive
You are authenticating as yourself, which is generally fine for BQ if you have the correct permissions. Using tables linked to Google Sheets often requires a service account. Create one (or have your BI/IT team create one), and then you will have to share the underlying Google Sheet with the service account. Finally, you will need to modify your python script to use the service account credentials and not your own.
The quick way around this is to use the BQ interface, select * from the Sheets-linked table, and save the results to a new table, and query that new table directly in your python script. This works well if this is a one-time upload/analysis. If the data in the sheets will be changing consistently and you will need to routinely query the data, this is not a long-term solution.
I solved problem by adding scope object to client.
from google.cloud import bigquery
import google.auth
credentials, project = google.auth.default(scopes=[
'https://www.googleapis.com/auth/drive',
'https://www.googleapis.com/auth/bigquery',
])
CLIENT = bigquery.Client(project='project', credentials=credentials)
https://cloud.google.com/bigquery/external-data-drive
import pandas as pd
from google.oauth2 import service_account
from google.cloud import bigquery
#from oauth2client.service_account import ServiceAccountCredentials
SCOPES = ['https://www.googleapis.com/auth/drive','https://www.googleapis.com/auth/bigquery']
SERVICE_ACCOUNT_FILE = 'mykey.json'
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
delegated_credentials = credentials.with_subject('myserviceaccountt#domain.iam.gserviceaccount.com')
client = bigquery.Client(credentials=delegated_credentials, project=project)
sql = 'SELECT * FROM `myModel`'
DF = client.query(sql).to_dataframe()
You can try to update your default credentials through the console:
gcloud auth application-default login --scopes=https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/cloud-platform
I want run a back-end process to get the list of channels from youtube without prompting for username password.I tried to do so with the following python code.
#!/usr/bin/python
from apiclient.discovery import build
from optparse import OptionParser
DEVELOPER_KEY = "MY API KEY"
YOUTUBE_API_SERVICE_NAME = "youtube"
YOUTUBE_API_VERSION = "v3"
youtube = build(YOUTUBE_API_SERVICE_NAME, YOUTUBE_API_VERSION,developerKey=DEVELOPER_KEY)
channels_response = youtube.channels().list(
part="contentDetails",
managedByMe="true",
onBehalfOfContentOwner=ownerdetail
).execute()
for channel in channels_response["items"]:
channel_id = channel["id"]
channel_title = channel["snippet"]["title"]
print "Channel details: %s - %s" % channel_id % channel_title
print "Done"
When I try to run this code I'm getting "Access Not Configured"> error in console.
My requirement is to run this successfully without prompting for username and password(since i want it as a back-end process). Any help is this would be really helpful since I'm new to this.
You can do so by getting a refresh token from OAuth2 Playground and setting it in your youtube object.
Here it explains a little more.
And a step by step video.