How to get a user's new Tweets with a Telegram Python bot while run_polling? - twitter

I'm currently developing a Telegram bot using telegram-python-bot and tweepy.
I want to create a feature that allows users of the bot to add their Twitter ID list via Telegram and have their new Tweets sent to them in real-time.
I want that the bot should be application.run_polling() to receive commands from the user, and at the same time, forwarding new tweets from Twitter users in users individual list.
When I read the tweepy documentation, I realized that I can get real-time tweets with fewer api requests if I fetch them through MyStream(auth=auth, listener=None).
But I don't know how to get both functions to work on the same file at the same time.
version
nest_asyncio-1.5.6 python_telegram_bot-20.0 tweepy-4.12.1
def main() -> None:
application = Application.builder().token("...").build()
add_list = ConversationHandler(
entry_points=[CallbackQueryHandler(input_id, pattern='input_id')],
states={ADD :[MessageHandler(filters.TEXT & ~filters.COMMAND, add)],},
fallbacks=[CallbackQueryHandler(button,pattern='back')])
application.add_handler(CommandHandler("on", on))
application.add_handler(add_list)
application.add_handler(CommandHandler("start", start))
application.add_handler(CommandHandler("list", list_setting))
application.add_handler(CommandHandler("admin", admin))
application.add_handler(CommandHandler("help", help_command))
application.add_handler(CallbackQueryHandler(button))
application.run_polling()
if __name__ == "__main__":
main()
This is my main statement and I made it work until the SIGINT(ctrl+c) came in via application.run_polling().
I want to combine the above code to run and do the following at the same time.
import tweepy
consumer_key = "..." # Twitter API Key
consumer_secret = "..." # Twitter API Secret Key
access_token = "..." # Twitter Access Key
access_token_secret = "..." # Twitter Access Secret Key
usernames = ['...']
auth = tweepy.OAuth1UserHandler(
consumer_key, consumer_secret, access_token, access_token_secret
)
# Convert screen names to user IDs
user_ids = []
for username in usernames:
user = tweepy.API(auth).get_user(screen_name=username)
user_ids.append(str(user.id))
# Create a custom stream class
class MyStream(tweepy.Stream):
def __init__(self, auth, listener=None):
super().__init__(consumer_key, consumer_secret, access_token, access_token_secret)
def on_status(self, status):
tweet_url = f"https://twitter.com/{status.user.screen_name}/status/{status.id_str}"
print(f"{status.user.screen_name} tweeted: {status.text}\n{tweet_url}")
# send message to telegram
# Create a stream object with the above class and authentication
myStream = MyStream(auth=auth, listener=None)
# Start streaming for the selected users
myStream.filter(follow=user_ids)
I also tried to use thread's interval function or python-telegram-bot's job_queue.run_repeating function,
but these seem problematic for forwarding messages in real time.
I'm desperately looking for someone to help me with this😢.

Related

Get/list all tweets of all users my account follow

I am using python tweepy to connect the twitter end point, and it's very simple to list all of any single user's tweets. It is also possible to read my account's "following" list, so technicly I can get the list of all the tweets by all of my followed users, thing is, it will be lot's and lot's of seperate API calls.
Is there a way to bulk this effectively?
You're not going to be able to get them all in one go, Twitter has rate limits to prevent that, but this script will get as many from each user as possible:
import tweepy
# Put your API keys here
CONSUMER_KEY = ""
CONSUMER_SECRET = ""
ACCESS_TOKEN = ""
ACCESS_TOKEN_SECRET = ""
# Authenticate to Tweepy and wait if you get rate limited
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth, wait_on_rate_limit=True)
# For each user that you follow.....
for user in tweepy.Cursor(api.friends, screen_name="pigeonburger").items():
# Get each user's username and print out 100 of their tweets
username = user._json['screen_name']
print(api.user_timeline(screen_name = username, count = 100))
# Do what you want with those tweets after

Trying to use Twitter API but my code wont run anything

I am trying to use the twitter API in order to stream all tweets that include Michigan State, Spartans, and MSU. After I can figure this out I want to use different big10 key words. However, I run this code and I can't get past
ln (*) no matter how long I wait nothing is happening. Is there any issue with my code? Or how to I get the display of this information so I can analyze it?
THANK YOU!
from tweepy.streaming import StreamListener
from tweepy import OAuthHandler
from tweepy import Stream
#Variables that contains the user credentials to access Twitter API
access_token = "ENTER YOUR ACCESS TOKEN"
access_token_secret = "ENTER YOUR ACCESS TOKEN SECRET"
consumer_key = "ENTER YOUR API KEY"
consumer_secret = "ENTER YOUR API SECRET"
#This is a basic listener that just prints received tweets to stdout.
class StdOutListener(StreamListener):
def on_data(self, data):
print data
return True
def on_error(self, status):
print status
if __name__ == '__main__':
#This handles Twitter authentification and the connection to Twitter Streaming API
l = StdOutListener()
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
stream = Stream(auth, l)
#This line filter Twitter Streams to capture data by the keywords: 'MichiganState', 'Spartans', 'MSU'
stream.filter(track=['MichiganState', 'Spartans', 'MSU'])'
Use on_status(self, status) in the listener class:
class StdOutListener(tweepy.StreamListener):
def on_status(self, status):
print status.text
print status.id

How do I migrate users from OpenID to Google OAuth2/OpenID Connect using Python Social Auth?

Google is deprecating the OpenID endpoint I was using (v1.0 I think, via the django_openid_auth module) and I need to update my app and migrate my users' accounts to use Google OAuth2.
I've changed the app to use python-social-auth and have it authenticating with social.backends.google.GoogleOAuth2 successfully.
I've written a pipeline function to find associated OpenID urls from the old table and this is working for the other backends I care about but Google:
def associate_legacy_user(backend, response, uid=None, user=None,
*args, **kwargs):
if uid and not user:
# Try to associate accounts registered in the old openid table
identity_url = None
if backend.name == 'google-oauth2':
# TODO: this isn't working
identity_url = response.get('open_id')
else:
# for all other backends, see if there is a claimed_id url
# matching the identity_url use identity_url instead of uid
# as uid may be the user's email or username
try:
identity_url = response.identity_url
except AttributeError:
identity_url = uid
if identity_url:
# raw sql as this is no longer an installed app
user_ids = sql_query.dbquery('SELECT user_id '
'FROM django_openid_auth_useropenid '
'WHERE claimed_id = %s',
(identity_url,))
if len(user_ids) == 1:
return {'user': User.objects.get(id=user_ids[0]['user_id'])}
As best I can tell from reading Google's migration guide, I need to add an openid.realm to the request, which I've done as follows in settings.py:
SOCIAL_AUTH_GOOGLE_OAUTH2_AUTH_EXTRA_ARGUMENTS \
= {'openid.realm': 'http://example.com/'}
But this doesn't seem to be returning the open_id value in the response passed into my pipeline function.
I seem to be stuck on Step 3:
I tried sub-classing the backend to change the RESPONSE_TYPE to add id_token but that returned an empty response:
import social.backends.google
class CustomGoogleOAuth2(social.backends.google.GoogleOAuth2):
RESPONSE_TYPE = 'code id_token'
I tried building an additional request to https://www.googleapis.com/oauth2/v3/token similar to this example, but I don't really know how to go about putting that together and debugging it.
Some more details:
My old claimed_ids for Google OpenID users look like: https://www.google.com/accounts/o8/id?id=AItOawmAW18QuHDdn6PZzaiI5BWUb84mZzNB9eo
I'm happy to use social.backends.google.GoogleOpenIdConnect or a similar alternative backend if that's an easier solution. And while it seems to be closer to what the Google docs are talking about, I wasn't able to get it to work when I tried:
I get a 400 Error: invalid_request Parameter not allowed for this message type: nonce
I can get past the nonce error using social.backends.google.GoogleOpenIdConnect by adding id_token to the RESPONSE_TYPE but then I get an AuthMissingParameter error in my /complete/google-openidconnect/ endpoint as the request's GET and POST are empty. (Tried 'code id_token', 'token id_token', 'id_token', ...)
I don't want to use social.backends.google.GooglePlusAuth as that doesn't integrate as nicely with my current login form.
Worst case, I should be able to use social.pipeline.social_auth.associate_by_email, but I only have email addresses for maybe 80% of the users so that leaves quite a few who will have a new account and need support to associate it manually.
Try as I might, I can't find any examples of people doing a similar migration with python-social-auth, but it must be happening to lots of people.
Any ideas?
Solution works for python social auth 0.1.26
In new versions (0.2.*) of python social auth, there is GoogleOpenIdConnect, but it does not work fine (at least I did not succeed). And my project has some legacy, so I can't use new version of social.
I wrote custom GoogleOpenIdConnect backend:
import datetime
from calendar import timegm
from jwt import InvalidTokenError, decode as jwt_decode
from social.backends.google import GoogleOAuth2
from social.exceptions import AuthTokenError
class GoogleOpenIdConnect(GoogleOAuth2):
name = 'google-openidconnect'
ACCESS_TOKEN_URL = 'https://www.googleapis.com/oauth2/v3/token'
DEFAULT_SCOPE = ['openid']
EXTRA_DATA = ['id_token', 'refresh_token', ('sub', 'id')]
ID_TOKEN_ISSUER = "accounts.google.com"
def user_data(self, access_token, *args, **kwargs):
return self.get_json(
'https://www.googleapis.com/plus/v1/people/me/openIdConnect',
params={'access_token': access_token, 'alt': 'json'}
)
def get_user_id(self, details, response):
return response['sub']
def request_access_token(self, *args, **kwargs):
"""
Retrieve the access token. Also, validate the id_token and
store it (temporarily).
"""
response = self.get_json(*args, **kwargs)
response['id_token_parsed'] = self.validate_and_return_id_token(response['id_token'])
return response
def validate_and_return_id_token(self, id_token):
"""
Validates the id_token according to the steps at
http://openid.net/specs/openid-connect-core-1_0.html#IDTokenValidation.
"""
try:
id_token = jwt_decode(id_token, verify=False)
except InvalidTokenError as err:
raise AuthTokenError(self, err)
# Verify the token was issued in the last 10 minutes
utc_timestamp = timegm(datetime.datetime.utcnow().utctimetuple())
if id_token['iat'] < (utc_timestamp - 600):
raise AuthTokenError(self, 'Incorrect id_token: iat')
return id_token
Notes:
get_user_id – An identifier for the user, unique among all Google accounts and never reused.
request_access_token – there is I add id_token_parsed to response, and it will be used in pipeline.
validate_and_return_id_token – validate of jwt is disabled, because in google developers console I have registered Client ID as web application so, I have no certificates for validate this data.
Then I created pipelines:
def social_user_google_backwards(strategy, uid, *args, **kwargs):
"""
Provide find user that was connect with google openID, but is logging with google oauth2
"""
result = social_user(strategy, uid, *args, **kwargs)
provider = strategy.backend.name
user = result.get('user')
if provider != 'google-openidconnect' or user is not None:
return result
openid_id = kwargs.get('response', {}).get('id_token_parsed', {}).get('openid_id')
if openid_id is None:
return result
social = _get_google_openid(strategy, openid_id)
if social is not None:
result.update({
'user': social.user,
'is_new': social.user is None,
'google_openid_social': social
})
return result
def _get_google_openid(strategy, openid_id):
social = strategy.storage.user.get_social_auth('openid', openid_id)
if social:
return social
return None
def associate_user(strategy, uid, user=None, social=None, *args, **kwargs):
result = social_associate_user(strategy, uid, user, social, *args, **kwargs)
google_openid_social = kwargs.pop('google_openid_social', None)
if google_openid_social is not None:
google_openid_social.delete()
return result
And changed my SOCIAL_AUTH_PIPELINE and AUTHENTICATION_BACKENDS settings:
AUTHENTICATION_BACKENDS = (
...
#'social.backends.open_id.OpenIdAuth' remove it
'social_extension.backends.google.GoogleOpenIdConnect', # add it
...
)
and
SOCIAL_AUTH_PIPELINE = (
'social.pipeline.social_auth.social_details',
'social.pipeline.social_auth.social_uid',
'social.pipeline.social_auth.auth_allowed',
# 'social.pipeline.social_auth.social_user', remove it
'social_extension.pipeline.social_user_google_backwards', # add it
'social.pipeline.user.get_username',
...
# 'social.pipeline.social_auth.associate_user', remove it
'social_extension.pipeline.associate_user', # add it
'social.pipeline.social_auth.load_extra_data',
...
)

how to get all tweets past and present with particular # tag or # tag

I am trying to make an application for an organization which will required to fetch all past as well present tweets with some particular hashtag like #airtel, #airtel etc, how should I get past tweet, I am able to fetch the present tweet with the following url : "https://api.twitter.com/1.1/search/tweets.json?q=%23airtel"
Thanks
You can get up to a maximum of 100 tweets with the twitter rest api see the following twitter documentation. The best you can do is use the count parameter https://api.twitter.com/1.1/search/tweets.json?q=%23airtel&count=100
After various search on Google I found some useful library for fetching tweets e.g:
TwitterSearch [https://github.com/ckoepp/TwitterSearch], you would I find the twitter profile # https://twitter.com/twittersearch
Tweepy [https://github.com/tweepy/tweepy], you could find more info at http://www.tweepy.org/
I have implemented using both. Tweepy implementation is as follows:
import tweepy
import json
consumer_key = "get"
consumer_secret = "From"
access_token = "Twitter"
access_token_secret = "site"
# Authenticate twitter Api
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
#made a cursor
c = tweepy.Cursor(api.search, q='%23Coursera')
c.pages(15) # you can change it make get tweets
#Lets save the selected part of the tweets inot json
tweetJson = []
for tweet in c.items():
if tweet.lang == 'en':
createdAt = str(tweet.created_at)
authorCreatedAt = str(tweet.author.created_at)
tweetJson.append(
{'tweetText':tweet.text,
'tweetCreatedAt':createdAt,
'authorName': tweet.author.name,
})
#dump the data into json format
print json.dumps(tweetJson)
If any one have problem, let me know, will provide git repo for this.
Thanks
Krishna

Twitter 3-legged authorization in Ruby

I am trying my hand ruby on rails. Mostly I have written code in Sinatra. Anyway this question may not have to do anything with framework. And this question may sound a very novice question. I am playing with Twitter 1.1 APIs and OAuth first time.
I have created an app XYZ and registered it with Twitter. I got XYZ's consumer key i.e., CONSUMER_KEY and consumer secret i.e. CONSUMER_SECRET. I also got XYZ's own access token i.e ACCESS_TOKEN and access secret i.e. ACCESS_SECRET
XYZ application type: Read, Write and Access direct messages
XYZ callback URL: http://www.mysite.com/cback
And I have checked: Allow this application to be used to Sign in with Twitter
What I am trying to do is very simple:
1) Users come to my website and click a link Link your twitter account (not signin with twitter)
2) That opens twitter popup where user grants permission to XYZ to perform actions on his/her behalf
3) Once user permits and popup gets closed, XYZ app gets user's access token and secret and save in the database.
4) Then XYZ uses that user's token and secret to perform actions in future.
I may be moron that such work flow has been implemented on several thousands sites and Twitter API documentations explain this 3-legged authentication, still I am unable to figure it out.
I have read https://dev.twitter.com/docs/auth/3-legged-authorization and https://dev.twitter.com/docs/auth/implementing-sign-twitter Unfortunately no ruby code found on internet that explains with step by step example.
What link should be used to open twitter authentication page when user clicks Link your twitter account.
Can anyone here, write some pseudo code with my pseduo credential above to achieve my goal from beging till end of this work flow? Thanks.
UPDATE:
I started with requesting request token as
require 'oauth'
consumer = OAuth::Consumer.new(CONSUMER_KEY, CONSUMER_SECRET,
{ site: "https://twitter.com"})
request_token = consumer.get_request_token oauth_callback: 'http://www.mysite.com/tauth'
redirect_to request_token.authorize_url
I'm not familiar with ROR but here is the workflow of the OAuth 'dance' that you need to follow when the user clicks your button:
Obtain an unauthorized request token from Twitter by sending a
request to
POST https://api.twitter.com/oauth/request_token
signing the request using your consumer secret. This will be done in the background and
will be transparent to the user.
You will receive am oauth_token and oauth_token_secret back from
twitter.
Redirect the user to
https://api.twitter.com/oauth/authorize?oauth_token=[token_received_from_twitter]
using the oauth token value you received from Twitter in step 2.
When the user authorizes your app they will be redirected to your
callback url with oauth_token and oauth_verifier appended to the
url. i.e.
http://www.mysite.com/cback?oauth_token=NPcudxy0yU5T3tBzho7iCotZ3cnetKwcTIRlX0iwRl0&oauth_verifer=uw7NjWHT6OJ1MpJOXsHfNxoAhPKpgI8BlYDhxEjIBY
Convert the request token into an access token by sending a signed
request along with the oauth_verifier to
POST
https://api.twitter.com/oauth/access_token
signing your request
with your consumer secret and the token secret received in step 2.
If everything goes ok, you will receive a new oauth_token and
oauth_token_secret from Twitter. This is your access token for the
user.
Using the access token and secret received in step 6 you can make
Twitter api calls on behalf the the user by sending signed requests
to the appropriate api endpoints.
Hope you solved your problem by this time, but I built this sample Sign in with Twitter ruby web app that provide all explanation you need to do this integration. Below there's a class that implements all necessary methods with comments:
require "net/https"
require "simple_oauth"
# This class implements the requests that should
# be done to Twitter to be able to authenticate
# users with Twitter credentials
class TwitterSignIn
class << self
def configure
#oauth = YAML.load_file(TWITTER)
end
# See https://dev.twitter.com/docs/auth/implementing-sign-twitter (Step 1)
def request_token
# The request to get request tokens should only
# use consumer key and consumer secret, no token
# is necessary
response = TwitterSignIn.request(
:post,
"https://api.twitter.com/oauth/request_token",
{},
#oauth
)
obj = {}
vars = response.body.split("&").each do |v|
obj[v.split("=").first] = v.split("=").last
end
# oauth_token and oauth_token_secret should
# be stored in a database and will be used
# to retrieve user access tokens in next requests
db = Daybreak::DB.new DATABASE
db.lock { db[obj["oauth_token"]] = obj }
db.close
return obj["oauth_token"]
end
# See https://dev.twitter.com/docs/auth/implementing-sign-twitter (Step 2)
def authenticate_url(query)
# The redirection need to be done with oauth_token
# obtained in request_token request
"https://api.twitter.com/oauth/authenticate?oauth_token=" + query
end
# See https://dev.twitter.com/docs/auth/implementing-sign-twitter (Step 3)
def access_token(oauth_token, oauth_verifier)
# To request access token, you need to retrieve
# oauth_token and oauth_token_secret stored in
# database
db = Daybreak::DB.new DATABASE
if dbtoken = db[oauth_token]
# now the oauth signature variables should be
# your app consumer keys and secrets and also
# token key and token secret obtained in request_token
oauth = #oauth.dup
oauth[:token] = oauth_token
oauth[:token_secret] = dbtoken["oauth_token_secret"]
# oauth_verifier got in callback must
# to be passed as body param
response = TwitterSignIn.request(
:post,
"https://api.twitter.com/oauth/access_token",
{:oauth_verifier => oauth_verifier},
oauth
)
obj = {}
vars = response.body.split("&").each do |v|
obj[v.split("=").first] = v.split("=").last
end
# now the we got the access tokens, store it safely
# in database, you're going to use it later to
# access Twitter API in behalf of logged user
dbtoken["access_token"] = obj["oauth_token"]
dbtoken["access_token_secret"] = obj["oauth_token_secret"]
db.lock { db[oauth_token] = dbtoken }
else
oauth_token = nil
end
db.close
return oauth_token
end
# This is a sample Twitter API request to
# make usage of user Access Token
# See https://dev.twitter.com/docs/api/1.1/get/account/verify_credentials
def verify_credentials(oauth_token)
db = Daybreak::DB.new DATABASE
if dbtoken = db[oauth_token]
# see that now we use the app consumer variables
# plus user access token variables to sign the request
oauth = #oauth.dup
oauth[:token] = dbtoken["access_token"]
oauth[:token_secret] = dbtoken["access_token_secret"]
response = TwitterSignIn.request(
:get,
"https://api.twitter.com/1.1/account/verify_credentials.json",
{},
oauth
)
user = JSON.parse(response.body)
# Just saving user info to database
user.merge! dbtoken
db.lock { db[user["screen_name"]] = user }
result = user
else
result = nil
end
db.close
return result
end
# Generic request method used by methods above
def request(method, uri, params, oauth)
uri = URI.parse(uri.to_s)
# always use SSL, you are dealing with other users data
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
# uncomment line below for debug purposes
#http.set_debug_output($stdout)
req = (method == :post ? Net::HTTP::Post : Net::HTTP::Get).new(uri.request_uri)
req.body = params.to_a.map { |x| "#{x[0]}=#{x[1]}" }.join("&")
req["Host"] = "api.twitter.com"
# Oauth magic is done by simple_oauth gem.
# This gem is enable you to use any HTTP lib
# you want to connect in OAuth enabled APIs.
# It only creates the Authorization header value for you
# and you can assign it wherever you want
# See https://github.com/laserlemon/simple_oauth
req["Authorization"] = SimpleOAuth::Header.new(method, uri.to_s, params, oauth)
http.request(req)
end
end
end
More detailed explanation at:
https://github.com/lfcipriani/sign_in_with_twitter_sample

Resources