Find where a t.co link goes to [closed] - twitter

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 11 months ago.
This post was edited and submitted for review 11 months ago and failed to reopen the post:
Original close reason(s) were not resolved
Improve this question
Given a "t.co" link, how can I find out what the link resolves to? For example, if I have "t.co/foo", I want a function or process that returns "domain.com/bar".

I would stay away from external APIs over which you have no control. That will simply introduce a dependency into your application that is a potential point of failure, and could cost you money to use.
CURL can do this quite nicely. Here's how I did it in PHP:
function unshorten_url($url) {
$ch = curl_init($url);
curl_setopt_array($ch, array(
CURLOPT_FOLLOWLOCATION => TRUE, // the magic sauce
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_SSL_VERIFYHOST => FALSE, // suppress certain SSL errors
CURLOPT_SSL_VERIFYPEER => FALSE,
));
curl_exec($ch);
return curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
}
I'm sure this could be adapted to other languages or even scripted with the curl command on UNIXy systems.
http://jonathonhill.net/2012-05-18/unshorten-urls-with-php-and-curl/

If you want to do it from the command line, curl's verbose option comes to the rescue:
curl -v <url>
gives you the HTTP reply. For t.co it seems to give you an HTTP/301 reply (permanently moved). Then, there's a Location field, which points to the URL behind the shortened one.

curl -s -o /dev/null --head -w "%{url_effective}\n" -L "https://t.co/6e7LFNBv"
--head or -I only downloads HTTP headers
-w or --write-out prints the specified string after the output
-L or --location follows location headers

Here is a Python solution.
import urllib2
class HeadRequest(urllib2.Request):
def get_method(self): return "HEAD"
def get_real(url):
res = urllib2.urlopen(HeadRequest(url))
return res.geturl()
Tested with an actual twitter t.co link:
url = "http://t.co/yla4TZys"
expanded = get_real(url)
expanded = http://twitter.com/shanselman/status/276958062156320768/photo/1
Wrap it up with a try-except and you are good to go.

Another Python solution, this time relying on the requests module instead of urllib2 (and all the rest of those libraries):
#!/usr/bin/env python
import requests
shorturl = raw_input("Enter the shortened URL in its entirety: ")
r = requests.get(shorturl)
print("""
The shortened URL forwards to:
%s
""" % r.url)

Here is an R solution, ported from other answers in this thread, and from example() code of the RCurl Package:
unshorten_url <- function(uri){
require(RCurl)
if(RCurl::url.exists(uri)){
# listCurlOptions()
opts <- list(
followlocation = TRUE, # resolve redirects
ssl.verifyhost = FALSE, # suppress certain SSL errors
ssl.verifypeer = FALSE,
nobody = TRUE, # perform HEAD request
verbose = FALSE
);
curlhandle = getCurlHandle(.opts = opts)
getURL(uri, curl = curlhandle)
info <- getCurlInfo(curlhandle)
rm(curlhandle) # release the curlhandle!
info$effective.url
} else {
# just return the url as-is
uri
}
}

Twitter expands the URL. Assume you have a single tweet using twitter API
encoded as json file.
import json
urlInfo=[]
tweet=json.loads(tweet)
keyList=tweet.keys() # list of all posssible keys
tweet['entities'] # gives us values linked to entities
You can observe that there is a value called 'urls'
tweet['entities']['urls'] # gives values mapped to key urls
urlInfo=tweet['entities']['expanded_url'] # move it to a list
# iterating over the list.. gives shortened URL
# and expanded URL
for item in urlInfo:
if "url" and "expanded_url" in urlInfo.keys():
print(item["url"] + " "+item["expanded_url"])

Related

Filename with a hash (#) not uploading

I'm attempting to send a file to OneDrive using the following code:
$uri = "/me/drive/items/$folderId/children('{$fileName}')/content";
$graph = $this->graph->create($user);
$client = $this->graph->createClient();
$item = $graph->createRequest("PUT", $uri)
->attachBody($fileContent)
->setReturnType(Model\DriveItem::class)
->execute($client);
This works great if $fileName is something like Test.doc
But for some reason, when the filename has a hash (#) in the filename, then I get an error:
object(Microsoft\Graph\Model\DriveItem)#1509 (1) {
["_propDict":protected]=>
array(1) {
["error"]=>
array(3) {
["code"]=>
string(10) "BadRequest"
["message"]=>
string(36) "Bad Request - Error in query syntax."
["innerError"]=>
array(2) {
["request-id"]=>
string(36) "ff3fe15f-b1ee-4e92-8abd-2400b1c1b5cf"
["date"]=>
string(19) "2018-10-04T14:30:51"
}
}
}
Can someone possibly clarify if this is a bug or actual behaviour (i.e. you cannot have a # in a filename)
Thanks
I guess you are utilizing Microsoft Graph Library for PHP, special characters such as # needs to be escaped.
So, either replace the hash with %23 (percent encoding) or use rawurlencode function as shown below:
$fileName = rawurlencode("Guide#.docx");
$requestUrl = "https://graph.microsoft.com/v1.0/drives/$driveId/root:/$fileName:/content";
try {
$item = $client->createRequest("PUT", $requestUrl)
->attachBody($fileContent)
->setReturnType(Model\DriveItem::class)
->execute();
} catch (\Microsoft\Graph\Exception\GraphException $ex) {
print $ex;
}
Although the file name have support # in name, but it doesn't mean the Product Team provide the API or adjust the existing API first time, the API you use may not have fully adjusted to suit thore latest naming rules. So it should be actual behavior now but not bug/or you can treat it as none-existed feature.
There are a related issue in the SharePoint dev issue list, although they aren't same one, but the suggestion is the same, vote the exising feature or submit an new one on UserVoice.

How to get youtube channel id using channel custom name?

My question is very similar to this one,
I want to get channel id using channel custom name.
The answer on the question mentioned above which is:
GET https://www.googleapis.com/youtube/v3/search?part=id%2Csnippet&q=annacavalli&type=channel&key={YOUR_API_KEY}
doesn't work on small channels, for ex. when I run it with this channel: https://www.youtube.com/AnnaShearerfashionfettish it returns nothing.
It's very easy, using curl and grep.
Command
channel_name='DOVASYNDROMEYouTubeOfficial' #change this as you like
curl --silent "https://www.youtube.com/c/${channel_name}/videos" |\
grep -o -P '(?<=canonical" href="https://www.youtube.com/channel/)[^"]*'
Output
UCq15_9MvmxT1r2-LLjtkokg
I didn't find a direct way to do this. I did a GET request to get the channel page HTML and parse it.
I used Jsoup to parse the html response.
val doc = Jsoup.parseBodyFragment(body)
val links = doc.select("link[rel=canonical]")
val channelUrl = links.first().attributes().get("href")
Did you try
https://www.googleapis.com/youtube/v3/channels?part=snippetforUsername={username}&key={your key}
Remember to change {your key} to your API key, and {username} to the desired username.

Hiding YouTube API for client using server

My inexperience has left me short of understanding how to hide an API Key. Sorry, but I've been away from web development for 15 years as I specialized in relational databases, and a lot has changed.
I've read a ton of articles, but don't understand how to take advantage of them. I want to put my YouTube API key(s) on the server, but have the client able to use them w/o exposure. I don't understand how setting an API Key on my server (ISP provided) enables the client to access the YouTube channel associated with the project. Can someone explain this to me?
I am not sure what you want to do but for a project I worked on I needed to get a specific playlist from YouTube and make the contents public to the visitors of the website.
What I did is a sort of proxy. I set up a php file contains the api key, and then have the end user get the YT content through this php file.
The php file gets the content form YT using curl.
I hope it helps.
EDIT 1
The way to hide the key is to put it in a PHP file on the server.
This PHP file will the one connecting to youtube and retrieving the data you want on your client page.
This example of code, with the correct api key and correct playlist id will get a json file with the 10 first tracks of the play list.
The $resp will have the json data. To extract it, it has to be decoded for example into an associative array. Once in the array it can be easily mixed in to the html that will be rendered on the client browser.
<?php
$apiKey = "AIza...";
$results = "10";
$playList = "PL0WeB6UKDIHRyXXXXXXXXXX...";
$request = "https://www.googleapis.com/youtube/v3/playlistItems?part=id,contentDetails,snippet&maxResults=" . $results .
"&fields=items(contentDetails%2FvideoId%2Cid%2Csnippet(position%2CpublishedAt%2Cthumbnails%2Fdefault%2Ctitle))" .
"&playlistId=" . $playList .
"&key=" . $apiKey;
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_RETURNTRANSFER => true,
CURLOPT_URL => $request,
CURLOPT_SSL_VERIFYPEER => false
));
$resp = curl_exec($curl);
if (curl_errno($curl)) {
$status = "CURL_ERROR";
}else{
// check the HTTP status code of the request
$resultStatus = curl_getinfo($curl, CURLINFO_HTTP_CODE);
if ($resultStatus == 200) {
$status = "OK";
//Do something with the $resp which is in JSON format.
//Like decoding it into an associative array
} else {
$status = "YT_ERROR";
}
}
curl_close($curl);
?>
<html>
<!-- your html here -->
</html>
Note: CURLOPT_SSL_VERIFYPEER is set to false. This is in development. For prod it should be true.
Also note that using the api this way, you can restrict the calls to your api key bounding them to your domain. You do that in the googla api console. (Tip for production)

Drupal 8 Guzzle Format Query String

Forgive me for my ignorance, this is my first attempt at Drupal 8 and I'm not a good php developer to begin with. But I've been reading and searching for hours. I'm trying to do a post using the new Guzzle that replaces the drupal_http_request(). I've done this using Curl but can't seem to get this going in the right direction here. I'm just not "getting it".
Here is a sample of the array I have that pulls data from a custom form. I also tried this with a custom variable where I built the string.
$fields = array(
"enroll_id" => $plan,
"notice_date" => $date,
"effective_date" => $date,
);
$client = \Drupal::httpClient();
$response = $client->post('myCustomURL', ['query' => $fields]);
$data = $response->getBody()->getContents();
try {
drupal_set_message($data);
} catch (RequestException $e) {
watchdog_exception('MyCustomForm', $e->getMessage());
}
This indeed returns the result of REJECTED from my API in $data below - but it doesn't append the URL to included the query => array. I've tried numerous combinations of this just putting the fully built URL in the post (that works with my API - tested) and I still receive the same result from my API. In the end what I'm trying to accomplish is
https://myCustomURL?enroll_id=value&notice_date=12/12/12&effective_date=12/12/12
Any direction or tips would be much appreciated.
Thanks for the responses guys. I was able to get it to work correctly by changing a few things in my post. First changing client -> post to a request('POST', XXX) and then changing "query" to "form_params" as "body" has been deprecated.
http://docs.guzzlephp.org/en/latest/quickstart.html#query-string-parameters
$client = \Drupal::httpClient();
$response = $client->request('POST','https://myURL.html', ['form_params' => $fields]);
$data = $response->getBody()->getContents();
Using $client->post will send a POST request. By looking at the URL that you tested directly you want a GET request.
Either use $client->get or $client->request with the GET parameter. More information and examples in the Guzzle documentation.

Slack clean all messages (~8K) in a channel [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 months ago.
The community reviewed whether to reopen this question 7 months ago and left it closed:
Original close reason(s) were not resolved
Improve this question
We currently have a Slack channel with ~8K messages all comes from Jenkins integration. Is there any programmatic way to delete all messages from that channel? The web interface can only delete 100 messages at a time.
I quickly found out there's someone already made a helper: slack-cleaner for this.
And for me it's just:
slack-cleaner --token=<TOKEN> --message --channel jenkins --user "*" --perform
I wrote a simple node script for deleting messages from public/private channels and chats. You can modify and use it.
https://gist.github.com/firatkucuk/ee898bc919021da621689f5e47e7abac
First, modify your token in the scripts configuration section then run the script:
node ./delete-slack-messages CHANNEL_ID
Get an OAuth token:
Go to https://api.slack.com/apps
Click 'Create New App', and name your (temporary) app.
In the side nav, go to 'Oauth & Permissions'
On that page, find the 'Scopes' section. Click 'Add an OAuth Scope' and add 'channels:history' and 'chat:write'. (see scopes)
At the top of the page, Click 'Install App to Workspace'. Confirm, and on page reload, copy the OAuth Access Token.
Find the channel ID
Also, the channel ID can be seen in the browser URL when you open slack in the browser. e.g.
https://mycompany.slack.com/messages/MY_CHANNEL_ID/
or
https://app.slack.com/client/WORKSPACE_ID/MY_CHANNEL_ID
default clean command did not work for me giving following error:
$ slack-cleaner --token=<TOKEN> --message --channel <CHANNEL>
Running slack-cleaner v0.2.4
Channel, direct message or private group not found
but following worked without any issue to clean the bot messages
slack-cleaner --token <TOKEN> --message --group <CHANNEL> --bot --perform --rate 1
or
slack-cleaner --token <TOKEN> --message --group <CHANNEL> --user "*" --perform --rate 1
to clean all the messages.
I use rate-limit of 1 second to avoid HTTP 429 Too Many Requests error because of slack api rate limit. In both cases, channel name was supplied without # sign
For anyone else who doesn't need to do it programmatic,
here's a quick way:
(probably for paid users only)
Open the channel in web or the desktop app, and click the cog (top right).
Choose "Additional options..." to bring up the archival menu. notes
Select "Set the channel message retention policy".
Set "Retain all messages for a specific number of days".
All messages older than this time are deleted permanently!
I usually set this option to "1 day" to leave the channel with some context, then I go back into the above settings, and set it's retention policy back to "default" to go continue storing them from now-on.
Notes:
Luke points out: If the option is hidden: you have to go to global workspace Admin settings, Message Retention & Deletion, and check "Let workspace members override these settings"
!!UPDATE!!
as #niels-van-reijmersdal metioned in comment.
This feature has been removed. See this thread for more info: twitter.com/slackhq/status/467182697979588608?lang=en
!!END UPDATE!!
Here is a nice answer from SlackHQ in twitter, and it works without any third party stuff.
https://twitter.com/slackhq/status/467182697979588608?lang=en
You can bulk delete via the archives (http://my.slack.com/archives )
page for a particular channel: look for "delete messages" in menu
Option 1 You can set a Slack channel to automatically delete messages after 1 day, but it's a little hidden. First, you have to go to your Slack Workspace Settings, Message Retention & Deletion, and check "Let workspace members override these settings". After that, in the Slack client you can open a channel, click the gear, and click "Edit message retention..."
Option 2 The slack-cleaner command line tool that others have mentioned.
Option 3 Below is a little Python script that I use to clear Private channels. Can be a good starting point if you want more programmatic control of deletion. Unfortunately Slack has no bulk-delete API, and they rate-limit the individual delete to 50 per minute, so it unavoidably takes a long time.
# -*- coding: utf-8 -*-
"""
Requirement: pip install slackclient
"""
import multiprocessing.dummy, ctypes, time, traceback, datetime
from slackclient import SlackClient
legacy_token = raw_input("Enter token of an admin user. Get it from https://api.slack.com/custom-integrations/legacy-tokens >> ")
slack_client = SlackClient(legacy_token)
name_to_id = dict()
res = slack_client.api_call(
"groups.list", # groups are private channels, conversations are public channels. Different API.
exclude_members=True,
)
print ("Private channels:")
for c in res['groups']:
print(c['name'])
name_to_id[c['name']] = c['id']
channel = raw_input("Enter channel name to clear >> ").strip("#")
channel_id = name_to_id[channel]
pool=multiprocessing.dummy.Pool(4) #slack rate-limits the API, so not much benefit to more threads.
count = multiprocessing.dummy.Value(ctypes.c_int,0)
def _delete_message(message):
try:
success = False
while not success:
res= slack_client.api_call(
"chat.delete",
channel=channel_id,
ts=message['ts']
)
success = res['ok']
if not success:
if res.get('error')=='ratelimited':
# print res
time.sleep(float(res['headers']['Retry-After']))
else:
raise Exception("got error: %s"%(str(res.get('error'))))
count.value += 1
if count.value % 50==0:
print(count.value)
except:
traceback.print_exc()
retries = 3
hours_in_past = int(raw_input("How many hours in the past should messages be kept? Enter 0 to delete them all. >> "))
latest_timestamp = ((datetime.datetime.utcnow()-datetime.timedelta(hours=hours_in_past)) - datetime.datetime(1970,1,1)).total_seconds()
print("deleting messages...")
while retries > 0:
#see https://api.slack.com/methods/conversations.history
res = slack_client.api_call(
"groups.history",
channel=channel_id,
count=1000,
latest=latest_timestamp,)#important to do paging. Otherwise Slack returns a lot of already-deleted messages.
if res['messages']:
latest_timestamp = min(float(m['ts']) for m in res['messages'])
print datetime.datetime.utcfromtimestamp(float(latest_timestamp)).strftime("%r %d-%b-%Y")
pool.map(_delete_message, res['messages'])
if not res["has_more"]: #Slack API seems to lie about this sometimes
print ("No data. Sleeping...")
time.sleep(1.0)
retries -= 1
else:
retries=10
print("Done.")
Note, that script will need modification to list & clear public channels. The API methods for those are channels.* instead of groups.*
As other answers allude, Slack's rate limits make this tricky - the rate limit is relatively low for their chat.delete API at ~50 requests per minute.
The best strategy that respects the rate limit is to retrieve messages from the channel you want to clear, then delete the messages in batches under 50 that run on a minutely interval.
I've built a project containing an example of this batching that you can easily fork and deploy on Autocode - it lets you clear a channel via slash command (and allows you restrict access to the command to just certain users of course!). When you run /cmd clear in a channel, it marks that channel for clearing and runs the following code every minute until it deletes all the messages in the channel:
console.log(`About to clear ${messages.length} messages from #${channel.name}...`);
let deletionResults = await async.mapLimit(messages, 2, async (message) => {
try {
await lib.slack.messages['#0.6.1'].destroy({
id: clearedChannelId,
ts: message.ts,
as_user: true
});
return {
successful: true
};
} catch (e) {
return {
successful: false,
retryable: e.message && e.message.indexOf('ratelimited') !== -1
};
}
});
You can view the full code and a guide to deploying your own version here: https://autocode.com/src/jacoblee/slack-clear-messages/
Tip: if you gonna use the slack cleaner https://github.com/kfei/slack-cleaner
You will need to generate a token: https://api.slack.com/custom-integrations/legacy-tokens
If you like Python and have obtained a legacy API token from the slack api, you can delete all private messages you sent to a user with the following:
import requests
import sys
import time
from json import loads
# config - replace the bit between quotes with your "token"
token = 'xoxp-854385385283-5438342854238520-513620305190-505dbc3e1c83b6729e198b52f128ad69'
# replace 'Carl' with name of the person you were messaging
dm_name = 'Carl'
# helper methods
api = 'https://slack.com/api/'
suffix = 'token={0}&pretty=1'.format(token)
def fetch(route, args=''):
'''Make a GET request for data at `url` and return formatted JSON'''
url = api + route + '?' + suffix + '&' + args
return loads(requests.get(url).text)
# find the user whose dm messages should be removed
target_user = [i for i in fetch('users.list')['members'] if dm_name in i['real_name']]
if not target_user:
print(' ! your target user could not be found')
sys.exit()
# find the channel with messages to the target user
channel = [i for i in fetch('im.list')['ims'] if i['user'] == target_user[0]['id']]
if not channel:
print(' ! your target channel could not be found')
sys.exit()
# fetch and delete all messages
print(' * querying for channel', channel[0]['id'], 'with target user', target_user[0]['id'])
args = 'channel=' + channel[0]['id'] + '&limit=100'
result = fetch('conversations.history', args=args)
messages = result['messages']
print(' * has more:', result['has_more'], result.get('response_metadata', {}).get('next_cursor', ''))
while result['has_more']:
cursor = result['response_metadata']['next_cursor']
result = fetch('conversations.history', args=args + '&cursor=' + cursor)
messages += result['messages']
print(' * next page has more:', result['has_more'])
for idx, i in enumerate(messages):
# tier 3 method rate limit: https://api.slack.com/methods/chat.delete
# all rate limits: https://api.slack.com/docs/rate-limits#tiers
time.sleep(1.05)
result = fetch('chat.delete', args='channel={0}&ts={1}'.format(channel[0]['id'], i['ts']))
print(' * deleted', idx+1, 'of', len(messages), 'messages', i['text'])
if result.get('error', '') == 'ratelimited':
print('\n ! sorry there have been too many requests. Please wait a little bit and try again.')
sys.exit()
Here is a great chrome extension to bulk delete your slack channel/group/im messages - https://slackext.com/deleter , where you can filter the messages by star, time range, or users.
BTW, it also supports load all messages in recent version, then you can load your ~8k messages as you need.
There is a slack tool to delete all slack messages on your workspace. Check it out: https://www.messagebender.com

Resources