I get different results when using a model to get image annotation predictions from web UI and from API. Specifically, using the web UI I actually get predictions, but using the API I get nothing - just empty output.
It's this one that gives nothing using the API: https://cloud.google.com/vision/automl/docs/predict#automl-nl-example-cli
Specifically, the return value is {} - an empty JS object. So, the call goes through just fine, there's just no output.
Any hints as to how to debug the issue?
By default only results with prediction score > 0.5 are returned by the API.
To get all predictions you will need to provide extra argument 'score_threshold' to predict request:
For the REST API:
{
"payload": {
"image": {
"imageBytes": "YOUR_IMAGE_BYTES"
},
"params": { "score_threshold": "0.0" },
}
}
For the python call:
payload = {'image': {'image_bytes': content }, "params": { "score_threshold": "0.0" }}
With this argument all predictions will be returned. The predictions will be ordered by the 'score'.
Hope that helps,
That doesn't work, at least at the moment.
Instead the params need to go at the same level as the payload. E.g.:
{
"payload": {
"image": {
"imageBytes": "YOUR_IMAGE_BYTES"
}
},
"params": { "score_threshold": "0.0" },
}
I'm using the following function score for outfits purchased:
{
"query": {
"function_score": {
"field_value_factor": {
"field": "purchased",
"factor": 1.2,
"modifier": "sqrt",
"missing": 1
}
}
}
}
However, when I create a search - I get the following error:
"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [purchased] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
The syntax is correct for the search as I've run it locally and it works perfectly. I'm now running it on my server and it's not workings. Do I need to define purchased as an integer somewhere or is this due to something else?
The purchased field is an analyzed string field, hence the error you see.
When indexing your documents, make sure that the numbers are not within double quotes, i.e.
Wrong:
{
"purchased": "324"
}
Right:
{
"purchased": 324
}
...or if you can't change the source documents (because you're not responsible for producing them), make sure that you create a mapping that defines the purchased field as being an integer field.
{
"your_type": {
"properties": {
"purchased": {
"type": "integer"
}
}
}
}
I would like to fetch details of a YouTube channel which has a custom URL, like https://www.youtube.com/c/pratiksinhchudasamaisawesome.
Custom channel URLs follow this format: https://www.youtube.com/c/{custom_channel_name}.
I can fetch the details of YouTube channels by Channel ID and username without any issues. Unfortunately, I need to use the custom channel URL which is the only time I encounter this issue.
I developed my app few months ago, and the custom channel URL was working up until a few days ago. Now, the YouTube data API does not return anything for the YouTube custom channel URL if I try get details using their custom name.
To get the details of this channel: https://www.youtube.com/user/thenewboston, for example, the request would be:
GET https://www.googleapis.com/youtube/v3/channels?part=snippet&forUsername=thenewboston&key={YOUR_API_KEY}
Response
200
- SHOW HEADERS -
{
"kind": "youtube#channelListResponse",
"etag": "\"zekp1FB4kTkkM-rWc1qIAAt-BWc/8Dz6-vPu69KX3yZxVCT3-M9YWQA\"",
"pageInfo": {
"totalResults": 1,
"resultsPerPage": 5
},
"items": [
{
"kind": "youtube#channel",
"etag": "\"zekp1FB4kTkkM-rWc1qIAAt-BWc/KlQLDlUPRAmACwKt9V8V2yrOfEg\"",
"id": "UCJbPGzawDH1njbqV-D5HqKw",
"snippet": {
"title": "thenewboston",
"description": "Tons of sweet computer related tutorials and some other awesome videos too!",
"publishedAt": "2008-02-04T16:09:31.000Z",
"thumbnails": {
"default": {
"url": "https://yt3.ggpht.com/--n5ELY2uT-U/AAAAAAAAAAI/AAAAAAAAAAA/d9JvaIEpstw/s88-c-k-no-rj-c0xffffff/photo.jpg"
},
"medium": {
"url": "https://yt3.ggpht.com/--n5ELY2uT-U/AAAAAAAAAAI/AAAAAAAAAAA/d9JvaIEpstw/s240-c-k-no-rj-c0xffffff/photo.jpg"
},
"high": {
"url": "https://yt3.ggpht.com/--n5ELY2uT-U/AAAAAAAAAAI/AAAAAAAAAAA/d9JvaIEpstw/s240-c-k-no-rj-c0xffffff/photo.jpg"
}
},
"localized": {
"title": "thenewboston",
"description": "Tons of sweet computer related tutorials and some other awesome videos too!"
}
}
}
]
}
It works perfectly.
Now we have to get details of these channels:
https://www.youtube.com/c/eretteretlenek
https://www.youtube.com/c/annacavalli
Then we get:
GET https://www.googleapis.com/youtube/v3/channels?part=snippet&forUsername=annacavalli&key={YOUR_API_KEY}
Response
200
- SHOW HEADERS -
{
"kind": "youtube#channelListResponse",
"etag": "\"zekp1FB4kTkkM-rWc1qIAAt-BWc/TAiG4jjJ-NTZu7gPKn7WGmuaZb8\"",
"pageInfo": {
"totalResults": 0,
"resultsPerPage": 5
},
"items": [
]
}
This can be easily reproduced using the API explorer.
Simplest solution, using API only, is to just use Search:list method of YouTube Data API. From what I can tell (mind you, this is from my own research, official docs say nothing on this subject!), if you search using the custom URL component, with "channel" result type filter and "relevance" (default) sorting, first result should be what you're looking for.
So the following query gets 16 results, with the first one being the one you're looking for. Same goes for all other custom channel URLs I tested, so I think this is the most reliable way of doing this.
GET https://www.googleapis.com/youtube/v3/search?part=id%2Csnippet&q=annacavalli&type=channel&key={YOUR_API_KEY}
The other idea is just scraping YouTube page at the custom URL, where you can find ChannelID in one of the meta tags in HTML code. But that's ineffective, unreliable and AFAIK in violation of YouTube terms of use.
Edit: Well, it returns no results for smaller channels, so it's not reliable at all.
Workaround
Expanding off of #jkondratowicz answer, using the search.list in combination with channels.list you can most of the time resolve the channel from the custom url value.
The channel resource has a property customUrl so if we take the channels from the search.list results and get that extra detail about them from the channels.list it is possible to try and match up the custom url value with the customUrl property.
A working JavaScript method here, just replace the api key with your own. Though it is still not perfect, this tries the first 50 channels returned. More could be done with paging and pageTokens.
function getChannel(customValue, callback) {
const API_KEY = "your_api_key"
$.ajax({
dataType: "json",
type: "GET",
url: "https://www.googleapis.com/youtube/v3/search",
data: {
key: API_KEY,
part: "snippet",
q: customValue,
maxResults: 50,
order: 'relevance',
type: 'channel'
}
}).done(function (res) {
const channelIds = [];
for (let i=0; i<res.items.length; i++) {
channelIds.push(res.items[i].id.channelId);
}
$.ajax({
dataType: "json",
type: "GET",
url: "https://www.googleapis.com/youtube/v3/channels",
data: {
key: API_KEY,
part: "snippet",
id: channelIds.join(","),
maxResults: 50
}
}).done(function (res) {
if (res.items) {
for (let i=0; i<res.items.length; i++) {
const item = res.items[i];
if (item.snippet.hasOwnProperty("customUrl") && customValue.toLowerCase() === item.snippet.customUrl.toLowerCase()) {
callback(item);
}
}
}
}).fail(function (err) {
logger.err(err);
});
}).fail(function (err) {
logger.err(err);
});
}
A good example using it with https://www.youtube.com/c/creatoracademy.
getChannel('creatoracademy', function (channel) {
console.log(channel);
});
However, it is still unreliable as it depends on if the channel comes back in the original search.list query. It seems possible that if the custom channel url is too generic that the actual channel may not come back in the search.list results. Though this method is much more reliable then depending on the first entry of search.list to be the right one as search results don't always come back in the same order.
Issue
There has been at least three feature requests to Google in the past year requesting for an additional parameter for this custom url value but they were all denied as being infeasible. Apparently it is too difficult to implement. It was also mentioned as not being on their roadmap.
https://issuetracker.google.com/issues/174903934 (Dec 2020)
https://issuetracker.google.com/issues/165676622 (Aug 2020)
https://issuetracker.google.com/issues/161718177 (Jul 2020)
Resources
Google: Understand your channel URLs
Search: list | YouTube Data API
Channels: list | YouTube Data API
An alternative way is to use a parser (PHP Simple HTLM DOM parser, for the example below : PHP Simple HTML DOM Parser) :
<?php
$media_url = 'https://www.youtube.com/c/[Channel name]';
$dom = new simple_html_dom();
$html = $dom->load(curl_get($media_url));
if (null !== ($html->find('meta[itemprop=channelId]',0))) {
$channelId = $html->find('meta[itemprop=channelId]',0)->content;
}
?>
(Using Youtube api's "search" method has a quota cost of 100)
It is possible the get the channel ID by the video ID, that can help depends on the need fo your application.
Here are an example:
$queryParams = [
'id' => 'UcDjWCEvZLM'
];
$response = $service->videos->listVideos('snippet', $queryParams)->getItems();
$channelId = $response[0]->snippet['channelId'];
$channelTitle = $response[0]->snippet['channelTitle'];
I'm assuming that only channels with videos uploaded by the channel owner will be of interest. This is accidentally convenient since my method doesn't work with 0 video channels anyway.
Given a channel's url, my method will get the beautifulsoup HTML object of that channel's videos tab, and scrape the HTML to find the unique channel id. It'll then reconstruct everything and give back the channel url with the unique channel id.
your_channel_url = 'Enter your channel url here'
channel_url = your_channel_url.strip("https://").strip("featured")
https = "https://"
channel_vids_tab = https + channel_url + '/videos'
import requests
from bs4 import BeautifulSoup
source = requests.get(channel_vids_tab).text
soup = BeautifulSoup(source, "html.parser")
a = soup.find('body').find('link')['href']
channel_id = a.split('/')[-1]
print(a)
print(channel_id)
This method bypass the headache of one channel having different /user and /c url (for example /user/vechz and /c/vechz vs /c/coreyms and /user/schafer5 leading to the same page). Though you need to manually enter the url at first, it can be easily automated.
I'm also fairly confident that if a channel has 0 videos, this line of thinking can also apply for the playlist created BY the channel owner, and only needs a little tweaking. But if there's 0 videos or playlist created by the channel ... who knows
As #jkondratowicz noted, there is no way to reliably get this from the API as small channels do not return at the top of the search results.
So here is a JS example of how to get the channel id by extracting it from the HTML channel page (h/t #Feign'):
export const getChannelIdForCustomUrl = async (customUrl: string) => {
const page = await axios.get(`https://www.youtube.com/c/${customUrl}`)
const chanId = page.data.match(/channelId":"(.*?)"/)[1]
return chanId
}
Here is the .NET approach to convert custom Channel name/URL to a ChannelId using search API as mentioned in the accepted answer.
// https://www.youtube.com/c/TheQ_original/videos
// they call custom URL ?
// https://stackoverflow.com/questions/37267324/how-to-get-youtube-channel-details-using-youtube-data-api-if-channel-has-custom
// https://developers.google.com/youtube/v3/docs/channels#snippet.customUrl
// GET https://www.googleapis.com/youtube/v3/search?part=id%2Csnippet&q=annacavalli&type=channel&key={YOUR_API_KEY}
//
/// <summary>
/// Returns ChannelID from old "Custom Channel Name/URL"
/// Support the following URLs format
/// https://www.youtube.com/c/MakeYourOWNCreation
/// </summary>
/// <param name="ChannelName"></param>
/// <returns></returns>
// 20220701
public async Task<string> CustomChannelNameToChannelId(String ChannelName)
{
var YoutubeService = YouTubeService();
//
List<YouTubeInfo> VideoInfos = new List<YouTubeInfo>();
//
// -) Step1: Retrieve 1st page of channels info
var SearchListRequest = YoutubeService.Search.List("snippet");
SearchListRequest.Q = ChannelName;
SearchListRequest.Type = "channel";
//
SearchListRequest.MaxResults = 50;
// Call the search.list method to retrieve results matching the specified query term.
var SearchListResponse = await SearchListRequest.ExecuteAsync();
// According to the SO post the custom channel will be the first one
var searchResult = SearchListResponse.Items[0];
//
// Return Channel Information, we care to obtain ChannelID
return searchResult.Id.ChannelId;
}
Let's use the TODO example. In TodoList it(Line 81) has a fragment composed as
todos(status: $status, first: $limit) {
edges {
node {
id,
${Todo.getFragment('todo')},
},
},
....
}
And now if I add a loop
this.props.viewer.todos.edges.map(edge =>
console.log(edge.node.text)
);
to function renderTodos() at Line 30, it will output undefined
The interesting thing is, if we add text to to the fragment like below
todos(status: $status, first: $limit) {
edges {
node {
id,
text,
${Todo.getFragment('todo')},
},
},
....
}
It actually "declared" text twice(declare in Todo component as well) and the loop works perfectly.
My question is, why it is not possible to get the "properties" back from a composition even though they are returned by Graphql Server?
Thanks to hueyp#7485 at #relay channel.
He pointed out it is an intended behaviour by design.
https://facebook.github.io/relay/docs/thinking-in-relay.html#data-masking
So I think I have done everything in order to server images and am unable to see an image even though I have followed the instructions precisely on this site:
http://jwage.com/2010/07/27/storing-files-with-mongodb-gridfs/
When I do:
$dm = $this->get('doctrine.odm.mongodb.document_manager');
$image = $dm->createQueryBuilder('MyBundle:Image')
->field('id')->equals($imageID) //I have verified imageID to be correct
->getQuery()
->getSingleResult();
var_dump($image->getFile()->getBytes());
I get:
string(194992) ""
Which tells me that the image is there, just that I am unable to get the bytes using the following as that comes null:
echo $image->getFile()->getBytes();
Please note that I am able to see the image using RockMongo etc so I know image is stored properly plus I can see it in the database stored as chucks. So what is wrong???
UPDATE: I HAVE ATTACHED THE VAR_DUMP OF THE file object.
object(Doctrine\MongoDB\GridFSFile)#427 (4) {
["mongoGridFSFile":"Doctrine\MongoDB\GridFSFile":private]=>
object(MongoGridFSFile)#430 (3) {
["file"]=>
array(7) {
["_id"]=>
object(MongoId)#431 (1) {
["$id"]=>
string(24) "503302b3c7e24c401a000004"
}
["name"]=>
string(6) "Image2"
["filename"]=>
string(14) "/tmp/phpBhZDAy"
["uploadDate"]=>
object(MongoDate)#432 (2) {
["sec"]=>
int(1345520307)
["usec"]=>
int(983000)
}
["length"]=>
float(194992)
["chunkSize"]=>
float(262144)
["md5"]=>
string(32) "5bbc9ede74f50f93a3f7d1f7babe3170"
}
["gridfs":protected]=>
object(MongoGridFS)#437 (5) {
["w"]=>
int(1)
["wtimeout"]=>
int(10000)
["chunks"]=>
object(MongoCollection)#438 (2) {
["w"]=>
int(1)
["wtimeout"]=>
int(10000)
}
["filesName":protected]=>
string(12) "assets.files"
["chunksName":protected]=>
string(13) "assets.chunks"
}
["flags"]=>
int(0)
}
["filename":"Doctrine\MongoDB\GridFSFile":private]=>
NULL
["bytes":"Doctrine\MongoDB\GridFSFile":private]=>
NULL
["isDirty":"Doctrine\MongoDB\GridFSFile":private]=>
bool(false)
}
null