I'm trying to compile channel statistics for a list of channels and for the first page it works, but when I invoke the next page using the token it gives me an error that the URI string is too long.
I'm using .NET core 2.2 and Google.Apis.YouTube.v2 V1.39.0.1572. And the code I use is really simple:
var youtubeService = new YouTubeService(new BaseClientService.Initializer()
{
ApiKey = Startup.Configuration["YTConfigurations:ApiKey"],
ApplicationName = this.GetType().ToString()
});
ChannelsResource.ListRequest channelsListRequest = youtubeService.Channels.List("snippet,statistics,brandingSettings,topicDetails");
channelsListRequest.Id = string.Join(",", channelEntities.Select(v => v.YTChannelId));
channelsListRequest.MaxResults = 50;
do
{
pageCounter++;
channelListResponse = channelsListRequest.Execute();//here is the error after page 1
foreach (var listResult in channelListResponse.Items)
{
channelEntity = channelEntities.FirstOrDefault(v => v.YTChannelId == listResult.Id);
channelEntity = Mapper.Map(listResult, channelEntity);
_repository.UpdateChannel(channelEntity);
}
if (channelListResponse.NextPageToken != null)
{
channelsListRequest.PageToken = channelListResponse.NextPageToken;
}
} while (channelListResponse.Items.Count == 50 && channelListResponse.NextPageToken != null);
When I execute this is what I get:
System.UriFormatException: Invalid URI: The Uri string is too long.
at System.UriHelper.EscapeString(String input, Int32 start, Int32 end, Char[] dest, Int32& destPos, Boolean isUriString, Char force1, Char force2, Char rsvd)
at System.Uri.EscapeDataString(String stringToEscape)
at Google.Apis.Requests.RequestBuilder.<>c.<BuildUri>b__25_0(KeyValuePair`2 x) in C:\Apiary\2019-05-01.11-08-18\Src\Support\Google.Apis.Core\Requests\RequestBuilder.cs:line 108
at System.Linq.Enumerable.SelectListIterator`2.ToArray()
at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
at Google.Apis.Requests.RequestBuilder.BuildUri() in C:\Apiary\2019-05-01.11-08-18\Src\Support\Google.Apis.Core\Requests\RequestBuilder.cs:line 107
at Google.Apis.Requests.RequestBuilder.CreateRequest() in C:\Apiary\2019-05-01.11-08-18\Src\Support\Google.Apis.Core\Requests\RequestBuilder.cs:line 332
at Google.Apis.Requests.ClientServiceRequest`1.CreateRequest(Nullable`1 overrideGZipEnabled) in C:\Apiary\2019-05-01.11-08-18\Src\Support\Google.Apis\Requests\ClientServiceRequest.cs:line 257
at Google.Apis.Requests.ClientServiceRequest`1.ExecuteUnparsedAsync(CancellationToken cancellationToken) in C:\Apiary\2019-05-01.11-08-18\Src\Support\Google.Apis\Requests\ClientServiceRequest.cs:line 229
at Google.Apis.Requests.ClientServiceRequest`1.Execute() in C:\Apiary\2019-05-01.11-08-18\Src\Support\Google.Apis\Requests\ClientServiceRequest.cs:line 167
at ChannelHarvester.Controllers.ChannelHarvester.extractStats(ICollection`1 channelEntities) in C:\Guayaba Projects\YT_ChannelHarvester\ChannelHarvester\Controllers\ChannelHarvester.cs:line 106
Am I doing something wrong? Please let me know if there is something I can fix on my end.
Thanks!
Well, I'm really sorry to have bothered you guys. I found what it was, in the following code I thought I was joining only 50 IDs, but apparently in some occasions I was sending a LOT more.
channelsListRequest.Id = string.Join(",", channelEntities.Select(v => v.YTChannelId));
So I guess we can close this
Related
I am trying to get on an AzureDevOps project the number of Warning for a build task. But I still only get the first 10 warnings. How do you do it all? I especially need the number of warning, not necessarily the detail of the warning
var credential = new VssBasicCredential(string.Empty, myPat);
var connection = new VssConnection(new Uri(myCollection), credential);
var buildClient = connection.GetClient<BuildHttpClient>();
var timeline = buildClient.GetBuildTimelineAsync(myProject, myBuildId).Result;
var vsTask= timeline.Records.FirstOrDefault(p => p?.Task?.Name == "VSBuild");
// always 10 utmost : warning and issues !!
var warning = vsTask.WarningCount;
var issues = vsTask.Issues;
This is really strange, but it looks that you cannot go above what is shown here
There is workaround (but it is a bit ugly). You can load log page and count warnings parsing the page.
static void Main(string[] args)
{
var credential = new VssBasicCredential(string.Empty, "PAT");
var connection = new VssConnection(new Uri("https://dev.azure.com/your-organization/"), credential);
var buildClient = connection.GetClient<BuildHttpClient>();
var timeline = buildClient.GetBuildTimelineAsync("yoyr project", 377).Result;
var vsTask = timeline.Records.FirstOrDefault(p => p?.Task?.Name == "VSBuild");
// always 10 utmost : warning and issues !!
var warning = vsTask.WarningCount;
var issues = vsTask.Issues;
Console.WriteLine(warning);
HttpClient client = new HttpClient();
var response = client.GetAsync(vsTask.Log.Url);
var pageContents = response.Result.Content.ReadAsStringAsync().Result;
var realNumberOfWarnings = AllIndexesOf(pageContents, "##[warning]");
Console.WriteLine(realNumberOfWarnings.Count);
Console.ReadLine();
}
public static List<int> AllIndexesOf(string str, string value)
{
if (String.IsNullOrEmpty(value))
throw new ArgumentException("the string to find may not be empty", "value");
List<int> indexes = new List<int>();
for (int index = 0; ; index += value.Length)
{
index = str.IndexOf(value, index);
if (index == -1)
return indexes;
indexes.Add(index);
}
}
I hope it will help.
Please check what Tingluo Huang [MSFT] wrote here:
We should only count and store 10 errors/warning (i guess we having
some counting bug, so we count to 11), those issues get stored in our
backend DB for driving the build summary page render. We don't want to
store too many errors/warning since most of the time the first few
errors/warnings are key to the problem, and the rest are just noise.
I am trying to do my batch insertion to an existing database but I got the following exception:
Exception in thread "GC-Monitor" java.lang.OutOfMemoryError: Java heap
space at java.util.Arrays.copyOf(Arrays.java:2245) at
java.util.Arrays.copyOf(Arrays.java:2219) at
java.util.ArrayList.grow(ArrayList.java:242) at
java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:216) at
java.util.ArrayList.ensureCapacityInternal(ArrayList.java:208) at
java.util.ArrayList.add(ArrayList.java:440) at
java.util.Formatter.parse(Formatter.java:2525) at
java.util.Formatter.format(Formatter.java:2469) at
java.util.Formatter.format(Formatter.java:2423) at
java.lang.String.format(String.java:2792) at
org.neo4j.kernel.impl.cache.MeasureDoNothing.run(MeasureDoNothing.java:64)
Fail: Transaction was marked as successful, but unable to commit
transaction so rolled back.
Here is the structure of my insertion code :
public void parseExecutionRecordFile(Node episodeVersionNode, String filePath, Integer insertionBatchSize) throws Exception {
Gson gson = new Gson();
BufferedReader reader = new BufferedReader(new FileReader(filePath));
String aDataRow = "";
List<ExecutionRecord> executionRecords = new LinkedList<>();
Integer numberOfProcessedExecutionRecords = 0;
Integer insertionCounter = 0;
ExecutionRecord lastProcessedExecutionRecord = null;
Node lastProcessedExecutionRecordNode = null;
Long start = System.nanoTime();
while((aDataRow = reader.readLine()) != null) {
JsonReader jsonReader = new JsonReader(new StringReader(aDataRow));
jsonReader.setLenient(true);
ExecutionRecord executionRecord = gson.fromJson(jsonReader, ExecutionRecord.class);
executionRecords.add(executionRecord);
insertionCounter++;
if(insertionCounter == insertionBatchSize || executionRecord.getType() == ExecutionRecord.Type.END_MESSAGE) {
lastProcessedExecutionRecordNode = appendEpisodeData(episodeVersionNode, lastProcessedExecutionRecordNode, executionRecords, lastProcessedExecutionRecord == null ? null : lastProcessedExecutionRecord.getTraceSequenceNumber());
executionRecords = new LinkedList<>();
lastProcessedExecutionRecord = executionRecord;
numberOfProcessedExecutionRecords += insertionCounter;
insertionCounter = 0;
}
}
}
public Node appendEpisodeData(Node episodeVersionNode, Node previousExecutionRecordNode, List<ExecutionRecord> executionRecordList, Integer traceCounter) {
Iterator<ExecutionRecord> executionRecordIterator = executionRecordList.iterator();
Node previousTraceNode = null;
Node currentTraceNode = null;
Node currentExecutionRecordNode = null;
try (Transaction tx = dbInstance.beginTx()) {
// some graph insertion
tx.success();
return currentExecutionRecordNode;
}
}
So basically, I read json object from a file (ca. 20,000 objects) and insert it to neo4j every 10,000 records. If I have only 10,000 JSON objects in the file, then it works fine. But when I have 20,000, it throws the exception.
Thanks in advance and any help would be really appreciated!
If with 10000 objects works, just try to at least duplicate the heap memory.
Take a look at the following site: http://neo4j.com/docs/stable/server-performance.html
The wrapper.java.maxmemory option could resolve your problem.
As you also insert several k properties all that tx state will be held in memory. So I think 10k batch size is just fine for that amount of heap.
You also don't close your JSON reader so it might linger around with the StringReader inside.
You should also use an ArrayList initialized at your batch-size and use list.clear() instead of recreation/reassignment.
Here's a code snippet I found from the gdata sample codes. I tried incorporating this, but to no effect.
public ContactsExample(ContactsExampleParameters parameters)
throws MalformedURLException, AuthenticationException {
projection = parameters.getProjection();
String url = parameters.getBaseUrl() + (parameters.isGroupFeed() ? "groups/" : "contacts/") + parameters.getUserName() + "/" + projection;
feedUrl = new URL(url);
service = new ContactsService("MYAPP");
String userName = parameters.getUserName();
String password = parameters.getPassword();
if (userName == null || password == null) {
return;
}
service.setUserCredentials(userName, password);
}
I have these doubts:
1. Is the name specified as 'MYAPP' any random name, or does it have any significance?
2. Which is the userName and password supposed to be used here?
I am trying to extract header and body information from email, the following code retrieves the header and body in their raw form. I have an email object that contains the fields from, subject, date, and body. I would like to extract these values from the email and assign them to the email object. How do I get around it? I have tried several ways like getting the header info and using a streamReader.ReadLine() to get a line but I got illegal path exceptions. I know I can use a library but I need to achieve it this way.
What I mean is this, IMAP command returns header information. And I want to extract subject value, date value, sender e-amil, etc. and assign them to my email objects corresponding values like
emailObject.subject = "subjectValue"
public class Imap
{
static void Main(string[] args)
{
try
{
path = Environment.CurrentDirectory + "\\emailresponse.txt";
if (System.IO.File.Exists(path))
System.IO.File.Delete(path);
sw = new System.IO.StreamWriter(System.IO.File.Create(path));
tcpc = new System.Net.Sockets.TcpClient("imap.gmail.com", 993);
ssl = new System.Net.Security.SslStream(tcpc.GetStream());
ssl.AuthenticateAsClient("imap.gmail.com");
receiveResponse("");
Console.WriteLine("username : ");
username = Console.ReadLine();
Console.WriteLine("password : ");
password = Console.ReadLine();
receiveResponse("$ LOGIN " + username + " " + password + " \r\n");
Console.Clear();
receiveResponse("$ LIST " + "\"\"" + " \"*\"" + "\r\n");
receiveResponse("$ SELECT INBOX\r\n");
receiveResponse("$ STATUS INBOX (MESSAGES)\r\n");
Console.WriteLine("enter the email number to fetch :");
int number = int.Parse(Console.ReadLine());
Console.WriteLine("*************Header************");
Console.WriteLine("");
// receiveResponse("$ FETCH " + number + " body[header]\r\n");
// BODY.PEEK[HEADER.FIELDS (SUBJECT)]
// StringBuilder sb = receiveResponse("$ FETCH " + number + " BODY.PEEK[HEADER.FIELDS (From Subject Date)]\r\n");
StringBuilder sb= receiveResponse("$ FETCH " + number + " body.peek[header]\r\n");
Console.WriteLine(sb);
Console.WriteLine("");
Console.WriteLine("Body");
sb = new StringBuilder();
sb=receiveResponse("$ FETCH " + number + " body[text]\r\n");
System.Text.ASCIIEncoding enc = new System.Text.ASCIIEncoding();
byte[] serverbuff = new Byte[1024];
int count = 0;
string retval = enc.GetString(serverbuff, 0, count);
Console.WriteLine(sb.ToString());
receiveResponse("$ LOGOUT\r\n");
}
catch (Exception ex)
{
Console.WriteLine("error: " + ex.Message);
}
finally
{
if (sw != null)
{
sw.Close();
sw.Dispose();
}
if (ssl != null)
{
ssl.Close();
ssl.Dispose();
}
if (tcpc != null)
{
tcpc.Close();
}
}
Console.ReadKey();
}
static StringBuilder receiveResponse(string command)
{
sb = new StringBuilder();
try
{
if (command != "")
{
if (tcpc.Connected)
{
dummy = Encoding.ASCII.GetBytes(command);
ssl.Write(dummy, 0, dummy.Length);
}
else
{
throw new ApplicationException("TCP CONNECTION DISCONNECTED");
}
}
ssl.Flush();
buffer = new byte[2048];
bytes = ssl.Read(buffer, 0, 2048);
sb.Append(Encoding.ASCII.GetString(buffer));
// Console.WriteLine(sb.ToString());
sw.WriteLine(sb.ToString());
// sb = new StringBuilder();
return sb;
}
catch (Exception ex)
{
throw new ApplicationException(ex.Message);
}
}
You said you do not want to use an IMAP library. This means that you will have to implement your own. You should start by reading RFC 3501 because there is no chance you could get the protocol right without reading the docs carefuly. In particular, you're issuing a STATUS command on the currently selected mailbox, which is explicitly forbidden by the protocol specification. The rest of the code supports the assumption that you have not read the RFC yet.
I'm comunicationg with a email gateway. That gateway has an specific ip and port.
The requests the gateway are JSON formated and the gateway normally responds first whith an proceeding state and then with a confirmation or error state, represented also in JSON.
The code to make the requests and receive the response is:
using System;
using System.IO;
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Collections.Generic;
using System.Threading;
using Microsoft.Win32;
public class TcpClientSample
{
public static void SendMessage(TcpClient client, string msg)
{
Console.WriteLine("REQUEST:" + msg);
NetworkStream stream = client.GetStream();
byte[] myWriteBuffer = Encoding.ASCII.GetBytes(msg);
stream.Write(myWriteBuffer, 0, myWriteBuffer.Length);
byte[] myWriteBuffer2 = Encoding.ASCII.GetBytes("\r\n");
stream.Write(myWriteBuffer2, 0, myWriteBuffer2.Length);
string gResponse = "";
BinaryReader r = new BinaryReader(stream);
int receivedMessages = 0;
while (true)
{
while (true)
{
char currentChar = r.ReadChar();
if (currentChar == '\n')
break;
else
gResponse = gResponse + currentChar;
}
if (gResponse != "")
{
Console.WriteLine("RESPONSE:" + gResponse);
receivedMessages = receivedMessages + 1;
}
if (receivedMessages == 2)
{
break;
}
}
}
public static void Main()
{
List<string> messages = new List<string>();
for (int i = 0; i < 1; i++)
{
String msg = "{ \"user\" : \"James\", \"email\" : \"james#domain.pt\" }";
messages.Add(msg);
}
TcpClient client = new TcpClient();
client.Connect("someIp", somePort);
int sentMessages = 0;
int receivedMessages = 0;
foreach (string msg in messages)
{
Thread newThread = new Thread(() =>
{
sentMessages = sentMessages + 1;
Console.WriteLine("SENT MESSAGES: " + sentMessages);
SendMessage(client, msg);
receivedMessages = receivedMessages + 1;
Console.WriteLine("RECEIVED MESSAGES: " + receivedMessages);
});
newThread.Start();
}
Console.ReadLine();
}
}
If I send few emails (up to 10) the network stream is OK.
But if I send thousands of emails I get messed chars lie
:{iyo"asn ooyes" "ncd" 0,"s_d:"4379" nme" 92729,"er_u" ,"ed_t_i" 2#" p cin_d:"921891010-11:11.725,"s" 4663175D0105E6912ADAAFFF6FDA393367" rpy:"rcein"
Why is this?
Don't worry I'm not a spammer :D
When you write a message to a TCP socket, it'll respond with the sent data. When the buffer is full, I expect it's 0, but you advance your send buffer anyway. You should advance it by the return value :)
Edit: it looks like you're using a stream abstraction which writes the internal buffer. The situation is the same. You are saying "the message has been completely sent" when the internal buffer state is not saying this, i.e. position does not equal limit. You need to keep sending until the remaining amount of buffer is 0 before moving on.
I solved this issue by having a single method just to read from the stream like this:
private TcpClient client;
private NetworkStream stream;
public void ListenFromGateway()
{
...
while (true)
{
byte[] bytes = new byte[client.ReceiveBufferSize];
//BLOCKS UNTIL AT LEAST ONE BYTE IS READ
stream.Read(bytes, 0, (int)client.ReceiveBufferSize);
//RETURNS THE DATA RECEIVED
string returndata = Encoding.UTF8.GetString(bytes);
//REMOVE THE EXCEDING CHARACTERS STARTING ON \r
string returndata = returndata.Remove(returndata.IndexOf('\r'));
...
}
Thanks for the help