OData Service not returning complete response - odata

Question

I am reading Sharepoint list data (>20000 entries) using Odata RESTful service as detailed here -http://blogs.msdn.com/b/ericwhite/archive/2010/12/09/getting-started-using-the-odata-rest-api-to-query-a-sharepoint-list.aspx
I am able to read data but I get only the first 1000 records. I also checked that List View Throttling is set to 5000 on sharepoint server. Kindly advise.
Update:
#Turker: Your answer is spot on!! Thank you very much. I was able to get the first 2000 records in first iteration. However, I am getting the same records in each iteration of while loop. My code is as follows-
...initial code...
int skipCount =0;
while (((QueryOperationResponse)query).GetContinuation() != null)
{
//query for the next partial set of customers
query = dc.Execute<CATrackingItem>(
((QueryOperationResponse)query).GetContinuation().NextLinkUri
);
//Add the next set of customers to the full list
caList.AddRange(query.ToList());
var results = from d in caList.Skip(skipCount)
select new
{
Actionable = Actionable,
}; Created = d.Created,
foreach (var res in results)
{
structListColumns.Actionable = res.Actionable;
structListColumns.Created= res.Created;
}
skipCount = caList.Count;
}//Close of while loop

Answer 1

Do you see a <link rel="next"> element at the end of the feed?
For example, if you look at
http://services.odata.org/Northwind/Northwind.svc/Customers/
you will see
<link rel="next" href="http://services.odata.org/Northwind/Northwind.svc/Customers/?$skiptoken='ERNSH'" />
at the end of the feed which means the service is implementing server side paging and you need to send the
http://services.odata.org/Northwind/Northwind.svc/Customers/?$skiptoken='ERNSH'
query to get the next set of results.

Answer 2

I don't see anything particularly wrong with your code. You can try to dump the URLs beign requested (either from the code, or using something like fiddler) to see if the client really sends the same queries (and thus getting same responses).
In any case, here is a sample code which does work (using the sample service):
DataServiceContext ctx = new DataServiceContext(new Uri("http://services.odata.org/Northwind/Northwind.svc"));
QueryOperationResponse<Customer> response = (QueryOperationResponse<Customer>)ctx.CreateQuery<Customer>("Customers").Execute();
do
{
foreach (Customer c in response)
{
Console.WriteLine(c.CustomerID);
}
DataServiceQueryContinuation<Customer> continuation = response.GetContinuation();
if (continuation != null)
{
response = ctx.Execute(continuation);
}
else
{
response = null;
}
} while (response != null);

Answer 3

I had the same problem, and wanted it to be a generic solution.
So I've extended DataServiceContext with a GetAlltems methode.
public static List<T> GetAlltems<T>(this DataServiceContext context)
{
return context.GetAlltems<T>(null);
}
public static List<T> GetAlltems<T>(this DataServiceContext context, IQueryable<T> queryable)
{
List<T> allItems = new List<T>();
DataServiceQueryContinuation<T> token = null;
EntitySetAttribute attr = (EntitySetAttribute)typeof(T).GetCustomAttributes(typeof(EntitySetAttribute), false).First();
// Execute the query for all customers and get the response object.
DataServiceQuery<T> query = null;
if (queryable == null)
{
query = context.CreateQuery<T>(attr.EntitySet);
}
else
{
query = (DataServiceQuery<T>) queryable;
}
QueryOperationResponse<T> response = query.Execute() as QueryOperationResponse<T>;
// With a paged response from the service, use a do...while loop
// to enumerate the results before getting the next link.
do
{
// If nextLink is not null, then there is a new page to load.
if (token != null)
{
// Load the new page from the next link URI.
response = context.Execute<T>(token);
}
allItems.AddRange(response);
}
// Get the next link, and continue while there is a next link.
while ((token = response.GetContinuation()) != null);
return allItems;
}