I just want to fetch all my liked videos ~25k items. as far as my research goes this is not possible via the YouTube v3 API.
I have already found multiple issues (issue, issue) on the same problem, though some claim to have fixed it, but it only works for them as they don't have < 5000 items in their liked video list.
playlistItems list API endpoint with playlist id set to "liked videos" (LL) has a limit of 5000.
videos list API endpoint has a limit of 1000.
Unfortunately those endpoints don't provide me with parameters that I could use to paginate the requests myself (e.g. give me all the liked videos between date x and y), so I'm forced to take the provided order (which I can't get past 5k entries).
Is there any possibility I can fetch all my likes via the API?
more thoughts to the reply from #Yarin_007
if there are deleted videos in the timeline they appear as "Liked https://...url" , the script doesnt like that format and fails as the underlying elements dont have the same structure as existing videos
can be easily fixed with a try catch
function collector(all_cards) {
var liked_videos = {};
all_cards.forEach(card => {
try {
// ignore Dislikes
if (card.innerText.split("\n")[1].startsWith("Liked")) {
....
}
}
catch {
console.log("error, prolly deleted video")
}
})
return liked_videos;
}
to scroll down to the bottom of the page ive used this simple script, no need to spin up something big
var millisecondsToWait = 1000;
setInterval(function() {
window.scrollTo(0, document.body.scrollHeight);
console.log("scrolling")
}, millisecondsToWait);
when more ppl want to retrive this kind of data, one could think about building a proper script that is more convenient to use. If you check the network requests you can find the desired data in the response of requests called batchexecute. One could copy the authentification of one of them provide them to a script that queries those endpoints and prepares the data like the other script i currently manually inject.
Hmm. perhaps Google Takeout?
I have verified the youtube data contains a csv called "liked videos.csv". The header is Video Id,Time Added, and the rows are
dQw4w9WgXcQ,2022-12-18 23:42:19 UTC
prvXCuEA1lw,2022-12-24 13:22:13 UTC
for example.
So you would need to retrieve video metadata per video ID. Not too bad though.
Note: the export could take a while, especially with 25k videos. (select only YouTube data)
I also had an idea that involves scraping the actual liked videos page (which would save you 25k HTTP Requests). But I'm unsure if it breaks with more than 5000 songs. (also, emulating the POST requests on that page may prove quite difficult, albeit not impossible. (they fetch /browse?key=..., and have some kind of obfuscated / encrypted base64 strings in the request-body, among other parameters)
EDIT:
Look. There's probably a normal way to get a complete dump of all you google data. (i mean, other than takeout. Email them? idk.)
anyway, the following is the other idea...
Follow this deep link to your liked videos history.
Scroll to the bottom... maybe with selenium, maybe with autoit, maybe put something on the "end" key of your keyboard until you reach your first liked video.
Hit f12 and run this in the developer console
// https://www.youtube.com/watch?v=eZPXmCIQW5M
// https://myactivity.google.com/page?utm_source=my-activity&hl=en&page=youtube_likes
// go over all "cards" in the activity webpage. (after scrolling down to the absolute bottom of it)
// create a dictionary - the key is the Video ID, the value is a list of the video's properties
function collector(all_cards) {
var liked_videos = {};
all_cards.forEach(card => {
// ignore Dislikes
if (card.innerText.split("\n")[1].startsWith("Liked")) {
// horrible parsing. your mileage may vary. I Tried to avoid using any gibberish class names.
let a_links = card.querySelectorAll("a")
let details = a_links[0];
let url = details.href.split("?v=")[1]
let video_length = a_links[3].innerText;
let time = a_links[2].parentElement.innerText.split(" • ")[0];
let title = details.innerText;
let date = card.closest("[data-date]").getAttribute("data-date")
liked_videos[url] = [title,video_length, date, time];
// console.log(title, video_length, date, time, url);
}
})
return liked_videos;
}
// https://stackoverflow.com/questions/57709550/how-to-download-text-from-javascript-variable-on-all-browsers
function download(filename, text, type = "text/plain") {
// Create an invisible A element
const a = document.createElement("a");
a.style.display = "none";
document.body.appendChild(a);
// Set the HREF to a Blob representation of the data to be downloaded
a.href = window.URL.createObjectURL(
new Blob([text], { type })
);
// Use download attribute to set set desired file name
a.setAttribute("download", filename);
// Trigger the download by simulating click
a.click();
// Cleanup
window.URL.revokeObjectURL(a.href);
document.body.removeChild(a);
}
function main() {
// gather relevant elements
var all_cards = document.querySelectorAll("div[aria-label='Card showing an activity from YouTube']")
var liked_videos = collector(all_cards)
// download json
download("liked_videos.json", JSON.stringify(liked_videos))
}
main()
Basically it gathers all the liked videos' details and creates a key: video_ID - Value: [title,video_length, date, time] object for each liked video.
It then automatically downloads the json as a file.
The documentation does not have any examples on how to add a subcollection to a document. I know how to add document to a collection and how to add data to a document, but how do I add a collection (subcollection) to a document?
Shouldn't there be some method like this:
dbRef.document("example").addCollection("subCollection")
Edit 13 Jan 2021:
According to the updated documentation regarding array membership, now it is possible to filter data based on array values using whereArrayContains() method. A simple example would be:
CollectionReference citiesRef = db.collection("cities");
citiesRef.whereArrayContains("regions", "west_coast");
This query returns every city document where the regions field is an array that contains west_coast. If the array has multiple instances of the value you query on, the document is included in the results only once.
Assuming we have a chat application that has a database structure that looks similar to this:
To write a subCollection in a document, please use the following code:
DocumentReference messageRef = db
.collection("rooms").document("roomA")
.collection("messages").document("message1");
Creating a messages collection and calling addDocument() 1000 times will be expensive for sure, but this is how Firestore works. You can switch to Firebase Realtime Database if you want where the number of writes doesn't matter. But regarding Supported Data Types in Firestore, in fact, you can use an array because it is supported. In Firebase Realtime database you could also use an array, but this is an anti-pattern. One of the many reasons Firebase recommends against using arrays is that it makes the security rules impossible to write.
Cloud Firestore can store arrays, but it does not support querying array members or updating single array elements. However, you can still model this kind of data by leveraging the other capabilities of the Cloud Firestore. Here is the documentation where it is very well explained.
You also cannot create a subcollection with 1000 messages, add all of them to the database, and expect it to be considered a single record. It will be considered one write operation for every message, in total 1000 operations. The picture above does not show how to retrieve data, it shows a database structure in which you have something like this:
collection -> document -> subCollection -> document
Here's a variation where the subcollection is storing ID values at the collection level, rather than within a document where the subcollection is a field there with additional data.
This is useful for connecting a 1-to-Many ID mapping w/out having to drill through an additional document:
function fireAddStudentToClassroom(studentUserId, classroomId) {
var db = firebase.firestore();
var studentsClassroomRef =
db.collection('student_class').doc(classroomId)
.collection('students');
studentsClassroomRef
.doc(studentUserId)
.set({})
.then(function () {
console.log('Document Added ');
})
.catch(function (error) {
console.error('Error adding document: ', error);
});
}
Thanks to #Alex's answer
This answer a bit off from the original question here, where it explicitly asks for adding a collection to a document. However, after searching for a solution for this scenario and not finding any mention in docs or on SO, this post seems like a reasonable place to share the findings
Here's my code:
firebase.firestore().collection($scope.longLanguage + 'Words').doc($scope.word).set(wordData)
.then(function() {
console.log("Collection added to Firestore!");
var promises = [];
promises.push(firebase.firestore().collection($scope.longLanguage + 'Words').doc($scope.word).collection('AudioSources').doc($scope.accentDialect).set(accentDialectObject));
promises.push(firebase.firestore().collection($scope.longLanguage + 'Words').doc($scope.word).collection('FunFacts').doc($scope.longLanguage).set(funFactObject));
promises.push(firebase.firestore().collection($scope.longLanguage + 'Words').doc($scope.word).collection('Translations').doc($scope.translationLongLanguage).set(translationObject));
Promise.all(promises).then(function() {
console.log("All subcollections were added!");
})
.catch(function(error){
console.log("Error adding subcollections to Firestore: " + error);
});
})
.catch(function(error){
console.log("Error adding document to Firestore: " + error);
});
This makes a collection EnglishWords, which has a document of. The document of has three subcollections: AudioSources (recordings of the word in American and British accents), FunFacts, and Translations. The subcollection Translations has one document: Spanish. The Spanish document has three key-value pairs, telling you that 'de' is the Spanish translation of 'of'.
The first line of the code creates the collection EnglishWords. We wait for the promise to resolve with .then, and then we create the three subcollections. Promise.all tells us when all three subcollections are set.
IMHO, I use arrays in Firestore when the entire array is uploaded and downloaded together, i.e., I don't need to access individual elements. For example, an array of the letters of the word 'of' would be ['o', 'f']. The user can ask, "How do I spell 'of'?" The user isn't going to ask, "What's the second letter in 'of'?"
I use collections when I need to access individual elements, a.k.a. documents. With the older Firebase Realtime Database, I had to download arrays and then iterate through the arrays with forEach to get the element I wanted. This was a lot of code, and with a deep data structure and/or large arrays I was downloading tons of data that I didn't need, and slowing my app running forEach loops on large arrays. Firestore puts the iterators in the database, on their end, so that I can request a single element and it sends me just that element, saving me bandwidth and making my app run faster. This might not matter for a web app, if your computer has a broadband connection, but for mobile apps with poor data connections and slow devices this is important.
Here are two pictures of my Firestore:
From the docs:
You do not need to "create" or "delete" collections. After you create the first document in a collection, the collection exists. If you delete all of the documents in a collection, it no longer exists.
Here i faced the same issue and solve with the answere of #Thomas David Kehoe
db.collection("First collection Name").doc("Id of the document").collection("Nested collection Name").add({
//your data
}).then((data) => {
console.log(data.id);
console.log("Document has added")
}).catch((err) => {
console.log(err)
})
too late for an answer but here is what worked for me,
mFirebaseDatabaseReference?.collection("conversations")?.add(Conversation("User1"))
?.addOnSuccessListener { documentReference ->
Log.d(TAG, "DocumentSnapshot written with ID: " + documentReference.id)
mFirebaseDatabaseReference?.collection("conversations")?.document(documentReference.id)?.collection("messages")?.add(Message(edtMessage?.text.toString()))
}?.addOnFailureListener { e ->
Log.w(TAG, "Error adding document", e)
}
add success listener for adding document and use firebase generated ID for a path.
Use this ID for the complete path for a new collection you want to add.
I.E. - dbReference.collection('yourCollectionName').document(firebaseGeneratedID).collection('yourCollectionName').add(yourDocumentPOJO/Object)
Okay so I recently faced a similar problem given the recent update in the firebase/firestore documentation.
And here is a solution that worked for me
const sendMessage = async () => {
await setDoc(doc(db, COLLECTION_NAME, projectId, SUB_COLLECTION_NAME, nanoid()), {
text:'this is a sample text',
createdAt: serverTimestamp(),
name: currentUser?.firstName + ' ' + currentUser?.lastName,
photoUrl: currentUser?.photoUrl,
userId: currentUser?.id,
});
}
You can find a similar example in the docs
https://firebase.google.com/docs/firestore/data-model#web-version-9_3
chat room
If you wish to listen for live update you can use a similar method as follows
const messagesRef = collection(db, COLLECTION_NAME, projectId, SUB_COLLECTION_NAME)
const liveUpdate = async () => {
const queryObj = query(messagesRef, orderBy("createdAt"), limit(25));
onSnapshot(queryObj, (querySnapshot) => {
const msgArr: any = [];
querySnapshot.forEach((doc) => {
msgArr.push({ id: doc.id, ...doc.data() })
});
console.log(msgArr);
});
}
There is no separate method to add sub-collection into the document.
You can just call the collection method itself.
If the collection exists it will reference that otherwise create a new one.
dbRef.document("example").collection("subCollection")
I am trying to figure out how to delete a local stored set of Entities which are excluded from what I receive from a server api. In this case we have let's say 10 entities stored locally in realm, and the server sends me 8 entities. 6 of them corresponds to entities I already have locally, 2 are new. Locally I have 4 entities which are "outdated" because the server doesn't provide them.
Of course I could delete everything and save the new entities, but I was trying to make things more efficient.
However I don't understand, once I created a swift Set which represents the difference between the local and remote situation, how can I delete the local objects, because realm tells me i can only delete objects which belong to the realm.
I've tried working with resultset, but I can't figure out a way to filter object with a "IN" clause to which is provided a set of objects, even if I recall doing something similar in the past.
let realm = try! Realm()
try! realm.write{
let oldEvents = Array(self.events!)
var newEvents = [Event]()
for event in events{
if let e = Mapper<Event>().map(JSONObject: event){
newEvents.append(e)
}
}
//here I create a set which contains the difference between remote and local entitites, and create an Array with them
let difference = Array(Set(newEvents).subtracting(Set(oldEvents)))
//this is obviously wrong
let objToDelete = realm.objects(Event.self).filter("event in %#", difference)
realm.delete(objToDelete)
//realm.delete(difference)->this gives the error that you can delete only objects that belong to the realm
//here I would like to create or update the new events
for newEvent in newEvents{
realm.create(Event.self, value: event, update: true)
}
}
Any hint would be greatly appreciated on how to deal with this situation. Thanks!
I'm having a hard time working with Firebase query results. With the following code:
ref.queryOrderedByChild("gender")
.queryEqualToValue("female")
.observeEventType(.ChildAdded, withBlock: { snapshot in
print("result: \(snapshot) ")
})
The "result" is printed 3 times. I would expect a single array of all of the results (similar to a query on Parse) versus this being printed 3 separate times.
The end goal here is to append all of the results to an Array. However, I don't know how to do that, since I can't see any way of knowing how many elements will come back from the server.
I assume there must be something simple I am missing here.
It appears it was something simple I was missing. Changing the event type from .ChildAdded to .Value resolves the issue. Hopefully this will help someone else...
var resultArray:[AnyObject] = []
ref.queryOrderedByChild("gender")
.queryEqualToValue("female")
.observeEventType(.Value, withBlock: { snapshot in
for item in snapshot.children{
resultArray.append(item)
}
print("Results Array: \(resultArray)")
print("Results Array Count: \(resultArray.count)")
})
I have a list of products that I download as json file from server. Each item contains a link to the image stored on server.
Now I want to be able to see the products when offline, so I store downloaded json file into forge.prefs http://docs.trigger.io/en/v1.3/modules/prefs.html and pull it out to display items on screen. It works nice but I also need to store images localy to be displayed correctly.
To achive this, I'm trying to use forge.file.cacheURL http://docs.trigger.io/en/v1.3/features/cache.html but can't handle the correct order of operations. To cache images I run the json file and for each line I call forge.file.cacheURL and store the url back to JSON item. But here is the problem as forge.file.cacheURL runs asynchronously so my loop running over the items and gathering the local images finishes and my code continues to display images(view items) but meantime the forge.file.cacheURL still gathers and caches the images because its asynchronous operation. I need somehow to detect that last item is being cached and then refresh the view on screen to use correct image urls ... or something else that will lead to what I need.
Hopefully you understand the concept. How should I handle this properly ?
Since v1.4.26 you've been able to permanently store images (see http://docs.trigger.io/en/v1.4/release-notes.html#v1-4-26), rather than just cache them. Depending on your needs, that might be a better option than forge.file.cacheURL and forge.file.isFile.
I don't follow exactly the situation you describe, but something like this will let you wait for several asynchronous things to finish before doing something:
// e.g.
var jsonCache = {
one: "http://example.com/one.jpg",
two: "http://example.com/two.jpg",
three: "http://example.com/three.jpg"
};
var cacheCount = 0;
for (var name in jsonCache) {
if (jsonCache.hasOwnProperty(name)) {
var imageURL = jsonCache[name];
cacheCount += 1;
forge.file.cacheURL(imageURL, function (file) {
forge.prefs.set(name, file, function () {
cacheCount -= 1; // race condition, but should be fine (!)
if (cacheCount <= 0) {
alert('all cached');
}
});
});
}
}