My app needs lots of data to display inside the flutter app offline. Previously I stored all the data in List format and looped through them to display items inside app but those data became very huge and exceeded more than 1000 lines and 100KB so I was wondering if it was fine or is there more efficient way to store data inside the app?
Lists Format I used:
const List<String> summaries = [
"Lots of texts",
"Another Lots of texts",
// Lots of other large items
]
const List QnA = [
[
{
'question':'My question?',
'answer':"""My Answer""",
},
{
'question':'My question?',
'answer':"""My Answer""",
}
]
// Lots of list items
];
Thank you!
I prefer converting allthose data's to a json format and place it inside asset folder and decoding it.
final data = rootBundle.loadString('assets/text_asset/data.json');
json.decode(data);
Related
I just want to fetch all my liked videos ~25k items. as far as my research goes this is not possible via the YouTube v3 API.
I have already found multiple issues (issue, issue) on the same problem, though some claim to have fixed it, but it only works for them as they don't have < 5000 items in their liked video list.
playlistItems list API endpoint with playlist id set to "liked videos" (LL) has a limit of 5000.
videos list API endpoint has a limit of 1000.
Unfortunately those endpoints don't provide me with parameters that I could use to paginate the requests myself (e.g. give me all the liked videos between date x and y), so I'm forced to take the provided order (which I can't get past 5k entries).
Is there any possibility I can fetch all my likes via the API?
more thoughts to the reply from #Yarin_007
if there are deleted videos in the timeline they appear as "Liked https://...url" , the script doesnt like that format and fails as the underlying elements dont have the same structure as existing videos
can be easily fixed with a try catch
function collector(all_cards) {
var liked_videos = {};
all_cards.forEach(card => {
try {
// ignore Dislikes
if (card.innerText.split("\n")[1].startsWith("Liked")) {
....
}
}
catch {
console.log("error, prolly deleted video")
}
})
return liked_videos;
}
to scroll down to the bottom of the page ive used this simple script, no need to spin up something big
var millisecondsToWait = 1000;
setInterval(function() {
window.scrollTo(0, document.body.scrollHeight);
console.log("scrolling")
}, millisecondsToWait);
when more ppl want to retrive this kind of data, one could think about building a proper script that is more convenient to use. If you check the network requests you can find the desired data in the response of requests called batchexecute. One could copy the authentification of one of them provide them to a script that queries those endpoints and prepares the data like the other script i currently manually inject.
Hmm. perhaps Google Takeout?
I have verified the youtube data contains a csv called "liked videos.csv". The header is Video Id,Time Added, and the rows are
dQw4w9WgXcQ,2022-12-18 23:42:19 UTC
prvXCuEA1lw,2022-12-24 13:22:13 UTC
for example.
So you would need to retrieve video metadata per video ID. Not too bad though.
Note: the export could take a while, especially with 25k videos. (select only YouTube data)
I also had an idea that involves scraping the actual liked videos page (which would save you 25k HTTP Requests). But I'm unsure if it breaks with more than 5000 songs. (also, emulating the POST requests on that page may prove quite difficult, albeit not impossible. (they fetch /browse?key=..., and have some kind of obfuscated / encrypted base64 strings in the request-body, among other parameters)
EDIT:
Look. There's probably a normal way to get a complete dump of all you google data. (i mean, other than takeout. Email them? idk.)
anyway, the following is the other idea...
Follow this deep link to your liked videos history.
Scroll to the bottom... maybe with selenium, maybe with autoit, maybe put something on the "end" key of your keyboard until you reach your first liked video.
Hit f12 and run this in the developer console
// https://www.youtube.com/watch?v=eZPXmCIQW5M
// https://myactivity.google.com/page?utm_source=my-activity&hl=en&page=youtube_likes
// go over all "cards" in the activity webpage. (after scrolling down to the absolute bottom of it)
// create a dictionary - the key is the Video ID, the value is a list of the video's properties
function collector(all_cards) {
var liked_videos = {};
all_cards.forEach(card => {
// ignore Dislikes
if (card.innerText.split("\n")[1].startsWith("Liked")) {
// horrible parsing. your mileage may vary. I Tried to avoid using any gibberish class names.
let a_links = card.querySelectorAll("a")
let details = a_links[0];
let url = details.href.split("?v=")[1]
let video_length = a_links[3].innerText;
let time = a_links[2].parentElement.innerText.split(" • ")[0];
let title = details.innerText;
let date = card.closest("[data-date]").getAttribute("data-date")
liked_videos[url] = [title,video_length, date, time];
// console.log(title, video_length, date, time, url);
}
})
return liked_videos;
}
// https://stackoverflow.com/questions/57709550/how-to-download-text-from-javascript-variable-on-all-browsers
function download(filename, text, type = "text/plain") {
// Create an invisible A element
const a = document.createElement("a");
a.style.display = "none";
document.body.appendChild(a);
// Set the HREF to a Blob representation of the data to be downloaded
a.href = window.URL.createObjectURL(
new Blob([text], { type })
);
// Use download attribute to set set desired file name
a.setAttribute("download", filename);
// Trigger the download by simulating click
a.click();
// Cleanup
window.URL.revokeObjectURL(a.href);
document.body.removeChild(a);
}
function main() {
// gather relevant elements
var all_cards = document.querySelectorAll("div[aria-label='Card showing an activity from YouTube']")
var liked_videos = collector(all_cards)
// download json
download("liked_videos.json", JSON.stringify(liked_videos))
}
main()
Basically it gathers all the liked videos' details and creates a key: video_ID - Value: [title,video_length, date, time] object for each liked video.
It then automatically downloads the json as a file.
The documentation does not have any examples on how to add a subcollection to a document. I know how to add document to a collection and how to add data to a document, but how do I add a collection (subcollection) to a document?
Shouldn't there be some method like this:
dbRef.document("example").addCollection("subCollection")
Edit 13 Jan 2021:
According to the updated documentation regarding array membership, now it is possible to filter data based on array values using whereArrayContains() method. A simple example would be:
CollectionReference citiesRef = db.collection("cities");
citiesRef.whereArrayContains("regions", "west_coast");
This query returns every city document where the regions field is an array that contains west_coast. If the array has multiple instances of the value you query on, the document is included in the results only once.
Assuming we have a chat application that has a database structure that looks similar to this:
To write a subCollection in a document, please use the following code:
DocumentReference messageRef = db
.collection("rooms").document("roomA")
.collection("messages").document("message1");
Creating a messages collection and calling addDocument() 1000 times will be expensive for sure, but this is how Firestore works. You can switch to Firebase Realtime Database if you want where the number of writes doesn't matter. But regarding Supported Data Types in Firestore, in fact, you can use an array because it is supported. In Firebase Realtime database you could also use an array, but this is an anti-pattern. One of the many reasons Firebase recommends against using arrays is that it makes the security rules impossible to write.
Cloud Firestore can store arrays, but it does not support querying array members or updating single array elements. However, you can still model this kind of data by leveraging the other capabilities of the Cloud Firestore. Here is the documentation where it is very well explained.
You also cannot create a subcollection with 1000 messages, add all of them to the database, and expect it to be considered a single record. It will be considered one write operation for every message, in total 1000 operations. The picture above does not show how to retrieve data, it shows a database structure in which you have something like this:
collection -> document -> subCollection -> document
Here's a variation where the subcollection is storing ID values at the collection level, rather than within a document where the subcollection is a field there with additional data.
This is useful for connecting a 1-to-Many ID mapping w/out having to drill through an additional document:
function fireAddStudentToClassroom(studentUserId, classroomId) {
var db = firebase.firestore();
var studentsClassroomRef =
db.collection('student_class').doc(classroomId)
.collection('students');
studentsClassroomRef
.doc(studentUserId)
.set({})
.then(function () {
console.log('Document Added ');
})
.catch(function (error) {
console.error('Error adding document: ', error);
});
}
Thanks to #Alex's answer
This answer a bit off from the original question here, where it explicitly asks for adding a collection to a document. However, after searching for a solution for this scenario and not finding any mention in docs or on SO, this post seems like a reasonable place to share the findings
Here's my code:
firebase.firestore().collection($scope.longLanguage + 'Words').doc($scope.word).set(wordData)
.then(function() {
console.log("Collection added to Firestore!");
var promises = [];
promises.push(firebase.firestore().collection($scope.longLanguage + 'Words').doc($scope.word).collection('AudioSources').doc($scope.accentDialect).set(accentDialectObject));
promises.push(firebase.firestore().collection($scope.longLanguage + 'Words').doc($scope.word).collection('FunFacts').doc($scope.longLanguage).set(funFactObject));
promises.push(firebase.firestore().collection($scope.longLanguage + 'Words').doc($scope.word).collection('Translations').doc($scope.translationLongLanguage).set(translationObject));
Promise.all(promises).then(function() {
console.log("All subcollections were added!");
})
.catch(function(error){
console.log("Error adding subcollections to Firestore: " + error);
});
})
.catch(function(error){
console.log("Error adding document to Firestore: " + error);
});
This makes a collection EnglishWords, which has a document of. The document of has three subcollections: AudioSources (recordings of the word in American and British accents), FunFacts, and Translations. The subcollection Translations has one document: Spanish. The Spanish document has three key-value pairs, telling you that 'de' is the Spanish translation of 'of'.
The first line of the code creates the collection EnglishWords. We wait for the promise to resolve with .then, and then we create the three subcollections. Promise.all tells us when all three subcollections are set.
IMHO, I use arrays in Firestore when the entire array is uploaded and downloaded together, i.e., I don't need to access individual elements. For example, an array of the letters of the word 'of' would be ['o', 'f']. The user can ask, "How do I spell 'of'?" The user isn't going to ask, "What's the second letter in 'of'?"
I use collections when I need to access individual elements, a.k.a. documents. With the older Firebase Realtime Database, I had to download arrays and then iterate through the arrays with forEach to get the element I wanted. This was a lot of code, and with a deep data structure and/or large arrays I was downloading tons of data that I didn't need, and slowing my app running forEach loops on large arrays. Firestore puts the iterators in the database, on their end, so that I can request a single element and it sends me just that element, saving me bandwidth and making my app run faster. This might not matter for a web app, if your computer has a broadband connection, but for mobile apps with poor data connections and slow devices this is important.
Here are two pictures of my Firestore:
From the docs:
You do not need to "create" or "delete" collections. After you create the first document in a collection, the collection exists. If you delete all of the documents in a collection, it no longer exists.
Here i faced the same issue and solve with the answere of #Thomas David Kehoe
db.collection("First collection Name").doc("Id of the document").collection("Nested collection Name").add({
//your data
}).then((data) => {
console.log(data.id);
console.log("Document has added")
}).catch((err) => {
console.log(err)
})
too late for an answer but here is what worked for me,
mFirebaseDatabaseReference?.collection("conversations")?.add(Conversation("User1"))
?.addOnSuccessListener { documentReference ->
Log.d(TAG, "DocumentSnapshot written with ID: " + documentReference.id)
mFirebaseDatabaseReference?.collection("conversations")?.document(documentReference.id)?.collection("messages")?.add(Message(edtMessage?.text.toString()))
}?.addOnFailureListener { e ->
Log.w(TAG, "Error adding document", e)
}
add success listener for adding document and use firebase generated ID for a path.
Use this ID for the complete path for a new collection you want to add.
I.E. - dbReference.collection('yourCollectionName').document(firebaseGeneratedID).collection('yourCollectionName').add(yourDocumentPOJO/Object)
Okay so I recently faced a similar problem given the recent update in the firebase/firestore documentation.
And here is a solution that worked for me
const sendMessage = async () => {
await setDoc(doc(db, COLLECTION_NAME, projectId, SUB_COLLECTION_NAME, nanoid()), {
text:'this is a sample text',
createdAt: serverTimestamp(),
name: currentUser?.firstName + ' ' + currentUser?.lastName,
photoUrl: currentUser?.photoUrl,
userId: currentUser?.id,
});
}
You can find a similar example in the docs
https://firebase.google.com/docs/firestore/data-model#web-version-9_3
chat room
If you wish to listen for live update you can use a similar method as follows
const messagesRef = collection(db, COLLECTION_NAME, projectId, SUB_COLLECTION_NAME)
const liveUpdate = async () => {
const queryObj = query(messagesRef, orderBy("createdAt"), limit(25));
onSnapshot(queryObj, (querySnapshot) => {
const msgArr: any = [];
querySnapshot.forEach((doc) => {
msgArr.push({ id: doc.id, ...doc.data() })
});
console.log(msgArr);
});
}
There is no separate method to add sub-collection into the document.
You can just call the collection method itself.
If the collection exists it will reference that otherwise create a new one.
dbRef.document("example").collection("subCollection")
i have a little problem with my Firebase-Storage.
In my storage I have some pictures like:
"apple.jpg", "blueberry.jpg", "melon.jpg"...
In my Swift-Code I have an Array with Fruit-Names like:
["Apple", "Blueberry", "Melon"]
In my function I loop through my Apple-Name-Array and if I print the fruitsRef I get the following result:
gs://myapp-5313d.appspot.com/sort/apple.jpg
gs://myapp-5313d.appspot.com/sort/blueberry.jpg
gs://myapp-5413d.appspot.com/sort/melon.jpg
but in my fruitsRef.downloadURL-Method I get random ordered results:
Optional(https://firebasestorage.googleapis.com/v0/b/myapp-5413d.appspot.com/o/sort%2Fblueberry.jpg?alt=media&token=5047d133-b37e-49d2-ae6c-59be5e7cafbf)
Optional(https://firebasestorage.googleapis.com/v0/b/myapp-5413d.appspot.com/o/sort%2Fmelon.jpg?alt=media&token=5047d133-b37e-49d2-ae6c-59be5e7cafbf)
Optional(https://firebasestorage.googleapis.com/v0/b/myapp-5413d.appspot.com/o/sort%2Fapple.jpg?alt=media&token=5047d133-b37e-49d2-ae6c-59be5e7cafbf)
func setUrlArray(sorts: Array<String>) {
for currentSort in sorts {
let storage = Storage.storage()
let storageRef = storage.reference()
let fruitsRef = storageRef.child("sort/" + currentSort.lowercased().replacingOccurrences(of: " ", with: "") + ".jpg")
print(fruitsRef)
fruitsRef.downloadURL { url, error in
print(url)
}
}
}
how is that possible?
kind regards,
doomsweb
The issue has to do with the asynchronous nature of the downloadURL function on your fruitsRef variable. When you are iterating the array, you are synchronously printing out the reference to the image (print(fruitsRef)), and then also kicking off an asynchronous task (downloadURL).
You've guaranteed that the for loop will print out the fruitsRef in a particular order, since you are looping through the array one at a time - however, you have no guarantees with regard to which order your Firebase storage will respond to your network requests.
From the server's perspective, you are asking for three different assets at basically the same time (since your app will loop through the array of three elements very quickly), and the server will try to respond to each request as soon as it's able. So as soon as any one of those assets is returned to your app, the print(url) line is hit.
I solved the issue now with a Little Trick. My fruits name Array was already sorted alphabetically. So I have sorted the array with the links now also alphabetically. Now the order is correct
I've found a ton of information on LocalStorage with HTML5 but they all focus on persistent single entries being saved.
I need to be able to have a contact form (simple name/email/phone) that gets saved to the iPad and then allows another person to submit an entry to save to the iPad locally (no Wifi/Internet available).
Then I want to be able to go in later and grab all the entries that were made in whatever format available.
Any direction & help would be appreciated.
I searched on Stackoverflow & Google but still couldn't find multiple entry tutorials or examples.
Thanks!
localStorage is a good fit for what you're after. As localStorage only supports strings, you will need to do some conversions to/from JSON to serialize the entries.
Here is some general code to hopefully get you started:
// read out any previous contact forms (initialise if it is empty)
var jsonEntries = localStorage["contactFormEntries"];
var contactFormEntries = JSON.parse(jsonEntries ? jsonEntries : '[]');
// ...
// sometime later... create a contact form record
var contactForm = {
'name': 'Timmy',
'email': 'timmy#example.com'
};
// add it to the entries array
contactFormEntries.push(contactForm);
// serialize all the contact form entries into local storage
localStorage["contactFormEntries"] = JSON.stringify(contactFormEntries);
// ...
// then, when you want to send the entries, read them all out again
var contactFormEntries = JSON.parse(localStorage["contactFormEntries"]);
I have a list of products that I download as json file from server. Each item contains a link to the image stored on server.
Now I want to be able to see the products when offline, so I store downloaded json file into forge.prefs http://docs.trigger.io/en/v1.3/modules/prefs.html and pull it out to display items on screen. It works nice but I also need to store images localy to be displayed correctly.
To achive this, I'm trying to use forge.file.cacheURL http://docs.trigger.io/en/v1.3/features/cache.html but can't handle the correct order of operations. To cache images I run the json file and for each line I call forge.file.cacheURL and store the url back to JSON item. But here is the problem as forge.file.cacheURL runs asynchronously so my loop running over the items and gathering the local images finishes and my code continues to display images(view items) but meantime the forge.file.cacheURL still gathers and caches the images because its asynchronous operation. I need somehow to detect that last item is being cached and then refresh the view on screen to use correct image urls ... or something else that will lead to what I need.
Hopefully you understand the concept. How should I handle this properly ?
Since v1.4.26 you've been able to permanently store images (see http://docs.trigger.io/en/v1.4/release-notes.html#v1-4-26), rather than just cache them. Depending on your needs, that might be a better option than forge.file.cacheURL and forge.file.isFile.
I don't follow exactly the situation you describe, but something like this will let you wait for several asynchronous things to finish before doing something:
// e.g.
var jsonCache = {
one: "http://example.com/one.jpg",
two: "http://example.com/two.jpg",
three: "http://example.com/three.jpg"
};
var cacheCount = 0;
for (var name in jsonCache) {
if (jsonCache.hasOwnProperty(name)) {
var imageURL = jsonCache[name];
cacheCount += 1;
forge.file.cacheURL(imageURL, function (file) {
forge.prefs.set(name, file, function () {
cacheCount -= 1; // race condition, but should be fine (!)
if (cacheCount <= 0) {
alert('all cached');
}
});
});
}
}