I am currently writing a full screen web application for the iPad. This application will likely write a small amount of data to a local cookie at an interval of once per second. A user will interact with the application for about 20 to 30 minutes at a time. I imagine they will use this application between one and four times a week so a maximum of 120 minutes a week or 7,200 cookie writes per week.
I was unable to find information regarding the number of write cycles that the iPad's internal flash memory can handle. I am concerned that my frequent cookie writes may shorten the lifespan of the iPad's internal flash memory. Is this a valid concern? If not, why not and at what number of writes should I be concerned?
I am not completely opposed to using other storage methods, such as HTML5 local storage if this mitigates the risk. However, I would prefer to use cookies as part of this application will be delivered on other browsers where HTML5 local storage is not supported.
Related
I've seen a couple examples like this:
service Service{
rpc updload(stream Data) returns (google.protobuf.Empty) {};
rpc download(google.protobuf.Empty) returns (stream Data) {};
}
message Data { bytes bytes = 1; }
What is the purpose of using stream, does it make the transfer more efficient?
In theory - yes - I obviously wan't to stream my file transfers but that's what happens over a connection... So, what is the actual benefit to this keyword, does it enforce some form of special buffering to reduce some overhead? Either way, the data is being transmitted, in full!
It's more efficient because, within a single call, multiple messages may be sent.
This avoids, not only re-establishing another (hopefully TLS i.e. even more work) connection with the server but also avoids spinning up client and server "stubs"; both the client and server are ready for more messages.
It's somewhat similar to being connected on a telephone call with your friend who, before hanging up, says "Oh, another thing...". Instead of hanging up the call and then, 5 minutes later, calling you back, interrupting dinner and causing you to pause a movie.
The answer is very similar to the gRPC + Image Upload question, although from a different perspective.
Doing a large download (10+ MB) as a single response message puts strong limits on the size of that download, as the entire response message is sent and processed at once. For most use cases, it is much better to chunk a 100 MB file into 1-10 MB chunks than require all 100 MB to be in memory at once. That also allows the downloader to begin processing the file before the entire file is acquired which reduces processing latency.
Without streaming, chunking would require multiple RPCs, which are annoying to coordinate and have performance complications. Because there is latency to complete RPCs, for reasonable performance you either have to do many RPCs in parallel (but how many?) or have a large batch size (but how big?). Multiple RPCs can also hit colder application caches, as each RPC goes to a different backend.
Using streaming provides the same throughput as the non-chunking approach without as many headaches of normal chunking approaches. Since streaming is pipelined (server can start sending next chunk as soon as previous chunk is sent) there's no added per-chunk latency between the client and server. This makes it much easier to choose a chunk size, as there is a wide range of "reasonable" sizes that will behave similarly and the system will naturally react as network performance varies.
While sending a message on an existing stream has less overhead than creating a new RPC, for many users the difference is negligible and it is generally better to structure your RPCs in a way that is architecturally beneficial to your application and not just to eek out small optimizations in gRPC. The reason to use the stream in this case is to make your application perform better at lower complexity.
I'm developing a C++ application in the ESP32-DevKitC board where I sense acceleration from an accelerometer. The application goal is to store the accelerometer data until storage is full and then send all the data through WiFi and start all again. The micro also goes to deep-sleep mode when is possible.
I'm currently using the ESP32 NVS library which is very well documented and pretty easy to use. The negative side of this is that the library uses Flash memory, therefore a lot of writings will end up degrading the drive.
I know that Espressif also offers some other storage libraries (FAT, SPIFFS, etc.) but, as far as I know (correct me if I'm wrong), they all use Flash drive.
Is there any other possibility of doing what I want to but without using the Flash storage?
Aclarations
Using Flash memory is not the problem itself, but degrading it.
Storage has to be non volatile or at least not being erased when the micro goes to deep-sleep mode.
I'm not using any Arduino library.
That's a great question that I wish more people would ask.
ESP32s use NOR flash storage, which is usually rated for between 10,000 to 100,000 write cycles (100,000 seems to be the standard these days). Flash can't write single bytes; instead of writes a "page" of bytes, which I believe is 256 bytes. So each 256 byte page is rated for at least 100,000 cycles. When a device is rated for 100,000 cycles it's likely to be usable for at least 10 times that, but the manufacturer is not going to make any promises beyond the 100,000.
SPIFFS (and LittleFS, now used on the ESP8266 Arduino Core) perform "wear leveling", to minimize the number of times a particular page is written. So if you modify the same section of a file repeatedly, it will automatically be written to different pages of flash. FAT is not designed to work well with flash storage; I would avoid it.
Whether SPIFFS with wear leveling will be adequate for your needs depends on your needed lifetime of the device versus how much data you'll be writing and how frequently.
NVS may perform some level of wear levelling, to an extent I'm unsure about. Here, in a forum post with 2 ESP employees, they both confirm that NVS does do some form of wear levelling. NVS is best used to persist things like configuration information that doesn't change frequently. It's not a great choice for storing information that's updated often.
You mentioned that the data just needs to survive deep sleep. If that's the case, your best option (if it's large enough) is to use the ESP32's RTC static RAM. This chunk of memory will survive restarts and deep sleep mode, but will lose its state if power is interrupted. It's real RAM so you won't wear it out by writing to it frequently, and it doesn't cost a lot of energy to write to. The catch is there's only 8KB of it.
If the 8KB of RTC RAM isn't enough and you're writing too much data too frequently to trust that SPIFFS will be okay, your best bet would be an SD card. The ESP32 can talk to an SD card adapter. SD cards use NAND flash, which has a much greater lifespan than NOR and can be safely overwritten many more times (which is why these kinds of cards are usable for filesystems in devices like Raspberry Pis).
Writing to flash also takes much more energy than writing to regular RAM. If your device is going to be battery powered, the RTC RAM is also a better choice than SPIFFS or an SD card from a power savings perspective.
Finally, if you use the RTC RAM I'd recommend starting to write it over wifi before it's full, as bringing up wifi and transmitting the data could easily take long enough that you might run out of space for some samples. Using it as a ring buffer and starting the transmit process when you hit a high water mark rather than when the buffer is full would probably be your best bet.
I know i'm late with this answer but you can buy ESP32 modules with external RAM even with 4-8mb. External ram is really fast ( at least much faster than the flash, it uses SPI interface to communicate ) and you can fit a lot of sensor readings in there.
I'm using an ESP32_WROVER_E module with 8mb external ram ( 4mb is usable with normal function calls ) and 16mb flash.
Here is a link of the module that i'm using at TME's site.
I have an AVX2 implementation of some workload.
I have determined that the vast majority of the execution time is occupied
by the memory loads and stores.
In an attempt to improve performance, I tried to change the conventional stores
to streaming (non-temporal) stores.
However, this change had little to no positive performance impact (I was expecting a sizeable performance increase).
What could be the reason for this?
The use of streaming stores can lead to a better performance under some circumstances:
The data "to be stored" is not read before writing: Streaming stores are write-through, which produces immediate bus traffic. The standard store uses a write back strategy which may delay the bus operation until a later time and avoids bus operations with multiple writes to the same cache line.
The time used for stores is smaller than the time used for calculation: A streaming store has to be finished before the next streaming store can be issued. Thus, ahving too liitle computation in between two streaming stores leads to some idle time for the processor in which no further computation can be executed. Where this problem may also be possible with standard stores, streaming stores even increase it.
The data "to be stored" is not needed shortly after being written: The streaming store surpasses caches while writing/storing. Thus, there is no copy of the data in the cache. When reading the data aftwerwards the data has to be loaded into the cache. Thus, you have no gain over a standard store. However, when using a standard store, the data is loaded into the cache, modified there, and maybe still there when a later access happens.
So you have to consider your code and problem, to these circumstances to know if streaming stores are worth a try. In an unfitting scenario your performance might even drop.
A blog entry with additional info and a benchmark can be found e.g. here.
I wrote a HTML5 web app (kind-of enterprise PWA) that needs to store a substantial amount of data offline (my users are aware of this).
The web app works fine in all major browsers (including Desktop, Android and iOS), however I'm experiencing problems on phones like the Lumia 640 (Edge 15 browser on Windows 10 Mobile).
It's hard to tell what the exact problem is, due to the lack of debugging capabilities of that browser. The app works fine when emulating a Lumia 650 in Edge Desktop, though.
I guess the problem is that I'm exceeding the "hard" storage limits described here, since I usually store about 25 mb of JSON data and that phone has 8 GB of storage AFAIK (meaning that I hit the 10 MB limit).
Is there a way to allow a single domain to store such a large amount of data in Edge Mobile? The page linked above mentions that those "limits are removed for UWP apps using JavaScript" - I don't really know what that means.
Or is there an alternative way for a web application to store very large JSON objects in Edge Mobile?
I think I'm running into the same boundaries, though for me the quota limit seems more like 5MB. For instance running the Browser Storage Abuser app at https://demo.agektmr.com/storage/ I can store 1 5MB file, 9 500KB files, etc and then things start failing.
My device has 8GB storage, so maybe it's storing things in UTF16 which explains why the size appears half of what they document.
What's strange is I've tried the same on an emulator instance with a 10GB partition and I see the same limits. From that table it seems like I should be able to store twice as much.
I'm choosing an analytics service for my iOS app. I want to track quite a lot of events and the app I'm developing is going to be used outdoors, so there will be no wi-fi connection available, and even the cellular connectivity can be of a poor quality.
Analytics is the only thing that requires network connectivity in my app. Recently I've checked how much traffic it consumes, and it consumes much more than I've expected. That was about 500KB for Google Analytics and about 2MB for Flurry, and that's just for a 2-minute long session with a few hundred events. It seems very inefficient to me. (Flurry logs a little bit more parameters, but definitely not 4 times more.)
I wonder — have anybody compared other popular analytics solutions for their bandwidth consumption? Which one is the slimmest one?
Thank you
If you don't need real time data (and you probably don't with outdoor app), you can get the best network compression for Analytics by dispatching more hits at once to benefit from batching and compression. To do that set the dispatch interval to 30 minutes. The maximum size of uncompressed hit that analytics will accept is about 8k so you should be sending less then that. With compression that would bring it down to ~25% of the original size for individual hit assuming mostly ascii data. To generate 500k of data you should be sending few hundred hits individually. With batching and compression the hits will shrink down more efficiently. Usually batch of 20 hits will compress to less then 10% of the uncompressed size or about 800 bytes per hit at most. For further network savings just send less data per event or fewer events. Btw, Analytics has a rate limit of 60 tokens that are replenished at a rate of 1 hit every 2 seconds. If you are sending few hundred events in short period of time your data is likely getting rate limited.
https://developers.google.com/analytics/devguides/collection/ios/limits-quotas#ios_sdk