I understand how a distributed ledger ensures integrity using a chained linked-list data model whereby each block is chained to all its previous ones.
I also understand how in a PoW/PoS/PoET/(insert any trustless consensus mechanism here) context, the verification process makes it difficult for a malicious (or a group of) individual to tamper with a block because they would not have the necessary resources to broadcast an instance of the ledger to all network members so that they update their version to a wrong one.
My question is, if let's say some one does manage to change a block, does an integrity checking process ever happen? Is it an actual part of the verification mechanism and if so, how far in history does it go?
Is it ever necessary to verify the integrity of i.e. block number 500 out of a total of 10000 and if so, how do I do that? Do I need to start from block 10000 and verify all blocks from there until block 500?
My question is, if let's say some one does manage to change a block, does an integrity checking process ever happen?
Change the block where? If you mean change my copy of the block in my computer, how would they do that? By breaking into my computer? And how would that affect anyone else?
Is it an actual part of the verification mechanism and if so, how far in history does it go? Is it ever necessary to verify the integrity of i.e. block number 500 out of a total of 10000 and if so, how do I do that? Do I need to start from block 10000 and verify all blocks from there until block 500?
The usual rule for most blockchains is that every full node checks every single block it ever receives to ensure that it follows every single system rule for validity.
While you could re-check every block you already checked to ensure that your particular copy hasn't been tampered with, this generally serves no purpose. Someone who could tamper with your local block storage could also tamper with your local checking code. So this kind of check doesn't usually provide any additional security.
To begin with, tampering with a block is not made almost-impossible because of resource shortage for broadcasting the wrong ledger to all nodes. Broadcasting is not necessarily resource intensive. It is a chain-reaction which you only have to trigger. The challenge with tampering a block-chained block arises from the difficulty of recomputing the valid hashes (meeting the block difficulty level) of all the successive blocks (the blocks that come after the one being tampered). Because altering a block alters its hash, which in turn changes the previous hash attribute of the next block, hence invalidating its previously correct hash, and so on till the latest block. If say the latest block index is 1000. And if you tamper the 990th block. This implies that you would have to re-mine (recalculate a valid hash by randomly changing the nonce value) blocks from 990 - 1000. That in itself is very difficult to do. But say somehow you manged to do that, but by the time you broadcast your updated blockchain, there would have been other blocks (of index say 1001, 1002) mined by other miners. So yours won't be the longest valid blockchain,and therefore would be rejected.
According to this article, when a new block is broadcasted to the network, the integrity of its transaction is validated and verified against the history of transactions. The trouble with integrity check arises only if a malicious longer block-chain is broadcasted to the network. In this case, the protocol forces the nodes to accept this longest chain. Note that, this longest chain maintains its own integrity and is verified of its own. But it is verified against its own version of truth. Mind you though, this can only be possible if the attacker has a hashing power that is at least 51% of the network's hashing power. And this kind of power is assumed practically impossible. https://medium.com/coinmonks/what-is-a-51-attack-or-double-spend-attack-aa108db63474
Related
I want to create SQS using code whenever it is required to send messages and delete it after all messages are consumed.
I just wanted to know if there is some delay required between creating an SQS using Java code and then sending messages to it.
Thanks.
Virendra Agarwal
You'll have to try it and make observations. SQS is a dostributed system, so there is a possibility that a queue might not immediately be usable, though I did not find a direct documentation reference for this.
Note the following:
If you delete a queue, you must wait at least 60 seconds before creating a queue with the same name.
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_CreateQueue.html
This means your names will always need to be different, but it also implies something about the internals of SQS -- deleting a queue is not an instantaneous process. The same might be true of creation, though that is not necessarily the case.
Also, there is no way to know with absolute certainty that a queue is truly empty. A long poll that returns no messages is a strong indication that there are no messages remaining, as long as there are also no messages in flight (consumed but not deleted -- these will return to visibility if the consumer resets their visibility or improperly handles an exception and does not explicitly reset their visibility before the visibility timeout expires).
However, GetQueueAttributes does not provide a fail-safe way of assuring a queue is truly empty, because many of the counter attributes are the approximate number of messages (visible, in-flight, etc.). Again, this is related to the distributed architecture of SQS. Certain rare, internal failures could potentially cause messages to be stranded internally, only to appear later. The significance of this depends on the importance of the messages and the life cycle of the queue, and the risks of any such an issue seem -- to me -- increased when a queue does not have an indefinite lifetime (i.e. when the plan for a queue is to delete it when it is "empty"). This is not to imply that SQS is unreliable, only to make the point that any and all systems do eventually behave unexpectedly, however rare or unlikely.
To our Streaming pipeline, we want to submit unique GCS files, each file containing multiple event information, each event also containing a key (for example, device_id). As part of the processing, we want to shuffle by this device_id so as to achieve some form of worker to device_id affinity (more background on why we want to do it is in this another SO question. Once all events from the same file are complete, we want to reduce (GroupBy) by their source GCS file (which we will make a property of the event itself, something like file_id) and finally write the output to GCS (could be multiple files).
The reason we want to do the final GroupBy is because we want to notify an external service once a specific input file has completed processing. The only problem with this approach is that since the data is shuffled by the device_id and then grouped at the end by the file_id, there is no way to guarantee that all data from a specific file_id has completed processing.
Is there something we could do about it? I understand that Dataflow provides exactly_once guarantees which means all the events will be eventually processed but is there a way to set a deterministic trigger to say all data for a specific key has been grouped?
EDIT
I wanted to highlight the broader problem we are facing here. The ability to mark
file-level completeness would help us checkpoint different stages of the data as seen by external consumers. For example,
this would allow us to trigger per-hour or per-day completeness which are critical for us to generate reports for that window. Given that these stages/barriers (hour/day) are clearly defined on the input (GCS files are date/hour partitioned), it is only natural to expect the same of the output. But with Dataflow's model, this seems impossible.
Similarly, although Dataflow guarantees exactly-once, there will be cases where the entire pipeline needs to be restarted since something went horribly wrong - in those cases, it is almost impossible to restart from the correct input marker since there is no guarantee that what was already consumed has been completely flushed out. The DRAIN mode tries to achieve this but as mentioned, if the entire pipeline is messed up and draining itself cannot make progress, there is no way to know which part of the source should be the starting point.
We are considering using Spark since its micro-batch based Streaming model seems to fit better. We would still like to explore Dataflow if possible but it seems that we wont be able to achieve it without storing these checkpoints externally from within the application. If there is an alternative way of providing these guarantees from Dataflow, it would be great. The idea behind broadening this question was to see if we are missing an alternate perspective which would solve our problem.
Thanks
This is actually tricky. Neither Beam nor Dataflow have a notion of a per-key watermark, and it would be difficult to implement that level of granularity.
One idea would be to use a stateful DoFn instead of the second shuffle. This DoFn would need to receive the number of elements expected in the file (from either a side-input or some special value on the main input). Then it could count the number of elements it had processed, and only output that everything has been processed once it had seen that number of elements.
This would be assuming that the expected number of elements can be determined ahead of time, etc.
I've got a file that I want to append to. It's pretty important to me that this is both fast (I'm calling it 50hz on an iPhone 4) and safe.
I've looked at atomic appending. It seems to me like I would have to copy the whole file, append to it, and then use the NSFileManager's replaceItemAtURL to move them over, which sounds rather slow.
On the other hand, I could simply suck up a non-atomic append, assuming that the failure conditions are strictly that some subset of bytes at the end of the data I'm trying to write are not written. My file format writes out the length of each chunk first, so if there's not enough space for the length data or the length data is bigger than the available bytes, I can detect a partial write and discard.
The question is, how feasible would it be to use an atomic append to rapidly atomically append small amounts of data (half a kilobyte or so at a time), and what exactly are the failure outcomes of a non-atomic append?
Edit: I am the only one appending to this file. I am concerned only with external failure conditions, e.g. process termination, device running out of power, disk full, etc. I am currently using a synchronous append.
POSIX gives no guarantees about atomicity of write(2) when writing to a file.
If the platform does not provide any other means of writing that grants additional characteristics (and I'm not aware of any such API in iOS) you basically have to live with the possibility that the write could be partial.
The workaround of many Cocoa APIs (like -[NSData writeToFile:atomically:]) is the mechanism you mentioned: Perform the work on a temporary file and then atomically link(2) the new over the old file. This strategy does not apply well to your use case as it requires a copy of the old contents.
I would suggest the non-atomic approach you already considered. Actually I once used a very similar mechanism in an iOS app where I had to write a transcript of user actions for crash recovery. The recovery code thoroughly tested the transcript for integrity and would bail out on unexpected errors. Yet, I never received a single report of a corrupt file.
Let's say I have some record like {my_table, Id, Value}.
I constantly overwrite the value so that it holds consecutive integers like 1, 2, 3, 4, 5 etc.
In a distributed environment, is it guaranteed that my event listeners will receive all of the values? (I don't care about ordering)
I haven't verified this by reading that part of the source yet, but it appears that sending a message out is part of the update process, so messages should always come out, even on very fast changes. (The alternative would be for Mnesia to either queue messages or queue changes and run them in batches. I'm almost positive this is not what happens -- it would be too hard to predict the variability of advantageous moments to start batching jobs or queueing messages. Sending messages is generally much cheaper than making a change in the db.)
Since Erlang guarantees delivery of messages to a live destination this is as close to a promise that every Mnesia change will eventually be seen as you're likely to get. The order of messages couldn't be guaranteed on the receiving end (as it appears you expect), and of course a network failure could make a set of messages get missed (rendering the destination something other than live from the perspective of the sender).
Short question but for me its difficult to understand.
Why exactly does ePoll scale better than Poll?
While Damon's reason is correct for the unusual case where you never block on a socket, in typical real-world programs, the reason is completely different. A typical program looks like this:
1) Do all the work we can do now.
2) Check if any network connections need service, blocking if there's nothing to do.
3) Service any network connections discovered.
4) Go to step 1.
Typically, because you just did all the work you can do, when you come around to step 2, there is no work for you to do. So you'll have to wait a bit. Now, imagine there are 800 sockets you are interested in. The kernel has to put on the wait queue for each of those 800 sockets. And, a split-second later when data arrives on one of those 800 sockets, the kernel has to remove you from those 800 wait queues. Placing a task on a wait queue requires creating a 'thunk' to link that task to that wait queue. No good optimizations are possible because the kernel has no idea which 800 sockets you'll be waiting for.
With epoll, the epoll socket itself has a wait queue, and the process is only put on that one single wait queue. A thunk is needed to link each of the 800 connections to the epoll wait queue, but that thunk is persistent. You create it by adding a socket to an epoll set, and it remains there until you remove the socket from the set.
When there's activity on the socket, the kernel handles it in the task that detects the activity. When you wait, the kernel already knows if there's a detected event and the kernel only has to put you on that one wait queue. When you wake, it only has to remove you from that one queue.
So it's not so much the copying that's the killer with select or poll, it's the fact that the kernel has to manipulate a massive number of wait queues on each blocking operation.
The poll system call needs to copy your list of file descriptors to the kernel each time. This happens only once with epoll_ctl, but not every time you call epoll_wait.
Also, epoll_wait is O(1) in respect of the number of descriptors watched1, which means it does not matter whether you wait on one descriptor or on 5,000 or 50,000 descriptors. poll, while being more efficient than select, still has to walk over the list every time (i.e. it is O(N) in respect of number of descriptors).
And lastly, epoll can in addition to the "normal" mode work in "edge triggered" mode, which means the kernel does not need keep track of how much data you've read after you've been signalled readiness. This mode is more difficult to grasp, but somewhat more efficient.
1As correctly pointed out by David Schwartz, epoll_wait is of course still O(N) in respect of events that occur. There is hardly a way it could be any different, with any interface. If N events happen on a descriptor that is watched, then the application needs to get N notifications, and needs to do N "things" in order to react on what's happening.
This is again slightly, but not fundamentally different in edge triggered mode, where you actually get M events with M <= N. In edge triggered mode, when the same event (say, POLLIN) happens several times, you will probably get fewer notifications, possibly only a single one. However, this doesn't change much about the big-O notation as such.
However, epoll_wait is irrespective of the number of descriptors watched. Under the assumption that it is used in the intended, "normal" way (that is, many descriptors, few events), this is what really matters, and here it is indeed O(1).
As an analogy, you can think of a hash table. A hash table accesses its content in O(1), but one could argue that calculating the hash is actually O(N) in respect of the key length. This is technically absolutely correct, and there probably exist cases where this is a problem, however, for most people, this just doesn't matter.