tag:blogger.com,1999:blog-6222839753802397496.post4823508289174596760..comments2010-05-20T13:12:47.804-07:00Comments on Mess With Your Junk v2: Scalable concurrent deterministic event processing via lazy topological sortingxDirtyPunkxhttp://www.blogger.com/profile/02274180099309125966noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-6222839753802397496.post-36931152016844813302010-05-20T13:12:47.804-07:002010-05-20T13:12:47.804-07:00> It really comes down to the storage vs the co...> It really comes down to the storage vs the cost of polling the extra events.<br /><br />I am not sure I get you. What extra storage are talking about? Note that "single-entry" events does not need to be stored in a global queue, they can be stored only in entry-specific queue.<br /><br /><br />> As to the amount of polling you have to do, you only really need to have a fairly limited number of events in that ordered list to be polled, enough to feed your consumers really<br /><br />I suspect that there may be some "worst cases", when one needs to pool a lot of events to find few ready-to-run.<br />It's like with GC. In some situations GC has to scan a whole huge heap to find that there is only few kb to reclaim. And such each time.<br /><br /><br />> if they match the "now processing" ticket already, the event can be dispatched without polling<br /><br />Cool! I guess in some systems (bank account processing) 99% of events can be dispatched that way.Dmitry Vyukovhttps://www.blogger.com/profile/10137998824493472445noreply@blogger.comtag:blogger.com,1999:blog-6222839753802397496.post-59099162846149341502010-05-20T09:02:00.230-07:002010-05-20T09:02:00.230-07:00Events that touch a single entity are indeed a spe...Events that touch a single entity are indeed a special case that you can optimize for, but I don't know how much of a need there will be in practice as I imagine they would be first to be processed. It really comes down to the storage vs the cost of polling the extra events.<br /><br />As to the amount of polling you have to do, you only really need to have a fairly limited number of events in that ordered list to be polled, enough to feed your consumers really; you could even do the first phase of the event ordering on demand to fill the list to a particular size. <br /><br />The other big thing is that the oldest events are the most likely to be ready, so you can keep them in age order and poll from the start of the list and then stop when you have dispatched enough. <br /><br />In the end, you still pay a cost for contention, although it's lower than what would be paid with transactions or locking. There is also the advantage that if you have lots of events that are not dependent on the entities being contended for, they can "flow around" them and still be processed. <br /><br />It also should be noted that when you take the initial tickets for an event, if they match the "now processing" ticket already, the event can be dispatched without polling. So in a low-contention system, you don't pay much price at all!xDirtyPunkxhttps://www.blogger.com/profile/02274180099309125966noreply@blogger.comtag:blogger.com,1999:blog-6222839753802397496.post-32307686805928648732010-05-20T08:07:00.908-07:002010-05-20T08:07:00.908-07:00Nice!
I guess the system may have a special suppor...Nice!<br />I guess the system may have a special support for events that touch only a single entity (if that's expected to be a common case). Each entity needs a SP/SC FIFO queue with such events. When processing of an event completes, the threads polls the queue while tickets are subsequent.<br />I am only a bit concerned with readiness determination. If I have zillions of pending events and only a few ready to run, I need to periodically scan all pending event in the hope of picking up a few new ready to run events.Dmitry Vyukovhttps://www.blogger.com/profile/10137998824493472445noreply@blogger.com