Help with Modelling a large system of probabilities

02052018, 02:07 AM
Post: #1




Help with Modelling a large system of probabilities
Apologies in advance; this really doesn't have much to do with HP calculators,
except that I'd love to develop an approximation model that would run on one.. I'm hoping for some expertise from someone smarter than me.. I'm trying to model a system that involves about 2400 hosts in a sharded database of sorts. Users enter queries, they get distributed to the 2400 systems simultaneously and when all have finished, the user gets the results. The overall system has an SLA (Service Level Agreement) that the queries finish in a certain length of time, 90% of the time. Obviously, the weakest link in the 2400 systems determines the length of time that a search completes in. A naive approach that assumes all queries on the backend are independent would mean we are looking for the probability that "at least one of" the backend systems is over SLA. To find that of course, we'd use the complement of probabilities; the intersection set of probabilities where all queries on the back end are within SLA. If they were all equal and independent, it might look something like: Probability of Search being within SLA == 0.99995^2400 Unfortunately, the queries on the backend aren't totally independent. Sometimes they are, such as a query hitting a system that is busy doing something else. However, sometimes the query performance rests with the complexity, number of terms etc, and will impact a number of backend systems. Nevertheless, there is a long tail of systems that are always within SLA. I'm wondering if there is a relatively simple way to model the behaviour such that we can "play with the numbers" and get approximate results. Eg. What percentage of the systems would we have to "fix" and by how much to maintain the user experience etc. As a sample, the worst systems are within SLA 95% of the time, while the best are within 100%. The user experience however is closer to 90% which is natural of course, as the probabilities would accumulate and be worse than the worst one.. Thoughts? Suggestions? Pointers? I'm stuck.. Thanks! 

02072018, 10:50 AM
(This post was last modified: 02072018 10:51 AM by pier4r.)
Post: #2




RE: Help with Modelling a large system of probabilities
Interesting problem (how many interesting problems are posted here, and I miss the majority of them).
I learned that some can proceed by large improvements at once, I have to go very slowly, step by step. At first I would check the real world reports, to get an idea about the success rate (being within the SLA) of each system. Even without real world stats one could work with assumptions. I mean in the Engineering course I learned that unless we need extra precision, it is ok to know a range of values where things works (or, conversely, don't). So you said that there are normal queries, that could be considered executed on each system independently from the others and then queries that depends on other systems. For the first type of queries (the independ ones), you already have the solution. The probability of going over the SLA is a Bernoulli trial. If the probability to succeed is P (assumed or computed), then to fail is (1P). Then the SLA fails when one system fails, when two system fails, when three, etc.. then all. This is: \[ \sum_{n=1}^{2400} P^{2400n} \cdot (1P)^n \] so the probability of succeeding is: \[ 1  \sum_{n=1}^{2400} P^{2400n} \cdot (1P)^n \] And this is for the simple queries. For the more complicated queries, either you go through the rabbit hole of dependent variables (I myself have little ideas about those) or again: simplify unless you need that high precision (or you want to know). So for depended queries again we need to set assumptions. How often do they happen? Often enough that we need to care for them? If they happen, can we model them like a chain of dependencies? Can be that one system may ask many others but each independently so the chain is at most of length 1 (longer lengths may be not so frequent)? Wikis are great, Contribute :) 

02082018, 12:28 PM
Post: #3




RE: Help with Modelling a large system of probabilities
Thanks pier4r, you've given me some ideas..
Appreciate the help. Cheers. 

05302020, 01:01 AM
Post: #4




RE: Help with Modelling a large system of probabilities
Hey @pier4r
Thanks so much for pointing me in the right direction! I've added a program for playing with the numbers while developing SLOs. Service Level Objectives Cheers. 

« Next Oldest  Next Newest »

User(s) browsing this thread: 1 Guest(s)