1. Kivy Python Example
2. Kivy Python Android
3. Kivy For Python 3.8

So I actually ended up making a slot machine module with a built-in mini-game that runs in the terminal. The slot machine is itself a class object that contains information about items and reels attached to it, allowing one to make customized machines and the mini-game is one such example.

Reinforcement learning has yet to reach the hype levels of its Supervised and Unsupervised learning cousins. Albeit, it is an exceptionally powerful approach aimed to solve a variety of problems in a completely different way. To learn reinforcement learning, it is best to start from its building blocks and progress from there. In this article, we will go through a Multi-Armed Bandit using Python to solve a business problem.

What are Multi-Armed Bandits?

Imagine you have 3 slot machines and you are given a set of rounds. In each round, you have to choose 1 slot machine and pull its arm and receive the reward (or no at all) from that slot machine. Then, you do it again, and again… Eventually, you figure out which slot machine gives you the most rewards and keep pulling it for each round.

This is the Multi-Armed Bandit problem also known as the k-Armed Bandit problem. K is the number of slot machines available and your reinforcement learning algorithm needs to figure out which slot machine to pull in order to maximize its rewards.

Multi-Armed bandits are a classical reinforcement learning example and it clearly exemplifies a well-known dilemma in reinforcement learning called the exploration-exploitation trade-off dilemma.

Keep in mind these important concepts. I will cover them in more detail in posts coming up. For now, let’s apply Multi-Armed bandits to a business problem.

Our Business Problem

We will apply the Multi-Armed Bandit method to a theoretical marketing business problem.

You are the Chief Marketing Officer promoting a new product and need to get the word out. To do this, you will be texting your target customers a message. Upon receiving the message, the customer will react one way or the other and generate a reward that you have access to.

You have 4 text messages to send but are not sure which one will work best.

Upper-Confidence-Bound Action Selection Formula

How do you choose the best action? Going back to my example in which you had to choose a slot machine at each round. Particularly at the beginning, you had to test them out correct? Either stick to one from the start (not the best option) or try a couple of them to find out the best one.

A common strategy is called the Upper-Confidence-Bound Action selection, in short, UCB.

If you are an optimist, you will like this one! It’s strategy is :

Optimism in the face of uncertainty.

This method selects the action according to its potential, captured in the Upper-Confidence interval. Balancing this out with how uncertain you are of its measurement.

The UCB formula is the following:

( A_{t} doteq argmax_{a} left [ Q_{t}(a) + csqrtfrac{lnt}{N_{t}(a)} right ] )

• t = the time (or round) we are currently at
• a = action selected (in our case the message chosen)
• Nt(a) = number of times action a was selected prior to the time t
• Qt(a) = average reward of action a prior to the time t
• c = a number greater than 0 that controls the degree of exploration
• ln t = natural logarithm of t

The idea behind this algorithm is that the value of the square root indicates the uncertainty of the value of action a.

If an action is not chosen, t increases Nt(a) does not which will make the uncertainty of that action to increase, increasing its chances of being chosen (exploration).

If an action is chosen both numerator increases. The numerators increase get smaller through time (due to the natural log) but the denominator does not, causing the uncertainty to decrease.

The value with the maximum UCB gets chosen at each round! Let’s go through an example of how to implement UCB in Python.

Multi-Armed Bandit – Generate Data

Let us begin implementing this classical reinforcement learning problem using python. As always, import the required libraries first.

The last line above is only needed if you are using a jupyter notebook for the implementation.

Next up, we will generate our dataset consisting of 5 columns. The first is the user we sent SMS messages to followed by m1 through m4 which holds the message that was sent. The value is the rewards for each combination of message and user. Each follows a normal distribution with a mean between 95 and 105 and a standard deviation of 5 or 10. We generate 10,000 samples.

You can see our dataset below.

To get a better sense of our average rewards for each message, let’s visualize the above dataset in a box plot.

Looking at the box-plot above, if you had to pick the best message which one would you choose?…. It would have to be m2 of course!

Do you think our multi-armed bandit algorithm will pick this up? We will find out soon.

Multi-Armed Bandit – UCB Method

In order to solve our Multi-Armed bandit problem using the Upper-Confidence Bound selection method, we need to iterate through each round, take an action (select and send a message), see its returns and pick again. Eventually, we will be selecting the best message.

To implement UCB in python first initialize our variables. Each is commented on below to aid in your understanding. These will help us solve the UCB formula previously shown.

Next, we will loop through each round represented by the variable t. At each round, the UCB_Values array will hold the UCB values as of that round and these get set to 0. Then we loop through each possible action. If an action has never been selected, it gets chosen. Otherwise, we calculate the UCB value for each action and store it in the UCB_Values array.

At the end of each round, the action containing the maximum UCB Value gets selected. The numpy argmax function returns the index of the maximum value in the array. This is stored in the action_selected variable.

The last block of code below, under “update Values as of round t” updates the values of our important variables with the information our system has seen thus far.

To allow us to perform additional analysis, I have added this additional piece to our code after “sum_rewards[action_selected] += reward”.

We store values at each round corresponding to the reward we would have received by choosing a message at random in each round. Obviously, we aim to do better than that through the UCB algorithm.

Let us first see if our method performs better than choosing an action randomly.

Kivy Python Example

Indeed it performs much better than selecting an action randomly!

Our variable, Nt_a, holds the number of times each action was selected. We said previously that for this exercise, M2 would be the better one as it’s distribution has the highest average. Plot each action and the number of times each was selected.

What do you think? Just as expected!

Conclusion

Kivy Python Android

Kivy For Python 3.8

We have now implemented the Upper-Confidence Bound Selection method to solve a classical reinforcement learning algorithm, the Multi-Armed Bandits. UCB is widely used to solve digital marketing problems. However, there exist other methods to solve these K-Armed bandits such as epsilon greedy and Thompson Sampling. Hope you enjoyed this article, stay tuned for more!