logistic回归

logistic回归

回归就是对已知公式的未知参数进行估计。比如已知公式是$y = a*x + b$,未知参数是a和b,利用多真实的(x,y)训练数据对a和b的取值去自动估计。估计的方法是在给定训练样本点和已知的公式后,对于一个或多个未知参数,机器会自动枚举参数的所有可能取值,直到找到那个最符合样本点分布的参数(或参数组合)。

logistic分布

设X是连续随机变量,X服从logistic分布是指X具有下列分布函数和密度函数:

$$F(x)=P(x \le x)=\frac 1 {1+e{-(x-\mu)/\gamma}}\ f(x)=F{‘}(x)=\frac {e^{-(x-\mu)/\gamma}} {\gamma(1+e{-(x-\mu)/\gamma})2}$$

其中,$MU$是位置参数,$Gamma$是形状参数。[en]Where $ mu$ is the position parameter and $ gamma$ is the shape parameter.

$f(x)$与$F(x)$图像如下,其中分布函数是以$(\mu, \frac 1 2)$为中心对阵,$\gamma$越小曲线变化越快。

logistic回归

logistic回归模型

二项Logistic回归模型如下:[en]The binomial logistic regression model is as follows:

$$P(Y=1|x)=\frac {exp(w \cdot x + b)} {1 + exp(w \cdot x + b)} \
P(Y=0|x)=\frac {1} {1 + exp(w \cdot x + b)}$$

其中,R^n$中的$x是输入,{0jue 1}$中的$Y是输出,w是权重向量,b是偏移量,$wot x$是w和x的内积。[en]Where $x in R ^ n $is the input, $Y in {0jue 1} $is the output, w is the weight vector, b is the offset, and $w cdot x $is the inner product of w and x.

参数估计

假设:[en]Suppose:

$$P(Y=1|x)=\pi (x), \quad P(Y=0|x)=1-\pi (x)$$

则似然函数为:[en]Then the likelihood function is:

$$\prod_{i=1}^N [\pi (x_i)]^{y_i} [1 – \pi(x_i)]^{1-y_i}
$$

求出对数似然函数:[en]Find the logarithmic likelihood function:

$$L(w) = \sum_{i=1}^N [y_i \log{\pi(x_i)} + (1-y_i) \log{(1 – \pi(x_i)})]\
=\sum_{i=1}^N [y_i \log{\frac {\pi (x_i)} {1 – \pi(x_i)}} + \log{(1 – \pi(x_i)})]$$

因此,得到最大值$L(W)$,并获得估计值$w$。[en]Thus, the maximum value of $L (w) $is obtained, and the estimated value of $w $is obtained.

求极值的方法有梯度下降法、梯度上升法等。[en]The method of finding the extreme value can be gradient descent method, gradient rise method and so on.

梯度上升确定回归系数

logistic回归的sigmoid函数:

$$\sigma (z) = \frac 1 {1 + e^{-z}}$$

假设LOGSTIC的函数为:[en]Suppose the function of logstic is:

$$y = w_0 + w_1 x_1 + w_2 x_2 + … + w_n x_n$$

它可以缩写为:[en]It can be abbreviated as:

$$y = w_0 + w^T x$$

梯度上升算法是按照上升最快的方向不断移动,每次都增加$\alpha \nabla_w f(w)$,

$$w = w + \alpha \nabla_w f(w) $$

其中,$$\nabla_w f(w)$$ 为函数导数, $\alpha$ 为增长的步长。

训练算法

该算法的主要思想是根据Logistic回归的Sigmoid函数将函数值映射到有限的空间。Sigmoid函数的取值范围是0到1,因此数据可以根据0和1分为两类。在算法中,我们首先将所有w个权值初始化为1,每次计算误差,然后根据误差调整w个权值。[en]The main idea of this algorithm is to map the function value to a limited space according to the sigmoid function of logistic regression. The value range of sigmoid function is 0 to 1, so the data can be divided into two categories according to 0 and 1. In the algorithm, we first initialize all the w weights to 1, calculate the error each time, and adjust the w weights according to the error.

  • 每个回归系数初始化为1[en]* each regression coefficient is initialized to 1
  • 重复N次[en]* repeat N times
  • 计算整个数据集合的梯度
  • 使用 $\alpha \cdot \nabla f(x)$ 来更新w向量
  • 返回回归系数
#!/usr/bin/env python
# encoding:utf-8

import math
import numpy
import time
import matplotlib.pyplot as plt

def sigmoid(x):
    return 1.0 / (1 + numpy.exp(-x))

def loadData():
    dataMat = []
    laberMat = []
    with open("test.txt", 'r') as f:
        for line in f.readlines():
            arry = line.strip().split()
            dataMat.append([1.0, float(arry[0]), float(arry[1])])
            laberMat.append(float(arry[2]))
    return numpy.mat(dataMat), numpy.mat(laberMat).transpose()

def gradAscent(dataMat, laberMat, alpha=0.001, maxCycle=500):
    """general gradscent"""
    start_time = time.time()
    m, n = numpy.shape(dataMat)
    weights = numpy.ones((n, 1))
    for i in range(maxCycle):
        h = sigmoid(dataMat * weights)
        error = laberMat - h
        weights += alpha * dataMat.transpose() * error
    duration = time.time() - start_time
    print "duration of time:", duration
    return weights

def stocGradAscent(dataMat, laberMat, alpha=0.01):
    start_time = time.time()
    m, n = numpy.shape(dataMat)
    weights = numpy.ones((n, 1))
    for i in range(m):
        h = sigmoid(dataMat[i] * weights)
        error = laberMat[i] - h
        weights += alpha * dataMat[i].transpose() * error
    duration = time.time() - start_time
    print "duration of time:", duration
    return weights

def betterStocGradAscent(dataMat, laberMat, alpha=0.01, numIter=150):
    """better one, use a dynamic alpha"""
    start_time = time.time()
    m, n = numpy.shape(dataMat)
    weights = numpy.ones((n, 1))
    for j in range(numIter):
        for i in range(m):
            alpha = 4 / (1 + j + i) + 0.01
            h = sigmoid(dataMat[i] * weights)
            error = laberMat[i] - h
            weights += alpha * dataMat[i].transpose() * error
    duration = time.time() - start_time
    print "duration of time:", duration
    return weights
    start_time = time.time()

def show(dataMat, laberMat, weights):
    m, n = numpy.shape(dataMat)
    min_x = min(dataMat[:, 1])[0, 0]
    max_x = max(dataMat[:, 1])[0, 0]
    xcoord1 = []; ycoord1 = []
    xcoord2 = []; ycoord2 = []
    for i in range(m):
        if int(laberMat[i, 0]) == 0:
            xcoord1.append(dataMat[i, 1]); ycoord1.append(dataMat[i, 2])
        elif int(laberMat[i, 0]) == 1:
            xcoord2.append(dataMat[i, 1]); ycoord2.append(dataMat[i, 2])
    fig = plt.figure()
    ax = fig.add_subplot(111)
    ax.scatter(xcoord1, ycoord1, s=30, c="red", marker="s")
    ax.scatter(xcoord2, ycoord2, s=30, c="green")
    x = numpy.arange(min_x, max_x, 0.1)
    y = (-weights[0] - weights[1]*x) / weights[2]
    ax.plot(x, y)
    plt.xlabel("x1"); plt.ylabel("x2")
    plt.show()

if __name__ == "__main__":
    dataMat, laberMat = loadData()
    #weights = gradAscent(dataMat, laberMat, maxCycle=500)
    #weights = stocGradAscent(dataMat, laberMat)
    weights = betterStocGradAscent(dataMat, laberMat, numIter=80)
    show(dataMat, laberMat, weights)

未优化程序的结果如下[en]The results of the unoptimized programs are as follows

logistic回归

随机梯度上升算法(减少迭代次数,算法速度快,但结果不够准确)的结果如下[en]The result of the random gradient rising algorithm (reduces the number of iterations, the algorithm is fast, but the result is not accurate enough) is as follows

logistic回归

对$\alpha$进行优化,动态调整步长(同样降低了迭代次数,但是由于代码采用动态调整alpha,提高了结果的准确性),结果如下

Original: https://www.cnblogs.com/coder2012/p/4598913.html
Author: cococo点点
Title: logistic回归

原创文章受到原创版权保护。转载请注明出处:https://www.johngo689.com/6046/

转载文章受原作者版权保护。转载请注明原作者出处!

(0)

大家都在看

最近整理资源【免费获取】:   👉 程序员最新必读书单  | 👏 互联网各方向面试题下载 | ✌️计算机核心资源汇总