Simple histogram modeling examples

Foreword

This notebook gives examples of histogram-based statistical modeling and classification, and assumes that the definitions in the notebook "statistical_modeling.nb" have already been evaluated. All user input sections have been highlighted as cell boxes.

Histogram modeling is a form of non-parametric modeling. Histogram models are very simple.They first quantize the space of possible data vectors into different regions (bins) and then simply compute the frequency of occurrence in the training data for each region.Vector-quantization,a data-based method of quantizing a real space into discrete regions,often accompanies histogram-based statistical models,although other quantization schemes are also possible.

Global user definitions

Display colors

Colors representing two classes

class1Color = Red;
class2Color = Blue;
centroidColor  =  Magenta;

Default point style definitions

style1  =  {class1Color, 
          PointSize[ .02]};
style2  =  {class2Color, 
          PointSize[ .02]};
centroidStyle  =  {centroidColor, PointSize[ .02]};

Default plot range

pr  =  {{-1, 1}, {-1, 1}};

Example A: Uniform distributions

Generate data from two uniform classes

Number of points in each class

n1  =  100;
n2  =  100;

Generate data

data1  =  
        Table[{ru2[{pr[[1, 1]], 0}], 
            ru2[pr[[2]]]}, {n1}];
data2  =  
        Table[{ru2[{0, pr[[1, 2]]}], 
            ru2[pr[[2]]]}, {n2}];

Visualize data

gd1  =  
        PlotData2D[data1, pr, style1, nS];
gd2  =  
        PlotData2D[data2, pr, style2, nS];
gd  =  
        Show[gd1, gd2,  yS,  FrameTicks -> None];

Uniform quantization

Number of centroids in each direction

nCentroids  =  4;

Compute uniformly distributed centroids

uniformCentroids  =  
        UniformQuantization[pr[[1]], pr[[2]], 
          nCentroids];

Visualize data

gCentroids =  
        PlotData2D[uniformCentroids, pr,  centroidStyle,  
          FrameTicks -> None];

Histograms for two classes

Compute histograms based on uniform centroids

h1  = Histogram[data1, uniformCentroids, pr, 50];
h2  = Histogram[data2, uniformCentroids, pr, 50];

Visualize two histograms

gh1  = 
        DrawHistogram[h1, pr,  nS,  
          Axes -> False];
gh2  = 
        DrawHistogram[h2, pr,  nS,  
          Axes -> False];
Show[GraphicsArray[{gh1, gh2}], yS,  imSize];

Classification of input space

gdnoFrame  =  
        Show[gd,  Frame -> False,  nS];
gclass  =  
        DrawHistogramClass[{h1, h2},  {class1Color, class2Color}, 
          nS];
Show[GraphicsArray[{gdnoFrame, gclass}], yS,  imSize];

Example B different uniform distributions

Generate data from two non-rectangular uniform classes

Number of points in each class

n1  =  100;
n2  =  100;

Specialized data generation

SpecialDataGeneration[ 
          flag_] := 
Module[{x, 
            y}, 
{x, y}  =  
            r2[1]; 
While[-flag*x  >  
              flag * y,  {x, y}  =  r2[1]]; {x, y}];

Generate data

data1  =  Table[SpecialDataGeneration[1], {n1}];
data2  =  Table[SpecialDataGeneration[-1],  {n2}];

Visualize data

gd1  =  
        PlotData2D[data1, pr, style1, nS];
gd2  =  
        PlotData2D[data2, pr, style2, nS];
gd  =  
        Show[gd1, gd2,  yS,  FrameTicks -> None];

Uniform quantization

Number of centroids in each direction

nCentroids  =  3;

Compute uniformly distributed centroids

uniformCentroids  =  
        UniformQuantization[pr[[1]], pr[[2]], 
          nCentroids];

Visualize data

gCentroids =  
        PlotData2D[uniformCentroids, pr,  centroidStyle,  
          FrameTicks -> None];

Histograms for two classes

Compute histograms based on uniform centroids

h1  = Histogram[data1, uniformCentroids, pr, 50];
h2  = Histogram[data2, uniformCentroids, pr, 50];

Visualize two histograms

gh1  = 
        DrawHistogram[h1, pr,  nS, 
          Axes -> False];
gh2  = 
        DrawHistogram[h2, pr,  nS, 
          Axes -> False];
Show[GraphicsArray[{gh1, gh2}], yS,  imSize];

Classification of input space

gdnoFrame  =  
        Show[gd,  Frame -> False,  nS];
gclass  =  
        DrawHistogramClass[{h1, h2},  {class1Color, class2Color}, 
          nS];
Show[GraphicsArray[{gdnoFrame, gclass}], yS,  imSize];

Data-dependent quantization (vector quantization)

Number of centroids for each class ((2^m))

m  =  2;

Compute data-dependent centroids

vqCentroids1 =  VQ[data1, m];
vqCentroids2  = VQ[data2, m];
vqCentroids  =  vqCentroids1~Join~vqCentroids2;

Visualize data

gCentroids =  
        PlotData2D[vqCentroids, pr,  centroidStyle,  
          FrameTicks -> None];

Visualize resultant partitioning of input space

gpart1  =  
        DrawPartitioning[vqCentroids,  pr, nS];
Show[gpart1, gCentroids, yS,  
        FrameTicks -> None];

New histograms for two classes

Compute histograms based on data-dependent centroids

h1  = Histogram[data1, vqCentroids, pr, 50];
h2  = Histogram[data2, vqCentroids, pr, 50];

Visualize two histograms

gh1  = 
        DrawHistogram[h1, pr,  nS,  
          Axes -> False];
gh2  = 
        DrawHistogram[h2, pr,  nS,  
          Axes -> False];
Show[GraphicsArray[{gh1, gh2}], yS,  imSize];

New classification of input space

gdnoFrame  =  
        Show[gd,  Frame -> False,  nS];
gclass  =  
        DrawHistogramClass[{h1, h2},  {class1Color, class2Color}, 
          nS];
Show[GraphicsArray[{gdnoFrame, gclass}], yS,  imSize];

Example C: Gaussian distributions

Generate data from two Gaussian classes

Definition of Gaussians

μ1  =  { .3,  .3};
Σ1  =   .07 {{1. , - .5}, {- .5,  .5}};

μ2  =  {- .5, - .5};
Σ2  =   .04 {{1. , - .5}, {- .5,  .5}};

Number of points in each class

n1  =  100;
n2  =  100;

Generate data

data1  =  
        Table[rgBounded[{μ1,  Σ1}, pr], {n1}];
data2  =  
        Table[rgBounded[{μ2,  Σ2}, 
            pr], {n2}];

Visualize data

gd1  =  
        PlotData2D[data1, pr, style1, nS];
gd2  =  
        PlotData2D[data2, pr, style2, nS];
gd  =  
        Show[gd1, gd2,  yS,  FrameTicks -> None];

Uniform quantization

Number of centroids in each direction

nCentroids  =  4;

Compute uniformly distributed centroids

uniformCentroids  =  
        UniformQuantization[pr[[1]], pr[[2]], 
          nCentroids];

Visualize data

gCentroids =  
        PlotData2D[uniformCentroids, pr,  centroidStyle, 
          FrameTicks -> None];

Histograms for two classes

Compute histograms based on uniform centroids

h1  = Histogram[data1, uniformCentroids, pr, 50];
h2  = Histogram[data2, uniformCentroids, pr, 50];

Visualize two histograms

gh1  = 
        DrawHistogram[h1, pr,  nS,  
          Axes -> False];
gh2  = 
        DrawHistogram[h2, pr,  nS,  
          Axes -> False];
Show[GraphicsArray[{gh1, gh2}], yS,  imSize];

Classification of input space

gdnoFrame  =  
        Show[gd,  Frame -> False,  nS];
gclass  =  
        DrawHistogramClass[{h1, h2},  {class1Color, class2Color}, 
          nS];
Show[GraphicsArray[{gdnoFrame, gclass}], yS,  imSize];

Data-dependent quantization (vector quantization)

Number of centroids for each class ((2^m))

m  =  3;

Compute data-dependent centroids

vqCentroids1 =  VQ[data1, m];
vqCentroids2  = VQ[data2, m];
vqCentroids  =  vqCentroids1~Join~vqCentroids2;

Visualize data

gCentroids =  
        PlotData2D[vqCentroids, pr,  centroidStyle,  
          FrameTicks -> None];

Visualize resultant partitioning of input space

gpart1  =  
        DrawPartitioning[vqCentroids,  pr, nS];
Show[gpart1, gCentroids, yS,  
        FrameTicks -> None];

New histograms for two classes

Compute histograms based on data-dependent centroids

h1  = Histogram[data1, vqCentroids, pr, 50];
h2  = Histogram[data2, vqCentroids, pr, 50];

Visualize two histograms

gh1  = 
        DrawHistogram[h1, pr,  nS, 
          Axes -> False];
gh2  = 
        DrawHistogram[h2, pr,  nS, 
          Axes -> False];
Show[GraphicsArray[{gh1, gh2}], yS,  imSize];

New classification of input space

gdnoFrame  =  
        Show[gd,  Frame -> False,  nS];
gclass  =  
        DrawHistogramClass[{h1, h2},  {class1Color, class2Color}, 
          nS];
Show[GraphicsArray[{gdnoFrame, gclass}], yS,  imSize];

Example D: different Gaussian distributions

Generate data from two Gaussian classes

Definition of Gaussians

μ1  =  { .3,  .3};
Σ1  =   .07 {{1. , - .5}, {- .5,  .5}};

μ2  =  {- .5, - .5};
Σ2  =   .06 {{1. ,  .5}, { .5,  .5}};

Number of points in each class

n1  =  100;
n2  =  100;

Generate data

data1  =  
        Table[rgBounded[{μ1,  Σ1}, pr], {n1}];
data2  =  
        Table[rgBounded[{μ2,  Σ2}, 
            pr], {n2}];

Visualize data

gd1  =  
        PlotData2D[data1, pr, style1, nS];
gd2  =  
        PlotData2D[data2, pr, style2, nS];
gd  =  
        Show[gd1, gd2,  yS,  FrameTicks -> None];

Uniform quantization

Number of centroids in each direction

nCentroids  =  4;

Compute uniformly distributed centroids

uniformCentroids  =  
        UniformQuantization[pr[[1]], pr[[2]], 
          nCentroids];

Visualize data

gCentroids =  
        PlotData2D[uniformCentroids, pr,  centroidStyle];

Histograms for two classes

Compute histograms based on uniform centroids

h1  = Histogram[data1, uniformCentroids, pr, 50];
h2  = Histogram[data2, uniformCentroids, pr, 50];

Visualize two histograms

gh1  = 
        DrawHistogram[h1, pr,  nS, 
          Axes -> False];
gh2  = 
        DrawHistogram[h2, pr,  nS, 
          Axes -> False];
Show[GraphicsArray[{gh1, gh2}], yS,  imSize];

Classification of input space

gdnoFrame  =  
        Show[gd,  Frame -> False,  nS];
gclass  =  
        DrawHistogramClass[{h1, h2},  {class1Color, class2Color}, 
          nS];
Show[GraphicsArray[{gdnoFrame, gclass}], yS,  imSize];

Data-dependent quantization (vector quantization)

Number of centroids for each class ((2^m))

m  =  3;

Compute data-dependent centroids

vqCentroids1 =  VQ[data1, m];
vqCentroids2  = VQ[data2, m];
vqCentroids  =  vqCentroids1~Join~vqCentroids2;

Visualize data

gCentroids =  
        PlotData2D[vqCentroids, pr,  centroidStyle,  
          FrameTicks -> None];

Visualize resultant partitioning of input space

gpart1  =  
        DrawPartitioning[vqCentroids,  pr, nS];
Show[gpart1, gCentroids, yS, FrameTicks -> None];

New histograms for two classes

Compute histograms based on data-dependent centroids

h1  = Histogram[data1, vqCentroids, pr, 50];
h2  = Histogram[data2, vqCentroids, pr, 50];

Visualize two histograms

gh1  = 
        DrawHistogram[h1, pr,  nS, 
          Axes -> False];
gh2  = 
        DrawHistogram[h2, pr,  nS, 
          Axes -> False];
Show[GraphicsArray[{gh1, gh2}], yS,  imSize];

New classification of input space

gdnoFrame  =  
        Show[gd,  Frame -> False,  nS];
gclass  =  
        DrawHistogramClass[{h1, h2},  {class1Color, class2Color}, 
          nS];
Show[GraphicsArray[{gdnoFrame, gclass}], yS,  imSize];