• Mathematics
    VectorsMatricesVector SpaceAlgebraCalculusLogicAnalytic GeometryStatistics & ProbabilityGeometrySpecial Functions
    Physics
    MechanicsUnits & Constants
    Electronics
    Circuits
    Computer Science
    EncodingComputerLanguagesComputer Networks
    Machine Learning
    ClusteringOptimizationRegressionKernels
    AI
    Neural Network
    Finance
    OptionsFixed IncomeMarket Analysis
    Help
    Contact usIndexSearchVersion historyFormula Syntax
    Practice Quiz
    LoginUser
  • Machine Learning
    Optimization: Cross-entropy Cost
    • Clustering
      • Optimization
        • Gradient Descent
        • Cross-entropy Cost
      • Regression
        • Kernels
          Cross-entropy Cost

          Given two probability distributions p and q over the same underlying set of events (domain) X, cross-entropy cost of distribution q relative to a
          distribution p over given set X is defined as     H (p,q) = - Ep[log(q)]   , where Ep[⋅] is an expected value of the distribution p.
          If p and q are discrete distributions, then     H (p,q) = -
          ∑
          x∈X          		p(x) log(q(x))

          If p and q are continuous distributions, then     H (p,q) = -
          ∫
          X          		p(x) log(q(x)) d x


          Please Note:

          • All values are calculated with the precision of 10-15, but are displayed with the precision of 10-9.
          • For the log function, we are using the natural logarithm ln.
          • If a relative distribution (either p(x) or q(x)) has a 0 value for some event, while its corresponding value of the base distribution, for the
            same event, is not 0, then this will cause the cross-entropy cost to be undefined, so such case will be omitted.
          • Multiple values for xi will be flagged as error.
          • If uploading values for xi, p(xi) and q(xi) from a file, the file must be an ASCII file. Values for xi must be in the first row of the file, p(xi) in the
            second, and q(xi) in the third row. The individual numbers in the row must be separated by either comma ( , ), semicolon ( ; ) or pipe ( | ) character.
          • If if you are calculating cross-entropy cost for the continuous distributions, please check the proper syntax for writing the formulas specifying the
            probability density functions p(x) and q(x)
          Distributions p and q are
          Please specify the events xi and their corresponding probabilities p(xi) and q(xi) :
          Events xi are specified by their
          xi
          p(xi)
          q(xi)
          Please specify the probability density functions for probability distributions p(x) and q(x) :
          p(X)=
          q(X)=
          Please specify the domain (a,b) of random variable X:
          from


          to


          Result:
          H (p , q)=0
          H (q , p)=0
          See Also:     Softmax & LogSumExp functions