The two algorithms for NMF are based on iterative updates of W and V. To find an approximate factorization for this problem, they defined cost functions that quantify the quality of the approximation. Such a cost function can be constructed using some measure of distance between two non-negative matrices A and B. The first measure used is the square of the Euclidean distance between A and B.
The other one is referred to as “divergence” and not “distance” because it’s not symmetric as the first one.
Both measures are lower bounded by zero, and vanish if and only if A = B.
With this, they considered two alternative formulations of NMF as the following optimization problems:
Prob. 1: Minimize || V – WH ||2 with respect to W and H, subject to the constraints W, H ≥ 0.
Prob. 2: Minimize D (V || WH) with respect to W and H, subject to the constraints W, H ≥ 0.
For solving problems 1 and 2, the following “multiplicative updating rules” are used:
Theorem 1 The Euclidean distance || V – WH ||2 is nonincreasing under the update rules:
The Euclidean distance is invariant under these updates if and only if W and H are at a stationary point of the distance.
Theorem 2 The divergence D (V || WH) is nonincreasing under the update rules:
The divergence is invariant under these updates if and only if W and H are at a stationary point of the divergence.
Dr. Masalmah uses Theorem 1 to update the endmembers matrix in the Gauss-Seidel algorithm.








