Scipy Lectures Notes   Basic Introduction to Scipy – cluster & optimize modules   Function Optimization With SciPy – Jason Brownlee   Stanford class – Introduction to Scientific Python  Scipy  jac = ‘cs’ – using complex derivative which gives more accurate results than other numerical methods (2-point and 3-point). In order to understand how complex derivative works, … Read more

Fundamental algorithms for scientific computing in Python: Scipy  SciPy User Guide  Scipy Optimize  Scipy minimize: algorithms  scipy.optimize.least_squares: lsq  Optimization and root finding: scipy  Three examples of nonlinear least-squares fitting in Python with SciPy: examples  Scipy Lecture Notes: notes  Python scipy.optimize.leastsq() Examples  minimize(method=’BFGS’): BFGS  Broyden–Fletcher–Goldfarb–Shanno algorithm: wikipedia  Limited-memory BFGS: wikipedia  Large-scale Bound-constrained Optimization: L-BFGS-B Newton’s method in … Read more

In order to clearly understand the algorithms behind ML, it is important to have clear numerical analysis understanding. First a good summary of gradient descent methods are explained in this paper: An overview of gradient descent optimization algorithms – arxiv Then, Michel Bierlaire from the EPFL in Switzerland wrote a good book on optimization, and … Read more