Computational Numerical Analysis

Part (1) — Introduction and Overview

In the past couple of semesters at my university, I decided that the mathematics course “Numerical Analysis” was my favorite math course ever. I found that course so close to real world problems, and it tried to solve those problems in the most efficient, yet realistic ways. What I liked the most about it is that it is the perfect combination of theory and applied science, and so I decided to explore what is beyond the presented in my undergraduate course.

But what is Numerical Analysis, exactly? What does it study? Honestly, when it comes to definitions, I like to go directly to wikipedia. It says that “Numerical Analysis is the study of algorithms that use numerical approximation (as opposed to general symbolic manipulations) for the problems of mathematical analysis (as distinguished from discrete mathematics)”.

So, this series of blog posts will be devoted to try to explain some of the topics that Numerical Analysis studies, and hopefully present them in an introductory way to Machine Learning (as this series of topics are in the core of theoretical Machine Learning research).

In this first article, I will present some of the topics that is studied by numerical analysis, and provide an overview on what to expect in the coming articles. I will also provide you with a list of all the references that I will be using in developing the material. I want to point out, that I am still a beginner, and that I decided to write these posts, in order to learn more about the topics in this field. If you found anything wrong, or you think of any suggestion that will make this a better learning experience please do not hesitate to contact me on any social network you find me on.

Introduction to Numerical Analysis

Why do we study that?

Generally, Numerical Analysis is a very broad topic, and it studies a lot of different sub-fields that are very important in solving numerical real life problems. Some of the topics that are studied includes:

Convergence and Series Acceleration:
This includes studying the rate of convergence in a sequence (that is the speed at which a convergent sequence approaches its limit), local and super convergence, etc.

Discretization:
Which is an extremely interesting topic that studies the process of converting continuous functions and mathematical models into discrete components. This is widely used in signal processing, system theory, and in control systems.

Solving Systems of Linear Equations:
This might be one of the most famous topics in Numerical Analysis. It basically studies methods that can solve systems of linear equations. These methods include the Gaussian Elimination, LU Decomposition, and many other methods and techniques.

Polynomial Interpolation and Extrapolation:
Interpolation is a method of getting new points within the range of a discrete set of known points. Extrapolation is the process of estimating, beyond the original observation range, the value of a variable on the basis of its relationship with another variable. Both topics are one of the most important topics not just in Numerical Analysis, but also in Machine Learning and Neural Networks.

Optimization:
Mathematical Optimization is the study of optimizing a specific function relative to a set of variables and constraints. Optimization can be easily looked at in your Calculus course, where you were asked to find the maximum or the minimum of a function. This topic is one of the most studied and used ideas in applied research and in solving real world problems. Every company, investment firm, factory, and many others wants to minimize or maximize either their yield, profit, productivity. Optimization problems are everywhere!

That’s not it! There are tons of other numerical analysis topics that are being studied in research centers and by companies and industry. Here is a list of Numerical Analysis topics from wikipedia: https://en.wikipedia.org/wiki/List_of_numerical_analysis_topics#Optimization

As you saw, Numerical Analysis, and the topics studied by it, is needed literally everywhere. Whether it is in industry or academia or anywhere! One of the biggest issues in our educational system (I’m talking about the Arabic world, and specifically in Jordan, but this might be everywhere) is that we take a lot of our courses in university or high school, without knowing “WHY” we take them. In my years in the university, one the most thing that I heard from other students was “Oh I hope they remove Calculus 2 from our curriculum” or “There is no need for Linear Algebra, Physics, or Applied Probability in our curriculum! Why do we even have that!”. And this is bad. Very bad. In this series, I will present a humble try of mine, to correct these ideas, and show that mathematics is one of the most important sciences in the 21st century. I will try to present a theoretical introduction followed by an applied perspective of the same concept, to show that theory and application are very much alike, and that they complete each other.

Overview

What is the curriculum gonna look like?

As I already mentioned, Numerical Analysis is a very broad topic, so in the upcoming series of posts, I will only cover some of the topics that I mentioned above. The following selection of topics is not final, and it is subject to change, but I hope it’ll be as close as possible to this.

First of all, I will start with Approximation Theory. Where we try to study how we can approximate complex functions with functions that are much simpler. We will talk about the Fourier Transformation, and we will talk about the Universal Approximation Theory. These topic will be the first part of a theoretical introduction to Machine Learning.

From there, I am going to move to Polynomial Interpolation and Extrapolation, I will talk in depth about the methods of them, the recursive (very interesting 😍) and the iterative methods, and how can we computationally implement that. We will follow some examples on that and try to solve different problems with interpolation and extrapolation.

After that, we will go to Optimization, and we will check out very cool algorithms and techniques. We will checkout Dynamic programming, Linear programming, Gradient Descent, Stochastic Gradient Descent and finally try to understand -together- the most powerful optimization algorithm, Adam.

Most probably, the topics mentioned above are ones that I am sure I will study and write on and anything other than that, will be something extra, where while reading some book or paper I found an interesting topic or concept that I would like to share with you, I will do that immediately!

I would like to note that all of the programs that we will write will be either in C++ or in Python. The reason for that is that I am most comfortable with those two languages, but you can always implement all the concepts in your favorite programming language.

Prerequisites

Although I will try to assume that on mathematical background is present, but being exposed to a previous one or two Calculus courses will definitely help. Of course, if you don’t know calculus, then do not worry about it.

The only thing that might be required is an advanced knowledge in programming. Since this series will be devoted -hopefully- to the computational applications of the topics presented. Programming language is not a constraint here, so any language is fine. Just make sure to at least have the mindset of a programmer or a computer scientist.

And of course, you will need the spirit of a learner. You want to learn, and that is why you are reading this series!

List of References

Currently, I am working on collecting and curating a list of structured Computational Numerical Analysis self studying curriculum, but it is still not complete. However, I will share with you the list of books and other references that I am following in creating this series. Any other reference that is used and not listed here will be listed in the original article of usage.

• Numerical Recipes: The Art of Scientific Computing, William H. Press et. al, Cambridge.
• Introduction to Numerical Analysis [2nd Edition], J. Stoer et. al, Springer-Verlag.
• Numerical Optimization [2nd Edition], Jorge Nocedal et. al, Springer.
• 3Blue1Brown: A youtube channel that presents several mathematical ideas in a very beautiful and visual way.

Finally, I hope you will like this journey. As I mentioned, I am still a beginner in all of this, and I wish we could learn more, together, and make an impact in our world and fields.