Optimization Theory

By Eleftherios Garyfallidis
284
2026

Description

Optimization is central to any problem involving decision making, whether in engineering or in economics. The task of decision making entails choosing between various alternatives. This choice is governed by our desire to make the best decision. The measure of goodness of the alternatives is described by an objective function or performance index. Optimization theory and methods deal with selecting the best alternative in the sense of the given objective function. The area of optimization has received enormous attention in recent years, primarily because of the rapid progress in computer technology, including the development and availability of user-friendly software, high-speed and parallel processors, and artificial neural networks. A clear example of this phenomenon is the wide accessibility of optimization software tools such as the Optimization Toolbox of MATLAB and the many other commercial software packages. There are currently several excellent graduate textbooks on optimization theory and methods as well as undergraduate textbooks on the subject with an emphasis on engineering design. However, there is a need for an introductory textbook on optimization theory and methods at a senior undergraduate or beginning graduate level. The present text was written with this goal in mind. Some of the exercises require using MATLAB. The student edition of MATLAB is sufficient for all of the MATLAB exercises included in the text. The MATLAB source listings for the MATLAB exercises are also included in the solutions manual. The purpose of the book is to give the reader a working knowledge of optimization theory and methods. To accomplish this goal include many examples that illustrate the theory and algorithms discussed in the text. However, it is not our intention to provide a cookbook of the most recent numerical techniques for optimization; rather, our goal is to equip the reader with sufficient background for further study of advanced topics in optimization. The field of optimization is still a very active research area. In recent years, various new approaches to optimization have been proposed. In this text, it has been tried to reflect at least some of the flavor of recent activity in the area.

About Author

Dr. Eleftherios Garyfallidis holds the position of Associate Professor of Intelligent Systems Engineering (ISE) of Indiana University (IU) School of Informatics, Computing and Engineering. He is also the founder and scientific lead of Diffusion Imaging in Python (DIPY), currently the largest open source project in the development of diffusion MRI methods. Diffusion MRI is a unique non-invasive MRI technique that is used to study the structural connectivity of the brain. Dr. Garyfallidis holds a PhD from the University of Cambridge, UK, and worked as a Postdoctoral researcher with Professor Maxime Descoteaux at the Sherbrooke Connectivity Imaging Lab (SCIL) of the University of Sherbrooke, CA. Dr. Garyfallidis has performed research and development at nearly all levels of diffusion MRI analysis. Recently, he started focusing more on the problems of segmentation and registration of tractography. Some of his most known inventions are Quick Bundles and Streamline-based Linear Registration (SLR). Prof. Garyfallidis is leading this group for Neuroengineering at ISE specializing in the developing of new methods and intelligent algorithms for medical imaging and brain mapping with applications to research, clinic and industry.

Table of Content

Preface
Chapter 1. Introduction to Optimization

Definition of Optimization
Types of Optimization Problems
Objective Function
Constructing Loss and Objective Functions
Computational Optimization Techniques
Applications of Optimization Theory
Constraints and Feasibility
Convex vs. Non-Convex Problems
Optimization in Real Life

Chapter 2. Linear Programming (LP)
Formulation of LP Problems
Graphical Method
Simplex Method
Duality in Linear Programming
Sensitivity Analysis
Applications of Linear Programming

Chapter 3. Nonlinear Programming (NLP)
Nonlinear Objective Functions
Karush–Kuhn–Tucker (KKT) Conditions
Gradient and Hessian
Convex NLP
Penalty and Barrier Methods
Common NLP Algorithms

Chapter 4. Convex Optimization
Convex Sets and Properties
Convex Optimization Problem
Optimality Conditions
Duality in Convex Optimization
Interior-Point Methods

Chapter 5. Unconstrained Optimization
Steepest Descent Method
Newton’s Method
Quasi-Newton Methods
Line Search Techniques
Conjugate Gradient Method
Trust Region Methods
Termination Criteria

Chapter 6. Constrained Optimization
Equality and Inequality Constraints 
Lagrange Multipliers
Sequential Quadratic Programming
Augmented Lagrangian Method
Feasible Direction Method
KKT Conditions

Chapter 7. Integer and Combinatorial Optimization
Integer Linear Programming
Branch and Bound
Cutting Plane Method
Dynamic Programming
Traveling Salesman Problem
Knapsack Problem

Chapter 8. Multi-Objective Optimization
Pareto Optimality
Weighted Sum Method
Goal Programming
Evolutionary Multi-Objective Algorithms
Chapter 9. Global Optimization
Local vs Global Optima
Deterministic Global Optimization
Simulated Annealing
Genetic Algorithms
Particle Swarm Optimization
Basin-Hopping Algorithm
Bibliography
Index