Other Optimizers¶

Efficient frontier methods involve the direct optimization of an objective subject to constraints. However, there are some portfolio optimization schemes that are completely different in character. PyPortfolioOpt provides support for these alternatives, while still giving you access to the same pre and post-processing API.

Note

As of v0.4, these other optimizers now inherit from BaseOptimizer or BaseConvexOptimizer, so you no longer have to implement pre-processing and post-processing methods on your own. You can thus easily swap out, say, EfficientFrontier for HRPOpt.

Hierarchical Risk Parity (HRP)¶

Hierarchical Risk Parity is a novel portfolio optimization method developed by Marcos Lopez de Prado [1]. Though a detailed explanation can be found in the linked paper, here is a rough overview of how HRP works:

From a universe of assets, form a distance matrix based on the correlation of the assets.
Using this distance matrix, cluster the assets into a tree via hierarchical clustering
Within each branch of the tree, form the minimum variance portfolio (normally between just two assets).
Iterate over each level, optimally combining the mini-portfolios at each node.

The advantages of this are that it does not require the inversion of the covariance matrix as with traditional mean-variance optimization, and seems to produce diverse portfolios that perform well out of sample.

The hierarchical_portfolio module seeks to implement one of the recent advances in portfolio optimization – the application of hierarchical clustering models in allocation.

All of the hierarchical classes have a similar API to EfficientFrontier, though since many hierarchical models currently don’t support different objectives, the actual allocation happens with a call to optimize().

Currently implemented:

HRPOpt implements the Hierarchical Risk Parity (HRP) portfolio. Code reproduced with permission from Marcos Lopez de Prado (2016).

class pypfopt.hierarchical_portfolio.HRPOpt(returns=None, cov_matrix=None)[source]¶

A HRPOpt object (inheriting from BaseOptimizer) constructs a hierarchical risk parity portfolio.

Instance variables:

Inputs
- n_assets - int
- tickers - str list
- returns - pd.DataFrame
Output:
- weights - np.ndarray
- clusters - linkage matrix corresponding to clustered assets.

Public methods:

optimize() calculates weights using HRP
portfolio_performance() calculates the expected return, volatility and Sharpe ratio for the optimized portfolio.
set_weights() creates self.weights (np.ndarray) from a weights dict
clean_weights() rounds the weights and clips near-zeros.
save_weights_to_file() saves the weights to csv, json, or txt.

__init__(returns=None, cov_matrix=None)[source]¶

Parameters:	returns (pd.DataFrame) – asset historical returns cov_matrix (pd.DataFrame.) – covariance of asset returns
Raises:	TypeError – if `returns` is not a dataframe

optimize(linkage_method='single')[source]¶

Construct a hierarchical risk parity portfolio, using Scipy hierarchical clustering (see here)

Parameters:	linkage_method (str) – which scipy linkage method to use
Returns:	weights for the HRP portfolio
Return type:	OrderedDict

portfolio_performance(verbose=False, risk_free_rate=0.02, frequency=252)[source]¶

After optimising, calculate (and optionally print) the performance of the optimal portfolio. Currently calculates expected return, volatility, and the Sharpe ratio assuming returns are daily

Parameters:	verbose (bool, optional) – whether performance should be printed, defaults to False risk_free_rate (float, optional) – risk-free rate of borrowing/lending, defaults to 0.02. The period of the risk-free rate should correspond to the frequency of expected returns. frequency (int, optional) – number of time periods in a year, defaults to 252 (the number of trading days in a year)
Raises:	ValueError – if weights have not been calculated yet
Returns:	expected return, volatility, Sharpe ratio.
Return type:	(float, float, float)

The Critical Line Algorithm¶

This is a robust alternative to the quadratic solver used to find mean-variance optimal portfolios, that is especially advantageous when we apply linear inequalities. Unlike generic convex optimization routines, the CLA is specially designed for portfolio optimization. It is guaranteed to converge after a certain number of iterations, and can efficiently derive the entire efficient frontier.

Tip

In general, unless you have specific requirements e.g you would like to efficiently compute the entire efficient frontier for plotting, I would go with the standard EfficientFrontier optimizer.

I am most grateful to Marcos López de Prado and David Bailey for providing the implementation [2]. Permission for its distribution has been received by email. It has been modified such that it has the same API, though as of v0.5.0 we only support max_sharpe() and min_volatility().

The cla module houses the CLA class, which generates optimal portfolios using the Critical Line Algorithm as implemented by Marcos Lopez de Prado and David Bailey.

class pypfopt.cla.CLA(expected_returns, cov_matrix, weight_bounds=(0, 1))[source]¶

Instance variables:

Inputs:
- n_assets - int
- tickers - str list
- mean - np.ndarray
- cov_matrix - np.ndarray
- expected_returns - np.ndarray
- lb - np.ndarray
- ub - np.ndarray
Optimization parameters:
- w - np.ndarray list
- ls - float list
- g - float list
- f - float list list
Outputs:
- weights - np.ndarray
- frontier_values - (float list, float list, np.ndarray list)

Public methods:

max_sharpe() optimizes for maximal Sharpe ratio (a.k.a the tangency portfolio)
min_volatility() optimizes for minimum volatility
efficient_frontier() computes the entire efficient frontier
portfolio_performance() calculates the expected return, volatility and Sharpe ratio for the optimized portfolio.
clean_weights() rounds the weights and clips near-zeros.
save_weights_to_file() saves the weights to csv, json, or txt.

__init__(expected_returns, cov_matrix, weight_bounds=(0, 1))[source]¶

Parameters:

expected_returns (pd.Series, list, np.ndarray) – expected returns for each asset. Set to None if optimising for volatility only.
cov_matrix (pd.DataFrame or np.array) – covariance of returns for each asset
weight_bounds (tuple (float, float) or (list/ndarray, list/ndarray) or list(tuple(float, float))) – minimum and maximum weight of an asset, defaults to (0, 1). Must be changed to (-1, 1) for portfolios with shorting.

Raises:

TypeError – if expected_returns is not a series, list or array
TypeError – if cov_matrix is not a dataframe or array

efficient_frontier(points=100)[source]¶

Efficiently compute the entire efficient frontier

Parameters:	points (int, optional) – rough number of points to evaluate, defaults to 100
Raises:	ValueError – if weights have not been computed
Returns:	return list, std list, weight list
Return type:	(float list, float list, np.ndarray list)

max_sharpe()[source]¶

Maximise the Sharpe ratio.

Returns:	asset weights for the max-sharpe portfolio
Return type:	OrderedDict

min_volatility()[source]¶

Minimise volatility.

Returns:	asset weights for the volatility-minimising portfolio
Return type:	OrderedDict

portfolio_performance(verbose=False, risk_free_rate=0.02)[source]¶

After optimising, calculate (and optionally print) the performance of the optimal portfolio. Currently calculates expected return, volatility, and the Sharpe ratio.

Parameters:	verbose (bool, optional) – whether performance should be printed, defaults to False risk_free_rate (float, optional) – risk-free rate of borrowing/lending, defaults to 0.02
Raises:	ValueError – if weights have not been calculated yet
Returns:	expected return, volatility, Sharpe ratio.
Return type:	(float, float, float)

set_weights(_)[source]¶

Utility function to set weights attribute (np.array) from user input

Parameters:	input_weights (dict) – {ticker: weight} dict

Implementing your own optimizer¶

Please note that this is quite different to implementing Custom optimization problems, because in that case we are still using the same convex optimization structure. However, HRP and CLA optimization have a fundamentally different optimization method. In general, these are much more difficult to code up compared to custom objective functions.

To implement a custom optimizer that is compatible with the rest of PyPortfolioOpt, just extend BaseOptimizer (or BaseConvexOptimizer if you want to use cvxpy), both of which can be found in base_optimizer.py. This gives you access to utility methods like clean_weights(), as well as making sure that any output is compatible with portfolio_performance() and post-processing methods.

The base_optimizer module houses the parent classes BaseOptimizer from which all optimizers will inherit. BaseConvexOptimizer is the base class for all cvxpy (and scipy) optimization.

Additionally, we define a general utility function portfolio_performance to evaluate return and risk for a given set of portfolio weights.

class pypfopt.base_optimizer.BaseOptimizer(n_assets, tickers=None)[source]¶

Instance variables:

n_assets - int
tickers - str list
weights - np.ndarray

Public methods:

set_weights() creates self.weights (np.ndarray) from a weights dict
clean_weights() rounds the weights and clips near-zeros.
save_weights_to_file() saves the weights to csv, json, or txt.

__init__(n_assets, tickers=None)[source]¶

Parameters:	n_assets (int) – number of assets tickers (list) – name of assets

clean_weights(cutoff=0.0001, rounding=5)[source]¶

Helper method to clean the raw weights, setting any weights whose absolute values are below the cutoff to zero, and rounding the rest.

Parameters:	cutoff (float, optional) – the lower bound, defaults to 1e-4 rounding (int, optional) – number of decimal places to round the weights, defaults to 5. Set to None if rounding is not desired.
Returns:	asset weights
Return type:	OrderedDict

save_weights_to_file(filename='weights.csv')[source]¶

Utility method to save weights to a text file.

Parameters:	filename (str) – name of file. Should be csv, json, or txt.

set_weights(input_weights)[source]¶

Utility function to set weights attribute (np.array) from user input

Parameters:	input_weights (dict) – {ticker: weight} dict

class pypfopt.base_optimizer.BaseConvexOptimizer(n_assets, tickers=None, weight_bounds=(0, 1), solver=None, verbose=False, solver_options=None)[source]¶

The BaseConvexOptimizer contains many private variables for use by cvxpy. For example, the immutable optimization variable for weights is stored as self._w. Interacting directly with these variables directly is discouraged.

Instance variables:

n_assets - int
tickers - str list
weights - np.ndarray
_opt - cp.Problem
_solver - str
_solver_options - {str: str} dict

Public methods:

add_objective() adds a (convex) objective to the optimization problem
add_constraint() adds a constraint to the optimization problem
convex_objective() solves for a generic convex objective with linear constraints
nonconvex_objective() solves for a generic nonconvex objective using the scipy backend. This is prone to getting stuck in local minima and is generally not recommended.
set_weights() creates self.weights (np.ndarray) from a weights dict
clean_weights() rounds the weights and clips near-zeros.
save_weights_to_file() saves the weights to csv, json, or txt.

__init__(n_assets, tickers=None, weight_bounds=(0, 1), solver=None, verbose=False, solver_options=None)[source]¶

Parameters:

weight_bounds (tuple OR tuple list, optional) – minimum and maximum weight of each asset OR single min/max pair if all identical, defaults to (0, 1). Must be changed to (-1, 1) for portfolios with shorting.
solver (str, optional. Defaults to "ECOS") – name of solver. list available solvers with: cvxpy.installed_solvers()
verbose (bool, optional) – whether performance and debugging info should be printed, defaults to False
solver_options (dict, optional) – parameters for the given solver

_map_bounds_to_constraints(test_bounds)[source]¶

Convert input bounds into a form acceptable by cvxpy and add to the constraints list.

Parameters:	test_bounds (tuple OR list/tuple of tuples OR pair of np arrays) – minimum and maximum weight of each asset OR single min/max pair if all identical OR pair of arrays corresponding to lower/upper bounds. defaults to (0, 1).
Raises:	TypeError – if `test_bounds` is not of the right type
Returns:	bounds suitable for cvxpy
Return type:	tuple pair of np.ndarray

_solve_cvxpy_opt_problem()[source]¶

Helper method to solve the cvxpy problem and check output, once objectives and constraints have been defined

Raises:	exceptions.OptimizationError – if problem is not solvable by cvxpy

add_constraint(new_constraint)[source]¶

Add a new constraint to the optimization problem. This constraint must satisfy DCP rules, i.e be either a linear equality constraint or convex inequality constraint.

Examples:

ef.add_constraint(lambda x : x[0] == 0.02)
ef.add_constraint(lambda x : x >= 0.01)
ef.add_constraint(lambda x: x <= np.array([0.01, 0.08, ..., 0.5]))

Parameters:	new_constraint – the constraint to be added

add_objective(new_objective, **kwargs)[source]¶

Add a new term into the objective function. This term must be convex, and built from cvxpy atomic functions.

Example:

def L1_norm(w, k=1):
    return k * cp.norm(w, 1)

ef.add_objective(L1_norm, k=2)

Parameters:	new_objective (cp.Expression (i.e function of cp.Variable)) – the objective to be added

add_sector_constraints(sector_mapper, sector_lower, sector_upper)[source]¶

Adds constraints on the sum of weights of different groups of assets. Most commonly, these will be sector constraints e.g portfolio’s exposure to tech must be less than x%:

sector_mapper = {
    "GOOG": "tech",
    "FB": "tech",,
    "XOM": "Oil/Gas",
    "RRC": "Oil/Gas",
    "MA": "Financials",
    "JPM": "Financials",
}

sector_lower = {"tech": 0.1}  # at least 10% to tech
sector_upper = {
    "tech": 0.4, # less than 40% tech
    "Oil/Gas": 0.1 # less than 10% oil and gas
}

Parameters:	sector_mapper ({str: str} dict) – dict that maps tickers to sectors sector_lower ({str: float} dict) – lower bounds for each sector sector_upper ({str:float} dict) – upper bounds for each sector

convex_objective(custom_objective, weights_sum_to_one=True, **kwargs)[source]¶

Optimize a custom convex objective function. Constraints should be added with ef.add_constraint(). Optimizer arguments must be passed as keyword-args. Example:

# Could define as a lambda function instead
def logarithmic_barrier(w, cov_matrix, k=0.1):
    # 60 Years of Portfolio Optimization, Kolm et al (2014)
    return cp.quad_form(w, cov_matrix) - k * cp.sum(cp.log(w))

w = ef.convex_objective(logarithmic_barrier, cov_matrix=ef.cov_matrix)

Parameters:	custom_objective (function with signature (cp.Variable, kwargs) -> cp.Expression) – an objective function to be MINIMISED. This should be written using cvxpy atoms Should map (w, kwargs) -> float. weights_sum_to_one (bool, optional) – whether to add the default objective, defaults to True
Raises:	OptimizationError – if the objective is nonconvex or constraints nonlinear.
Returns:	asset weights for the efficient risk portfolio
Return type:	OrderedDict

nonconvex_objective(custom_objective, objective_args=None, weights_sum_to_one=True, constraints=None, solver='SLSQP', initial_guess=None)[source]¶

Optimize some objective function using the scipy backend. This can support nonconvex objectives and nonlinear constraints, but may get stuck at local minima. Example:

# Market-neutral efficient risk
constraints = [
    {"type": "eq", "fun": lambda w: np.sum(w)},  # weights sum to zero
    {
        "type": "eq",
        "fun": lambda w: target_risk ** 2 - np.dot(w.T, np.dot(ef.cov_matrix, w)),
    },  # risk = target_risk
]
ef.nonconvex_objective(
    lambda w, mu: -w.T.dot(mu),  # min negative return (i.e maximise return)
    objective_args=(ef.expected_returns,),
    weights_sum_to_one=False,
    constraints=constraints,
)

Parameters:	objective_function (function with signature (np.ndarray, args) -> float) – an objective function to be MINIMISED. This function should map (weight, args) -> cost objective_args (tuple of np.ndarrays) – arguments for the objective function (excluding weight) weights_sum_to_one (bool, optional) – whether to add the default objective, defaults to True constraints (dict list) – list of constraints in the scipy format (i.e dicts) solver (string) – which SCIPY solver to use, e.g “SLSQP”, “COBYLA”, “BFGS”. User beware: different optimizers require different inputs. initial_guess (np.ndarray) – the initial guess for the weights, shape (n,) or (n, 1)
Returns:	asset weights that optimize the custom objective
Return type:	OrderedDict

References¶

[1]	López de Prado, M. (2016). Building Diversified Portfolios that Outperform Out of Sample. The Journal of Portfolio Management, 42(4), 59–69.

[2]	Bailey and Loópez de Prado (2013). An Open-Source Implementation of the Critical-Line Algorithm for Portfolio Optimization