Efficient Frontier Optimisation

The implementation of efficient frontier optimisation in PyPortfolioOpt is separated into the objective_functions and efficient_frontier modules. It was designed this way because in my mind there is a clear conceptual separation between the optimisation objective and the actual optimisation method – if we wanted to use something other than mean-variance optimisation via quadratic programming, these objective functions would still be applicable.

It should be noted that while efficient frontier optimisation is technically a very specific method, I tend to use it as a blanket term (interchangeably with mean-variance optimisation) to refer to anything similar, such as minimising variance.

Optimisation

PyPortfolioOpt uses scipy.optimize. I realise that most python optimisation projects use cvxopt instead, but I do think that scipy.optimize is far cleaner and much more readable (as per the Zen of Python, “Readability counts”). That being said, scipy.optimize arguably has worse documentation, though ultimately I felt that it was intuitive enough to justify the lack of explained examples. Because they are both based on LAPACK, I don’t see why performance should differ significantly, but if it transpires that cvxopt is faster by an order of magnitude, I will definitely consider switching.

Tip

If you would like to plot the efficient frontier, take a look at the The Critical Line Algorithm.

The efficient_frontier module houses the EfficientFrontier class, which generates optimal portfolios for various possible objective functions and parameters.

class pypfopt.efficient_frontier.EfficientFrontier(expected_returns, cov_matrix, weight_bounds=(0, 1), gamma=0)

An EfficientFrontier object (inheriting from BaseScipyOptimizer) contains multiple optimisation methods that can be called (corresponding to different objective functions) with various parameters.

Instance variables:

  • Inputs:

    • cov_matrix
    • n_assets
    • tickers
    • bounds
  • Optimisation parameters:

    • initial_guess
    • constraints
  • Output: weights

Public methods:

  • max_sharpe() optimises for maximal Sharpe ratio (a.k.a the tangency portfolio)
  • min_volatility() optimises for minimum volatility
  • custom_objective() optimises for some custom objective function
  • efficient_risk() maximises Sharpe for a given target risk
  • efficient_return() minimises risk for a given target return
  • portfolio_performance() calculates the expected return, volatility and Sharpe ratio for the optimised portfolio.
__init__(expected_returns, cov_matrix, weight_bounds=(0, 1), gamma=0)
Parameters:
  • expected_returns (pd.Series, list, np.ndarray) – expected returns for each asset. Set to None if optimising for volatility only.
  • cov_matrix (pd.DataFrame or np.array) – covariance of returns for each asset
  • weight_bounds (tuple, optional) – minimum and maximum weight of an asset, defaults to (0, 1). Must be changed to (-1, 1) for portfolios with shorting.
  • gamma (float, optional) – L2 regularisation parameter, defaults to 0. Increase if you want more non-negligible weights
Raises:
  • TypeError – if expected_returns is not a series, list or array
  • TypeError – if cov_matrix is not a dataframe or array

Note

As a rule of thumb, any parameters that can apply to all optimisers are instance variables (passed when you are initialising the object).

efficient_return(target_return, market_neutral=False)

Calculate the ‘Markowitz portfolio’, minimising volatility for a given target return.

Parameters:
  • target_return (float) – the desired return of the resulting portfolio.
  • market_neutral (bool, optional) – whether the portfolio should be market neutral (weights sum to zero), defaults to False. Requires negative lower weight bound.
Raises:

ValueError – if target_return is not a positive float

Returns:

asset weights for the Markowitz portfolio

Return type:

dict

efficient_risk(target_risk, risk_free_rate=0.02, market_neutral=False)

Calculate the Sharpe-maximising portfolio for a given volatility (i.e max return for a target risk).

Parameters:
  • target_risk (float) – the desired volatility of the resulting portfolio.
  • risk_free_rate (float, optional) – risk-free rate of borrowing/lending, defaults to 0.02. The period of the risk-free rate should correspond to the frequency of expected returns.
  • market_neutral – whether the portfolio should be market neutral (weights sum to zero), defaults to False. Requires negative lower weight bound.
  • market_neutral – bool, optional
Raises:
  • ValueError – if target_risk is not a positive float
  • ValueError – if risk_free_rate is non-numeric
Returns:

asset weights for the efficient risk portfolio

Return type:

dict

max_sharpe(risk_free_rate=0.02)

Maximise the Sharpe Ratio. The result is also referred to as the tangency portfolio, as it is the tangent to the efficient frontier curve that intercepts the risk-free rate.

Parameters:risk_free_rate (float, optional) – risk-free rate of borrowing/lending, defaults to 0.02. The period of the risk-free rate should correspond to the frequency of expected returns.
Raises:ValueError – if risk_free_rate is non-numeric
Returns:asset weights for the Sharpe-maximising portfolio
Return type:dict
min_volatility()

Minimise volatility.

Returns:asset weights for the volatility-minimising portfolio
Return type:dict
portfolio_performance(verbose=False, risk_free_rate=0.02)

After optimising, calculate (and optionally print) the performance of the optimal portfolio. Currently calculates expected return, volatility, and the Sharpe ratio.

Parameters:
  • verbose (bool, optional) – whether performance should be printed, defaults to False
  • risk_free_rate (float, optional) – risk-free rate of borrowing/lending, defaults to 0.02. The period of the risk-free rate should correspond to the frequency of expected returns.
Raises:

ValueError – if weights have not been calcualted yet

Returns:

expected return, volatility, Sharpe ratio.

Return type:

(float, float, float)

Caution

If you pass an unreasonable target into efficient_risk() or efficient_return(), the optimiser will fail silently and return weird weights. Caveat emptor applies!

Objective functions

The objective_functions module provides optimisation objectives, including the actual objective functions called by the EfficientFrontier object’s optimisation methods. These methods are primarily designed for internal use during optimisation (via scipy.optimize), and each requires a certain signature (which is why they have not been factored into a class). For obvious reasons, any objective function must accept weights as an argument, and must also have at least one of expected_returns or cov_matrix.

Because scipy.optimize only minimises, any objectives that we want to maximise must be made negative.

Currently implemented:

  • negative mean return
  • (regularised) negative Sharpe ratio
  • (regularised) volatility
  • CVaR (expected shortfall)
pypfopt.objective_functions.negative_cvar(weights, returns, s=10000, beta=0.95, random_state=None)

Calculate the negative CVaR. Though we want the “min CVaR portfolio”, we actually need to maximise the expected return of the worst q% cases, thus we need this value to be negative.

Parameters:
  • weights (np.ndarray) – asset weights of the portfolio
  • returns (pd.DataFrame or np.ndarray) – asset returns
  • s (int, optional) – number of bootstrap draws, defaults to 10000
  • beta (float, optional) – “significance level” (i. 1 - q), defaults to 0.95
  • random_state (int, optional) – seed for random sampling, defaults to None
Returns:

negative CVaR

Return type:

float

pypfopt.objective_functions.negative_mean_return(weights, expected_returns)

Calculate the negative mean return of a portfolio

Parameters:
  • weights (np.ndarray) – asset weights of the portfolio
  • expected_returns (pd.Series) – expected return of each asset
Returns:

negative mean return

Return type:

float

pypfopt.objective_functions.negative_sharpe(weights, expected_returns, cov_matrix, gamma=0, risk_free_rate=0.02)

Calculate the negative Sharpe ratio of a portfolio

Parameters:
  • weights (np.ndarray) – asset weights of the portfolio
  • expected_returns (pd.Series) – expected return of each asset
  • cov_matrix (pd.DataFrame) – the covariance matrix of asset returns
  • gamma (float, optional) – L2 regularisation parameter, defaults to 0. Increase if you want more non-negligible weights
  • risk_free_rate (float, optional) – risk-free rate of borrowing/lending, defaults to 0.02. The period of the risk-free rate should correspond to the frequency of expected returns.
Returns:

negative Sharpe ratio

Return type:

float

pypfopt.objective_functions.volatility(weights, cov_matrix, gamma=0)

Calculate the volatility of a portfolio. This is actually a misnomer because the function returns variance, which is technically the correct objective function when minimising volatility.

Parameters:
  • weights (np.ndarray) – asset weights of the portfolio
  • cov_matrix (pd.DataFrame) – the covariance matrix of asset returns
  • gamma (float, optional) – L2 regularisation parameter, defaults to 0. Increase if you want more non-negligible weights
Returns:

portfolio variance

Return type:

float

One of the experimental features implemented in PyPortfolioOpt is the L2 regularisation parameter gamma, which is discussed below.

L2 Regularisation

As has been discussed in the User Guide, efficient frontier optimisation often results in many weights being negligible, i.e the efficient portfolio does not end up including most of the assets. This is expected behaviour, but it may be undesirable if you need a certain number of assets in your portfolio.

In order to coerce the efficient frontier optimiser to produce more non-negligible weights, I have added what can be thought of as a “small weights penalty” to all of the objective functions, parameterised by \(\gamma\) (gamma). Considering the minimum variance objective for instance, we have:

\[\underset{w}{\text{minimise}} ~ \left\{w^T \Sigma w \right\} ~~~ \longrightarrow ~~~ \underset{w}{\text{minimise}} ~ \left\{w^T \Sigma w + \gamma w^T w \right\}\]

Note that \(w^T w\) is the same as the sum of squared weights (I didn’t write this explicitly to reduce confusion caused by \(\Sigma\) denoting both the covariance matrix and the summation operator). This term reduces the number of negligible weights, because it has a minimum value when all weights are equally distributed, and maximum value in the limiting case where the entire portfolio is allocated to one asset. I refer to it as L2 regularisation because it has exactly the same form as the L2 regularisation term in machine learning, though a slightly different purpose (in ML it is used to keep weights small while here it is used to make them larger).

Note

In practice, \(\gamma\) must be tuned to achieve the level of regularisation that you want. However, if the universe of assets is small (less than 20 assets), then gamma=1 is a good starting point. For larger universes, or if you want more non-negligible weights in the final portfolio, increase gamma.

Custom objectives

Though it is simple enough to modify objective_functions.py to implement a custom objective (indeed, this is the recommended approach for long-term use), I understand that most users would find it much more convenient to pass a custom objective into the optimiser without having to edit the source files.

Thus, v0.2.0 introduces a simple API within the EfficientFrontier object for optimising your own objective function.

The first step is to define the objective function, which must take an array of weights as input (with optional additional arguments), and return a single float corresponding to the cost. As an example, we will pretend that L2 regularisation is not built-in and re-implement it below:

def my_objective_function(weights, cov_matrix, k):
    variance = np.dot(weights.T, np.dot(cov_matrix, weights))
    return variance + k * (weights ** 2).sum()

Next, we instantiate the EfficientFrontier object, and pass the objectives function (and all required arguments) into custom_objective():

ef = EfficientFrontier(mu, S)
weights = ef.custom_objective(my_objective_function, ef.cov_matrix, 0.3)

Caution

It is assumed that the objective function you define will be solvable by sequential quadratic programming. If this isn’t the case, you may experience silent failure.