General Efficient Frontier¶
The meanvariance optimization methods described previously can be used whenever you have a vector of expected returns and a covariance matrix. The objective and constraints will be some combination of the portfolio return and portfolio volatility.
However, you may want to construct the efficient frontier for an entirely different type of risk model (one that doesn’t depend on covariance matrices), or optimize an objective unrelated to portfolio return (e.g tracking error). PyPortfolioOpt comes with several popular alternatives and provides support for custom optimization problems.
Efficient Semivariance¶
Instead of penalising volatility, meansemivariance optimization seeks to only penalise downside volatility, since upside volatility may be desirable.
There are two approaches to the meansemivariance optimization problem. The first is to use a
heuristic (i.e “quick and dirty”) solution: pretending that the semicovariance matrix
(implemented in risk_models
) is a typical covariance matrix and doing standard
meanvariance optimization. It can be shown that this does not yield a portfolio that
is efficient in meansemivariance space (though it might be a goodenough approximation).
Fortunately, it is possible to write meansemivariance optimization as a convex problem (albeit one with many variables), that can be solved to give an “exact” solution. For example, to maximise return for a target semivariance \(s^*\) (longonly), we would solve the following problem:
Here, B is the \(T \times N\) (scaled) matrix of excess returns:
B = (returns  benchmark) / sqrt(T)
. Additional linear equality constraints and
convex inequality constraints can be added.
PyPortfolioOpt allows users to optimize along the efficient semivariance frontier
via the EfficientSemivariance
class. EfficientSemivariance
inherits from
EfficientFrontier
, so it has the same utility methods
(e.g add_constraint()
, portfolio_performance()
), but finds portfolios on the meansemivariance
frontier. Note that some of the parent methods, like max_sharpe()
and min_volatility()
are not applicable to meansemivariance portfolios, so calling them returns NotImplementedError
.
EfficientSemivariance
has a slightly different API to EfficientFrontier
. Instead of passing
in a covariance matrix, you should past in a dataframe of historical/simulated returns (this can be constructed
from your price dataframe using the helper method expected_returns.returns_from_prices()
). Here
is a full example, in which we seek the portfolio that minimises the semivariance for a target
annual return of 20%:
from pypfopt import expected_returns, EfficientSemivariance
df = ... # your dataframe of prices
mu = expected_returns.mean_historical_returns(df)
historical_returns = expected_returns.returns_from_prices(df)
es = EfficientSemivariance(mu, historical_returns)
es.efficient_return(0.20)
# We can use the same helper methods as before
weights = es.clean_weights()
print(weights)
es.portfolio_performance(verbose=True)
The portfolio_performance
method outputs the expected portfolio return, semivariance,
and the Sortino ratio (like the Sharpe ratio, but for downside deviation).
Interested readers should refer to Estrada (2007) [1] for more details. I’d like to thank Philipp Schiele for authoring the bulk of the efficient semivariance functionality and documentation (all errors are my own). The implementation is based on Markowitz et al (2019) [2].
Caution
Finding portfolios on the meansemivariance frontier is computationally harder
than standard meanvariance optimization: our implementation uses 2T + N
optimization variables,
meaning that for 50 assets and 3 years of data, there are about 1500 variables.
While EfficientSemivariance
allows for additional constraints/objectives in principle,
you are much more likely to run into solver errors. I suggest that you keep EfficientSemivariance
problems small and minimally constrained.

class
pypfopt.efficient_frontier.
EfficientSemivariance
(expected_returns, returns, frequency=252, benchmark=0, weight_bounds=(0, 1), solver=None, verbose=False, solver_options=None)[source]¶ EfficientSemivariance objects allow for optimization along the meansemivariance frontier. This may be relevant for users who are more concerned about downside deviation.
Instance variables:
Inputs:
n_assets
 inttickers
 str listbounds
 float tuple OR (float tuple) listreturns
 pd.DataFrameexpected_returns
 np.ndarraysolver
 strsolver_options
 {str: str} dict
Output:
weights
 np.ndarray
Public methods:
min_semivariance()
minimises the portfolio semivariance (downside deviation)max_quadratic_utility()
maximises the “downside quadratic utility”, given some risk aversion.efficient_risk()
maximises return for a given target semideviationefficient_return()
minimises semideviation for a given target returnadd_objective()
adds a (convex) objective to the optimization problemadd_constraint()
adds a constraint to the optimization problemconvex_objective()
solves for a generic convex objective with linear constraintsportfolio_performance()
calculates the expected return, semideviation and Sortino ratio for the optimized portfolio.set_weights()
creates self.weights (np.ndarray) from a weights dictclean_weights()
rounds the weights and clips nearzeros.save_weights_to_file()
saves the weights to csv, json, or txt.

efficient_return
(target_return, market_neutral=False)[source]¶ Minimise semideviation for a given target return.
Parameters:  target_return (float) – the desired return of the resulting portfolio.
 market_neutral (bool, optional) – whether the portfolio should be market neutral (weights sum to zero), defaults to False. Requires negative lower weight bound.
Raises:  ValueError – if
target_return
is not a positive float  ValueError – if no portfolio can be found with return equal to
target_return
Returns: asset weights for the optimal portfolio
Return type: OrderedDict

efficient_risk
(target_semideviation, market_neutral=False)[source]¶ Maximise return for a target semideviation (downside standard deviation). The resulting portfolio will have a semideviation less than the target (but not guaranteed to be equal).
Parameters:  target_semideviation (float) – the desired maximum semideviation of the resulting portfolio.
 market_neutral – whether the portfolio should be market neutral (weights sum to zero), defaults to False. Requires negative lower weight bound.
 market_neutral – bool, optional
Returns: asset weights for the efficient risk portfolio
Return type: OrderedDict

max_quadratic_utility
(risk_aversion=1, market_neutral=False)[source]¶ Maximise the given quadratic utility, using portfolio semivariance instead of variance.
Parameters:  risk_aversion (positive float) – risk aversion parameter (must be greater than 0), defaults to 1
 market_neutral – whether the portfolio should be market neutral (weights sum to zero), defaults to False. Requires negative lower weight bound.
 market_neutral – bool, optional
Returns: asset weights for the maximumutility portfolio
Return type: OrderedDict

min_semivariance
(market_neutral=False)[source]¶ Minimise portfolio semivariance (see docs for further explanation).
Parameters:  market_neutral – whether the portfolio should be market neutral (weights sum to zero), defaults to False. Requires negative lower weight bound.
 market_neutral – bool, optional
Returns: asset weights for the volatilityminimising portfolio
Return type: OrderedDict

portfolio_performance
(verbose=False, risk_free_rate=0.02)[source]¶ After optimising, calculate (and optionally print) the performance of the optimal portfolio, specifically: expected return, semideviation, Sortino ratio.
Parameters:  verbose (bool, optional) – whether performance should be printed, defaults to False
 risk_free_rate (float, optional) – riskfree rate of borrowing/lending, defaults to 0.02. The period of the riskfree rate should correspond to the frequency of expected returns.
Raises: ValueError – if weights have not been calculated yet
Returns: expected return, semideviation, Sortino ratio.
Return type: (float, float, float)
Efficient CVaR¶
The conditional valueatrisk (a.k.a expected shortfall) is a popular measure of tail risk. The CVaR can be thought of as the average of losses that occur on “very bad days”, where “very bad” is quantified by the parameter \(\beta\).
For example, if we calculate the CVaR to be 10% for \(\beta = 0.95\), we can be 95% confident that the worstcase average daily loss will be 3%. Put differently, the CVaR is the average of all losses so severe that they only occur \((1\beta)\%\) of the time.
While CVaR is quite an intuitive concept, a lot of new notation is required to formulate it mathematically (see the wiki page for more details). We will adopt the following notation:
 w for the vector of portfolio weights
 r for a vector of asset returns (daily), with probability distribution \(p(r)\).
 \(L(w, r) =  w^T r\) for the loss of the portfolio
 \(\alpha\) for the portfolio valueatrisk (VaR) with confidence \(\beta\).
The CVaR can then be written as:
This is a nasty expression to optimize because we are essentially integrating over VaR values. The key insight of Rockafellar and Uryasev (2001) [3] is that we can can equivalently optimize the following convex function:
where \([x]^+ = \max(x, 0)\). The authors prove that minimising \(F_\beta(w, \alpha)\) over all \(w, \alpha\) minimises the CVaR. Suppose we have a sample of T daily returns (these can either be historical or simulated). The integral in the expression becomes a sum, so the CVaR optimization problem reduces to a linear program:
This formulation introduces a new variable for each datapoint (similar to Efficient Semivariance), so you may run into performance issues for long returns dataframes. At the same time, you should aim to provide a sample of data that is large enough to include tail events.
I am grateful to Nicolas Knudde for the initial draft (all errors are my own). The implementation is based on Rockafellar and Uryasev (2001) [3].

class
pypfopt.efficient_frontier.
EfficientCVaR
(expected_returns, returns, beta=0.95, weight_bounds=(0, 1), solver=None, verbose=False, solver_options=None)[source]¶ The EfficientCVaR class allows for optimization along the meanCVaR frontier, using the formulation of Rockafellar and Ursayev (2001).
Instance variables:
Inputs:
n_assets
 inttickers
 str listbounds
 float tuple OR (float tuple) listreturns
 pd.DataFrameexpected_returns
 np.ndarraysolver
 strsolver_options
 {str: str} dict
Output:
weights
 np.ndarray
Public methods:
min_cvar()
minimises the CVaRefficient_risk()
maximises return for a given CVaRefficient_return()
minimises CVaR for a given target returnadd_objective()
adds a (convex) objective to the optimization problemadd_constraint()
adds a constraint to the optimization problemportfolio_performance()
calculates the expected return and CVaR of the portfolioset_weights()
creates self.weights (np.ndarray) from a weights dictclean_weights()
rounds the weights and clips nearzeros.save_weights_to_file()
saves the weights to csv, json, or txt.

efficient_return
(target_return, market_neutral=False)[source]¶ Minimise CVaR for a given target return.
Parameters:  target_return (float) – the desired return of the resulting portfolio.
 market_neutral (bool, optional) – whether the portfolio should be market neutral (weights sum to zero), defaults to False. Requires negative lower weight bound.
Raises:  ValueError – if
target_return
is not a positive float  ValueError – if no portfolio can be found with return equal to
target_return
Returns: asset weights for the optimal portfolio
Return type: OrderedDict

efficient_risk
(target_cvar, market_neutral=False)[source]¶ Maximise return for a target CVaR. The resulting portfolio will have a CVaR less than the target (but not guaranteed to be equal).
Parameters:  target_cvar (float) – the desired conditional value at risk of the resulting portfolio.
 market_neutral – whether the portfolio should be market neutral (weights sum to zero), defaults to False. Requires negative lower weight bound.
 market_neutral – bool, optional
Returns: asset weights for the efficient risk portfolio
Return type: OrderedDict

min_cvar
(market_neutral=False)[source]¶ Minimise portfolio CVaR (see docs for further explanation).
Parameters:  market_neutral – whether the portfolio should be market neutral (weights sum to zero), defaults to False. Requires negative lower weight bound.
 market_neutral – bool, optional
Returns: asset weights for the volatilityminimising portfolio
Return type: OrderedDict

portfolio_performance
(verbose=False)[source]¶ After optimising, calculate (and optionally print) the performance of the optimal portfolio, specifically: expected return, CVaR
Parameters: verbose (bool, optional) – whether performance should be printed, defaults to False Raises: ValueError – if weights have not been calculated yet Returns: expected return, CVaR. Return type: (float, float)
EfficientCDaR¶
The conditional drawdown at risk (CDaR) is a more exotic measure of tail risk. It tries to alleviate the problems with Efficient Semivariance and Efficient CVaR in that it accounts for the timespan of material decreases in value. The CDaR can be thought of as the average of losses that occur on “very bad periods”, where “very bad” is quantified by the parameter \(\beta\). The drawdown is defined as the difference in noncompounded return to the previous peak.
Put differently, the CDaR is the average of all drawdowns so severe that they only occur \((1\beta)\%\) of the time. When \(\beta = 1\) CDaR is simply the maximum drawdown.
While drawdown is quite an intuitive concept, a lot of new notation is required to formulate it mathematically (see the wiki page for more details). We will adopt the following notation:
 w for the vector of portfolio weights
 r for a vector of cumulative asset returns (daily), with probability distribution \(p(r(t))\).
 \(D(w, r, t) = \max_{\tau<t}(w^T r(\tau))w^T r(t)\) for the drawdown of the portfolio
 \(\alpha\) for the portfolio drawdown (DaR) with confidence \(\beta\).
The CDaR can then be written as:
This is a nasty expression to optimise because we are essentially integrating over VaR values. The key insight of Chekhlov, Rockafellar and Uryasev (2005) [4] is that we can can equivalently optimise a convex function, which can be transformed to a linear problem (in the same manner as for CVaR).

class
pypfopt.efficient_frontier.
EfficientCDaR
(expected_returns, returns, beta=0.95, weight_bounds=(0, 1), solver=None, verbose=False, solver_options=None)[source]¶ The EfficientCDaR class allows for optimisation along the meanCDaR frontier, using the formulation of Chekhlov, Ursayev and Zabarankin (2005).
Instance variables:
Inputs:
n_assets
 inttickers
 str listbounds
 float tuple OR (float tuple) listreturns
 pd.DataFrameexpected_returns
 np.ndarraysolver
 strsolver_options
 {str: str} dict
Output:
weights
 np.ndarray
Public methods:
min_cdar()
minimises the CDaRefficient_risk()
maximises return for a given CDaRefficient_return()
minimises CDaR for a given target returnadd_objective()
adds a (convex) objective to the optimisation problemadd_constraint()
adds a (linear) constraint to the optimisation problemportfolio_performance()
calculates the expected return and CDaR of the portfolioset_weights()
creates self.weights (np.ndarray) from a weights dictclean_weights()
rounds the weights and clips nearzeros.save_weights_to_file()
saves the weights to csv, json, or txt.

efficient_return
(target_return, market_neutral=False)[source]¶ Minimise CDaR for a given target return.
Parameters:  target_return (float) – the desired return of the resulting portfolio.
 market_neutral (bool, optional) – whether the portfolio should be market neutral (weights sum to zero), defaults to False. Requires negative lower weight bound.
Raises:  ValueError – if
target_return
is not a positive float  ValueError – if no portfolio can be found with return equal to
target_return
Returns: asset weights for the optimal portfolio
Return type: OrderedDict

efficient_risk
(target_cdar, market_neutral=False)[source]¶ Maximise return for a target CDaR. The resulting portfolio will have a CDaR less than the target (but not guaranteed to be equal).
Parameters:  target_cdar (float) – the desired maximum CDaR of the resulting portfolio.
 market_neutral – whether the portfolio should be market neutral (weights sum to zero), defaults to False. Requires negative lower weight bound.
 market_neutral – bool, optional
Returns: asset weights for the efficient risk portfolio
Return type: OrderedDict

min_cdar
(market_neutral=False)[source]¶ Minimise portfolio CDaR (see docs for further explanation).
Parameters:  market_neutral – whether the portfolio should be market neutral (weights sum to zero), defaults to False. Requires negative lower weight bound.
 market_neutral – bool, optional
Returns: asset weights for the volatilityminimising portfolio
Return type: OrderedDict

portfolio_performance
(verbose=False)[source]¶ After optimising, calculate (and optionally print) the performance of the optimal portfolio, specifically: expected return, CDaR
Parameters: verbose (bool, optional) – whether performance should be printed, defaults to False Raises: ValueError – if weights have not been calculated yet Returns: expected return, CDaR. Return type: (float, float)
I am grateful to Nicolas Knudde for implementing this feature.
Custom optimization problems¶
We have seen previously that it is easy to add constraints to EfficientFrontier
objects (and
by extension, other general efficient frontier objects like EfficientSemivariance
). However, what if you aren’t interested
in anything related to max_sharpe()
, min_volatility()
, efficient_risk()
etc and want to
set up a completely new problem to optimize for some custom objective?
For example, perhaps our objective is to construct a basket of assets that best replicates a particular index, in other words, to minimise the tracking error. This does not fit within a meanvariance optimization paradigm, but we can still implement it in PyPortfolioOpt:
from pypfopt.base_optimizer import BaseConvexOptimizer
from pypfopt.objective_functions import ex_post_tracking_error
historic_rets = ... # dataframe of historic asset returns
benchmark_rets = ... # pd.Series of historic benchmark returns (same index as historic)
opt = BaseConvexOptimizer(
n_assets=len(historic_returns.columns),
tickers=historic_returns.columns,
weight_bounds=(0, 1)
)
opt.convex_objective(
ex_post_tracking_error,
historic_returns=historic_rets,
benchmark_returns=benchmark_rets,
)
weights = opt.clean_weights()
The EfficientFrontier
class inherits from BaseConvexOptimizer
. It may be more convenient
to call convex_objective
from an EfficientFrontier
instance than from BaseConvexOptimizer
,
particularly if your objective depends on the mean returns or covariance matrix.
You can either optimize some generic convex_objective
(which must be built using cvxpy
atomic functions – see here)
or a nonconvex_objective
, which uses scipy.optimize
as the backend and thus has a completely
different API. For more examples, check out this cookbook recipe.
 class
pypfopt.base_optimizer.
BaseConvexOptimizer
¶
BaseConvexOptimizer.
convex_objective
(custom_objective, weights_sum_to_one=True, **kwargs)¶Optimize a custom convex objective function. Constraints should be added with
ef.add_constraint()
. Optimizer arguments must be passed as keywordargs. Example:# Could define as a lambda function instead def logarithmic_barrier(w, cov_matrix, k=0.1): # 60 Years of Portfolio Optimization, Kolm et al (2014) return cp.quad_form(w, cov_matrix)  k * cp.sum(cp.log(w)) w = ef.convex_objective(logarithmic_barrier, cov_matrix=ef.cov_matrix)
Parameters:
 custom_objective (function with signature (cp.Variable, **kwargs) > cp.Expression) – an objective function to be MINIMISED. This should be written using cvxpy atoms Should map (w, **kwargs) > float.
 weights_sum_to_one (bool, optional) – whether to add the default objective, defaults to True
Raises: OptimizationError – if the objective is nonconvex or constraints nonlinear.
Returns: asset weights for the efficient risk portfolio
Return type: OrderedDict
BaseConvexOptimizer.
nonconvex_objective
(custom_objective, objective_args=None, weights_sum_to_one=True, constraints=None, solver='SLSQP', initial_guess=None)¶Optimize some objective function using the scipy backend. This can support nonconvex objectives and nonlinear constraints, but may get stuck at local minima. Example:
# Marketneutral efficient risk constraints = [ {"type": "eq", "fun": lambda w: np.sum(w)}, # weights sum to zero { "type": "eq", "fun": lambda w: target_risk ** 2  np.dot(w.T, np.dot(ef.cov_matrix, w)), }, # risk = target_risk ] ef.nonconvex_objective( lambda w, mu: w.T.dot(mu), # min negative return (i.e maximise return) objective_args=(ef.expected_returns,), weights_sum_to_one=False, constraints=constraints, )
Parameters:
 objective_function (function with signature (np.ndarray, args) > float) – an objective function to be MINIMISED. This function should map (weight, args) > cost
 objective_args (tuple of np.ndarrays) – arguments for the objective function (excluding weight)
 weights_sum_to_one (bool, optional) – whether to add the default objective, defaults to True
 constraints (dict list) – list of constraints in the scipy format (i.e dicts)
 solver (string) – which SCIPY solver to use, e.g “SLSQP”, “COBYLA”, “BFGS”. User beware: different optimizers require different inputs.
 initial_guess (np.ndarray) – the initial guess for the weights, shape (n,) or (n, 1)
Returns: asset weights that optimize the custom objective
Return type: OrderedDict
References¶
[1]  Estrada, J (2007). MeanSemivariance Optimization: A Heuristic Approach. 
[2]  Markowitz, H.; Starer, D.; Fram, H.; Gerber, S. (2019). Avoiding the Downside. 
[3]  (1, 2) Rockafellar, R.; Uryasev, D. (2001). Optimization of conditional valueatrisk 
[4]  Chekhlov, A.; Rockafellar, R.; Uryasev, D. (2005). Drawdown measure in portfolio optimization 