How to Stop Measuring Your Portfolio Against Its All-Time High

When people discuss the “4% rule,” the question often arises — 4% of what amount? Many investors naturally anchor to their portfolio’s all-time high, assuming that figure represents their true wealth. But as recent market cycles remind us, that mindset can be misleading. Stock markets can drop 50% or more, instantly changing what “4%” actually means. Instead of thinking you’ve “lost” money when values retreat from peak levels, it’s wiser to accept volatility as part of investing and base your long-term withdrawal plan on sustainable averages, not temporary highs. Anchoring to record numbers can distort reality — investing success depends on staying steady through both peaks and plunges.

I’m currently reading the new book Rethinking Investing: A Very Short Guide to Very Long-Term Investing by Charles Ellis. He’s been in the industry a long time, but may be best known for his bestselling book Winning the Loser’s Game, first published in 1985.

In the book, Ellis proposed a potentially better way to set your Spending Rule in retirement. It’s not based on the most recent value of your portfolio, and definitely not the all-time high of your portfolio. Instead, he wants you to use the rolling average over the last 5-7 years. Then, you can add the 4% rule (or whatever).

In designing your own spending rule, first, average the year-end values of your assets over the prior several years (preferably more than five years) to dampen the impact of market fluctuations. Next, calculate what would be a prudent withdrawal of the averaged assets—likely 4–5%—to determine what dollar amount you can prudently withdraw from your current portfolio each year to cover some of your expenses.

This has the effect of smoothing out your annual withdrawals:

Averaging your assets over multiple years makes the funds available for your spending far more consistent and predictable. If, for example, you settle on a 5% rate of withdrawal and a six-year moving average of the year-end value of your assets, a 30% drop in the stock market would lead to only a 5% reduction in your payout that year (and much of that reduction likely would be provided by your consistent dividend income).

(Side note: This supports the idea of me tracking my consistent dividend income…)

Let’s take a look at Vanguard LifeStrategy Growth Fund (VASGX), an all-in-one fund that is diversified similarly to their Target Date Retirement Funds, but a handy benchmark since it is a constant 80% stocks/20% bonds. Here’s a Growth of $10,000 chart for the last 5 years ending 4/4:

Instead of seeing that you are about 11% off your all-time high value of about $18,300, you might appreciate that you are still above January 2024 levels, and that your 5-year rolling average of year-end values is about $15,400. If you based your 4% withdrawal rate on that value, you would be much calmer now.

I like this strategy, and I believe it should be applied even when you are still accumulating for retirement. Don’t anchor yourself the all-time high of your portfolio and make it your new “If it ever goes below this, I’ll be sad!” value. Instead, mentally track a rolling average of your net worth. I’ll look to add this concept to my portfolio updates, hopefully it’ll reduce my stress levels during volatile times.

Finally, Ellis points out another potential benefit:

Importantly, by following such a Spending Rule, you are then free to concentrate on achieving significantly higher long-term returns without the need to be overinvested in bonds. Stabilizing the investor’s income with a responsible Spending Rule frees the investment portfolio to invest more in equities and produces, over time, a higher and more rapidly rising portfolio value and income stream.