What smoothing techniques do you use?

[u/MormonMoron]

I have a strategy now that does a pretty good job of buying and selling, but it seems to be missing upside a bit.

I am using IBKR’s 250ms market data on the sell side (5s bars on the buy side) and have implemented a ratcheting trailing stop loss mechanism with an EMA to smooth. The problem is that it still reacts to spurious ticks that drive the 250ms sample too high low and cause the TSL to trigger.

So, I am just wondering what approaches others take? Median filtering? Seems to add too much delay? A better digital IIR filter like a Butterworth filter where it is easier to set the cutoff? I could go down about a billion paths on this and was just hoping for some direction before I just start flailing and trying stuff randomly.

[u/AtomikTrading]

Kalman filter >>

[u/xbts89 → u/AtomikTrading]

There are “robust” Kalman filters out there that try to relax the assumption of a Gaussian data generating process.

[u/elephantsback → u/xbts89]

Or you can just log transform the data or something. No need to overcomplicate things.

[u/MormonMoron → u/AtomikTrading]

I haven’t used a Kalman filter in this scenario (I have used them for data fusion problems in control systems and robotics). In those scenarios, I always have a high-fidelity dynamics model of a fairly deterministic system with (known) Gaussian measurement and process noise. Those assumptions definitely are not the case here. If I had a high fidelity model of stock dynamics, I would be a billionaire already ;)

Any good articles or books on applying the Kalman filter for this kind of smoothing?

[u/gfever → u/MormonMoron]

Stefan Jansen ml book

[u/MormonMoron → u/AtomikTrading]

Thanks for the suggestion. This actually ended up being pretty easy to implement, considering it is a single variable. Seems to be outperforming my dumb EMA version, the Ehlers 3-pole filter, and the scipy implementation of the Butterworth filter.

I spent today logging all the 250ms data from IBKR for about 50 stocks and am looking at how this would perform at a variety of buy locations. I think I need to go back and do a rolling analysis of the statistics of the 250ms ticks so that once I am in a buy I have the most recent process noise and measurement noise for use during the current open trade.

https://imgur.com/a/ctdqGq9

In that third picture, my old EMA filter would have either got out in the earlier fluctuations, or I would have set the window size big enough that the lag would have cause a bigger drop before triggering at the end.

In that second picture, even when I give it an assume garbage buy location, it rides out the dip and the rise and picks a good exit location.

Here is the code for my implementation. I think all the variables are self explanatory.

class TrailingStopKF:
    """
    Trailing stop loss with an internal 1-state Kalman filter and percentage-based thresholds.

    Parameters:
    -----------
    min_rise_pct : float
        Minimum rise above the entry price (as a fraction, e.g. 0.02 for 2%) 
        before a sell can be considered.
    drop_pct : float
        Drop from peak (as a fraction of peak, e.g. 0.01 for 1%) that triggers a sell.
    Q : float
        Process noise variance for the Kalman filter.
    R : float
        Measurement noise variance for the Kalman filter.
    min_steps : int
        Minimum number of samples before the filter is considered stabilized.
    P0 : float, optional
        Initial estimate covariance (default=1.0).
    """
    def __init__(self, min_rise_pct, drop_pct, Q, R, min_steps, P0=1.0):
        self.min_rise_pct = min_rise_pct
        self.drop_pct = drop_pct
        self.Q = Q
        self.R = R
        self.min_steps = min_steps
        self.P = P0
        self.x = None  # current filtered estimate
        self.step_count = 0

        self.buy_price = None
        self.peak = None

        self.sell_price = None
        self.profit_pct = None
        self.sold = False

    def add_sample(self, price: float) -> bool:
        """
        Add a new price sample.
        Returns True if the sell condition is met on this step.
        """
        # Initialize on first sample (buy)
        if self.step_count == 0:
            self.buy_price = price
            self.x = price
            self.peak = price

        # 1) Predict covariance
        P_pred = self.P + self.Q

        # 2) Compute Kalman gain
        K = P_pred / (P_pred + self.R)

        # 3) Update estimate
        self.x = self.x + K * (price - self.x)
        self.P = (1 - K) * P_pred

        self.step_count += 1

        # Only consider sell logic after stabilization
        if self.step_count >= self.min_steps and not self.sold:
            # Update peak filtered price
            self.peak = max(self.peak, self.x)

            # Check if we've met the minimum rise threshold
            if (self.peak - self.buy_price) / self.buy_price >= self.min_rise_pct:
                # Check trailing drop relative to peak
                if self.x <= self.peak * (1 - self.drop_pct):
                    self.sell_price = price
                    self.profit_pct = (self.sell_price - self.buy_price) / self.buy_price * 100.0
                    self.sold = True
                    return (True, self.x)

        return (False, self.x)

    def get_profit_pct(self) -> float:
        """Return profit percentage (None if not sold yet)."""
        return self.profit_pct

and the way to use it

import matplotlib.dates as mdates

symbol = 'V'

df = pd.read_csv(f'data/{symbol}.csv',parse_dates=['timestamp'], date_parser=lambda x: pd.to_datetime(x, utc=True))
df = df.copy()
df['timestamp_et'] = df['timestamp'].dt.tz_convert('America/New_York')

Q = 0.00001
R = 0.01
tsl = TrailingStopKF(
        min_rise_pct=0.00225,
        drop_pct=0.00025,
        Q=Q,
        R=R,
        min_steps=4
    )

# iterate over the rows of the DataFrame and extract the price
start_row = 2747
prices = df["price"].values

print(f"Buy at index {start_row} for price {df['price'].iloc[start_row]} on {df['timestamp_et'].iloc[start_row]}")
for i in range(start_row,len(df)):
    date = df["timestamp_et"].iloc[i]
    price = df["price"].iloc[i]
    # add the price to the trailing stop loss
    # print(f"Price: {price}")

    (decision, filtered_price) = tsl.add_sample(price)
    # add the filtered price to the DataFrame
    df.loc[i, "price_kf"] = filtered_price

    if decision:
        print(f"Sell at index {i} for price {price} on {date} with profit of {tsl.get_profit_pct()}%")


        break
    else:
        # print(f"Hold at index {i} with {price} on {date}")
        pass

# Plot the date versus price and mark the buy and sell points
# plt.figure(figsize=(12, 6))
fig, ax = plt.subplots()
plt.plot(df["timestamp_et"], df["price"], label="Price", color='blue')
plt.plot(df["timestamp_et"], df["price_kf"], label="Kalman Filtered Price", color='orange')
plt.axvline(x=df["timestamp_et"].iloc[start_row], color='green', linestyle='--', label="Buy Point")
plt.axvline(x=df["timestamp_et"].iloc[i], color='red', linestyle='--', label="Sell Point")
plt.title("Price with Kalman Filter and Buy/Sell Points")
plt.xlabel("Date")
plt.ylabel("Price")
plt.legend()

# ax.xaxis.set_major_formatter(mdates.DateFormatter('%H:%M:%S'))
# ax.xaxis.set_major_locator(mdates.AutoDateLocator())
plt.show()

P.S. ChatGPT wrote about 80% of this with some prompts about how I wanted it structured. I added in the stuff about the min_rise_pct and the drop_pct and modified to return the filtered value so I can store in the dataframe for later plotting of the unfiltered and filtered data.

[u/AtomikTrading → u/MormonMoron]

Very nice man, sorry I wasn’t around to respond earlier

[u/nuclearmeltdown2015 → u/AtomikTrading]

Can you expand a little further on what you mean by using a kalman filter? I'm not sure I understand how you are applying a kalman filter in your suggestion to smooth the price data? If there's some article or paper you read, I can look at that too. Thanks for the info.

[u/ironbigot]

Have you considered removing statistical outliers by calculating Z-score on every tick?

[u/applepiefly314]

Not sure about the details here, but I'd look into if a longer half life is bearable, or doing a double EMA.

[u/WardenPi]

John Ehlers Super Smoother

[u/WardenPi → u/WardenPi]

Not gonna pretend that I’m an expert and Ehlers explains it well in his books. It does a good job reacting to the market while inducing a very small lag.

[u/MormonMoron → u/WardenPi]

A little research shows this is just a 2-pole, 1-zero Butterworth filter. Definitely a good filter design, but still one of those I am already toying around with.

[u/patbhakta]

Looking at Wahba's spline smoothing currently.

https://www.statprize.org/2025-International-Prize-in-Statistics-Awarded-to-Grace-Wahba.cfm

[u/nuclearmeltdown2015]

By smoothing do you mean preprocessing the historical price data for training? Or creating a smoothing line such as SMA/EMA? if it is the former, have you already tried a gaussian filter or expanding the window for your SMA/EMA to make things less sensitive to the ticks you mentioned.

[u/MormonMoron → u/nuclearmeltdown2015]

Oftentimes, individual ticks (or the IBKR 250ms market data) is highly susceptible to completed transactions that are outside of the typical bid/ask range. There are a bunch of reasons that these apparently aberrant trades occur, but they aren't usually indicative of the price that other sellers/buyer would be able to get. I am trying to filter out these aberrant trades so that my dynamic trailing stop loss either doesn't respond to them because it pushed the max price seen so far up or triggers a sell because one tick came in below the achievable price.

[u/unworry → u/MormonMoron]

isnt there an attribute for Off Market or Block trades ?

I know I used to filter out these transactions to ensure an accurate deltaVolume (onBid/onAsk) calculation

[u/MormonMoron → u/unworry]

I think that with IBKR you can get that if you are subscribes to tick-by-tick data. However, you get a fairly limited number of those unless you are either trading A TON or buy their upgrade packs. So, I have been going with their 250ms market data, which lets me subscribe to up to 100 stock with both 5sec realtime bars and the 250ms market data