Yellowstone gRPC

Using Compressed Filters

Update Yellowstone account subscriptions without re-sending the full key list.

Previously, changing a Yellowstone subscription involved re-sending the full list of public keys you subscribed to.

We've added a compressed filter (a Cuckoo filter) that allows you to efficiently update your existing Yellowstone subscription by adding and removing public keys.

This works by creating a small (but unique) fingerprint for each public key in your subscription. These fingerprints take up about 90% less space than storing the public keys themselves.

This lets us support Yellowstone subscriptions to a larger number of concurrent accounts, without compromising on speed.

Application-side, you can now update your subscription without re-sending the entire public key list.

Using Compressed Yellowstone Filters

First, you store the public keys that you wish to subscribe to:

Store public keys

import (
    "fmt"
    "github.com/Gealber/cuckoo"
    "github.com/mr-tron/base58"
    pb "yourmodule/proto"
)

func buildCuckooFilter(pubkeys []string) (*pb.CuckooFilter, error) {
    capacity := max(uint(len(pubkeys)), 10_000)
    cf := cuckoo.New(capacity)

    for _, pk := range pubkeys {
        raw, err := base58.Decode(pk)
        if err != nil || len(raw) != 32 {
            return nil, fmt.Errorf("invalid pubkey %q", pk)
        }
        if !cf.Insert(raw) {
            return nil, fmt.Errorf("filter full, increase capacity")
        }
    }

    data := cf.Bytes()
    return &pb.CuckooFilter{
        Data:             data,
        BucketCount:      uint32(len(data) / 8),
        EntriesPerBucket: 4,
        FingerprintBits:  16,
        HashSeed:         cf.Seed(),
        HashAlgorithm:    pb.CuckooHashAlgorithm_SIP_HASH,
    }, nil
}

tracked := []string{
    "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v", // USDC mint
    "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA",  // Token Program
    // ... thousands more
}

// Exact set for client-side false-positive filtering
exact := make(map[string]struct{}, len(tracked))
for _, pk := range tracked {
    exact[pk] = struct{}{}
}

filter, err := buildCuckooFilter(tracked)
if err != nil {
    return err
}

Then you can update your subscription like this:

Update subscription

import (
    "context"
    "crypto/tls"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials"
    "google.golang.org/grpc/metadata"
    pb "yourmodule/proto"
)

conn, err := grpc.NewClient("https://yellowstone.eu.fluxrpc.com",
    grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{})))
if err != nil {
    return err
}
defer conn.Close()

ctx := metadata.AppendToOutgoingContext(context.Background(), "x-token", "YOUR_FLUX_RPC_API_KEY")
stream, err := pb.NewGeyserClient(conn).Subscribe(ctx)
if err != nil {
    return err
}

commitment := pb.CommitmentLevel_PROCESSED
req := &pb.SubscribeRequest{
    Accounts: map[string]*pb.SubscribeRequestFilterAccounts{
        "tracked_accounts": {CuckooAccountsFilter: filter},
    },
    Commitment: &commitment,
}
if err := stream.Send(req); err != nil {
    return err
}

To add a new account to your subscription at any time:

Add pubkey

// Keep exact set in memory (for false-positive filtering)
exact := make(map[string]struct{})
cf := cuckoo.New(10_000)

// Add new account
newPubkey := "So11111111111111111111111111111111111111112" // wSOL
exact[newPubkey] = struct{}{}
raw, _ := base58.Decode(newPubkey)
cf.Insert(raw)  // O(1) operation!

// Rebuild filter and send updated subscription
data := cf.Bytes()
filter := &pb.CuckooFilter{
    Data:             data,
    BucketCount:      uint32(len(data) / 8),
    EntriesPerBucket: 4,
    FingerprintBits:  16,
    HashSeed:         cf.Seed(),
    HashAlgorithm:    pb.CuckooHashAlgorithm_SIP_HASH,
}

commitment := pb.CommitmentLevel_PROCESSED
req := &pb.SubscribeRequest{
    Accounts: map[string]*pb.SubscribeRequestFilterAccounts{
        "tracked_accounts": {CuckooAccountsFilter: filter},
    },
    Commitment: &commitment,
}
stream.Send(req)  // Send on same stream — no reconnect!

To remove an account from your subscription:

Remove pubkey

// Remove account
oldPubkey := "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v" // USDC
delete(exact, oldPubkey)
raw, _ := base58.Decode(oldPubkey)
cf.Delete(raw)  // O(1) operation!

// Send updated subscription (same as above)
stream.Send(req)

Finally, if you would like to check membership of a public key in your local set, you can:

Check membership

// When you receive an update, check against exact set
for {
    update := stream.Recv()
    acc := update.GetAccount()
    if acc == nil {
        continue
    }

    pubkey := base58.Encode(acc.Account.Pubkey)

    // First: check if it matches the filter (may have false positives)
    raw, _ := base58.Decode(pubkey)
    inFilter := cf.Lookup(raw)

    // Second: check exact set (definitive)
    _, inExact := exact[pubkey]

    if !inExact {
        if inFilter {
            log.Printf("⚠ FALSE POSITIVE: %s (dropped)", pubkey)
        }
        continue  // Drop it
    }

    // Exact match — process it
    log.Printf("✓ MATCH: %s", pubkey)
}

The Cuckoo filter is probabilistic, it can only tell you one of the following:

The account is definitely not in the set
The account might be in the set

However, the full list of accounts is stored client-side, so when performing this membership check, it will not return false positives. It will tell you with certainty if it is present or not.