Automated HDBSCAN parameter selection

[danellecline]

Picking parameters can be challenging.

This paper has an interesting automated choice for DBSCAN.

Something like the following could be used to find epsilon for a given minpts/k (untested)

def calculate_eps_minpts(data, k):
    # Fit NearestNeighbors model
    neighbors = NearestNeighbors(n_neighbors=k)
    neighbors_fit = neighbors.fit(data)

    # Compute the k-distance for each point
    distances, indices = neighbors_fit.kneighbors(data)

    # Take the k-th nearest distance (i.e., the k-distance)
    k_distances = distances[:, k - 1]

    # Sort the k-distances in ascending order
    k_distances = np.sort(k_distances)

    # Plotting the k-distances to visualize the knee
    plt.plot(k_distances)
    plt.xlabel("Points sorted by distance")
    plt.ylabel(f"Distance to {k}-th nearest neighbor")
    plt.title(f"{k}-distance Graph")
    plt.show()

    # Use KneeLocator to find the knee point
    kneedle = KneeLocator(range(len(k_distances)), k_distances, curve="convex", direction="increasing")

    # The knee point corresponds to the optimal epsilon
    eps = k_distances[kneedle.knee]

    return eps