Sentiment Balance Score (SBS): A Technical Specification for Topic-Level Sentiment Measurement and Benchmarking

Abstract

---

Sentiment Balance Score (SBS) is a topic-level, polarity-balanced metric derived from open-ended feedback. SBS converts unstructured comments into standardized aspect/topic mentions, assigns aspect-level polarity, applies confidence and intensity weighting, stabilizes estimates for low-volume topics using Bayesian smoothing, and enables peer-cohort benchmarking and trend/volatility analysis. This document defines the end-to-end pipeline and provides implementable formulas and data structures.

1. Data Model and Notation

Let each written comment be a document $d$ with metadata:

• builder (or entity) $b(d)$
• timestamp $t(d)$

The pipeline transforms each comment into a set of mentions (aka aspect annotations). A mention $i$i is a record:

$$i = \langle b_i, \tau_i, \ell_i, s_i, m_i, c_i \rangle$$

Where:

• $b_i$​: builder/entity id
• $\tau_i$​: time bucket (e.g., day/week/month)
• $\ell_i$​: canonical topic label (e.g., paint_quality)
• $s_i \in \{-1,0,+1\}$: polarity (negative, neutral, positive)
• $m_i \in [0,1]$: intensity/strength (how strong the sentiment is)
• $c_i \in [0,1]$: model confidence (probability-like; calibration discussed later)

A mention has a derived weight:$$w_i = m_i \cdot c_i$$

---

2. End-to-End Pipeline (Per Comment)

Each comment goes through deterministic stages. You can implement this as an event-driven pipeline.

Step 0 – Ingest

Store raw comment text and metadata:

• comment_id, builder_id, created_at, text, source, etc.

Step 1 – Preprocessing

Recommended (not strictly required):

• language detection
• sentence segmentation
• PII masking (optional, governance-driven)

Step 2 – Aspect Extraction (Topic Discovery)

Goal: identify spans that correspond to “what the person is talking about.”

Output: a set of extracted aspect spans:$$A(d) = \{a_1, a_2, \dots, a_K\}$$

Each aspect span $a_k$​ should include:

• raw span text
• candidate topic label(s)
• evidence / anchor phrase boundaries (if your model provides them)

Implementation notes

• You can do this with LLM structured extraction, ABSA models, or hybrid (LLM + taxonomy mapping).
• A taxonomy constraint improves standardization (critical for benchmarking).

Step 3 – Aspect-Level Sentiment (No Inheritance)

For each extracted aspect $a_k$, predict:

• polarity $s$
• intensity $m$
• confidence $c$

This avoids the inheritance error where a comment-level label is applied to all topics.

Mixed sentiment

If a span expresses mixed sentiment (rare but real), options:

• split into two mentions (preferred)
• or assign $s=0$ and track a mixed flag

Step 4 – Canonical Topic Mapping

Map each aspect’s free-text label to a canonical topic $\ell$.

Two-stage mapping recommended:

1. alias dictionary / rules (deterministic)
2. embedding similarity to canonical topic embeddings with thresholding

Unmatched topics go to OTHER for later curation.

Step 5 – Persist Mentions

Insert mention rows:

• polarity $s_i$
• intensity $m_i$
• confidence $c_i$
• weight $w_i$
• canonical topic $\ell_i$
• builder/time metadata

---

3. Topic-Level Aggregation

For a given builder $b$, topic $\ell$, time window $\tau$, aggregate weighted “votes”:$$P_{b,\ell,\tau} = \sum_{i \in (b,\ell,\tau), \ s_i=+1} w_i$$$$N_{b,\ell,\tau} = \sum_{i \in (b,\ell,\tau), \ s_i=-1} w_i$$$$U_{b,\ell,\tau} = \sum_{i \in (b,\ell,\tau), \ s_i=0} w_i$$

Define weighted voting mass:$$V_{b,\ell,\tau} = P_{b,\ell,\tau} + N_{b,\ell,\tau}$$

Neutral is tracked but non-voting by default (analogous to NPS passives). You can optionally include it for other diagnostics (e.g., “clarity/decisiveness”).

---

4. Core SBS Metric

4.1 Raw (unsmoothed) SBS

If $V > 0$:$$SBS^{raw}_{b,\ell,\tau} = 100 \cdot \frac{P_{b,\ell,\tau} – N_{b,\ell,\tau}}{P_{b,\ell,\tau} + N_{b,\ell,\tau}}$$

Range: $[-100, +100]$

If $V=0$, SBS is undefined; return null and show “insufficient voting signal.”

This is the closest analogue to “%positive − %negative” in the two-class voting universe.

---

4.2 Bayesian-Smoothed SBS (recommended)

Low-volume topics produce unstable rates. Stabilize using Beta priors.

Interpret $P$ and $N$ as weighted pseudo-counts. Use a symmetric prior by default:$$\alpha > 0, \ \beta > 0$$

Smoothed positive and negative rates:$$\hat{p}_{b,\ell,\tau} = \frac{P_{b,\ell,\tau} + \alpha}{V_{b,\ell,\tau} + \alpha + \beta}$$$$\hat{n}_{b,\ell,\tau} = \frac{N_{b,\ell,\tau} + \beta}{V_{b,\ell,\tau} + \alpha + \beta}$$

Then:$$SBS_{b,\ell,\tau} = 100 \cdot (\hat{p}_{b,\ell,\tau} – \hat{n}_{b,\ell,\tau})$$

Equivalent closed form:$$SBS_{b,\ell,\tau} = 100 \cdot \frac{(P_{b,\ell,\tau} – N_{b,\ell,\tau}) + (\alpha – \beta)}{V_{b,\ell,\tau} + \alpha + \beta}$$

With symmetric priors $\alpha=\beta$, this becomes:$$SBS_{b,\ell,\tau} = 100 \cdot \frac{P_{b,\ell,\tau} – N_{b,\ell,\tau}}{V_{b,\ell,\tau} + 2\alpha}$$

Choosing priors

• $\alpha=\beta=1$: Laplace smoothing (simple, common)
• $\alpha=\beta=k/2$: “k pseudo-votes” stabilizer. Pick $k$ based on desired damping.

---

5. Confidence and Uncertainty

You should report SBS alongside a confidence measure.

5.1 Vote-Mass Confidence (simple, executive-friendly)

Map $V$ to $[0,1]$:$$Conf_{b,\ell,\tau} = 1 – e^{-V_{b,\ell,\tau}/\kappa}$$

$\kappa$ sets the scale of stability. Example: $\kappa=20$.

5.2 Credible Interval (statistician-friendly)

Let the posterior for positive rate be:$$p \sim \mathrm{Beta}(P+\alpha, N+\beta)$$

A credible interval for $p$ yields an interval for SBS via transformation:$$SBS = 100 \cdot (2p – 1)$$

Compute:

• $p_{low} = \mathrm{BetaInvCDF}(0.025, P+\alpha, N+\beta)$
• $p_{high} = \mathrm{BetaInvCDF}(0.975, P+\alpha, N+\beta)$

Then:$$SBS_{low} = 100(2p_{low}-1), \quad SBS_{high} = 100(2p_{high}-1)$$

---

6. Benchmarking Methodology

Compute SBS for peer cohorts.

Let cohort $C$ be defined by filters (region, price tier, product line, delivery model, etc.). Aggregate across all builders $b \in C$:$$P_{C,\ell,\tau} = \sum_{b \in C} P_{b,\ell,\tau}$$$$N_{C,\ell,\tau} = \sum_{b \in C} N_{b,\ell,\tau}$$

Compute cohort SBS the same way:$$SBS_{C,\ell,\tau} = 100 \cdot \frac{P_{C,\ell,\tau} – N_{C,\ell,\tau}}{(P_{C,\ell,\tau}+N_{C,\ell,\tau}) + 2\alpha}$$

Then define benchmark delta:$$\Delta_{b,\ell,\tau} = SBS_{b,\ell,\tau} – SBS_{C,\ell,\tau}$$

Percentiles

For each topic $\ell$ in cohort $C$, compute the empirical distribution of $SBS_{b,\ell,\tau}$ and report:

• percentile rank
• quartiles
• z-score (optional, though SBS isn’t guaranteed normal)

---

7. Trend and Volatility

Compute SBS over rolling windows (e.g., monthly). Let $SBS_{b,\ell,t}$ be time series.

Trend slope

Fit OLS on last $n$ periods:$$SBS_{b,\ell,t} = a + bt + \epsilon$$

Report:

• slope $b$ (points/month)
• $R^2$ (signal strength)

Volatility

Compute standard deviation of SBS or of the posterior mean over last $n$ periods:$$Vol_{b,\ell} = \mathrm{StdDev}(SBS_{b,\ell,t})$$

High volatility + low confidence often indicates insufficient volume or operational instability.

---

8. Programmatic Implementation Notes

8.1 Storage

Maintain:

• mentions table (atomic)
• rollup table keyed by (builder, topic, time bucket):
  • pos_weight_sum
  • neg_weight_sum
  • neutral_weight_sum
  • vote_weight_sum
  • counts
• model/prompt versions for reproducibility

8.2 Idempotency

Store comment_processing keyed by:

• comment_id
• model_version
• prompt_version
  and only process once per version.

8.3 Topic Governance

Maintain:

• topic taxonomy
• alias mappings
• “OTHER queue” review process
• periodic embedding re-indexing

8.4 Calibration

Confidence $c_i$ should ideally be calibrated (temperature scaling or isotonic regression) against a labeled validation set. If not, treat it as relative and keep a QA program that measures drift.

---

9. Summary of the SBS Definition

Atomic unit: aspect mention $i$ with $(\ell_i, s_i, m_i, c_i)$

Weight:$$w_i = m_i \cdot c_i$$

Aggregates:$$P = \sum w_i \mathbb{1}[s_i=+1], \quad N = \sum w_i \mathbb{1}[s_i=-1]$$

Smoothed SBS:$$SBS = 100 \cdot \frac{P – N}{P + N + 2\alpha} \quad \text{(for symmetric priors)}$$

Benchmark delta:$$\Delta = SBS_{builder} – SBS_{cohort}$$

Confidence:$$Conf = 1 – e^{-(P+N)/\kappa} \quad \text{or credible intervals via Beta posterior}$$