Modeling product-usage metrics
PostHog/posthogSource at 326da88 (opens GitHub)
Nothing to cut.
We built a cheaper version and ran both on the same tasks. It wasn't better, so the original stays as it is.
- We tried prepared facts, and didn't keep it because it didn't use fewer tokens than the original.
What AIR found
Not checked yetSecurity review
AIR hasn't reviewed this skill yet. Check it to run the review.
On a smaller model
On a smaller model, the results dropped.
A separate result from the token saving: this is the cost of running the same skill on a smaller model, and the two are never added together.
| Task | Results | Cost |
|---|---|---|
| Retention mobile app config | Lower results | 35% cheaper |
| Metrics map product questions | Not compared | Not compared |