macobo Github contribution chart
macobo Github Stats
macobo Most Used Languages

Activity

29 Sep 2022

Macobo

fix(historical-exports): Fix resume logic (#12031)

Previously, historical exports v2 would consistently try to resume chunks that didn't need resuming.

This is because the start date of the export kept overwriting the current date.

This fixes that issue

Pushed On 29 Sep 2022 at 12:01:33
Pull Request

Macobo

fix(historical-exports): Fix resume logic

Created On 29 Sep 2022 at 12:01:32

Macobo

fix(historical-exports): Fix resume logic

Previously, historical exports v2 would consistently try to resume chunks that didn't need resuming.

This is because the start date of the export kept overwriting the current date.

This fixes that issue

Pushed On 29 Sep 2022 at 11:41:25

Macobo

fix(historical-exports): Fix resume logic

Previously, historical exports v2 would consistently try to resume chunks that didn't need resuming.

This is because the start date of the export kept overwriting the current date.

This fixes that issue

Pushed On 29 Sep 2022 at 11:39:48
Pull Request

Macobo

fix(historical-exports): Fix resume logic

Created On 29 Sep 2022 at 11:29:57
Create Branch
Macobo In PostHog/posthog Create Branchfix-resume-logic

Macobo

🦔 PostHog provides open-source product analytics, session recording, feature flagging and a/b testing that you can self-host.

On 29 Sep 2022 at 11:29:49

Macobo

feat(apps): inject javascript from decide

Changes

When the /decide endpoints contains code in the format:

{
    inject: [{ 
        id: 9,
        source: 'export function inject({ config, posthog }) { console.log("hello"); }', 
        config: { key: 'value' } ,
    }]
} 

then posthog-js will execute it.

This is needed to get https://github.com/PostHog/posthog/pull/12003 working.

What do we send and when is still up for debate in that linked issue, though getting this in will greatly help local dev and testing for apps at the hackathon here in Rome.

Checklist

Merged On 29 Sep 2022 at 10:39:29

Macobo

YOLO

Commented On 29 Sep 2022 at 10:39:29

Macobo

chore(historical-exports): add debug logging for resume code (#12026)

Testing V2 exports live there seem to be issues around resuming. This debug log will help me figure out why

Pushed On 29 Sep 2022 at 09:59:07
Pull Request

Macobo

chore(historical-exports): add debug logging for resume code

Created On 29 Sep 2022 at 09:59:06
Pull Request

Macobo

chore(historical-exports): add debug logging for resume code

Created On 29 Sep 2022 at 09:35:26
Create Branch
Macobo In PostHog/posthog Create Branchadd-debug-logging-v2-exports

Macobo

🦔 PostHog provides open-source product analytics, session recording, feature flagging and a/b testing that you can self-host.

On 29 Sep 2022 at 09:35:17
Issue Comment

Macobo

Proposal: PostHog standard fields - to reduce time to spark joy

Is your feature request related to a problem?

A hurdle to getting value out of PostHog is the lengthy process it takes to define actions, figure out what's important to track, create filters on an insight, and then add insights to a dashboard.

The default dashboard is interesting (and gets a step closer), but the action creation process still has to be completed first, then the user needs to revisit the report to configure their insights (by editing default dashboard insights).

Describe the solution you'd like

This proposal is for defining a set of standard fields/events that, when a customer matches their actions to these fields, allows us to do some magical things. ✨

  1. Automatic insight creation
  2. Automatic dashboard creation
  3. Signals for positive/negative events (correlation hints, provided by the customer)

Mapping actions to standard fields

Roughly using the AARRR tutorial as a guide, I abstracted the most common fields and added them to this list (which would need to be debated/finalized):

  • Row 1 shows a field already mapped
  • Row 3 shows the hover state, which gives the option to map the action
  • The final row shows the options I'm proposing (which would be valuable either for current use or future automatic analysis)

Proposed list of fields

Acquisition

  • Visited website (traffic to domain)
  • Conversion funnel event (any event involved in a conversion funnel - order can be set later)

Activation

  • KPI (array)

Retention

  • Signed in
  • Closed account

Revenue

  • Created subscription
  • Canceled subscription

Field mapping when sending a custom event

Imagine being able to send a parameter that matches the string of the standard field, which would eliminate the need to manually map them in Data Management.

In the toolbar

This process doesn't have to take place en masse in Data Management, since it can be added directly to the Toolbar.

Detecting unused fields and offering suggestions

When a relevant standard field is mapped but unused (think: onboarding), we could prompt to create a standard insight using the data.

All events labeled Conversion funnel event could appear, which leaves the customer only to drag and re-order (or remove superfluous steps) to very quickly create a conversion funnel.

Dashboards

Using the sample dashboard as inspiration, we can auto-populate these with the data provided. In the sample conversion insight we already include, we'd just need to replace the dummy funnel steps with the real ones (as defined in the example above).

The same can be done for pageviews, traffic sources, top pages, and more.

Note: If we also helped map event properties, we'd be able to automatically create useful insights like breakdowns by plan, etc.

Extending the use case

We don't do a great job at offering an apples to apples service with GA for website analytics, and with the number of GA refugees, it leaves open an opportunity. A relatively easy way to address this is to create a marketing dashboard template.

Here's a rough list at what I think would make for a compelling marketing dashboard, all of which could be generated between built-in events and field mapped-events.

| HIGH LEVEL STATS | | |--- |--- | | Visitors | count | | Pageviews | | | Avg session length | percent | | | | | TRAFFIC | | | Breakdown by device | graph, pill | | Source | bar | | Referrers by source | table | | Referrers by UTM source | table | | | | | CONTENT | | | Top pages | table | | Top converting pages | table | | | | | USERS | | | Session per user | count | | Users by plan | | | Geography | map | | | | | OTHER | | | New Twitter followers | bar, graph |

Using the above, it wouldn't be too hard to also make a "live" dashboard, showing what's currently happening on a customer's website. (Refreshing the insights every 10-15 seconds should feel enough.)

Summary

By allowing users to provide more context to their events, we can get them to insights faster, while collecting data we can use to train an ML model about what success and failure looks like, and potentially open up to a ripe market without too much additional work.

Forked On 29 Sep 2022 at 07:33:02

Macobo

Question on 'Mapping actions to standard fields': why not do this for custom events as well?

Commented On 29 Sep 2022 at 07:33:02

Macobo

feat(ingestion): Protect against undefined, verify $anon_distinct_id/alias as well (#12010)

  • chore(ingestion): Don't allow merging via user id undefined

Ticket: https://github.com/PostHog/posthog/issues/12008

  • Verify alias/$anon_distinct_id is valid before merging

  • Update tests to match style of rest of file

  • Attempt at follow-up

  • Fix a typo

Pushed On 29 Sep 2022 at 07:31:15
Pull Request

Macobo

feat(ingestion): Protect against `undefined`, verify `$anon_distinct_id`/`alias` as well

Created On 29 Sep 2022 at 07:31:14
Merge

Macobo

feat(ingestion): Protect against `undefined`, verify `$anon_distinct_id`/`alias` as well

Problem

https://github.com/PostHog/posthog/issues/12008

Changes

  • undefined as a distinct_id is now banned from merging
  • We now verify $anon_distinct_id/`alias as well
  • Improved test coverage around identify code

How did you test this code?

See person-state tests.

Forked On 29 Sep 2022 at 07:30:42

Macobo

It's guaranteed
On 29 Sep 2022 at 07:30:42

Macobo

feat(ingestion): Protect against `undefined`, verify `$anon_distinct_id`/`alias` as well

Problem

https://github.com/PostHog/posthog/issues/12008

Changes

  • undefined as a distinct_id is now banned from merging
  • We now verify $anon_distinct_id/`alias as well
  • Improved test coverage around identify code

How did you test this code?

See person-state tests.

Merged On 29 Sep 2022 at 07:30:42

Macobo

Commented On 29 Sep 2022 at 07:30:42

Macobo

Fix a typo

Pushed On 29 Sep 2022 at 07:13:12
Issue Comment

Macobo

Async migration 0007_persons_and_groups_on_events_backfill get stcuk at 82%

Bug description

When running the async migration 0007_persons_and_groups_on_events_backfill vie the user interface, the migration gets stucked at 82%. It progress superfast to 82 and then gets stucked. image

How to reproduce

  1. Upgrade selfhosted helm to 1.40.0
  2. Run 0007_persons_and_groups_on_events_backfill in the async migration UI
  3. Wait forever?

Environment

  • [ ] PostHog Cloud
  • [x] self-hosted PostHog, version/commit: 1.40.0

Additional context

  • When the plugins pod restarts it crashes in a restart loop because migrations are currently running. Stopped it after 12h.
  • We only have 18 Mio vents, so this should not impacting.
  • The same happened with 0006_persons_and_groups_on_events_backfill in the oprevious version. Just ignored running it

Forked On 29 Sep 2022 at 06:43:16

Macobo

This isn't necessarily a bug - this step might take a while as it's waiting for the data to finish backfilling.

Could you follow this guide to connect to ClickHouse and run the following queries for me?

  • SELECT count(*) FROM events
  • SELECT count(*) FROM events WHERE empty(person_id)
  • SELECT count(*) FROM events WHERE person_created_at = toDateTime(0)
  • SELECT toStartOfWeek(timestamp) as week, count(*) FROM events WHERE person_properties = '' GROUP BY week ORDER BY week
-- Measuring mutation progress (e.g. the backfill). Note that progress estimation is very hit-or-miss since different parts have different sizes.
WITH (
    select count() from system.parts where table = 'sharded_events' and active
) AS active_parts
SELECT 1 - (parts_to_do / active_parts) as progress, parts_to_do, * 
FROM system.mutations
WHERE not is_done and table = 'sharded_events' 
select table, round(elapsed) as elapsed, round(progress, 3) as progress, round((elapsed / progress) - elapsed) as estimation_todo, is_mutation, * from system.merges 
-- Dictionary stats. Useful for verifying cache hit rates.
select name, query_count, hit_rate, found_rate, element_count, load_factor, formatReadableSize(bytes_allocated) as allocated from system.dictionaries; 
SELECT * FROM system.disks

select * from system.asynchronous_metrics where metric LIKE '%IO%' 

These should help us gauge whether the mutation is making progress or if it's stuck.

When the plugins pod restarts it crashes in a restart loop because migrations are currently running. Stopped it after 12h.

You can set ASYNC_MIGRATIONS_BLOCK_UPGRADE to false under instance settings or via an env var. This way pods restarting will not block behind the async migration.

Commented On 29 Sep 2022 at 06:43:16
Merge

Macobo

feat: person state test coverage increase

Problem

Test coverage for $identify and $create_alias calls was limited, but it's an important and complex part of our ingestion.

Changes

We've had problems, where we don't update Postgres properly or ClickHouse or the personContainer so just being exhaustive in verifying all of them have the right values.

WIP: The distinct_id for clickhouse wait doesn't guarantee we did all the processing needed (during merges etc)

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

Forked On 29 Sep 2022 at 06:16:25

Macobo

This is redundant given no updates occur
On 29 Sep 2022 at 06:16:25

Macobo

feat: person state test coverage increase

Problem

Test coverage for $identify and $create_alias calls was limited, but it's an important and complex part of our ingestion.

Changes

We've had problems, where we don't update Postgres properly or ClickHouse or the personContainer so just being exhaustive in verifying all of them have the right values.

WIP: The distinct_id for clickhouse wait doesn't guarantee we did all the processing needed (during merges etc)

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

Merged On 29 Sep 2022 at 06:16:26

Macobo

Commented On 29 Sep 2022 at 06:16:26
Merge

Macobo

feat: person state test coverage increase

Problem

Test coverage for $identify and $create_alias calls was limited, but it's an important and complex part of our ingestion.

Changes

We've had problems, where we don't update Postgres properly or ClickHouse or the personContainer so just being exhaustive in verifying all of them have the right values.

WIP: The distinct_id for clickhouse wait doesn't guarantee we did all the processing needed (during merges etc)

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

Forked On 29 Sep 2022 at 06:15:56

Macobo

Note: We have so much comments per test that this comment just gets lost in the noise. Is this an important aspect of the test?
On 29 Sep 2022 at 06:15:56

Macobo

feat: person state test coverage increase

Problem

Test coverage for $identify and $create_alias calls was limited, but it's an important and complex part of our ingestion.

Changes

We've had problems, where we don't update Postgres properly or ClickHouse or the personContainer so just being exhaustive in verifying all of them have the right values.

WIP: The distinct_id for clickhouse wait doesn't guarantee we did all the processing needed (during merges etc)

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

Merged On 29 Sep 2022 at 06:15:56

Macobo

Commented On 29 Sep 2022 at 06:15:56
Merge

Macobo

feat: person state test coverage increase

Problem

Test coverage for $identify and $create_alias calls was limited, but it's an important and complex part of our ingestion.

Changes

We've had problems, where we don't update Postgres properly or ClickHouse or the personContainer so just being exhaustive in verifying all of them have the right values.

WIP: The distinct_id for clickhouse wait doesn't guarantee we did all the processing needed (during merges etc)

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

Forked On 29 Sep 2022 at 06:12:16

Macobo

This test description is confusing as it's describing the end state but not the data going in with terminology not used anywhere in the code (anon person). Given these tests are ~50 lines long it's really hard to find what's the important thing in this code it's critical we get descriptions correct. ```suggestion it('adds distinct_id and updates properties when $anon_distinct_id is passed and does not exist', async () => { ```
On 29 Sep 2022 at 06:12:16

Macobo

feat: person state test coverage increase

Problem

Test coverage for $identify and $create_alias calls was limited, but it's an important and complex part of our ingestion.

Changes

We've had problems, where we don't update Postgres properly or ClickHouse or the personContainer so just being exhaustive in verifying all of them have the right values.

WIP: The distinct_id for clickhouse wait doesn't guarantee we did all the processing needed (during merges etc)

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

Merged On 29 Sep 2022 at 06:12:17

Macobo

Commented On 29 Sep 2022 at 06:12:17

Macobo

feat: person state test coverage increase

Problem

Test coverage for $identify and $create_alias calls was limited, but it's an important and complex part of our ingestion.

Changes

We've had problems, where we don't update Postgres properly or ClickHouse or the personContainer so just being exhaustive in verifying all of them have the right values.

WIP: The distinct_id for clickhouse wait doesn't guarantee we did all the processing needed (during merges etc)

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

Merged On 29 Sep 2022 at 06:09:21

Macobo

Commented On 29 Sep 2022 at 06:09:21