absoludity Github contribution chart
absoludity Github Stats
absoludity Most Used Languages

Activity

30 Sep 2022

Absoludity

Add overview documentation for the benchmarks, to be filled in for other benchmarks.

Pushed On 30 Sep 2022 at 06:48:56

Absoludity

Update defaults to be more useful (even if different from Flink Nexmark)

Created On 30 Sep 2022 at 02:55:40
Create Branch
Absoludity In vmware/database-stream-processor Create Branchnexmark-better-defaults

Absoludity

Streaming and Incremental Computation Framework

On 30 Sep 2022 at 02:17:00
Issue Comment

Absoludity

Replace `ansi_term` dependency in Pinniped-proxy

Summary Replace ansi_term dependency in the Pinniped-proxy service.

Background and rationale Dependabot is warning that Rust's dependency ansi_term is unmaintained. Project hasn't been updated for at least three years. It is not a direct dependency in pinniped-proxy, but a dependency of clap v2, which in turn is a dependency of structopt. Structopt is not maintained either (see their pinned issue).

Dependabot is warning that it is a critical issue. image

Description A feasible solution is to migrate structopt to clap v3. There is a migration guide here.

Acceptance criteria ansi_term is not used anymore.

Forked On 29 Sep 2022 at 09:07:04

Absoludity

Yep, with the work I've been doing with https://github.com/vmware/database-stream-processor/ I've been using clap v3 directly after seeing structopt's functionality was now included there, and had noted to come back and do the same here. Thanks!

Commented On 29 Sep 2022 at 09:07:04

Absoludity

Add charts

Pushed On 29 Sep 2022 at 06:55:38

Absoludity

Clarify use of local VMs.

Pushed On 29 Sep 2022 at 04:55:56

Absoludity

Add playbook for DBSP setup.

Pushed On 29 Sep 2022 at 04:49:32

Absoludity

Add section for debugging current nexmark issues

Pushed On 27 Sep 2022 at 08:12:52

Absoludity

`OrderedColumnLeaf` work

  • Switches OrdKeyBatch and OrdValBatch to use OrderedColumnLeaf
  • Adds some OrderedColumnLeaf benchmarks
  • Switches UnorderedColumnLeafBuilder to use columns of values instead of a Vec<(K, R)>

Forked On 27 Sep 2022 at 12:02:35

Absoludity

I also enhanced nexmark query parsing somewhat, where previously invalid query names passed to --query would result in a silent failure (that is, using --query something_invalid would simply do nothing), it now knows what the available queries are and therefore can validate them, which leads to this behavior now

λ  cargo bench --bench nexmark ... --query something_invalid
error: "something_invalid" isn't a valid value for '--query <QUERY>'
        [possible values: q0, q1, q2, q3, q4, q5, q6, q7, q8, q9, q12, q13, q14, q15, q16, q17, q18, q19, q20, q21, q22] 

Excellent, thanks @Kixiron .

I also took advantage of using mimalloc to get more/richer/better stats on each query

Nice, thanks! Yeah, I did look at using mimalloc when I saw you use it in a PR the other week, but I didn't spend enough time looking through your use of it to understand it well enough to use here.

From @ryzhyk's comment above, looks like he was just waiting for the performance numbers, which you've shown now to be minimally different?

Commented On 27 Sep 2022 at 12:02:35

Absoludity

Initial playbook for running Nexmark Flink benchmarks.

Signed-off-by: absoludity absoludity@gmail.com

Pushed On 26 Sep 2022 at 12:58:37

Absoludity

Automated setup for nexmark flink benchmark tests, and possibly DBSP.

On 23 Sep 2022 at 05:09:28

Absoludity

Automated setup for nexmark flink benchmark tests, and possibly DBSP.

On 23 Sep 2022 at 05:09:28

Absoludity

Clippy

Pushed On 20 Sep 2022 at 10:27:24

Absoludity

[nexmark] Add q22 - Flink's SPLIT_INDEX but a trivial split for Rust

Created On 20 Sep 2022 at 09:01:47
Create Branch

Absoludity

Streaming and Incremental Computation Framework

On 20 Sep 2022 at 08:58:35

Absoludity

Clippy

Pushed On 20 Sep 2022 at 07:30:34

Absoludity

[nexmark] Add Nexmark q21 demoing case-when and regular exprs.

Created On 20 Sep 2022 at 06:56:54
Create Branch

Absoludity

Streaming and Incremental Computation Framework

On 20 Sep 2022 at 06:55:02

Absoludity

Don't depend on Bid model ordering

Pushed On 20 Sep 2022 at 12:28:40

Absoludity

[nexmark] Add Nexmark q19 - Illustrating a TOP-N query.

Signed-off-by: Michael Nelson minelson@vmware.com

Using Fold, as suggested. Suffers from high memory usage, so that again, I can run with 10M events:

cargo bench --bench nexmark --features with-nexmark -- --first-event-rate=10000000 --max-events=10000000 --cpu-cores 8 --query q19 --num-event-generators 6 --source-buffer-size 10000 --input-batch-size 40000      
   Compiling dbsp v0.1.0 (/home/michael/dev/vmware/database-stream-processor)
    Finished bench [optimized + debuginfo] target(s) in 3m 40s
     Running benches/nexmark/main.rs (target/release/deps/nexmark-733304a45ec676db)
Starting q19 bench of 10000000 events...
10,000,000 / 10,000,000 [===================================================================================================================================================================================================] 100 % 1846403.3887/s 0s
┌───────┬────────────┬───────┬─────────┬─────────────────┬──────────────────┬───────────────┬───────────────┬─────────────┐
│ Query │ #Events    │ Cores │ Elapsed │ Cores * Elapsed │ Throughput/Cores │ Total Usr CPU │ Total Sys CPU │ Max RSS(Kb) │
├───────┼────────────┼───────┼─────────┼─────────────────┼──────────────────┼───────────────┼───────────────┼─────────────┤
│ q19   │ 10,000,000 │ 8     │ 5.417s  │ 43.336s         │ 230.753 K/s      │ 35.194s       │ 1.454s        │ 2,100,076   │
└───────┴────────────┴───────┴─────────┴─────────────────┴──────────────────┴───────────────┴───────────────┴─────────────┘ 

But not 100M:

cargo bench --bench nexmark --features with-nexmark -- --first-event-rate=10000000 --max-events=100000000 --cpu-cores 8 --query q19 --num-event-generators 6 --source-buffer-size 10000 --input-batch-size 40000 
    Finished bench [optimized + debuginfo] target(s) in 0.10s
     Running benches/nexmark/main.rs (target/release/deps/nexmark-733304a45ec676db)
Starting q19 bench of 100000000 events...
81,760,000 / 100,000,000 [===============================================================================================================================================================>-----------------------------------] 82 % 2165128.9594/s 8serror: bench failed, to rerun pass `--bench nexmark`

Caused by:
  process didn't exit successfully: `/home/michael/dev/vmware/database-stream-processor/target/release/deps/nexmark-733304a45ec676db --first-event-rate=10000000 --max-events=100000000 --cpu-cores 8 --query q19 --num-event-generators 6 --source-buffer-size 10000 --input-batch-size 40000 --bench` (signal: 9, SIGKILL: kill) 

Forked On 20 Sep 2022 at 12:05:46

Absoludity

Sorry, just noticed that the sorted behavior currently relies on the order of the `Bid` Model (it just so happened that my tests all used the same bidder, so it worked). Fixing...
On 20 Sep 2022 at 12:05:46

Absoludity

[nexmark] Add Nexmark q19 - Illustrating a TOP-N query.

Signed-off-by: Michael Nelson minelson@vmware.com

Using Fold, as suggested. Suffers from high memory usage, so that again, I can run with 10M events:

cargo bench --bench nexmark --features with-nexmark -- --first-event-rate=10000000 --max-events=10000000 --cpu-cores 8 --query q19 --num-event-generators 6 --source-buffer-size 10000 --input-batch-size 40000      
   Compiling dbsp v0.1.0 (/home/michael/dev/vmware/database-stream-processor)
    Finished bench [optimized + debuginfo] target(s) in 3m 40s
     Running benches/nexmark/main.rs (target/release/deps/nexmark-733304a45ec676db)
Starting q19 bench of 10000000 events...
10,000,000 / 10,000,000 [===================================================================================================================================================================================================] 100 % 1846403.3887/s 0s
┌───────┬────────────┬───────┬─────────┬─────────────────┬──────────────────┬───────────────┬───────────────┬─────────────┐
│ Query │ #Events    │ Cores │ Elapsed │ Cores * Elapsed │ Throughput/Cores │ Total Usr CPU │ Total Sys CPU │ Max RSS(Kb) │
├───────┼────────────┼───────┼─────────┼─────────────────┼──────────────────┼───────────────┼───────────────┼─────────────┤
│ q19   │ 10,000,000 │ 8     │ 5.417s  │ 43.336s         │ 230.753 K/s      │ 35.194s       │ 1.454s        │ 2,100,076   │
└───────┴────────────┴───────┴─────────┴─────────────────┴──────────────────┴───────────────┴───────────────┴─────────────┘ 

But not 100M:

cargo bench --bench nexmark --features with-nexmark -- --first-event-rate=10000000 --max-events=100000000 --cpu-cores 8 --query q19 --num-event-generators 6 --source-buffer-size 10000 --input-batch-size 40000 
    Finished bench [optimized + debuginfo] target(s) in 0.10s
     Running benches/nexmark/main.rs (target/release/deps/nexmark-733304a45ec676db)
Starting q19 bench of 100000000 events...
81,760,000 / 100,000,000 [===============================================================================================================================================================>-----------------------------------] 82 % 2165128.9594/s 8serror: bench failed, to rerun pass `--bench nexmark`

Caused by:
  process didn't exit successfully: `/home/michael/dev/vmware/database-stream-processor/target/release/deps/nexmark-733304a45ec676db --first-event-rate=10000000 --max-events=100000000 --cpu-cores 8 --query q19 --num-event-generators 6 --source-buffer-size 10000 --input-batch-size 40000 --bench` (signal: 9, SIGKILL: kill) 

Merged On 20 Sep 2022 at 12:05:46

Absoludity

Commented On 20 Sep 2022 at 12:05:46

Absoludity

[nexmark] Add Nexmark q19 - Illustrating a TOP-N query.

Signed-off-by: Michael Nelson minelson@vmware.com

Using Fold, as suggested. Suffers from high memory usage, so that again, I can run with 10M events:

cargo bench --bench nexmark --features with-nexmark -- --first-event-rate=10000000 --max-events=10000000 --cpu-cores 8 --query q19 --num-event-generators 6 --source-buffer-size 10000 --input-batch-size 40000      
   Compiling dbsp v0.1.0 (/home/michael/dev/vmware/database-stream-processor)
    Finished bench [optimized + debuginfo] target(s) in 3m 40s
     Running benches/nexmark/main.rs (target/release/deps/nexmark-733304a45ec676db)
Starting q19 bench of 10000000 events...
10,000,000 / 10,000,000 [===================================================================================================================================================================================================] 100 % 1846403.3887/s 0s
┌───────┬────────────┬───────┬─────────┬─────────────────┬──────────────────┬───────────────┬───────────────┬─────────────┐
│ Query │ #Events    │ Cores │ Elapsed │ Cores * Elapsed │ Throughput/Cores │ Total Usr CPU │ Total Sys CPU │ Max RSS(Kb) │
├───────┼────────────┼───────┼─────────┼─────────────────┼──────────────────┼───────────────┼───────────────┼─────────────┤
│ q19   │ 10,000,000 │ 8     │ 5.417s  │ 43.336s         │ 230.753 K/s      │ 35.194s       │ 1.454s        │ 2,100,076   │
└───────┴────────────┴───────┴─────────┴─────────────────┴──────────────────┴───────────────┴───────────────┴─────────────┘ 

But not 100M:

cargo bench --bench nexmark --features with-nexmark -- --first-event-rate=10000000 --max-events=100000000 --cpu-cores 8 --query q19 --num-event-generators 6 --source-buffer-size 10000 --input-batch-size 40000 
    Finished bench [optimized + debuginfo] target(s) in 0.10s
     Running benches/nexmark/main.rs (target/release/deps/nexmark-733304a45ec676db)
Starting q19 bench of 100000000 events...
81,760,000 / 100,000,000 [===============================================================================================================================================================>-----------------------------------] 82 % 2165128.9594/s 8serror: bench failed, to rerun pass `--bench nexmark`

Caused by:
  process didn't exit successfully: `/home/michael/dev/vmware/database-stream-processor/target/release/deps/nexmark-733304a45ec676db --first-event-rate=10000000 --max-events=100000000 --cpu-cores 8 --query q19 --num-event-generators 6 --source-buffer-size 10000 --input-batch-size 40000 --bench` (signal: 9, SIGKILL: kill) 

Forked On 19 Sep 2022 at 11:57:49

Absoludity

Not sure yet if there's a better way, but because the Fold needs an `&mut VecDeque<Bid>`, the vec here is also a reference, which means that without dereferencing the vec (and then requiring a clone :/ ), the output type is also a referenc: ```diff diff --git a/src/nexmark/queries/q19.rs b/src/nexmark/queries/q19.rs index 729f6f0..3f45aa5 100644 --- a/src/nexmark/queries/q19.rs +++ b/src/nexmark/queries/q19.rs @@ -52,7 +52,7 @@ pub fn q19(input: NexmarkStream) -> Q19Stream { top.push_back(val.clone()); }, )) - .flat_map(|(_, vec)| -> VecDeque<Bid> { (*vec).clone() }) + .flat_map(|(_, vec)| vec) } #[cfg(test)] ``` ``` error[E0308]: mismatched types --> src/nexmark/queries/q19.rs:45:5 | 39 | pub fn q19(input: NexmarkStream) -> Q19Stream { | --------- expected `Stream<circuit_builder::Circuit<()>, zset_batch::OrdZSet<nexmark::model::Bid, isize>>` because of return type ... 45 | / bids_by_auction 46 | | .aggregate::<(), _>(Fold::new( 47 | | VecDeque::with_capacity(TOP_BIDS), 48 | | |top: &mut VecDeque<Bid>, val: &Bid, _w| { ... | 54 | | )) 55 | | .flat_map(|(_, vec)| vec) | |_________________________________^ expected struct `nexmark::model::Bid`, found `&nexmark::model::Bid` | = note: expected struct `Stream<_, zset_batch::OrdZSet<nexmark::model::Bid, _>>` found struct `Stream<_, zset_batch::OrdZSet<&nexmark::model::Bid, _>>` For more information about this error, try `rustc --explain E0308`. error: could not compile `dbsp` due to previous error ```
On 19 Sep 2022 at 11:57:49

Absoludity

[nexmark] Add Nexmark q19 - Illustrating a TOP-N query.

Signed-off-by: Michael Nelson minelson@vmware.com

Using Fold, as suggested. Suffers from high memory usage, so that again, I can run with 10M events:

cargo bench --bench nexmark --features with-nexmark -- --first-event-rate=10000000 --max-events=10000000 --cpu-cores 8 --query q19 --num-event-generators 6 --source-buffer-size 10000 --input-batch-size 40000      
   Compiling dbsp v0.1.0 (/home/michael/dev/vmware/database-stream-processor)
    Finished bench [optimized + debuginfo] target(s) in 3m 40s
     Running benches/nexmark/main.rs (target/release/deps/nexmark-733304a45ec676db)
Starting q19 bench of 10000000 events...
10,000,000 / 10,000,000 [===================================================================================================================================================================================================] 100 % 1846403.3887/s 0s
┌───────┬────────────┬───────┬─────────┬─────────────────┬──────────────────┬───────────────┬───────────────┬─────────────┐
│ Query │ #Events    │ Cores │ Elapsed │ Cores * Elapsed │ Throughput/Cores │ Total Usr CPU │ Total Sys CPU │ Max RSS(Kb) │
├───────┼────────────┼───────┼─────────┼─────────────────┼──────────────────┼───────────────┼───────────────┼─────────────┤
│ q19   │ 10,000,000 │ 8     │ 5.417s  │ 43.336s         │ 230.753 K/s      │ 35.194s       │ 1.454s        │ 2,100,076   │
└───────┴────────────┴───────┴─────────┴─────────────────┴──────────────────┴───────────────┴───────────────┴─────────────┘ 

But not 100M:

cargo bench --bench nexmark --features with-nexmark -- --first-event-rate=10000000 --max-events=100000000 --cpu-cores 8 --query q19 --num-event-generators 6 --source-buffer-size 10000 --input-batch-size 40000 
    Finished bench [optimized + debuginfo] target(s) in 0.10s
     Running benches/nexmark/main.rs (target/release/deps/nexmark-733304a45ec676db)
Starting q19 bench of 100000000 events...
81,760,000 / 100,000,000 [===============================================================================================================================================================>-----------------------------------] 82 % 2165128.9594/s 8serror: bench failed, to rerun pass `--bench nexmark`

Caused by:
  process didn't exit successfully: `/home/michael/dev/vmware/database-stream-processor/target/release/deps/nexmark-733304a45ec676db --first-event-rate=10000000 --max-events=100000000 --cpu-cores 8 --query q19 --num-event-generators 6 --source-buffer-size 10000 --input-batch-size 40000 --bench` (signal: 9, SIGKILL: kill) 

Merged On 19 Sep 2022 at 11:57:50

Absoludity

Commented On 19 Sep 2022 at 11:57:50

Absoludity

[nexmark] Add Nexmark q19 - Illustrating a TOP-N query.

Created On 19 Sep 2022 at 11:52:32
Create Branch

Absoludity

Streaming and Incremental Computation Framework

On 19 Sep 2022 at 11:50:12

Absoludity

[nexmark] Add q20 - illustrating filter-join. (#192)

  • [nexmark] Add q20 - illustrating filter-join.

Signed-off-by: Michael Nelson minelson@vmware.com

  • Update to filter by category.

Signed-off-by: Michael Nelson minelson@vmware.com

Pushed On 19 Sep 2022 at 11:42:50

Absoludity

[nexmark] Add q20 - illustrating filter-join.

Created On 19 Sep 2022 at 11:42:50

Absoludity

[nexmark] Add q18 - illustrating deduplicate query. (#191)

  • [nexmark] Add q18 - illustrating deduplicate query.

Signed-off-by: Michael Nelson minelson@vmware.com

  • Remove unnecessary join

Signed-off-by: Michael Nelson minelson@vmware.com

Pushed On 19 Sep 2022 at 11:26:03

Absoludity

[nexmark] Add q20 - illustrating filter-join.

Signed-off-by: Michael Nelson minelson@vmware.com

Pushed On 19 Sep 2022 at 11:26:03

Absoludity

Update to filter by category.

Pushed On 19 Sep 2022 at 11:26:03