bennnjamin Github contribution chart
bennnjamin Github Stats
bennnjamin Most Used Languages

Activity

29 Aug 2022

Issue Comment

Bennnjamin

Running an async task in a synchronous application

Hi,

I am looking to do what I hope is a very simple task (at least in my head) but I'm struggling to wrap my head around how amphp is supposed to work. I have a synchronous application with a problem that I thought amphp would be perfect to solve. Picture the following:

  1. A client uploads a video to our site via AJAX.
  2. Our site sends the video to a 3rd-party site which automatically generates a thumbnail for the video.
  3. The 3rd-party site takes some time to generate the thumbnail depending on the video size/length etc. Unfortunately, there's no way to track this and the 3rd-party site does not inform us when it is ready either.

Given the above scenario, our site basically has to query the 3rd-party site every 5 seconds for the thumbnail from the moment the video is successfully uploaded to the 3rd-party site. This process could take anywhere from a few seconds to minutes. I figured doing this particular task via an async PHP script/function would be the best solution and that's where amphp comes in. So the above scenario with amphp involved becomes this:

  1. A client uploads a video to our site via AJAX.
  2. Our site sends the video to a 3rd-party site which automatically generates a thumbnail for the video.
  3. An amphp async task is created to get the thumbnail from the 3rd-party site.
  4. The AJAX response is returned.
  5. Sometime later... the DB is updated with the thumbnail results from the async task in step 3. I also plan to send a ZMQ request back to the client if they're still online to update the thumbnail.

I thought I could do the above by following this example (from amphp.org/amp/promises/):

use Amp\Loop;

function asyncMultiply($x, $y)
{
    // Create a new promisor
    $deferred = new Amp\Deferred;

    // Resolve the async result 30s from now
    Loop::delay($msDelay = 30000, function () use ($deferred, $x, $y) {
        $deferred->resolve($x * $y);
    });

    return $deferred->promise();
}

// Some code here...
$promise = asyncMultiply(6, 7);
$result = Amp\Promise\wait($promise);

var_dump($result); // int(42)
// More code here... 

However, the above example must have gone right over my head because when I ran that directly in my synchronous application, it seemed to just wait for the 30 seconds. My expectation was that this was an async task, and the Some code here and More code here would execute without waiting for the result from the asyncMultiply(6, 7) since that was an async task. What am I missing?

Furthermore, presuming I'm not far off from achieving my task, would this approach be a problem if there were a few clients simultaneously uploading a video to our site via AJAX (i.e. our application would be running multiple async tasks at the same time) ?

Note: Not sure if relevant but the application is running in a WordPress/PHP-FPM environment.

Forked On 29 Aug 2022 at 10:00:36

Bennnjamin

I'm not familiar with React's versioning semantics so you may need to open an issue with them. react-adapter allows you to run any React PHP library the expects a LoopInterface on the Amp event loop. If you are using React, it might be simpler to stick with only react libraries for now (dropping any Amp libs), or create a simple project that uses react-adapter to run React libraries on the Amp event loop.

Commented On 29 Aug 2022 at 10:00:36
Issue Comment

Bennnjamin

Running an async task in a synchronous application

Hi,

I am looking to do what I hope is a very simple task (at least in my head) but I'm struggling to wrap my head around how amphp is supposed to work. I have a synchronous application with a problem that I thought amphp would be perfect to solve. Picture the following:

  1. A client uploads a video to our site via AJAX.
  2. Our site sends the video to a 3rd-party site which automatically generates a thumbnail for the video.
  3. The 3rd-party site takes some time to generate the thumbnail depending on the video size/length etc. Unfortunately, there's no way to track this and the 3rd-party site does not inform us when it is ready either.

Given the above scenario, our site basically has to query the 3rd-party site every 5 seconds for the thumbnail from the moment the video is successfully uploaded to the 3rd-party site. This process could take anywhere from a few seconds to minutes. I figured doing this particular task via an async PHP script/function would be the best solution and that's where amphp comes in. So the above scenario with amphp involved becomes this:

  1. A client uploads a video to our site via AJAX.
  2. Our site sends the video to a 3rd-party site which automatically generates a thumbnail for the video.
  3. An amphp async task is created to get the thumbnail from the 3rd-party site.
  4. The AJAX response is returned.
  5. Sometime later... the DB is updated with the thumbnail results from the async task in step 3. I also plan to send a ZMQ request back to the client if they're still online to update the thumbnail.

I thought I could do the above by following this example (from amphp.org/amp/promises/):

use Amp\Loop;

function asyncMultiply($x, $y)
{
    // Create a new promisor
    $deferred = new Amp\Deferred;

    // Resolve the async result 30s from now
    Loop::delay($msDelay = 30000, function () use ($deferred, $x, $y) {
        $deferred->resolve($x * $y);
    });

    return $deferred->promise();
}

// Some code here...
$promise = asyncMultiply(6, 7);
$result = Amp\Promise\wait($promise);

var_dump($result); // int(42)
// More code here... 

However, the above example must have gone right over my head because when I ran that directly in my synchronous application, it seemed to just wait for the 30 seconds. My expectation was that this was an async task, and the Some code here and More code here would execute without waiting for the result from the asyncMultiply(6, 7) since that was an async task. What am I missing?

Furthermore, presuming I'm not far off from achieving my task, would this approach be a problem if there were a few clients simultaneously uploading a video to our site via AJAX (i.e. our application would be running multiple async tasks at the same time) ?

Note: Not sure if relevant but the application is running in a WordPress/PHP-FPM environment.

Forked On 29 Aug 2022 at 06:28:22

Bennnjamin

You do need to ensure you are using amphp/react-adapter as @kelunik mentioned. This is because your application can only have one event loop running. Both Amp and React provide their own event loop implementations. The adapter allows you to mix both Amp and React libraries in the same application, which is great!

You could also try just getting the example to work with React's own Promise library (https://github.com/reactphp/promise) since it seems you already have an event loop running under React.

Commented On 29 Aug 2022 at 06:28:22
Issue Comment

Bennnjamin

Running an async task in a synchronous application

Hi,

I am looking to do what I hope is a very simple task (at least in my head) but I'm struggling to wrap my head around how amphp is supposed to work. I have a synchronous application with a problem that I thought amphp would be perfect to solve. Picture the following:

  1. A client uploads a video to our site via AJAX.
  2. Our site sends the video to a 3rd-party site which automatically generates a thumbnail for the video.
  3. The 3rd-party site takes some time to generate the thumbnail depending on the video size/length etc. Unfortunately, there's no way to track this and the 3rd-party site does not inform us when it is ready either.

Given the above scenario, our site basically has to query the 3rd-party site every 5 seconds for the thumbnail from the moment the video is successfully uploaded to the 3rd-party site. This process could take anywhere from a few seconds to minutes. I figured doing this particular task via an async PHP script/function would be the best solution and that's where amphp comes in. So the above scenario with amphp involved becomes this:

  1. A client uploads a video to our site via AJAX.
  2. Our site sends the video to a 3rd-party site which automatically generates a thumbnail for the video.
  3. An amphp async task is created to get the thumbnail from the 3rd-party site.
  4. The AJAX response is returned.
  5. Sometime later... the DB is updated with the thumbnail results from the async task in step 3. I also plan to send a ZMQ request back to the client if they're still online to update the thumbnail.

I thought I could do the above by following this example (from amphp.org/amp/promises/):

use Amp\Loop;

function asyncMultiply($x, $y)
{
    // Create a new promisor
    $deferred = new Amp\Deferred;

    // Resolve the async result 30s from now
    Loop::delay($msDelay = 30000, function () use ($deferred, $x, $y) {
        $deferred->resolve($x * $y);
    });

    return $deferred->promise();
}

// Some code here...
$promise = asyncMultiply(6, 7);
$result = Amp\Promise\wait($promise);

var_dump($result); // int(42)
// More code here... 

However, the above example must have gone right over my head because when I ran that directly in my synchronous application, it seemed to just wait for the 30 seconds. My expectation was that this was an async task, and the Some code here and More code here would execute without waiting for the result from the asyncMultiply(6, 7) since that was an async task. What am I missing?

Furthermore, presuming I'm not far off from achieving my task, would this approach be a problem if there were a few clients simultaneously uploading a video to our site via AJAX (i.e. our application would be running multiple async tasks at the same time) ?

Note: Not sure if relevant but the application is running in a WordPress/PHP-FPM environment.

Forked On 29 Aug 2022 at 05:28:12

Bennnjamin

In that case you definitely have the right setup for what you're trying to achieve. This line $result = Amp\Promise\wait($promise); will cause your application to block until the promise resolves so if your server has this code exactly as written, any code after that line will be executed 30 seconds later. Instead, you want to use

$promise->onResolve(function ($exception, $value) {
  //handle $exception, or $value here
}); 

which is a callback that will be ran after your promise completes, so 30 seconds later in your case. This will not block the event loop.

exec/shell_exec is not the most scalable solution for the reasons you mentioned. Since you already have a separate PHP process running an event loop, I would not recommend it. Amp or React is a far superior solution. To answer the question though 1) Passing data to the script is done by serializing it, typically with json_encode and json_decode 2) yes you would get two background processes and it may not be scalable depending on your client load. There are more robust solutions to allow X number of jobs to run simultaneously, etc.

Commented On 29 Aug 2022 at 05:28:12
Issue Comment

Bennnjamin

Running an async task in a synchronous application

Hi,

I am looking to do what I hope is a very simple task (at least in my head) but I'm struggling to wrap my head around how amphp is supposed to work. I have a synchronous application with a problem that I thought amphp would be perfect to solve. Picture the following:

  1. A client uploads a video to our site via AJAX.
  2. Our site sends the video to a 3rd-party site which automatically generates a thumbnail for the video.
  3. The 3rd-party site takes some time to generate the thumbnail depending on the video size/length etc. Unfortunately, there's no way to track this and the 3rd-party site does not inform us when it is ready either.

Given the above scenario, our site basically has to query the 3rd-party site every 5 seconds for the thumbnail from the moment the video is successfully uploaded to the 3rd-party site. This process could take anywhere from a few seconds to minutes. I figured doing this particular task via an async PHP script/function would be the best solution and that's where amphp comes in. So the above scenario with amphp involved becomes this:

  1. A client uploads a video to our site via AJAX.
  2. Our site sends the video to a 3rd-party site which automatically generates a thumbnail for the video.
  3. An amphp async task is created to get the thumbnail from the 3rd-party site.
  4. The AJAX response is returned.
  5. Sometime later... the DB is updated with the thumbnail results from the async task in step 3. I also plan to send a ZMQ request back to the client if they're still online to update the thumbnail.

I thought I could do the above by following this example (from amphp.org/amp/promises/):

use Amp\Loop;

function asyncMultiply($x, $y)
{
    // Create a new promisor
    $deferred = new Amp\Deferred;

    // Resolve the async result 30s from now
    Loop::delay($msDelay = 30000, function () use ($deferred, $x, $y) {
        $deferred->resolve($x * $y);
    });

    return $deferred->promise();
}

// Some code here...
$promise = asyncMultiply(6, 7);
$result = Amp\Promise\wait($promise);

var_dump($result); // int(42)
// More code here... 

However, the above example must have gone right over my head because when I ran that directly in my synchronous application, it seemed to just wait for the 30 seconds. My expectation was that this was an async task, and the Some code here and More code here would execute without waiting for the result from the asyncMultiply(6, 7) since that was an async task. What am I missing?

Furthermore, presuming I'm not far off from achieving my task, would this approach be a problem if there were a few clients simultaneously uploading a video to our site via AJAX (i.e. our application would be running multiple async tasks at the same time) ?

Note: Not sure if relevant but the application is running in a WordPress/PHP-FPM environment.

Forked On 29 Aug 2022 at 04:33:31

Bennnjamin

Amp is great for this but it is non-trivial to integrate into conventional PHP applications. Since you mentioned AJAX I am assuming you have conventional PHP deployment where each PHP process handles a single web request. In this typical execution context, you have a PHP script that is executed in it's entirety for a single HTTP request, usually returning an HTTP response, and exiting.

Amp is an event loop that will run until there is nothing else to run on the loop. So once you start an Amp event loop, it blocks until completion, and provides an API for you to run tasks on the event loop. If you re-create your example with a single PHP script and run it from the command line, you will see the exact same behavior: your command line script runs synchronously for 30 seconds and exits. The event loop has one task to run after a delay of 30 seconds, and exits after all tasks complete. The asynchronous power of Amp comes when you use it to run your entire application, allowing any task (including handling HTTP requests/responses) to be run on the event loop. This lets you do things like immediately return a response to the client, and simultaneously kick off a long-running job on the event loop.

Now with your existing application, you have a couple options.

  1. Since you're already using a message-broker (ZMQ), that is an ideal solution for handling async/long-running jobs in conventional PHP applications. You can use ZMQ to communicate between your current application, and an Amp event loop to run jobs.
  2. You can also just use shell_exec to kick off background process as well right after the upload completes. Like shell_exec("poll_api.php &")
  3. You can skip Amp for now and create a PHP script that polls for jobs from ZMQ, looking for messages to run that job (see https://github.com/zeromq/php-zmq/blob/master/examples/simple-server.php).

Commented On 29 Aug 2022 at 04:33:31
Pull Request

Bennnjamin

Sort binary results v2

Created On 18 Aug 2022 at 08:14:37
Issue Comment

Bennnjamin

Sort binary results v2

This is an alternate way to sort the results using the intended method of prefilling the array keys with any null column values. My only issue with this is SplFixedArray::offsetGet did not work properly with null values in the array. PHP would think it was not set if you do something like $splArray[1] = null;. It is not clear based on the PHP docs whether this is intended behavior, or not. But I got around it by using "\0" instead.

Forked On 18 Aug 2022 at 08:14:37

Bennnjamin

Implementation has issues, don't merge.

Commented On 18 Aug 2022 at 08:14:37
Pull Request

Bennnjamin

Sort binary results v2

Created On 18 Aug 2022 at 08:09:43
Create Branch
Bennnjamin In bennnjamin/amphp-mysql Create Branchfix-column-order-v2

Bennnjamin

Async MySQL client for PHP based on Amp.

On 18 Aug 2022 at 08:07:32
Pull Request

Bennnjamin

Sort binary result sets correctly

Created On 18 Aug 2022 at 08:07:15
Create Branch
Bennnjamin In bennnjamin/amphp-mysql Create Branchfix-column-order

Bennnjamin

Async MySQL client for PHP based on Amp.

On 18 Aug 2022 at 07:24:15
Issue Comment

Bennnjamin

MySQL time value type binary decoding now properly increments offset

Fixes #119 by incrementing offset when decoding MySQL time value column types. Also expands tests to ensure the test database has a table with a DATETIME column type.

Forked On 18 Aug 2022 at 06:54:31

Bennnjamin

Continuing to investigate this and so far I have found the issue is with prepared statements and NULL columns in a table. If you use db->query() to SELECT a table containing some NULL columns in the results, it works. But if you use db->prepare() and statement->execute() on the exact same SQL, the result set is ordered incorrectly - all of the NULL columns are at the beginning of the result array for each row, as if the row's columns are sorted with NULL first. When different rows have different columns that are NULL, the result set is completely out of order where each row has a different order of columns.

Commented On 18 Aug 2022 at 06:54:31
Issue Comment

Bennnjamin

MySQL time value type binary decoding now properly increments offset

Fixes #119 by incrementing offset when decoding MySQL time value column types. Also expands tests to ensure the test database has a table with a DATETIME column type.

Forked On 18 Aug 2022 at 04:59:39

Bennnjamin

After you made the changes to exception handling, I'm having an issue access the result iterator. Maybe I am just using it wrong, but this snippet fails:

 $db = new MysqlConnectionPool(MysqlConfig::fromAuthority($host, $user, $pass, $dbName));
  
    $result = $db->query("SELECT * FROM users WHERE email = $email");
    $next = $result->getNextResult();
    print_r($next);

    $db->close(); 

The primary exception:

PHP Fatal error:  Uncaught Error: Event loop terminated without resuming the current suspension:

#0 /vendor/revolt/event-loop/src/EventLoop/Internal/DriverSuspension.php:83 Fiber::suspend()
#1 /vendor/amphp/pipeline/src/Internal/QueueState.php:277 Revolt\EventLoop\Internal\DriverSuspension->suspend()
#2 /vendor/amphp/pipeline/src/Queue.php:87 Amp\Pipeline\Internal\QueueState->push() 

My understanding is that Amp & the Event Loop should internally handle suspending/resuming the Fiber while the iterator is blocking waiting for the next result. If that's not correct, I'm happy to learn more about the internals.

Commented On 18 Aug 2022 at 04:59:39
Issue Comment

Bennnjamin

MySQL time value type binary decoding now properly increments offset

Fixes #119 by incrementing offset when decoding MySQL time value column types. Also expands tests to ensure the test database has a table with a DATETIME column type.

Forked On 15 Aug 2022 at 03:39:59

Bennnjamin

Great work, I will update today. I haven't really looked at amphp/pipeline in the midst of migrating to v3 but at first glance it reminds me a bit of Rx while also providing async collections as well. I also just found this collection library loophp/collection that had some async support using v2 of Amp. If you haven't come across it before, definitely check it out.

Commented On 15 Aug 2022 at 03:39:59
Issue Comment

Bennnjamin

MySQL time value type binary decoding now properly increments offset

Fixes #119 by incrementing offset when decoding MySQL time value column types. Also expands tests to ensure the test database has a table with a DATETIME column type.

Forked On 11 Aug 2022 at 05:53:44

Bennnjamin

Yeah that was exactly my thoughts.

Also I wanted to mention something I found when discovering this bug. Not sure if it's intended behavior or not. This line here https://github.com/amphp/mysql/blob/173e94a58be645109f63e6c58ad0fd9262a29ad7/src/Internal/ConnectionProcessor.php#L1142 would throw an exception because $i is trying to access a column past the array boundary (since $offset is never incremented, there's another loop iteration for Time columns). This exception is swallowed and maybe thats intentional, but if that exception bubbled up to the EventLoop it would have made it much easier to catch. Instead I would sometimes see a very obscure error message after the event loop terminates about can't restart a fiber that's already suspended. You can move that exception up to the previous line so that is throws every time, and the EventLoop will not crash. Unfortunately I don't know enough about the details of Fibers and Revolt to know if that's intended behavior or even the accepted approach to handling exceptions.

Commented On 11 Aug 2022 at 05:53:44
Issue Comment

Bennnjamin

MySQL time value type binary decoding now properly increments offset

Fixes #119 by incrementing offset when decoding MySQL time value column types. Also expands tests to ensure the test database has a table with a DATETIME column type.

Forked On 10 Aug 2022 at 10:56:50

Bennnjamin

Oh I hadn't come across that. That's real helpful, thanks. I was also looking at cycle/orm.

Commented On 10 Aug 2022 at 10:56:50
Issue Comment

Bennnjamin

MySQL time value type binary decoding now properly increments offset

Fixes #119 by incrementing offset when decoding MySQL time value column types. Also expands tests to ensure the test database has a table with a DATETIME column type.

Forked On 10 Aug 2022 at 10:29:47

Bennnjamin

I am hoping to use it in production, so I will let you know if I discover any more issues. I'm hoping to put together an ORM layer as well, but that's optimistic. Great work so far on v3 though. I had begun development on v2 and fibers are a game-changer for async I/O, and refactoring the dev environment to v3 was a breeze.

Commented On 10 Aug 2022 at 10:29:47

Bennnjamin

php_no_xdebug permissions back to 755

Pushed On 10 Aug 2022 at 09:14:53

Bennnjamin

push permisisons back for MysqlDataType.php

Pushed On 10 Aug 2022 at 09:13:30
Issue Comment

Bennnjamin

MySQL time value type binary decoding now properly increments offset

Fixes #119 by incrementing offset when decoding MySQL time value column types. Also expands tests to ensure the test database has a table with a DATETIME column type.

Forked On 10 Aug 2022 at 09:12:15

Bennnjamin

Good catch. Added the same line to TIME as well.

Commented On 10 Aug 2022 at 09:12:15

Bennnjamin

fixed offset for Time columns

Pushed On 10 Aug 2022 at 09:11:19
Issue Comment

Bennnjamin

MySQL time value type binary decoding now properly increments offset

Fixes #119 by incrementing offset when decoding MySQL time value column types. Also expands tests to ensure the test database has a table with a DATETIME column type.

Forked On 10 Aug 2022 at 09:05:41

Bennnjamin

If you want a clean history, might be easier for me to just create a new PR - there aren't that many changes to the codebase.

Commented On 10 Aug 2022 at 09:05:41
Issue Comment

Bennnjamin

MySQL time value type binary decoding now properly increments offset

Fixes #119 by incrementing offset when decoding MySQL time value column types. Also expands tests to ensure the test database has a table with a DATETIME column type.

Forked On 10 Aug 2022 at 09:04:39

Bennnjamin

Just noticed that - it was not my intention to push those. Should be fixed now.

Commented On 10 Aug 2022 at 09:04:39

Bennnjamin

reset permissions back

Pushed On 10 Aug 2022 at 09:03:42
Pull Request

Bennnjamin

MySQL time value type binary decoding now properly increments offset

Created On 10 Aug 2022 at 09:01:47
Issue Comment

Bennnjamin

Prepared Statements don't work with DATETIME columns

UPDATE: I have done some more testing and noticed that it works when I specify column names.

I have a very simple example that I'm testing v3 on with my own development database. The following code runs with no errors, but the result is empty since nothing is printed. If I use the db->query(...) function and don't use prepared statements, but instead pass values directly in the sql string it works. Any ideas how to troubleshoot this or any guidance on prepared statements would be greatly appreicated.

$email = "mytestemail";

$db = new MysqlConnectionPool(MysqlConfig::fromAuthority($host, $user, $pass, $database));

$stmt = $db->prepare("SELECT * FROM users WHERE email = ?");


$result = $stmt->execute([$email]);
foreach ($result as $row) {
    print_r($row);
}

$result2 = $db->execute("SELECT * FROM users WHERE email = ?", [$email]);
foreach ($result2 as $row) {
    print_r($row);
} 

Forked On 10 Aug 2022 at 09:00:27

Bennnjamin

Turns out that any columns of type DATETIME, TIME, TIMESTAMP are not decoded properly, causing this issue.

Commented On 10 Aug 2022 at 09:00:27
Create Branch
Bennnjamin In bennnjamin/amphp-mysql Create Branchfix-datetime

Bennnjamin

Async MySQL client for PHP based on Amp.

On 10 Aug 2022 at 08:58:44

Bennnjamin

Async MySQL client for PHP based on Amp.

Forked On 09 Aug 2022 at 10:51:11

Bennnjamin

started

Started On 03 Aug 2022 at 04:23:33