Skip to content

Diffbot client not closing TCP connections until client exits #58

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jonathantullett opened this issue Dec 1, 2017 · 1 comment
Open

Comments

@jonathantullett
Copy link

When making a call to create or update a job, the client is not closing TCP connections after the request to the api endpoint.

Running the following code demonstrates the issue:

    $diffbot = new Diffbot("xxxxxxxxx");
    $job = $diffbot->crawl("jonathan_test");
    $job->setSeeds(["http://www.example.com"])->setMaxToCrawl(100)
        ->setMaxToProcess(100)->setMaxRounds(1)
        ->setOnlyProcessIfNew(1)->setMaxHops(3);

    $api
        = $diffbot->createArticleAPI('crawl')->setMeta(true)->setDiscussion(false)
        ->setQuerystring(true)
    ;

    $job->setApi($api);
    $x = $job->call();
sleep(100);

Now the socket remains open until the process quits:

$ netstat -an|  grep 443                                                                                                       
tcp        0      0 192.168.22.214:50844    35.192.184.37:443       TIME_WAIT

While not too bad for a single socket, if you create a lot of diffbot objects using new Diffbot(), you can quickly run out of open files on the system as the sockets aren't closed even when the object falls out of scope.

@jonathantullett
Copy link
Author

I wrote a wrapper for the Diffbot client which implemented a singleton and this has resolved the issue for me.

<?php
namespace DDA;
class Diffbot
{
    private static $instance;
    private function __construct() {}
    private static $apiKey = "sadffddsfafafdfdsdfsdfs";
    /**
     * @return \Swader\Diffbot\Diffbot
     * @throws \Zend_Exception
     */
    public static function getInstance(): \Swader\Diffbot\Diffbot
    {
        if(self::$instance === null) {
            self::$instance = new \Swader\Diffbot\Diffbot(self::$apiKey);
        }
        return self::$instance;
    }
}

So I would propose that rather than calling new Diffbot(),each time you call the wrapped static method getInstance() which returns the singleton instance.

While the root cause - dangling TCP connections - is still there, it's only a single one, from the single Diffbot object, being re-used extensively, even when updating thousands of searches.

I would propose a similar approach is taken within your library.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant