Skip to content

Conversation

@tarnfeld
Copy link

@tarnfeld tarnfeld commented Oct 1, 2014

When downloading a lot of files from various data nodes, it's valuable to use keep-alive connections to each node and pool those sockets.

This will happen automatically when using a session object, and it will also maintain a pool of pools for each host.

@tarnfeld tarnfeld changed the title Make use of a Session() object to get connection pooling Make use of a Session() object for connection pooling Oct 1, 2014
@stevendgonzales
Copy link
Contributor

Thank you very much for the PR. I think this will be a valuable addition.

This change will require changes in https://github.com/ProjectMeniscus/pywebhdfs/blob/master/pywebhdfs/tests/test_webhdfs.py to ensure that unit tests remain passing.

Specifically, in the test classes, you would probably want to replace the session object with a magic mock, and then set side effects and return values for the requests. Also the _get, _put, _post, and _delete methods would require unit tests.

Please let me know if this is something you can add to this PR. If not, I can try and make some time to add these tests before releasing another version to pypi

@tarnfeld
Copy link
Author

tarnfeld commented Oct 4, 2014

Thanks for the comments! I'll be sure to add tests over the next couple of days and get everything ready for merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants