Security News
Research
Data Theft Repackaged: A Case Study in Malicious Wrapper Packages on npm
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Write a classic sequential program. Then convert it into a parallel one.
It runs faster.
Don't use it.
Before:
.. code:: python
for image in images:
create_thumbnail(image)
After:
.. code:: python
from fork import fork
for image in images:
fork(create_thumbnail, image)
As usual:
.. code:: python
result = fork(my_func, *args, **kwargs)
It's a proxy object that behaves almost exactly like the real return value of my_func
except that
it's lazy.
Very lazy. You can even add, multiply, etc. such proxy results without blocking which come in
quite handy, especially in loops. Use fork.await
, str
, print
, etc. to force evaluation
and get the real and non-lazy value back.
.. code:: python
sizes = 0
for image in images:
sizes += fork(create_thumbnail, image) # lazy evaluation
print(sizes) # forces evaluation
You don't need to bother. fork will take care of that for you.
You can assist fork by decorating your functions; not decorating defaults to fork.cpu_bound
:
.. code:: python
@io_bound
def call_remote_webservice():
# implementation
@cpu_bound
def heavy_computation(n):
# implementation
Original (sequential) tracebacks are preserved. That should make debugging easier. However, don't try to catch exceptions. You better want to exit and see them. When you force evaluation potential exceptions will be raised.
If you really need more control over the type of execution, use fork.process
or fork.thread
.
They work just like fork.fork
but enforce the corresponding type of background execution.
.. code:: python
import pkg_resources
for worker_function in pkg_resources.iter_entry_points(group='worker'):
process(worker_function)
You can shorten your programs by using fork.map
. It works like fork.fork
but submits
a function multiple times for each item given by an iterable.
.. code:: python
results = fork.map(create_thumbnail, images)
fork.map_process
and fork.map_thread
work accordingly and force a specific type of
execution. Use those if really necessary.
Otherwise, just use fork.map
. fork take care for you in this case again.
In order to wait for the completion of a set of result proxies, use fork.await_all
. If you want to
unblock by the first unblocking result proxy, call fork.await_any
.
There are also blocking variants available: fork.block_map
, fork.block_map_process
and
fork.block_map_thread
; in case you need some syntactic sugar:
.. code:: python
fork.await_all(fork.map(create_thumbnail, images))
# equals
fork.block_map(create_thumbnail, images)
Good
Bad
weird calling syntax (no syntax support)
type(result) == ResultProxy
not working with lambdas due to PickleError
needs fix:
cannot fix efficiently:
ideas are welcome :-)
.. _FORK: https://pypi.python.org/pypi/xfork .. _asyncio: https://docs.python.org/3/library/asyncio.html
FAQs
submitting cpu-bound tasks to processes and io-bound tasks to threads
We found that xfork demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.