Research
Security News
Quasar RAT Disguised as an npm Package for Detecting Vulnerabilities in Ethereum Smart Contracts
Socket researchers uncover a malicious npm package posing as a tool for detecting vulnerabilities in Etherium smart contracts.
scramjet-framework-py
Advanced tools
⭐ Star us on GitHub — it motivates us a lot! 🚀
Scramjet is a simple reactive stream programming framework. The code is written by chaining functions that transform the streamed data, including well known map, filter and reduce.
The main advantage of Scramjet is running asynchronous operations on your data streams concurrently. It allows you to perform the transformations both synchronously and asynchronously by using the same API - so now you can "map" your stream from whatever source and call any number of API's consecutively.
Originally written on top of node.js object streams, Scramjet is now being ported into Python. This is what is happening in this repository.
Tested with Python 3.8.10 and Ubuntu 20.04.
Scramjet Framework is available on PyPI, You can install it with simple pip command:
pip install scramjet-framework-py
Let's say we have a fruits.csv
file like this:
orange,sweet,1
lemon,sour,2
pigface,salty,5
banana,sweet,3
cranberries,bitter,6
and we want to write the names of the sweet fruits to a separate file. To do this, write an async function like this:
from scramjet import streams
import asyncio
async def sweet_stream():
with open("fruits.csv") as file_in, open("sweet.txt", "w") as file_out:
await (
streams.Stream
.read_from(file_in)
.map(lambda line: line.split(','))
.filter(lambda record: record[1] == "sweet")
.map(lambda record: f"{record[0]}\n")
.write_to(file_out)
)
asyncio.run(sweet_stream())
output saved in sweet.txt:
orange
banana
and that's it!
Basic building block of Scramjet is the Stream
class. It reads input in
chunks, performs operations on these chunks and produces an iterable output
that can be collected and written somewhere.
Creating a stream is done using read_from
class method. It accepts
any iterable or an object implementing .read() method as the input, and returns
a Stream
instance.
Transforming a stream:
map
- transform each chunk in a stream using specified function.filter
- keep only chunks for which specified function evaluates to True
.flatmap
- run specified function on each chunk, and return all of its results as separate chunks.batch
- convert a stream of chunks into a stream of lists of chunks.Each of these methods return the modified stream, so they can be chained like
this: some_stream.map(...).filter(...).batch(...)
Collecting data from the stream (asynchronous):
write_to
- write all resulting stream chunks into a target.to_list
- return a list with all stream chunks.reduce
- combine all chunks using specified function.You can find more examples in hello_datastream.py
file. They don't require any additional dependencies, just the standard library,
so you can run them simply with:
python hello_datastream.py
Anything missing? Or maybe there is something which would make using Scramjet Framework much easier or efficient? Don't hesitate to fill up a new feature request! We really appreciate all feedback.
If you have found a bug, inconsistent or confusing behavior please fill up a new bug report.
You can contribute to this project by giving us feedback (reporting bugs and requesting features) and also by writing code yourself!
The easiest way is to create a fork of this repository and then create a pull request with all your changes. In most cases, you should branch from and target main
branch.
Please refer to Development Setup section on how to setup this project.
Install Python3 interpreter on your computer. Refer to official docs.
Install git
version control system. Refer to official docs.
Clone this repository:
git clone git@github.com:scramjetorg/framework-python.git
sudo apt install python3-virtualenv
virtualenv -p python3 venv
.venv/bin/activate
$ python --version
Python 3.8.10
pip install -r dev-requirements.txt
pytest
:bulb: HINT: add a filename if you want to limit which tests are run
PYFCA_DEBUG=1 # debug pyfca
DATASTREAM_DEBUG=1 # debug datastream
SCRAMJET_DEBUG=1 # debug both
FAQs
Scramjet is a simple reactive stream programming framework.
We found that scramjet-framework-py demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Socket researchers uncover a malicious npm package posing as a tool for detecting vulnerabilities in Etherium smart contracts.
Security News
Research
A supply chain attack on Rspack's npm packages injected cryptomining malware, potentially impacting thousands of developers.
Research
Security News
Socket researchers discovered a malware campaign on npm delivering the Skuld infostealer via typosquatted packages, exposing sensitive data.