Welcome to Fiber
Distributed Computing
for AI Made
Simple.
Fiber is a Python-based distributed computing library for modern computer clusters. Instead of programming your desktop or laptop, now you can program the whole computer cluster. It was developed to power large scale parallel scientific computation projects like POET.
Fiber allows you to write programs that run on a computer cluster level without the need to dive into the details of computer cluster.
Fiber provides the same API as Python's standard multiprocessing library that you are familiar with. If you know how to use multiprocessing, you can program a computer cluster with Fiber.
Fiber's communication backbone is built on top of Nanomsg which is a high-performance asynchronous messaging library to allow fast and reliable communication.
You don't need to deploy Fiber on computer clusters. You run it as the same way as running a normal application on a computer cluster and Fiber handles the rest for you.
Fiber has built-in error handling when you are running a pool of workers. Users can focus on writing the actual application code instead of dealing with crashed workers.
Fiber can dynamically allocate resources from computer clusters including CPU/Memory/GPU etc. It can scale up and down according to the computation needed by the user.
Installing Fiber is simple, just do pip install fiber and you are ready to get started.
Whether you decide to convert an exiting program to run on a computer or you are starting from scratch, you will find Fiber's multiprocessing compatible API easy to use.
Fiber uses Docker to encapsulate your code together with all the dependencies so that you can run your code anywhere without worrying about mismatching dependencies.
To run on a computer cluster (Kubernetes) is also easy, simple do fiber run python code_with_fiber.py to launch your code on a computer cluster, and fiber will handle the rest.
from fiber import Pool
import random
NUM_SAMPLES = int(1e6)
def is_inside(p):
x, y = random.random(), random.random()
return x * x + y * y < 1
def main():
pool = Pool(processes=4)
pi = 4.0 * sum(pool.map(is_inside, range(0, NUM_SAMPLES))) / NUM_SAMPLES
print("Pi is roughly {}".format(pi))
if __name__ == '__main__':
main()