Welcome to Fiber

Distributed Computing
for AI Made
Simple.

Scroll Down

About Fiber

Fiber is a Python-based distributed computing library for modern computer clusters. Instead of programming your desktop or laptop, now you can program the whole computer cluster. It was developed to power large scale parallel scientific computation projects like POET.

Easy to use

Fiber allows you to write programs that run on a computer cluster level without the need to dive into the details of computer cluster.

Easy to learn

Fiber provides the same API as Python's standard multiprocessing library that you are familiar with. If you know how to use multiprocessing, you can program a computer cluster with Fiber.

Fast

Fiber's communication backbone is built on top of Nanomsg which is a high-performance asynchronous messaging library to allow fast and reliable communication.

Batteries included

You don't need to deploy Fiber on computer clusters. You run it as the same way as running a normal application on a computer cluster and Fiber handles the rest for you.

Reliable

Fiber has built-in error handling when you are running a pool of workers. Users can focus on writing the actual application code instead of dealing with crashed workers.

Dynamic scaling

Fiber can dynamically allocate resources from computer clusters including CPU/Memory/GPU etc. It can scale up and down according to the computation needed by the user.

How Fiber Works?

Install

Installing Fiber is simple, just do pip install fiber and you are ready to get started.

Develop

Whether you decide to convert an exiting program to run on a computer or you are starting from scratch, you will find Fiber's multiprocessing compatible API easy to use.

Build

Fiber uses Docker to encapsulate your code together with all the dependencies so that you can run your code anywhere without worrying about mismatching dependencies.

Run

To run on a computer cluster (Kubernetes) is also easy, simple do fiber run python code_with_fiber.py to launch your code on a computer cluster, and fiber will handle the rest.

Code example: Estimating Pi with Fiber

from fiber import Pool import random NUM_SAMPLES = int(1e6) def is_inside(p): x, y = random.random(), random.random() return x * x + y * y < 1 def main(): pool = Pool(processes=4) pi = 4.0 * sum(pool.map(is_inside, range(0, NUM_SAMPLES))) / NUM_SAMPLES print("Pi is roughly {}".format(pi)) if __name__ == '__main__': main()