# Python SDK Image Upload stuck using multiprocessing

**URL:** <https://discuss.roboflow.com/t/python-sdk-image-upload-stuck-using-multiprocessing/4218>\
**Category:** 🛠️ Feature Reqs\
**Tags:** bugs\
**Created:** [January 16, 2024, 8:41pm UTC](https://discuss.roboflow.com/t/python-sdk-image-upload-stuck-using-multiprocessing/4218 "2024-01-16T20:41:04Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kevin\_Deloria](https://yyz1.discourse-cdn.com/flex029/user_avatar/discuss.roboflow.com/kevin_deloria/32/3649_2.png) [@Kevin\_Deloria](https://discuss.roboflow.com/u/Kevin_Deloria)\
**Post date:** [January 16, 2024, 8:41pm UTC](https://discuss.roboflow.com/t/python-sdk-image-upload-stuck-using-multiprocessing/4218/1 "2024-01-16T20:41:04Z")

</div>

I wrote a simple script that uploads photos (batched and tagged properly) using the Python SDK.

```auto
from roboflow import Roboflow

rf = Roboflow(api_key=API_KEY)

workspaceId = WORKSPACE_ID
projectId = PROJECT_ID
project = rf.workspace(workspaceId).project(projectId)

def upload_image(image_path, batch_name, tag):
    try:
        print(image_path, batch_name, tag)
        project.upload(
            image_path=image_path,
            batch_name=batch_name,
            num_retry_uploads=3,
            tag=tag
        )
        filename = image_path.split('\\')[-1]
        print(filename)
        # move the file to the uploaded images folder
        os.rename(image_path, f"{uploaded_images_path}\\{filename}")
    except Exception as e:
        print("Something went wrong...")
        print(e)

    print("Done")

```

and it works as expected

```auto
for file, batch, tag in args:
    upload_image(file, batch, tag)

```

However I would like to do multiple uploads simultaneously using multiprocessing to save time on larger datasets.

```auto
# DOES NOT WORK
with Pool(processes=multiprocessing.cpu_count() - 1) as pool:
    L = pool.starmap(upload_image, args)
    pool.close()
    pool.join()

```

This does not work and is stuck indefinitely.

---

<div class="post-metadata">

**Author:** ![tonylampada](https://yyz1.discourse-cdn.com/flex029/user_avatar/discuss.roboflow.com/tonylampada/32/3487_2.png) [@tonylampada](https://discuss.roboflow.com/u/tonylampada)\
**Post date:** [January 18, 2024, 2:01pm UTC](https://discuss.roboflow.com/t/python-sdk-image-upload-stuck-using-multiprocessing/4218/2 "2024-01-18T14:01:17Z")

</div>

Hi Kevin.  
Your multiprocessing code looks correct. I can’t think of a reason for it not to work, especially when the sequential for loop does.

Couple questions:

- Where does Pool come from? is it `from multiprocessing import Pool` ?
- Can you try a fixed amount of workers: with Pool(processes=10) as pool
- Can you show a little bit of what’s in `args`?
- Can you add some print()s before/after pool.starmap and provide the output you get in the console?

---

<div class="post-metadata">

**Author:** ![Kevin\_Deloria](https://yyz1.discourse-cdn.com/flex029/user_avatar/discuss.roboflow.com/kevin_deloria/32/3649_2.png) [@Kevin\_Deloria](https://discuss.roboflow.com/u/Kevin_Deloria)\
**Post date:** [January 22, 2024, 8:00pm UTC](https://discuss.roboflow.com/t/python-sdk-image-upload-stuck-using-multiprocessing/4218/3 "2024-01-22T20:00:14Z")

</div>

Thanks @tonylampada

- Where does Pool come from? is it `from multiprocessing import Pool` ?  
**Yes**
- Can you try a fixed amount of workers: with Pool(processes=10) as pool’  
Yes - I’ve tried hardcoding
- Can you show a little bit of what’s in `args`?  
This is a list of tuples to match the function signature `upload_image(image_path, batch_name, tag)`

```auto
[('C:\\Users\\kevin\\Documents\\IBI Training\\Unsorted\\pID-25968-msid-101460--3cf5b920-ae51-4173-a4c8-985ea9766df1.jpg', '25968', '25968'), ('C:\\Users\\kevin\\Documents\\IBI Training\\Unsorted\\pID-25968-msid-103233--c4f5df3a-28e8-40ee-89a4-c7e8b815ca41.jpg', '25968', '25968')]
Running...
[('C:\\Users\\kevin\\Documents\\IBI Training\\Unsorted\\pID-25968-msid-103233--c4f5df3a-28e8-40ee-89a4-c7e8b815ca41.jpg', '25968', '25968'), ('C:\\Users\\kevin\\Documents\\IBI Training\\Unsorted\\pID-25968-msid-101460--3cf5b920-ae51-4173-a4c8-985ea9766df1.jpg', '25968', '25968')]

```

This is fine since I use this synchronously.

- Can you add some print()s before/after pool.starmap and provide the output you get in the console?

Yep - it’s being invoked.

I realised that this is most likely not related to Roboflow API and more on the limitation of `multiprocessing`.

> <https://stackoverflow.com/questions/41385708/multiprocessing-example-giving-attributeerror/42383397#42383397>

I’ll try to play around with it more since I really need to upload multiple images simultaneously.

---

<div class="post-metadata">

**Author:** ![tonylampada](https://yyz1.discourse-cdn.com/flex029/user_avatar/discuss.roboflow.com/tonylampada/32/3487_2.png) [@tonylampada](https://discuss.roboflow.com/u/tonylampada)\
**Post date:** [March 28, 2024, 12:30pm UTC](https://discuss.roboflow.com/t/python-sdk-image-upload-stuck-using-multiprocessing/4218/4 "2024-03-28T12:30:35Z")

</div>

Late follow-up:

You can now do this more easily (no coding needed) by using the python CLI (run `roboflow import` on the terminal with `-c` parameter for concurrency).

Documentation:

> **[Introduction | Roboflow Docs](https://docs.roboflow.com/roboflow-cli/introduction)**
>
> Use the Roboflow CLI to manage and interact with projects.