this post was submitted on 02 Nov 2024
263 points (98.2% liked)

Python

6337 readers
8 users here now

Welcome to the Python community on the programming.dev Lemmy instance!

๐Ÿ“… Events

PastNovember 2023

October 2023

July 2023

August 2023

September 2023

๐Ÿ Python project:
๐Ÿ’“ Python Community:
โœจ Python Ecosystem:
๐ŸŒŒ Fediverse
Communities
Projects
Feeds

founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[โ€“] [email protected] 4 points 1 day ago

As for data science using Python, something tells me that this has to do with memory heap capacities. I'm not sure about Python's max memory heap, but Javascript through Node.js seems to have only 512MB. I've been using Node.js to deal with big datasets and my most recent experimentation stumbled across the need of loading 100 million numbers to the RAM: while my PC has a fair amount of physical RAM (12GB) and a great part of it was available, it'll simply error when filling an array. I needed an additional parameter, --max-old-space-size, so Node.js could deal with such amount of data. I didn't try the same task with Python because I'm used to Javascript (yet I'm done some things in Python), but I wonder how much memory can Python hold until an error like "out of memory" happens, because ML models (for example, those hosted and served in HuggingFace) loads training weights with dozens of GBs