• python
  • javascript
  • reactjs
  • sql
  • c#
  • java
Facebook Twitter Instagram
Devs Fixed
  • python
  • javascript
  • reactjs
  • sql
  • c#
  • java
Devs Fixed
Home ยป Resolved: Load gzip data fast with Python

Resolved: Load gzip data fast with Python

0
By Isaac Tonny on 17/06/2022 Issue
Share
Facebook Twitter LinkedIn

Question:

I have a large file (~500 Mo) that contains a dictionary of dictionaries like this :
This file is saved as gzip (using Python) to optimize memory. Now I query a lot this file, so when I want to get just ‘data2’, I need to load it all and look for data2, which takes time in loading & memory. Is there a way to load only the element we want ? Or at least to load faster the file ?
Thank you,

Answer:

You can’t start reading from the middle of a gzip stream. You need to start from the beginning and read until you find what you’re looking for. However you do not need to read the whole thing into memory. Just read small chunks at a time, giving the chunk size to read().
For fast access, I would recommend copying your data into an sqlite3 database.

If you have better answer, please add a comment about this, thank you!

gzip load python query-optimization
Share. Facebook Twitter LinkedIn

Related Posts

Resolved: TYPO3 SQL error: Field ‘tx_imagezoom_set’ doesn’t have a default value

26/03/2023

Resolved: std::regex_replace to replace multiple combinations

26/03/2023

Resolved: How can I copy files using the ansible.builtin.copy module and avoid conflicting file names?

26/03/2023

Leave A Reply

© 2023 DEVSFIX.COM

Type above and press Enter to search. Press Esc to cancel.