Python websocket multithread/ multiprocess storing the received JSON data in one thread questions
For my development, I’m trying to have two threads.
One threads is for receiving real time data and put them into a list/dictionary(which one should I use to store the JSON I received?)
And the other thread is to loop through the list/dictionary and print out the data in the JSON object/string we need.(not all the fields are needed in the received JSON data)
But the second thread doesn’t start.
The reason to use multithread is because we don't want to miss out on updated data when the program is printing the data(eventually, the "print" action will be replaced by updated our database)
What is the problem?
I could provide my code if it's needed.
Thank you.
python
#technology
websockets
json
Hi
@tobywong
You can probably search on the internet for multithreaded examples in Python that suit your requirements. When I do this, I explicitly search for an example to achieve multi-threaded processing.
For example:
# Example threading using the thread-safe 'queue.Queue' class which allows you to put
# data into the queue on one thread and retrieve it on another - asynchronously
import threading
import queue
# Create the queue
data_queue = queue.Queue()
In thread 1, when you receive your data, simply store it within the queue
def receive_data_cb(data):
data_queue.put(data)
In thread 2, process the data when it becomes available
def process_data():
while True:
data = data_queue.get() # Block until data arrives
print(data) # This is where you can store in a DB
To put it all together, create 2 threads and start each:
thread1 = threading.Thread(target=receive_data_cb)
thread2 = threading.Thread(target=process_data)
thread1.start()
thread2.start()
The above is the general algorithm I found by researching. However, I would personally experiment and try some simple tests. For example, you may not need to explicitly create 2 distinct threads here but instead just one which handle your processing of the data. The main thread, where you start your application, could handle the data coming from the WebSocket server.
Hi
@tobywong
Queue class is much more complex than list and designed to operate efficiently in multi-thread context.
This article compares list and Queue:
https://www.troyfawkes.com/learn-python-multithreading-queues-basics/
Behind the scene, Queue is managing a deque or a list, but if you check Queue.put() and Queue.get(), you'll see that these functions are using:
a Semaphoe to protect simultaneous call to both functions
wait() function to let the GIL to suspend the current thread and let others restart
With your implementation, if you don't use async function, the thread that manages the list can't be suspended by the GIL so others threads are waiting infinitely to run