Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I've looked around a bit but I can't seem to solve this problem I have. I'd like to execute a python script within a view of my django app. I've placed the code I'd like to execute inside a django management command so it can be accessed via command line python manage.py command-name . I then tried to run this command using subprocess.Popen("python manage.py command-name",shell=True) .

However, this command could take some time to execute so I'd like the view to continue and allow the script to execute in the background. Using subprocess.Popen alone seems to cause the view to hang until the script has finished, so I tried using a thread (following another SA question):

class SubprocessThread(threading.Thread):
def __init__(self, c):
    self.command = c
    self.stdout = None
    self.stderr = None
    threading.Thread.__init__(self)
def run(self):
    p = subprocess.Popen(self.command,
                         shell=True,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.PIPE)
    self.stdout, self.stderr = p.communicate()
and then executing it:
t = SubprocessThread("python manage.py command-name")
t.setDaemon(True)
t.start()
t.join()
However, the view still hangs: the cursor has a busy symbol and the AJAX on the page does not load.  Otherwise the page's html seems to load fine and commands in the view after the thread call appear to finish normally (before the script finishes).  Can someone please help me? I'd like the script to execute and do its own thing without holding up the view or AJAX calls on the page.
                I regularly use a similar pattern (with a subprocess, because a thread is killed when the view function exits), and I regularly forget to return something in the view function, making the browser hang. Then, I blame the subporcess, while I just forgot to return an HttpResponse. Check that!
– Dunatotatos
                Oct 31, 2017 at 8:23
I wasted a lot of time trying to implement something similar, but had the same problems as you. Eventually, I gave up and implemented a beanstalk queue to handle the work. 
http://kr.github.com/beanstalkd/
I put an id on the queue in the Django view, and then have a management command to run the consumer (watched by supervisord). 
Using a queue means you could expand to multiple consumers, and allows to better manage the load (pausing the consumer if necessary without losing the work required).  
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.