Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I want to run a python script and capture the output on a text file as well as want to show on console.

I want to specify it as a property of the python script itself. NOT to use the command echo "hello world" | tee test.txt on command prompt every time.

Within script I tried:

sys.stdout = open('log.txt','w')
But this does not show the stdout output on screen.
I have heard about logging module but I could not get luck using that module to do the job.
python foo_bar.py > file
This will write all results being printed on stdout from the Python source to file to the logfile.
Or if you want logging from within the script:
import sys
class Logger(object):
    def __init__(self):
        self.terminal = sys.stdout
        self.log = open("logfile.log", "a")
    def write(self, message):
        self.terminal.write(message)
        self.log.write(message)  
    def flush(self):
        # this flush method is needed for python 3 compatibility.
        # this handles the flush command by doing nothing.
        # you might want to specify some extra behavior here.
sys.stdout = Logger()
Now you can use:
print "Hello"
This will write "Hello" to both stdout and the logfile.
                Hi Amith,I do not want to use this as it need manual interaction to do this  (> file). Is there something which I can do from with in script as well or once execution has completed then what ever has came on console, takes and push to a file??
– user2033758
                Feb 16, 2013 at 4:31
                @user2033758: There are two suggestions in this answer, and the second does not need any manual interaction.  You can either use the command line or you can use the class in your code.  I tested this and the class sends output to both console and file, without any special parameters on the command line.
– JDM
                May 4, 2013 at 10:49
                How do I get the normal sys.stdout behavior back? Say I want this, but only in the startup phase of my program, and then after a while I want to just let the program run, but without saving everything
– Toke Faurby
                Jan 30, 2017 at 19:38
                @TokeFaurbyYou could save the previous sys.stdout in Logger, and create a function which reverts it to the original state.
– Gábor Fekete
                May 8, 2017 at 8:07
                @UmairAyub If you also want python error logging just add sys.stderr = sys.stdout after sys.stdout = Logger()
– insaneinvader
                Apr 6 at 19:38
I got the way to redirect the out put to console as well as to a text file as well simultaneously:
te = open('log.txt','w')  # File where you need to keep the logs
class Unbuffered:
   def __init__(self, stream):
       self.stream = stream
   def write(self, data):
       self.stream.write(data)
       self.stream.flush()
       te.write(data)    # Write the data of stdout here to a text file as well
sys.stdout=Unbuffered(sys.stdout)
                I ran a python process and unable to see the data in log.txt file unless until i kill the process. Looks like log is getting buffered and then dumped at the end.
– Mukesh Kumar
                Jan 21, 2021 at 9:34
                Hello @MukeshKumar, I to faced the same problem. I added te.flush() to the end of file to force it to stop buffer the data and write it. Also, I had problems deleting the data because it was still opened in the application, so I added te.close(). As a third problem, my cell in Jupyter kept running indefinitely until I kill the kernel. I solved it by capturing the original state of stdout (orig_stdout = sys.stdout) and assigned it again in the end of script (sys.stdout = orig_stdout). After that, everything was running very well.
– Arnold Souza
                Jan 25, 2021 at 17:56
                Hi @ArnoldSouza Thanks for the reply.  How does te.flush() works ? when does it flush the data to the log file  ? is it based in time or buffer size ? can you please share your code as well in case i am missing something
– Mukesh Kumar
                Jan 26, 2021 at 12:55
                I posted the code as an answer in this stack. About obj.flush(), I believe the function open works based in  buffer size. More about it here: stackoverflow.com/questions/3167494/…
– Arnold Souza
                Jan 26, 2021 at 18:50
                Hi Arnold, may I know, if I were to stop the streaming capture, do I add these commands at the end of the script? te.flush\ t.close()\ sys.stdout = orig_stdout? My jupyter is still running indefinitely.
– user3782604
                Sep 5, 2021 at 4:28
Use logging module to debug and follow your app
Here is how I managed to log to file and to console / stdout
import logging
logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s - %(levelname)s - %(message)s',
                    filename='logs_file',
                    filemode='w')
# Until here logs only to file: 'logs_file'
# define a new Handler to log to console as well
console = logging.StreamHandler()
# optional, set the logging level
console.setLevel(logging.INFO)
# set a format which is the same for console use
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
# tell the handler to use this format
console.setFormatter(formatter)
# add the handler to the root logger
logging.getLogger('').addHandler(console)
# Now, we can log to both ti file and console
logging.info('Jackdaws love my big sphinx of quartz.')
logging.info('Hello world')
read it from source:
https://docs.python.org/2/howto/logging-cookbook.html
                Not cool to copy-paste others-own answers ttps://stackoverflow.com/a/9321890/7646049 just give the reference on them.
– Evgeny Kuznetsov
                Jan 12, 2021 at 18:38
                @EvgenyKuznetsov  I posted a link to the python docs, which is where I copied from; Have you thought that the other stackoverflow answer maybe copied from the same place?
– Radu Gabriel
                Jan 14, 2021 at 11:35
Based on Amith Koujalgi's answer, here's a simple module you can use for logging - 
transcript.py:
Transcript - direct print output to a file, in addition to terminal.
Usage:
    import transcript
    transcript.start('logfile.log')
    print("inside file")
    transcript.stop()
    print("outside file")
import sys
class Transcript(object):
    def __init__(self, filename):
        self.terminal = sys.stdout
        self.logfile = open(filename, "a")
    def write(self, message):
        self.terminal.write(message)
        self.logfile.write(message)
    def flush(self):
        # this flush method is needed for python 3 compatibility.
        # this handles the flush command by doing nothing.
        # you might want to specify some extra behavior here.
def start(filename):
    """Start transcript, appending print output to given filename"""
    sys.stdout = Transcript(filename)
def stop():
    """Stop transcript and return print functionality to normal"""
    sys.stdout.logfile.close()
    sys.stdout = sys.stdout.terminal
print('This is not in the output file.')        
with closing(Tee("outputfile.log", "w", channel="stdout")) as outputstream:
    print('This is written to the output file and the console.')
    # raise Exception('The file "outputfile.log" is closed anyway.')
print('This is not written to the output file.')   
# Output on console:
# This is not in the output file.
# This is written to the output file and the console.
# This is not written to the output file.
# Content of file outputfile.txt:
# This is written to the output file and the console.
The Tee class in IPython.utils.io does what you want, but it lacks the __enter__ and __exit__ methods needed to call it in the with-statement. Those are added by contextlib.closing.
I devised an easier solution. Just define a function that will print to file or to screen or to both of them. In the example below I allow the user to input the outputfile name as an argument but that is not mandatory:
OutputFile= args.Output_File
OF = open(OutputFile, 'w')
def printing(text):
    print text
    if args.Output_File:
        OF.write(text + "\n")
After this, all that is needed to print a line both to file and/or screen is:
    printing(Line_to_be_printed)
                This works. However, note that the input 'text' to 'printing' function has to be explicit text. It won't support any additional arguments that print supports.
– rrlamichhane
                Jul 25, 2018 at 23:58
                My issue for supporting non-explicit text (like datetime.datetime) was in the write function.  I just surrounded text with str(text) to convert to string.
– tzg
                Nov 28, 2018 at 18:52
Here is a simple context manager that prints to the console and writes the same output to an file. It also writes any exceptions to the file.
import traceback
import sys
# Context manager that copies stdout and any exceptions to a log file
class Tee(object):
    def __init__(self, filename):
        self.file = open(filename, 'w')
        self.stdout = sys.stdout
    def __enter__(self):
        sys.stdout = self
    def __exit__(self, exc_type, exc_value, tb):
        sys.stdout = self.stdout
        if exc_type is not None:
            self.file.write(traceback.format_exc())
        self.file.close()
    def write(self, data):
        self.file.write(data)
        self.stdout.write(data)
    def flush(self):
        self.file.flush()
        self.stdout.flush()
To use the context manager:
print("Print")
with Tee('test.txt'):
    print("Print+Write")
    raise Exception("Test")
print("Print")
                Good answer! In Python3 I got an error because of "isatty" missing. To fix it add the following: def isatty(self):         return False
– matt3o
                Apr 25, 2020 at 7:59
I've tried a few solutions here and didn't find the one that writes into file and into console at the same time. So here is what I did (based on this answer)
class Logger(object):
    def __init__(self):
        self.terminal = sys.stdout
    def write(self, message):
        with open ("logfile.log", "a", encoding = 'utf-8') as self.log:            
            self.log.write(message)
        self.terminal.write(message)
    def flush(self):
        #this flush method is needed for python 3 compatibility.
        #this handles the flush command by doing nothing.
        #you might want to specify some extra behavior here.
sys.stdout = Logger()   
This solution uses more computing power, but reliably saves all of the data from stdout into logger file and uses less memeory. For my needs I've added time stamp into self.log.write(message) aswell. Works great.
                For every log line you open a file, write a line, close the file... There's no need for that.
– warchantua
                Jun 21, 2021 at 12:47
This way worked very well in my situation. I just added some modifications based on other code presented in this thread.
import sys, os 
orig_stdout = sys.stdout  # capture original state of stdout
te = open('log.txt','w')  # File where you need to keep the logs
class Unbuffered:
   def __init__(self, stream):
       self.stream = stream
   def write(self, data):
       self.stream.write(data)
       self.stream.flush()
       te.write(data)    # Write the data of stdout here to a text file as well
sys.stdout=Unbuffered(sys.stdout)
#######################################
##  Feel free to use print function  ##
#######################################
print("Here is an Example =)")
#######################################
##  Feel free to use print function  ##
#######################################
# Stop capturing printouts of the application from Windows CMD
sys.stdout = orig_stdout  # put back the original state of stdout
te.flush()  # forces python to write to file
te.close()  # closes the log file
# read all lines at once and capture it to the variable named sys_prints
with open('log.txt', 'r+') as file:
    sys_prints = file.readlines() 
# erase the file contents of log file
open('log.txt', 'w').close()
The following works on Linux where tee is natively available. To also use it on Windows, either use unxutils or find another tee-like option. Run your script using,
python -u <YOUR_SCRIPT>.py | tee log.txt
This will print in the console as well as log to the file.
Also, don't forget to use the -u flag otherwise, you won't see any output in the console.
Based on Brian Burns edited answer, I created a single class that is easier to call:
class Logger(object):
    Class to log output of the command line to a log file
    Usage:
    log = Logger('logfile.log')
    print("inside file")
    log.stop()
    print("outside file")
    log.start()
    print("inside again")
    log.stop()
    def __init__(self, filename):
        self.filename = filename
    class Transcript:
        def __init__(self, filename):
            self.terminal = sys.stdout
            self.log = open(filename, "a")
        def __getattr__(self, attr):
            return getattr(self.terminal, attr)
        def write(self, message):
            self.terminal.write(message)
            self.log.write(message)
        def flush(self):
    def start(self):
        sys.stdout = self.Transcript(self.filename)
    def stop(self):
        sys.stdout.log.close()
        sys.stdout = sys.stdout.terminal
To redirect output to a file and a terminal without modifying how your Python script is used outside, you could use pty.spawn(itself):
#!/usr/bin/env python
"""Redirect stdout to a file and a terminal inside a script."""
import os
import pty
import sys
def main():
    print('put your code here')
if __name__=="__main__":
    sentinel_option = '--dont-spawn'
    if sentinel_option not in sys.argv:
        # run itself copying output to the log file
        with open('script.log', 'wb') as log_file:
            def read(fd):
                data = os.read(fd, 1024)
                log_file.write(data)
                return data
            argv = [sys.executable] + sys.argv + [sentinel_option]
            rc = pty.spawn(argv, read)
    else:
        sys.argv.remove(sentinel_option)
        rc = main()
    sys.exit(rc)
If pty module is not available (on Windows) then you could replace it with teed_call() function that is more portable but it provides ordinary pipes instead of a pseudo-terminal -- it may change behaviour of some programs.
The advantage of pty.spawn and subprocess.Popen -based solutions over replacing sys.stdout with a file-like object is that they can capture the output at a file descriptor level e.g., if the script starts other processes that can also produce output on stdout/stderr. See my answer to the related question: Redirect stdout to a file in Python?
Based on @Arnold Suiza's answer, here is a function you can run once in the beginning and afterwards all will be immediately printed to stdout & file:
def print_to_file(filename):
    orig_stdout = sys.stdout  # capture original state of stdout
    class Unbuffered:
        def __init__(self, filename):
            self.stream = orig_stdout
            self.te = open(filename,'w')  # File where you need to keep the logs
        def write(self, data):
            self.stream.write(data)
            self.stream.flush()
            self.te.write(data)    # Write the data of stdout here to a text file as well
            self.te.flush()
    sys.stdout=Unbuffered(filename)
Now just run print_to_file('log.txt') at program start and you're good to go!
                addendum to the error I posted for anyone else hit by it: it appears this was a library calling the flush (I was using click.echo()) -> I just added an empty flush() method to print_to_file() to fix.
– Robert Muil
                Mar 29, 2022 at 13:06
The answer provided by user2033758 is concise; however, it does not work if the code involves libraries using sys.stdout methods such as sys.stdout.flush(). I slightly modified this answer by adding the flush() method which simply calls the sys.stdout's flush() method, and I also moved the file object into the Unbuffered class:
import sys
class Unbuffered(object):
    def __init__(self, stream, filepath):
        self.stream = stream
        self.file = open(filepath, 'w')  # File where you need to keep the logs
    def write(self, data):
        self.stream.write(data)
        self.stream.flush()
        self.file.write(data)    # Write the data of stdout here to a text file as well
    def flush(self):
        self.stream.flush()
sys.stdout=Unbuffered(sys.stdout, filepath='console_text.txt')
    def __init__(self, filename):
        self.terminal = sys.stdout, sys.stderr
        self.logfile = open(filename, "a")
    def write(self, message):
        self.terminal.write(message)
        self.logfile.write(message)
    def flush(self):
        # this flush method is needed for python 3 compatibility.
        # this handles the flush command by doing nothing.
        # you might want to specify some extra behavior here.
def start(filename):
    """Start transcript, appending print output to given filename"""
    sys.stdout = Transcript(filename)
def stop():
    """Stop transcript and return print functionality to normal"""
    sys.stdout.logfile.close()
    sys.stdout = sys.stdout.terminal
    sys.stderr = sys.stderr.terminal
you can redirect the output to a file by using  >>  python with print rint's "chevron" syntax as indicated in the docs
let see,
fp=open('test.log','a')   # take file  object reference 
print >> fp , "hello world"            #use file object with in print statement.
print >> fp , "every thing will redirect to file "
fp.close()    #close the file 
checkout the file test.log you will have the data 
and to print on console just use plain print statement .
                This is valid in python2.7. >> can be used with print statement to redirect string to given file object. Please verify by trying my solution by yourself
– neotam
                Oct 2, 2019 at 10:55